Semantic navigation method, object picking and delivery method and robot based on large model

Abstract

The present application relates to a semantic navigation method, an item picking and delivery method, and a robot based on a large model. The method uses the powerful learning ability of the large model to improve environmental adaptability, so that the robot can perceive and adapt to complex and changing environments in real time, and dynamically adjust the navigation strategy to deal with emergencies or unknown obstacles. The high environmental adaptability not only improves the navigation accuracy, but also enhances the robustness of the robot in various scenarios. Secondly, the large model can parse the semantics and contextual information of human language more deeply, accurately understand the user's command intentions, so that the robot can be closer to the user's actual needs when performing tasks, and improve the accuracy and satisfaction of task completion. Furthermore, the decision-making ability of the large model in complex scenarios enables the robot to flexibly respond to various emergencies and task requirements, and quickly make reasonable and efficient decisions, which not only improves the robot's work efficiency, but also enhances its ability to deal with uncertain factors.

Description

Technical Field

[0001] This application relates to the field of robot navigation technology, and in particular to a semantic navigation method based on a large model, an item retrieval and delivery method, and a robot. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, robotics is moving towards intelligence and autonomy. In robotic applications, semantic navigation and object retrieval are two crucial capabilities. These not only require robots to accurately understand and respond to human commands but also to navigate autonomously in complex and ever-changing environments, precisely locating and delivering objects. However, traditional methods face numerous challenges in achieving these functions, such as limitations in environmental perception, ambiguity in command understanding, and the complexity of decision-making and control.

[0003] Traditional robot navigation systems often rely on predefined maps and simple obstacle detection, making it difficult to achieve efficient and accurate navigation in complex, dynamically changing environments rich in semantic information. For example, in environments such as homes, hospitals, or shopping malls, robots need to understand and follow instructions such as "go to the nearest coffee shop" or "avoid crowded areas." While deep learning has demonstrated powerful policy optimization capabilities in many fields, effectively designing reward functions, handling high-dimensional state spaces, and achieving secure and efficient online learning remain challenges in robot semantic navigation. Large models perform well in semantic parsing, but are often accompanied by high computational costs and slow response times, which is detrimental to robot navigation systems requiring real-time responses.

[0004] To overcome these challenges, researchers have begun exploring large-model-based control methods to improve robots' semantic understanding and autonomous navigation accuracy. Large models, through their rich knowledge representations learned from massive amounts of data and sophisticated feature extraction capabilities, provide robots with a more intelligent and flexible decision-making foundation.

[0005] In the fields of robotic semantic navigation and object retrieval, existing technologies largely rely on fixed maps and path planning algorithms. When the environment changes, robots often fail to adapt in time, leading to navigation failures or inappropriate path selections, impacting task completion efficiency and safety. Traditional machine learning models, limited by the scale and diversity of training data, struggle to capture the rich connotations and subtle nuances of human language. Robots are prone to misunderstanding or omissions when understanding natural language instructions, hindering their ability to accurately execute complex tasks. Other methods utilize machine learning techniques to improve a robot's environmental perception and decision-making capabilities, but limitations in model size and training data often prevent them from achieving ideal performance. Existing decision-making processes are based on preset rules or algorithms, lacking sufficient flexibility and creativity. When faced with unexpected situations or the need for flexible strategy adjustments, robots struggle to make optimal decisions, affecting task completion quality. Complex system architectures and high coupling between modules require significant modifications to add new features or fix errors, increasing development and maintenance costs and reducing system scalability and maintainability. Summary of the Invention

[0006] Therefore, it is necessary to provide a semantic navigation method, an item retrieval and delivery method, and a robot based on a large model to address the aforementioned technical problems.

[0007] A semantic navigation method based on a large model, comprising:

[0008] Get the user's input text.

[0009] A large model is used to perform semantic understanding and parsing of the input text, and corresponding task instructions are generated based on the obtained semantic parsing results, and the robot is controlled to execute the task instructions.

[0010] The navigation system acquires images of the robot's surrounding environment, uses feature matching methods to identify and track key points in the environment, achieves self-localization, intelligently plans its route, avoids obstacles, and identifies important navigational landmarks.

[0011] Navigation systems reconstruct a 3D model of the surrounding environment using 3D mapping, and then use this model to understand the complex environment and achieve accurate positioning and navigation.

[0012] In one embodiment, the large model includes an encoder and a decoder; the large model is used to perform semantic understanding and parsing of the input text, and generates corresponding task instructions based on the obtained semantic parsing results, and controls the robot to execute the task instructions, including:

[0013] An encoder is used to encode the input text into a vector sequence; the vector sequence includes the semantic information of the text.

[0014] A decoder is used to combine contextual information, grammatical rules, and prior knowledge to understand and parse the context of a vector sequence, thus obtaining semantic parsing results.

[0015] Based on the semantic parsing results, corresponding task instructions are generated, and the robot is controlled to execute the task instructions.

[0016] In one embodiment, the feature matching method is the SuperGlub algorithm.

[0017] In one embodiment, the navigation system reconstructs a 3D model of the surrounding environment using 3D mapping, understands the complex environment based on the 3D model, and achieves accurate positioning and navigation, including:

[0018] The navigation system denoises, filters, and compresses the point cloud data generated by the lidar sensor, and extracts key features from the processed data; key features include edges, corners, and planes.

[0019] Based on key features, a preset reconstruction algorithm is used to reconstruct a three-dimensional model of the surrounding environment.

[0020] By understanding the complex surrounding environment based on 3D models, precise positioning and navigation can be achieved.

[0021] A method for picking up and delivering items, the method comprising:

[0022] An agent is constructed using a reinforcement learning framework. The agent continuously monitors real-time status information from sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information. The agent learns the optimal strategy for detecting and grasping targets by interacting with the target object and the robotic arm.

[0023] The robot's navigation system navigates to its destination using the aforementioned large-model-based semantic navigation method.

[0024] After the robot reaches its destination, the navigation system acquires the real-time status information of the intelligent agent detection and the target detected by the target detection algorithm. Then, it calculates the three-dimensional coordinates of the target and corrects them according to the positions of the camera and the robot arm. The navigation system sends the output information to the robot arm control system. The robot arm control system parses the received output information and generates control instructions to guide the robot arm to grasp the target. The output information includes path planning results, current position information, and environmental parameters.

[0025] The robotic arm control system executes control commands to complete the picking and delivering of items.

[0026] In one embodiment, a reinforcement learning framework is used to construct an agent that continuously monitors real-time state information from sensors carried by the robot and the environment. Based on the real-time state information, the agent selects and executes the optimal grasping action, including:

[0027] The process of a robot's robotic arm grasping an object is modeled as a Markov decision process; the Markov decision process is a set containing four elements. Where S is the state space and A is the action space. For the transfer function, For the reward function, Represents the current state. For action, Represents the agent's current state Execute action The new state that may be reached later.

[0028] Intelligent agents are constructed using a reinforcement learning framework.

[0029] During agent training, the agent receives state information and selects the optimal action; the environment rewards or punishes actions based on their effects, and the agent uses the rewards to update its decision-making strategy, iteratively optimizing until it can accurately grasp the target, thus obtaining the optimal strategy; the state information includes the image and the position of the robotic arm; the optimal state value function under the optimal strategy satisfies the Bellman optimality equation:

[0030] ;

[0031] The optimal action-value function under the optimal policy satisfies the Bellman optimality equation:

[0032] ;

[0033] in, , These are the optimal state value function and the optimal action value function, respectively. This represents the discount factor.

[0034] After training, the agent is deployed in the actual environment. The agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information.

[0035] In one embodiment, the robotic arm control system executes control commands to complete the item retrieval and delivery actions, including:

[0036] In the target grasping process, the robotic arm control system converts the information into the robotic arm's base coordinate system to ensure accurate positioning. Then, path planning is performed, and the robotic arm moves according to the planned path, activating the end effector to prepare for grasping. During the grasping process, real-time feedback and adjustments are made. After successful grasping, the robotic arm places the object at the designated location.

[0037] In one embodiment, after the robot reaches its destination, the navigation system acquires real-time status information from the intelligent agent detection and the target detected by the target detection algorithm. It then calculates the target's three-dimensional coordinates and corrects them based on the positions of the camera and the robotic arm. The navigation system sends the output information to the robotic arm control system, which parses the received output information and generates control instructions to guide the robotic arm in grasping the target, including:

[0038] After the robot reaches its destination, the navigation system obtains the real-time status information of the intelligent agent detection and the target detected by the target detection algorithm. Then, it calculates the three-dimensional coordinates of the target and corrects them according to the positions of the camera and the robotic arm.

[0039] The navigation system sends the path planning results, current location information, and environmental parameters to the robot arm control system.

[0040] The robotic arm control system receives path planning results, current position information, and environmental parameters, performs real-time analysis, and generates corresponding control commands based on the analysis results. The control commands include parameters describing the robotic arm's motion trajectory, speed, and force.

[0041] The robotic arm control system sends control commands to the robotic arm, guiding it to perform specific actions.

[0042] During execution, the robotic arm will provide real-time feedback on the execution status and results to the robotic arm control system.

[0043] When encountering obstacles or changes in the environment, the robot control system reassesses the current situation based on feedback from the robot and new inputs from the detection algorithm, and adjusts control commands and parameters accordingly.

[0044] In one embodiment, the target detection algorithm is the YOLOv8 algorithm.

[0045] A robot includes a navigation system, a 3D mapping module, a robotic arm control system, and a robotic arm. The robot uses the aforementioned object retrieval and delivery method to grasp objects.

[0046] The aforementioned semantic navigation method, item retrieval and delivery method, and robot based on large-scale models leverage the powerful learning capabilities of these models to enhance environmental adaptability. This allows the robot to perceive and adapt to complex and ever-changing environments in real time, dynamically adjusting its navigation strategy to cope with unexpected situations or unknown obstacles. This high level of environmental adaptability not only improves navigation accuracy but also enhances the robot's robustness across various scenarios. Secondly, large-scale models can more deeply analyze the semantics and contextual information of human language, thereby accurately understanding the user's instructions and intentions. This allows the robot to better meet the user's actual needs when performing tasks, improving task completion accuracy and satisfaction. Furthermore, the decision-making capabilities of large-scale models in complex scenarios enable the robot to flexibly respond to various unexpected situations and task requirements, quickly making reasonable and efficient decisions. This not only improves the robot's work efficiency but also enhances its ability to cope with uncertainties. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a semantic navigation method based on a large model in one embodiment;

[0048] Figure 2 This is a flowchart illustrating an item retrieval and delivery method in one embodiment;

[0049] Figure 3 This is a schematic diagram illustrating the interaction between an agent and its environment in one embodiment;

[0050] Figure 4 This is a schematic diagram of the model architecture of the YOLOv8 algorithm in another embodiment;

[0051] Figure 5 This is a schematic diagram of the control process for a robot to pick up and deliver items in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] Traditional robot navigation may rely on pre-set maps or fixed path planning, but this approach is not flexible or accurate enough in complex and ever-changing environments. Semantic navigation technology based on large models needs to address how to enable robots to understand and recognize semantic information in the environment (such as rooms, corridors, doors, etc.) and perform dynamic path planning accordingly to achieve more accurate and flexible navigation.

[0054] Large models, by learning statistical regularities and patterns from massive amounts of data, can capture more complex and refined feature information, thereby providing robots with more accurate and reliable decision support. Furthermore, large models possess powerful generalization capabilities, enabling them to adapt to the needs of different scenarios and tasks.

[0055] To address the shortcomings of existing technologies, this application proposes a semantic navigation method and an item retrieval and delivery method based on a large model. The technical solution mainly includes the following aspects:

[0056] 1) Large-scale model construction and training: Utilize deep learning technology to build large-scale neural network models, and train the models with a large amount of processed data to enable them to have powerful semantic understanding and environmental perception capabilities.

[0057] 2) Semantic Navigation Algorithm Design: Based on a pre-trained large model, a semantic navigation algorithm is designed and implemented. This algorithm can receive human natural language commands and convert them into navigation goals and path planning information that the robot can understand. Simultaneously, the algorithm can also perceive changes in the surrounding environment in real time and dynamically adjust the navigation strategy and path planning.

[0058] 3) Optimization of Item Grasping Strategy: Combining semantic navigation algorithms and robotic arm control technology, an efficient item grasping strategy was designed and implemented. This strategy can automatically plan the retrieval and delivery path and grasping actions based on the location and shape information of the target item, ensuring that the item can be accurately and safely retrieved and delivered to the designated location.

[0059] In one embodiment, such as Figure 1 As shown, a semantic navigation method based on a large model is presented. This method addresses several key technical issues, including the accuracy of semantic navigation, semantic understanding and command execution, object recognition and localization, dynamic environment adaptability, multi-sensor fusion, and real-time feedback, thereby improving the performance and reliability of robots in practical applications. The method includes the following steps:

[0060] Step 100: Obtain the user's input text.

[0061] Specifically, it is used to input relevant information according to specific task requirements, and the input method is text input.

[0062] Step 102: Use a large model to perform semantic understanding and parsing on the input text, generate corresponding task instructions based on the obtained semantic parsing results, and control the robot to execute the task instructions.

[0063] Specifically, after the robot system receives the user's input text, it invokes the large model module. This module runs a large model used for semantic understanding and parsing of the input text. The large model is responsible for semantic understanding and parsing of the user's input text. This includes identifying keywords, phrases, and sentence structures in the text, as well as understanding the logical relationships and intentions between them.

[0064] The Qwen2-1.5B model uses a byte-pair-based tokenizer to segment the input text into words or sub-words and converts them into numerical codes. Then, the model extracts semantic, syntactic, and semantic features of the text through a self-attention mechanism and feedforward neural network layers, generating a deep semantic representation. In the decoder stage, Qwen2-1.5B utilizes these extracted feature vectors and contextual information to generate new text sequences. To further improve model performance, Qwen2-1.5B has also undergone instruction fine-tuning, enhancing its ability to understand and execute specific instructions.

[0065] The large model Qwen2-1.5B is used to perform semantic understanding and parsing of the input text. Qwen2-1.5B parses these inputs into token sequences that the model can process. Then, the question is transformed into a query in the knowledge graph, matching relevant content, and using entities, attributes and relationships in the graph for reasoning and association, providing accurate and comprehensive results. Based on the user's intent, corresponding task instructions are generated and the robot is controlled to execute the task instructions.

[0066] Robots need to be able to understand instructions in human natural language, such as "go to the kitchen and get a cup," and translate these instructions into concrete actions. This involves Natural Language Processing (NLP) technology and how to combine language instructions with the robot's actual actions. Control methods based on large language models can provide more powerful language understanding and generation capabilities, enabling robots to better understand and execute complex instructions.

[0067] Step 104: The navigation system acquires images of the robot's surrounding environment, uses feature matching methods to identify and track key points in the environment, achieves self-localization, intelligently plans its route, avoids obstacles, and identifies important navigational landmarks.

[0068] Specifically, the navigation system enables mobile robots to navigate effectively and autonomously in their environments by combining image-to-image matching and image-to-semantic matching techniques. The system uses cameras to capture images and employs feature matching to track specific targets or feature points, while also using deep learning models for target detection to understand image content. This information helps the robot locate and plan paths in complex environments, avoid obstacles, identify landmarks, and manage egocentric views.

[0069] In practical applications, the environment faced by robots is dynamically changing. People moving around, objects shifting, and other factors can all affect the robot's navigation and object retrieval tasks. A large-scale model-based control system needs to be able to perceive environmental changes in real time and make rapid adjustments to ensure the smooth execution of tasks.

[0070] To improve a robot's perception and navigation accuracy, multiple sensors (such as LiDAR, cameras, and inertial navigation systems) are typically used. Large-scale model-based control systems need to effectively fuse information from these sensors to provide more comprehensive and accurate environmental perception.

[0071] Step 106: The navigation system reconstructs a 3D model of the surrounding environment through 3D mapping, understands the complex environment based on the 3D model, and achieves accurate positioning and navigation.

[0072] Specifically, navigation systems rely on 3D mapping to better understand the complex surrounding environment and achieve precise positioning and navigation. First, environmental information is captured using sensors such as LiDAR and cameras, and then optimized to improve data quality and reduce computational complexity. Next, local environment identification and positioning estimation are performed, comparing the current environment with existing map data to estimate the robot's precise location on the map. Following this, newly acquired environmental data is fused and updated with existing maps. Furthermore, loop closure detection and global optimization are necessary to eliminate accumulated errors and ensure the global consistency and accuracy of the map. 3D mapping provides richer environmental information, helping the system identify obstacles, plan paths, and avoid collisions, thereby improving navigation accuracy and safety.

[0073] In the aforementioned semantic navigation method based on large models, the method leverages the powerful learning capabilities of large models to enhance environmental adaptability. This enables the robot to perceive and adapt to complex and ever-changing environments in real time, dynamically adjusting its navigation strategy to cope with unexpected situations or unknown obstacles. This high level of environmental adaptability not only improves navigation accuracy but also enhances the robot's robustness in various scenarios. Secondly, large models can more deeply analyze the semantics and contextual information of human language, thereby accurately understanding the user's instructions and intentions. This allows the robot to better meet the user's actual needs when performing tasks, improving the accuracy and satisfaction of task completion. Furthermore, the decision-making capabilities of large models in complex scenarios enable the robot to flexibly respond to various unexpected situations and task requirements, quickly making reasonable and efficient decisions. This not only improves the robot's work efficiency but also enhances its ability to cope with uncertainties.

[0074] In one embodiment, the large model includes an encoder and a decoder; step 102 includes: encoding the input text into a vector sequence using the encoder; the vector sequence includes the semantic information of the text; using the decoder to combine contextual information, grammatical rules and prior knowledge to understand and parse the context to obtain a semantic parsing result; generating corresponding task instructions based on the semantic parsing result, and controlling the robot to execute the task instructions.

[0075] Specifically, an encoder-decoder structure is used for semantic parsing. The encoder of the Qwen2-1.5B model consists of multiple layers with the same structure, each layer containing a self-attention mechanism and a feedforward neural network. The self-attention mechanism helps the model pay attention to information at different positions while processing text, capturing contextual dependencies in the text. The feedforward neural network further processes the output of the self-attention layers. The encoder is responsible for encoding the input text into a series of vectors, which contain the semantic information of the text. The decoder also consists of multiple layers, each with a similar structure to the encoder, but with the addition of a masked self-attention mechanism. Through masking operations, the decoder's self-attention layers ensure that the model can only access previous words when generating the current word, and cannot access subsequent words. The decoder also includes an encoder-decoder attention mechanism, allowing the decoder to pay attention to the encoder's output, thereby incorporating information from the input sequence into the output sequence. The decoder then generates a structured representation step by step based on the encoder's output. During the decoding process, the model considers contextual information, grammatical rules, and prior knowledge to understand and reason about the context in order to generate an accurate and context-appropriate structured representation. After understanding the user's intent, the large model generates corresponding task instructions based on the intent and controls the robot to execute these instructions.

[0076] In one embodiment, the feature matching method in step 104 is based on the SuperGlue algorithm.

[0077] Specifically, the navigation system acquires images of the robot's surrounding environment. Based on the SuperGlub algorithm, it uses feature matching to identify and track key points in the environment. An attention-based context aggregation mechanism allows SuperGlub to infer the 3D scene and feature assignments. Feature matching uses nearest neighbor search to match feature points extracted from the model's images and uses domain consensus to filter out incorrect matches, enabling the robot to self-localize, intelligently plan its route, avoid obstacles, and identify important navigational landmarks.

[0078] In one embodiment, step 106 includes: the navigation system denoises, filters, and compresses the point cloud data generated by the lidar sensor, and extracts key features from the processed data; the key features include edges, corners, and planes; a preset reconstruction algorithm is used to reconstruct a three-dimensional model of the surrounding environment based on the key features; and the complex surrounding environment is understood based on the three-dimensional model to achieve accurate positioning and navigation.

[0079] In one embodiment, such as Figure 2 As shown, a method for picking up and delivering items is provided, the method including:

[0080] Step 200: Construct an agent using a reinforcement learning framework. The agent continuously monitors real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information. The agent learns the optimal strategy for detecting and grasping the target by interacting with the target object and the robotic arm.

[0081] Step 202: The robot's navigation system navigates to its destination using the large model-based semantic navigation method described above.

[0082] Step 204: After the robot reaches the destination, the navigation system obtains the real-time status information of the intelligent agent detection and the target detected by the target detection algorithm. Then, it calculates the three-dimensional coordinates of the target and corrects them according to the positions of the camera and the robot arm. The navigation system sends the output information to the robot arm control system. The robot arm control system parses the received output information and generates control instructions to guide the robot arm to grasp the target. The output information includes path planning results, current position information and environmental parameters.

[0083] Step 206: The robotic arm control system executes control commands to complete the item retrieval and delivery actions.

[0084] Among the aforementioned methods for retrieving and delivering items, the advantage lies in the comprehensive improvement of the robot's intelligence level in semantic autonomous navigation and item retrieval and delivery through the introduction of large-scale model control technology. These advantages not only improve the robot's work efficiency and task completion quality but also enhance its environmental adaptability, semantic understanding ability, decision-making flexibility, scalability, and maintainability, providing strong support for the further development and application of robotics technology.

[0085] In one embodiment, step 200 includes: modeling the process of the robot's robotic arm grasping an object as a Markov decision process; the Markov decision process is a set containing four elements. Where S is the state space and A is the action space. For the transfer function, For the reward function, Represents the current state. For action, Represents the agent's current state Execute action The new state that may be reached later, Represents the new state The action selected by the agent; the agent is constructed using a reinforcement learning framework; during the agent's training process, the agent receives state information and selects the optimal action; the environment provides rewards or penalties based on the action's effect, and the agent uses the rewards to update its decision-making strategy, iteratively optimizing until it can accurately grasp the target, thus obtaining the optimal strategy; the state information includes the image and the position of the robotic arm; the optimal state value function under the optimal strategy satisfies the Bellman optimality equation:

[0086] (1)

[0087] The optimal action-value function under the optimal policy satisfies the Bellman optimality equation:

[0088] (2)

[0089] in, , These are the optimal state value function and the optimal action value function, respectively. This represents the discount factor.

[0090] After training, the agent is deployed in the actual environment. The agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information.

[0091] Specifically, a reinforcement learning framework is used to build an agent that learns how to detect and grasp targets by interacting with them and a robotic arm. During training, the agent receives state information, including images and the position of the robotic arm, and selects the optimal action. The environment rewards or penalizes the action based on its effectiveness; a successful grasp earns a positive reward, while failure or collision results in a penalty. The agent uses the rewards to update its decision-making strategy, iteratively optimizing until it can grasp accurately. After training, the agent is deployed in a real-world environment, selecting and executing the optimal action based on real-time status, while continuously monitoring environmental feedback to adjust its strategy to adapt to changes. A diagram illustrating the interaction between the agent and the environment is shown below. Figure 3 As shown.

[0092] In a Markov Decision Process (MDP), the entity that makes the decision is called an agent. An MDP can be described as a set containing four elements. S represents the state space. A represents the action space. It is a transfer function, defined in the current state. Take action Afterwards, transition to the new state. The probability of. It is a reward function that maps state-action pairs to a real number, representing the state at a given time. Execute action The instant reward received. This represents the discount factor, used to calculate cumulative returns for a given strategy. Its state value function Indicator from state Start and follow the strategy It can satisfy the Bellman equation:

[0093] (3)

[0094] Its action value function It refers to the state Take action And follow the strategy It can satisfy the Bellman equation:

[0095] (4)

[0096] In the two equations above, Represents the agent's current state Execute action The new state that may be reached later. This refers to the new state The action chosen by the agent. It is a strategy Below, in state Execute action The conditional probability.

[0097] Strategy The expected return is greater than or equal to the expected return in all states. If the expected return is [amount], then this strategy [is effective]. Defined as superior to or equal to strategy For all states , ≥ If and only if MDP guarantees that there exists at least one policy whose performance is no worse than any other policy; this policy is called the optimal policy. It is typically used... The optimal strategy shares a single value function, called the optimal state-value function and the optimal action-value function, which are denoted as follows: and Optimal strategy The optimal state-value function satisfies the Bellman optimality equation as shown in equation (1), and the optimal policy is... The optimal action value function satisfies the Bellman optimal equation as shown in formula (2).

[0098] The fixed point in the equation shown in formula (2) corresponds to the optimal value function. Given all the state transition equations and reward functions, the optimal value function can be directly approximated by iteratively solving the corresponding fixed point equations. Once the optimal value function is obtained, it can be used to derive the optimal policy for each state.

[0099] In one embodiment, step 206 includes: in the target grasping process, the robot control system converts the information to the robot arm base coordinate system to ensure accurate positioning, then performs path planning, the robot arm moves according to the planned path, activates the end effector to prepare for grasping, provides real-time feedback and adjustment during the grasping process, and after successful grasping, the robot arm places the object at the designated location.

[0100] Specifically, in the target grasping process, the information provided by the algorithm model is first converted to the robotic arm's base coordinate system to ensure accurate positioning. Then, path planning is performed to find a collision-free path. The robotic arm moves according to the planned path, activating the end effector to prepare for grasping. Real-time feedback and adjustments are made during the grasping process to ensure stability and accuracy. After successful grasping, the robotic arm places the object at the designated location and evaluates the task execution, continuously optimizing the algorithm to improve grasping efficiency and accuracy.

[0101] In one embodiment, step 204 includes: after the robot reaches its destination, the navigation system acquires real-time status information from the intelligent agent detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them based on the positions of the camera and the robot arm; the navigation system sends the path planning results, current position information, and environmental parameters to the robot arm control system; the robot arm control system receives the path planning results, current position information, and environmental parameters, performs real-time analysis, and generates corresponding control commands based on the analysis results; the control commands include parameters describing the robot arm's trajectory, speed, and force; the robot arm control system sends the control commands to the robot arm to guide the robot arm to perform specific actions; during execution, the robot arm provides real-time feedback on the execution status and results to the robot arm control system; when encountering obstacles or environmental changes, the robot arm control system reassesses the current situation based on the robot arm's feedback and the new input from the detection algorithm, and adjusts the control commands and parameters.

[0102] Specifically, after the navigation system reaches its destination, it acquires the target detected by the YOLOv8 detection algorithm, calculates its 3D coordinates, and performs position correction based on the relative positions of the camera and the robotic arm. The YOLOv8 algorithm uses a convolutional neural network to extract feature information from the image and then matches these features with pre-defined detection boxes. Through gridded prediction and bounding box regression techniques, the algorithm can accurately locate the target object. Simultaneously, it classifies the target and evaluates its confidence level, ensuring the accuracy and stability of the detection results. The navigation system is responsible for planning the optimal path from the starting point to the destination and perceiving environmental changes in real time, such as obstacle positions and road conditions. This information is processed by high-precision sensors and algorithms to form a comprehensive understanding of the environment. Meanwhile, the detection algorithm focuses on real-time monitoring of dynamic changes in the environment, ensuring that the navigation system's decisions can quickly adapt to new situations. Subsequently, this information needs to be integrated and parsed to provide accurate control commands to the robotic arm. During this process, a stable communication channel needs to be established between the navigation system and the robotic arm, using a unified communication protocol and interface to ensure accurate data transmission. Through the designed interface, the navigation system sends key data such as path planning results, current location information, and environmental parameters to the robotic arm control system. Upon receiving this information, the robotic arm control system immediately analyzes it and generates corresponding control commands to grasp the target. These commands detail how the robotic arm should move, including parameters such as trajectory, speed, and force. Subsequently, these commands are sent to the robotic arm via communication protocols and interfaces, guiding it to execute specific actions. During execution, the robotic arm provides real-time feedback on its execution status and results to the control system. This information is crucial for the dynamic adjustment of the control system. When encountering obstacles or environmental changes, the control system reassesses the current situation based on the robotic arm's feedback and new inputs from the detection algorithm, and may adjust control commands and parameters accordingly. This closed-loop feedback mechanism ensures that the robotic arm can flexibly handle various complex situations, ensuring the successful completion of the task.

[0103] The navigation system and the robotic arm use a unified communication protocol and interface for data transmission.

[0104] In one embodiment, the target detection algorithm is the YOLOv8 algorithm.

[0105] Specifically, the model architecture of the YOLOv8 algorithm is as follows: Figure 4 As shown, Figure 4 The C2F module, by concatenating the outputs of different modules with the original feature map, can better aggregate multi-scale information. The CBS module is used for feature extraction and transformation. Convolution operations superimpose small-sized filters onto the image to extract local features; the calculation formula is as follows:

[0106] (5)

[0107] (6)

[0108] The size of the input image is (H, W, C) (H represents the height of the input, and W represents the width of the input). Indicates the height of the convolution kernel. This represents the width of the convolution kernel, and includes padding P and stride S.

[0109] It should be understood that, although Figure 1 , Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0110] In a specific embodiment, such as Figure 5 As shown, the specific steps of the robot's item retrieval and delivery method include:

[0111] Step T1: User Input. Users input text based on their needs, such as "deliver the documents to the office," "deliver the snacks to the lobby," or other similar actions.

[0112] Step T2: Large Model Invocation. The system invokes the large model module in this module.

[0113] Step T3: Large Model Instruction Parsing. The word vectors containing dense features generated in the semantic understanding module and the sparse features generated using the user input text are input together into the entity extraction and intent recognition joint extraction model to obtain the entities and related intents from the user input text.

[0114] Step T4: Navigation System. By integrating image matching and semantic parsing technologies, visual information is acquired using cameras, and key points in the environment are identified and tracked through feature matching algorithms. The system can accurately locate itself, intelligently plan its route, skillfully avoid obstacles, and identify important navigational landmarks.

[0115] Step T5: 3D Mapping. The 3D mapping module processes point cloud data generated by sensors such as LiDAR through denoising, filtering, and compression. It extracts key features, such as edges, corners, and planes, from the processed data. Using these extracted features and algorithms, it reconstructs a 3D model of the surrounding environment to achieve accurate obstacle recognition and path planning.

[0116] Step T6: Motion Control. Correction is performed by combining the camera and the robot's position. Through stable communication, the navigation system sends path, position, and environmental data to the robot's control system. The control system parses the data and generates control commands to guide the robot to grasp the target.

[0117] Step T7: Reinforcement Learning. The agent will continuously monitor real-time state information from sensors and the environment, and based on this information, quickly select and execute the optimal grasping action.

[0118] Step T8: Target Grasping. The algorithm model output information is converted to the robotic arm's base coordinate system for fine-grained path planning. The robotic arm moves along the planned path, activating the end effector to prepare for grasping. During grasping, real-time feedback and adjustments ensure stability and accuracy. Upon successful grasping, the robotic arm smoothly transports the object to the designated position, maintaining high stability and precision. After task completion, the system evaluates the grasping success rate, path efficiency, and motion smoothness.

[0119] Step T9: Detection Algorithm. The detection algorithm acquires image frames to be detected through a camera, performs preprocessing, extracts features using a convolutional neural network, identifies and locates objects in the image, calculates the three-dimensional coordinates of the detected objects, and outputs the detection results to the robotic arm.

[0120] In one embodiment, a robot is provided, which includes a navigation system, a 3D mapping module, a robotic arm control system, and a robotic arm. The robot uses any of the above-described object retrieval and delivery methods to grasp objects.

[0121] Specifically, the robot proposed in this application, through its modular design and loosely coupled system architecture, enhances the independence and replaceability of its modules, thereby reducing the overall complexity of the system. This design makes adding new functions and fixing errors more convenient, while also reducing development and maintenance costs. This is of great significance for promoting the popularization and application of robotics technology.

[0122] Robot semantic navigation and object grasping technology based on large model control has broad application prospects and significant research value. It can not only enhance the intelligence level of robots in fields such as home services, logistics delivery, and medical care, but also provide strong support for the further development of artificial intelligence technology. Through the implementation of this invention, the shortcomings of traditional robots in semantic understanding and autonomous navigation can be effectively addressed, promoting the development of robotics technology towards greater intelligence, autonomy, and flexibility.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A semantic navigation method based on a large model, characterized in that: The method comprises: Get the user's input text; The input text is semantically understood and parsed using a large model, the input text is encoded into a vector sequence using an encoder of the large model, the context is understood and parsed using a decoder of the large model in combination with context information, grammatical rules and prior knowledge on the vector sequence, a semantic parsing result is obtained, and a corresponding task instruction is generated according to the obtained semantic parsing result, and the robot is controlled to execute the task instruction; the large model is a large model Qwen2-1.5B, including an encoder and a decoder; the vector sequence includes semantic information of the text; The navigation system acquires the image of the robot's surrounding environment, uses a feature matching method to identify and track key points in the environment, achieves self-positioning, intelligently plans the action route, avoids obstacles, and identifies important navigation signs; the feature matching method is the SuperGlue algorithm; The navigation system reconstructs a 3D model of the surrounding environment through 3D mapping, understands the complex surrounding environment based on the 3D model, and achieves accurate positioning and navigation; The navigation system reconstructs a 3D model of the surrounding environment through 3D mapping, understands the complex surrounding environment based on the 3D model, and achieves accurate positioning and navigation, including: The navigation system removes noise, filters, and compresses the point cloud data generated by the LiDAR sensor, and extracts key features from the processed data; the key features include edges, corners, and planes; Reconstructing a three-dimensional model of the surrounding environment using a preset reconstruction algorithm according to the key features; The complex surrounding environment is understood based on the three-dimensional model to achieve accurate positioning and navigation.

2. A method for picking up and delivering items, characterized in that: The method comprises: An intelligent agent is constructed using a reinforcement learning framework. The intelligent agent continuously monitors the real-time status information from the sensors and environment carried by the robot, and selects and executes the optimal grasping action based on the real-time status information. The intelligent agent learns the optimal strategy for detecting and grasping the target by interacting with the target object and the robotic arm. The robot's navigation system navigates to the destination through the semantic navigation method based on the large model described in claim 1; After the robot reaches the destination, the navigation system obtains the real-time status information of the intelligent body detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator. The navigation system sends the output information to the manipulator control system, which parses the received output information and generates control instructions to guide the manipulator to grasp the target; the output information includes the path planning result, the current position information and the environmental parameters; The robot control system executes the control instructions to complete the object picking and delivering action.

3. The article picking and delivering method according to claim 2, characterized in that: The intelligent agent is constructed using a reinforcement learning framework. The intelligent agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information, including: The process of the robot's mechanical arm grasping an object is modeled as a Markov decision process; the Markov decision process is a set of four elements , where S is the state space and A is the action space. is the transfer function, is the reward function, Represents the current state, For action, Represents the agent in its current state Execute an action the new state that may be achieved later; Adopt reinforcement learning framework to build intelligent agents; During the training process of the agent, the agent receives state information and selects the optimal action. The environment gives rewards or penalties according to the effect of the action. The agent uses the rewards to update the decision strategy and iterates the optimization until it can accurately grasp and obtain the optimal strategy. The state information includes the image and the position of the robot arm. The optimal state value function under the optimal strategy satisfies the Bellman optimal equation: The optimal action value function under the optimal strategy satisfies the Bellman optimal equation: in, , are the optimal state value function and the optimal action value function, respectively. represents the discount factor; After the training is completed, the intelligent agent is deployed in the actual environment. The intelligent agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information.

4. The article picking and delivering method according to claim 2, characterized in that: The robot control system executes the control instructions to complete the object picking and delivery action, including: In the target grasping process, the robot control system converts the information to the robot arm base coordinate system to ensure accurate positioning, and then performs path planning. The robot arm moves according to the planned path and activates the end effector to prepare for grasping. Real-time feedback and adjustments are made during the grasping process. After a successful grasp, the robot arm places the object at the designated location.

5. The article picking and delivering method according to claim 2, characterized in that: After the robot reaches the destination, the navigation system obtains the real-time status information of the intelligent body detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects it according to the position of the camera and the manipulator. The navigation system sends the output information to the manipulator control system, which parses the received output information and generates control instructions to guide the manipulator to grasp the target, including: After the robot reaches the destination, the navigation system obtains the real-time status information detected by the intelligent agent and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator; The navigation system sends the path planning results, current position information and environmental parameters to the manipulator control system; The manipulator control system receives the path planning results, current position information and environmental parameters, performs real-time analysis, and generates corresponding control instructions according to the analysis results; the control instructions include parameters describing the movement trajectory, speed and strength of the manipulator; The manipulator control system sends the control instruction to the manipulator to instruct the manipulator to perform specific actions; During the execution process, the robot will feedback the execution status and results to the robot control system in real time; When encountering obstacles or environmental changes, the manipulator control system re-evaluates the current situation and adjusts control instructions and parameters based on the manipulator's feedback and new inputs from the detection algorithm.

6. The article picking and delivering method according to claim 2, characterized in that: The target detection algorithm is the YOLOv8 algorithm.

7. A robot, characterized in that: The robot comprises a navigation system, a three-dimensional mapping module, a manipulator control system and a manipulator arm, and the robot adopts the object picking and delivering method according to any one of claims 2 to 6 to grasp objects.

Images