Semantic navigation method based on large model, article taking and delivering method and robot

Abstract

The invention relates to a semantic navigation method based on a large model, an article taking and delivering method and a robot, and the method employs the powerful learning capability of the large model to improve the environmental adaptability, enables the robot to sense and adapt to a complex and changeable environment in real time, dynamically adjusts a navigation strategy to deal with an emergency or an unknown obstacle, and improves the navigation efficiency. The high environmental adaptability not only improves the navigation accuracy, but also enhances the robustness of the robot in various scenes. And secondly, the large model can more deeply analyze semantic and contextual information of human languages and accurately understand user instruction intentions, so that the robot can be closer to actual demands of users when executing tasks, and the accuracy and satisfaction degree of task completion are improved. Moreover, the decision-making ability of the large model in a complex scene enables the robot to flexibly cope with various emergencies and task requirements, and rapidly make reasonable and efficient decisions, thereby not only improving the working efficiency of the robot, but also enhancing the ability of the robot to cope with uncertain factors.

Description

Technical Field

[0001] The present application relates to the field of robot navigation technology, and in particular to a semantic navigation method based on a large model, an object picking and delivering method, and a robot. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, robotics is moving towards intelligence and autonomy. In robotic applications, semantic navigation and object pickup and delivery are two crucial capabilities. They not only require robots to accurately understand and respond to human commands, but also require autonomous navigation in complex and changing environments, precise positioning and object pickup and delivery. However, traditional methods face many challenges in achieving these functions, such as limitations in environmental perception, ambiguity in command understanding, and complexity in decision-making and control.

[0003] Traditional robot navigation systems often rely on predefined maps and simple obstacle detection, and it is difficult to achieve efficient and accurate navigation in complex environments with dynamic changes and rich semantic information. For example, in environments such as homes, hospitals, or shopping malls, robots need to understand and follow instructions such as "go to the nearest coffee shop" and "avoid crowded areas." Although deep learning has demonstrated powerful policy optimization capabilities in many fields, in robot semantic navigation, how to effectively design reward functions, handle high-dimensional state spaces, and achieve safe and efficient online learning remains a challenge. Large models perform well in semantic parsing, but are often accompanied by problems of large computational complexity and slow response speed, which is not conducive to robot navigation systems that require real-time response.

[0004] To overcome these challenges, researchers have begun to explore control methods based on large models to improve the robot's semantic understanding ability and autonomous navigation accuracy. Large models provide robots with a more intelligent and flexible decision-making basis through rich knowledge representation and complex feature extraction capabilities learned from massive data.

[0005] In the field of robot semantic navigation and item delivery, existing technologies mostly rely on fixed maps and path planning algorithms. When the environment changes, robots often cannot adapt in time, resulting in navigation failure or unreasonable path selection, affecting the efficiency and safety of task completion. Traditional machine learning models are limited by the scale and diversity of training data, and it is difficult to capture the rich connotations and nuances of human language. Robots are prone to misunderstanding or omission when understanding natural language instructions, and cannot accurately perform complex tasks. Other methods use machine learning technology to improve the robot's environmental perception and decision-making capabilities, but they are often difficult to achieve ideal performance due to the limitations of model scale and training data. The decision-making process of existing technologies is based on preset rules or algorithms, lacking sufficient flexibility and creativity. When faced with emergencies or when flexible adjustments to strategies are required, it is difficult for robots to make optimal decisions, affecting the quality of task completion. The system architecture is complex, the coupling between modules is high, and adding new functions or fixing errors requires major changes, which increases development and maintenance costs and reduces the scalability and maintainability of the system. Summary of the invention

[0006] Based on this, it is necessary to provide a semantic navigation method, item picking and delivery method and robot based on a large model to address the above technical problems.

[0007] A semantic navigation method based on a large model, the method comprising: Get the user's input text.

[0008] A large model is used to semantically understand and parse the input text, generate corresponding task instructions based on the semantic analysis results, and control the robot to execute the task instructions.

[0009] The navigation system acquires images of the robot's surroundings and uses feature matching methods to identify and track key points in the environment, achieve self-positioning, intelligently plan routes, avoid obstacles, and identify important navigation signs.

[0010] The navigation system reconstructs a three-dimensional model of the surrounding environment through three-dimensional mapping, understands the complex surrounding environment based on the three-dimensional model, and achieves accurate positioning and navigation.

[0011] In one embodiment, the large model includes: an encoder and a decoder; the large model is used to semantically understand and parse the input text, and corresponding task instructions are generated according to the obtained semantic parsing results, and the robot is controlled to execute the task instructions, including: An encoder is used to encode the input text into a vector sequence; the vector sequence includes the semantic information of the text.

[0012] The decoder combines the context information, grammatical rules and prior knowledge of the vector sequence to understand and parse the context and obtain the semantic parsing result.

[0013] Generate corresponding task instructions based on the semantic analysis results, and control the robot to execute the task instructions.

[0014] In one embodiment, the feature matching method is the SuperGlub algorithm.

[0015] In one embodiment, the navigation system reconstructs a three-dimensional model of the surrounding environment through a three-dimensional mapping method, understands the complex surrounding environment based on the three-dimensional model, and achieves accurate positioning and navigation, including: The navigation system denoises, filters, and compresses the point cloud data generated by the lidar sensor, and extracts key features from the processed data; key features include edges, corners, and planes.

[0016] The preset reconstruction algorithm is used according to the key features to reconstruct the three-dimensional model of the surrounding environment.

[0017] Understand the complex surrounding environment based on the three-dimensional model to achieve precise positioning and navigation.

[0018] A method for picking up and delivering an item, the method comprising: The intelligent agent is constructed using a reinforcement learning framework. The intelligent agent continuously monitors the real-time status information from the sensors and environment carried by the robot, and selects and executes the optimal grasping action based on the real-time status information. The intelligent agent learns the optimal strategy for detecting and grasping the target by interacting with the target object and the robotic arm. The robot's navigation system navigates to the destination through the above-mentioned semantic navigation method based on the large model.

[0019] After the robot arrives at the destination, the navigation system obtains the real-time status information of the intelligent body detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator. The navigation system sends the output information to the manipulator control system, which parses the received output information and generates control instructions to guide the manipulator to grasp the target; the output information includes path planning results, current position information and environmental parameters.

[0020] The robot control system executes control instructions to complete the object picking and delivery actions.

[0021] In one embodiment, a reinforcement learning framework is used to construct an intelligent agent, which continuously monitors real-time status information from sensors and the environment carried by the robot, and selects and executes the optimal grasping action based on the real-time status information, including: The process of the robot's mechanical arm grasping an object is modeled as a Markov decision process; the Markov decision process is a set of four elements , where S is the state space and A is the action space. is the transfer function, is the reward function, Represents the current state, For action, Represents the agent in its current state Execute an action A new state that may be achieved later.

[0022] The agent is constructed using a reinforcement learning framework.

[0023] During the training process of the agent, the agent receives state information and selects the optimal action. The environment gives rewards or penalties based on the effect of the action. The agent uses the rewards to update the decision strategy and iterates and optimizes until it can accurately grasp and obtain the optimal strategy. The state information includes the image and the position of the robot arm. The optimal state value function under the optimal strategy satisfies the Bellman optimal equation: ; The optimal action value function under the optimal strategy satisfies the Bellman optimality equation: ; in, , are the optimal state value function and the optimal action value function, respectively. Represents the discount factor.

[0024] After training is completed, the intelligent agent is deployed in the actual environment. The intelligent agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information.

[0025] In one embodiment, the robot control system executes the control instruction to complete the object picking and delivering action, including: In the target grasping process, the robot control system converts the information to the robot arm base coordinate system to ensure accurate positioning, and then performs path planning. The robot arm moves according to the planned path and activates the end effector to prepare for grasping. Real-time feedback and adjustments are made during the grasping process. After a successful grasp, the robot arm places the object at the designated location.

[0026] In one embodiment, after the robot reaches the destination, the navigation system obtains the real-time status information detected by the intelligent agent and the target detected by the target detection algorithm, and then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator. The navigation system sends the output information to the manipulator control system, and the manipulator control system parses the received output information and generates control instructions to guide the manipulator to grasp the target, including: After the robot reaches the destination, the navigation system obtains the real-time status information of the intelligent body detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator.

[0027] The navigation system sends the path planning results, current position information and environmental parameters to the manipulator control system.

[0028] The robot control system receives the path planning results, current position information and environmental parameters, performs real-time analysis, and generates corresponding control instructions based on the analysis results; the control instructions include parameters that describe the robot's motion trajectory, speed and strength.

[0029] The robot control system sends control instructions to the robot to instruct the robot to perform specific actions.

[0030] During the execution process, the robot will feedback the execution status and results to the robot control system in real time.

[0031] When encountering obstacles or environmental changes, the manipulator control system re-evaluates the current situation and adjusts control instructions and parameters based on the manipulator's feedback and new inputs from the detection algorithm.

[0032] In one embodiment, the target detection algorithm is the YOLOv8 algorithm.

[0033] A robot comprises a navigation system, a three-dimensional mapping module, a manipulator control system and a manipulator arm. The robot adopts the above-mentioned object picking and delivering method to grab objects.

[0034] The semantic navigation method, item picking and delivery method and robot based on the large model mentioned above use the powerful learning ability of the large model to improve environmental adaptability, so that the robot can perceive and adapt to complex and changing environments in real time, dynamically adjust the navigation strategy to deal with emergencies or unknown obstacles, and the high environmental adaptability not only improves the accuracy of navigation, but also enhances the robustness of the robot in various scenarios. Secondly, the large model can parse the semantics and contextual information of human language more deeply, so as to accurately understand the user's instruction intention, so that the robot can be closer to the user's actual needs when performing tasks, and improve the accuracy and satisfaction of task completion. Furthermore, the decision-making ability of the large model in complex scenarios enables the robot to flexibly respond to various emergencies and task requirements, and quickly make reasonable and efficient decisions, which not only improves the robot's work efficiency, but also enhances its ability to deal with uncertain factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flowchart of a semantic navigation method based on a large model in an embodiment; Figure 2 A schematic diagram of a flow chart of a method for picking up and delivering items in one embodiment; Figure 3 A schematic diagram of the interaction between an intelligent agent and an environment in one embodiment; Figure 4 A schematic diagram of a model architecture of a YOLOv8 algorithm in another embodiment; Figure 5 The figure is a schematic diagram of the control flow of a robot picking up and delivering items in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] Traditional robot navigation may rely on preset maps or fixed path planning, but this method is not flexible and accurate enough in complex and changing environments. Semantic navigation technology based on large models needs to solve how to enable robots to understand and recognize semantic information in the environment (such as rooms, corridors, doors, etc.), and perform dynamic path planning based on this to achieve more accurate and flexible navigation.

[0038] By learning statistical laws and patterns in large amounts of data, large models can capture more complex and detailed feature information, thereby providing robots with more accurate and reliable decision support. In addition, large models also have strong generalization capabilities and can adapt to the needs of different scenarios and tasks.

[0039] In view of the shortcomings of the existing technology, this application proposes a semantic navigation method and an item picking and delivery method based on a large model. The technical solution mainly includes the following aspects: 1) Large model construction and training: Use deep learning technology to build a large-scale neural network model, and train the model with a large amount of processed data to enable it to have strong semantic understanding and environmental perception capabilities.

[0040] 2) Semantic navigation algorithm design: Based on the trained large model, a semantic navigation algorithm is designed and implemented. The algorithm can receive human natural language instructions and convert them into navigation goals and path planning information that the robot can understand. At the same time, the algorithm can also perceive changes in the surrounding environment in real time and dynamically adjust navigation strategies and path planning.

[0041] 3) Optimization of object grabbing strategy: Combining semantic navigation algorithm and robot arm control technology, an efficient object grabbing strategy is designed and implemented. This strategy can automatically plan the pick-up and delivery path and grabbing action according to the location and shape information of the target object, ensuring that the object can be accurately and safely picked up and delivered to the designated location.

[0042] In one embodiment, Figure 1 As shown, a semantic navigation method based on a large model is provided, which solves multiple key technical problems such as the accuracy of semantic navigation, semantic understanding and instruction execution, object recognition and positioning, dynamic environment adaptability, multi-sensor fusion and real-time feedback, so as to improve the performance and reliability of robots in practical applications. The method includes the following steps: Step 100: Get the user's input text.

[0043] Specifically, it is used to input relevant information according to specific task requirements, and the input method is text input.

[0044] Step 102: Use the large model to perform semantic understanding and analysis on the input text, generate corresponding task instructions based on the obtained semantic analysis results, and control the robot to execute the task instructions.

[0045] Specifically, after the robot system obtains the user's input text, it calls the big model module. The big model module runs the big model for semantic understanding and parsing of the input text. The big model is responsible for semantic understanding and parsing of the user's input text. This includes identifying keywords, phrases, and sentence structures in the text, and understanding the logical relationships and intentions between them.

[0046] The Qwen2-1.5B model uses a byte pair encoding-based tokenizer to segment the input text into vocabulary or subword units and convert them into digital codes. The model then deeply extracts the semantic, grammatical, and syntactic features of the text through a self-attention mechanism and a feedforward neural network layer to generate a deep semantic representation. In the decoder stage, Qwen2-1.5B uses these extracted feature vectors and contextual information to generate new text sequences. To further improve the performance of the model, Qwen2-1.5B has also undergone instruction fine-tuning to enhance its ability to understand and execute specific instructions.

[0047] The large model Qwen2-1.5B is used to semantically understand and parse the input text. Qwen2-1.5B parses these inputs into token sequences that the model can process, then converts the questions into queries in the knowledge graph, matches relevant content, and uses the entities, attributes, and relationships in the graph for reasoning and association, providing accurate and comprehensive results, generating corresponding task instructions based on user intentions, and controlling the robot to execute task instructions.

[0048] Robots need to be able to understand instructions in human natural language, such as "go to the kitchen and get a cup", and convert these instructions into specific actions. This involves natural language processing (NLP) technology and how to combine language instructions with the actual actions of the robot. Control methods based on large language models can provide more powerful language understanding and generation capabilities, enabling robots to better understand and execute complex instructions.

[0049] Step 104: The navigation system acquires an image of the robot's surrounding environment and uses a feature matching method to identify and track key points in the environment, thereby achieving self-positioning, intelligently planning a route of action, avoiding obstacles, and identifying important navigation landmarks.

[0050] Specifically, the navigation system enables the mobile robot to perform effective autonomous navigation in the environment by combining image-to-image matching and image-to-semantic matching technologies. The system uses a camera to capture images and tracks specific targets or feature points through feature matching technology, while using a deep learning model for target detection to understand the image content. This information helps the robot locate and plan paths in complex environments, avoid obstacles, identify landmarks, and process egocentric views.

[0051] In practical applications, the environment faced by robots is dynamically changing. The movement of people and objects may affect the robot's navigation and object delivery tasks. The control system based on the large model needs to be able to perceive environmental changes in real time and make adjustments quickly to ensure the smooth progress of the task.

[0052] In order to improve the perception ability and navigation accuracy of robots, multiple sensors (such as lidar, cameras, inertial navigation, etc.) are usually used. The control system based on the large model needs to be able to effectively fuse the information of these sensors to provide more comprehensive and accurate environmental perception.

[0053] Step 106: The navigation system reconstructs a three-dimensional model of the surrounding environment through three-dimensional mapping, understands the complex surrounding environment based on the three-dimensional model, and achieves accurate positioning and navigation.

[0054] Specifically, the navigation system needs to use three-dimensional mapping to better understand the complex environment around it and achieve accurate positioning and navigation. First, use sensors such as lidar and cameras to capture environmental information and optimize it to improve data quality and reduce computational complexity. Subsequently, local environment identification and positioning estimation are performed to estimate the exact position of the robot in the map by comparing the current environment with existing map data. Next, the newly acquired environmental data is integrated and updated with the existing map. In addition, loop closure detection and global optimization are required to eliminate cumulative errors and ensure the global consistency and accuracy of the map. Three-dimensional mapping can provide richer environmental information, which helps the system identify obstacles, plan paths and avoid collisions. This improves the accuracy and safety of navigation.

[0055] In the above semantic navigation method based on the large model, the method uses the powerful learning ability of the large model to improve environmental adaptability, so that the robot can perceive and adapt to complex and changing environments in real time, and dynamically adjust the navigation strategy to deal with emergencies or unknown obstacles. The high environmental adaptability not only improves the accuracy of navigation, but also enhances the robustness of the robot in various scenarios. Secondly, the large model can parse the semantics and contextual information of human language more deeply, so as to accurately understand the user's instruction intention, so that the robot can be closer to the user's actual needs when performing tasks, and improve the accuracy and satisfaction of task completion. Furthermore, the decision-making ability of the large model in complex scenarios enables the robot to flexibly respond to various emergencies and task requirements, and quickly make reasonable and efficient decisions, which not only improves the robot's work efficiency, but also enhances its ability to deal with uncertain factors.

[0056] In one embodiment, the large model includes: an encoder and a decoder; step 102 includes: using the encoder to encode the input text into a vector sequence; the vector sequence includes semantic information of the text; using the decoder to understand and parse the context of the vector sequence in combination with context information, grammatical rules and prior knowledge to obtain a semantic parsing result; generating corresponding task instructions according to the semantic parsing result, and controlling the robot to execute the task instructions.

[0057] Specifically, an encoder-decoder structure is used for semantic parsing. The encoder of the Qwen2-1.5B model consists of multiple layers of the same structure, each of which contains a self-attention mechanism and a feedforward neural network. The self-attention mechanism helps the model to focus on information at different positions while processing text and capture contextual dependencies in the text. The feedforward neural network further processes the output of the self-attention layer. The encoder is responsible for encoding the input text into a series of vectors that contain the semantic information of the text. The decoder also consists of multiple layers, and the structure of each layer is similar to that of the encoder, but with the addition of a masked self-attention mechanism. Through masking operations, the self-attention layer of the decoder ensures that the model can only access previous words when generating the current word, but cannot access subsequent words. The decoder also contains an encoder-decoder attention mechanism, which allows the decoder to focus on the output of the encoder, thereby integrating information from the input sequence into the output sequence. The decoder gradually generates a structured representation based on the output of the encoder. During the decoding process, the model considers contextual information, grammatical rules, and prior knowledge, understands and infers the context, and generates accurate and contextually appropriate structured representations. After understanding the user's intention, the large model generates corresponding task instructions based on the intention and controls the robot to execute these instructions.

[0058] In one embodiment, the feature matching method in step 104 is based on the SuperGlue algorithm.

[0059] Specifically, the navigation system obtains images of the robot's surrounding environment, uses feature matching methods based on the SuperGlub algorithm to identify and track key points in the environment, and uses a context aggregation mechanism based on the attention mechanism to allow SuperGlub to infer 3D scenes and feature allocation. Feature matching uses the nearest neighbor search to match feature points with the image features extracted by the model, and uses domain consensus to filter incorrect matches, achieving self-positioning, intelligently planning routes, avoiding obstacles, and identifying important navigation signs.

[0060] In one embodiment, step 106 includes: the navigation system denoises, filters, and compresses the point cloud data generated by the lidar sensor, and extracts key features from the processed data; the key features include edges, corners, and planes; a preset reconstruction algorithm is used based on the key features to reconstruct a three-dimensional model of the surrounding environment; and the complex surrounding environment is understood based on the three-dimensional model to achieve precise positioning and navigation.

[0061] In one embodiment, Figure 2 As shown, a method for picking up and delivering an item is provided, the method comprising: Step 200: An intelligent agent is constructed using a reinforcement learning framework. The intelligent agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information. The intelligent agent learns the optimal strategy for detecting and grasping the target by interacting with the target object and the robotic arm.

[0062] Step 202: The robot's navigation system navigates through the above-mentioned semantic navigation method based on the large model to reach the destination.

[0063] Step 204: After the robot arrives at the destination, the navigation system obtains the real-time status information of the intelligent body detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator. The navigation system sends the output information to the manipulator control system, and the manipulator control system parses the received output information and generates control instructions to guide the manipulator to grasp the target; the output information includes path planning results, current position information and environmental parameters.

[0064] Step 206: The robot control system executes the control instruction to complete the object picking and delivering action.

[0065] The advantage of the above-mentioned method of picking up and delivering items is that it comprehensively improves the robot's intelligence level in the field of semantic autonomous navigation and picking up and delivering items by introducing large model control technology. These advantages not only improve the robot's work efficiency and task completion quality, but also enhance its environmental adaptability, semantic understanding ability, decision-making flexibility, and scalability and maintainability, providing strong support for the further development and application of robot technology.

[0066] In one embodiment, step 200 includes: modeling the process of the robot's mechanical arm grasping an object as a Markov decision process; the Markov decision process is a set of four elements: , where S is the state space and A is the action space. is the transfer function, is the reward function, Represents the current state, For action, Represents the agent in its current state Execute an action The new state that may be achieved later, Represents in new state The action selected by the agent under the action; the reinforcement learning framework is used to build the agent; during the agent training process, the agent receives state information and selects the optimal action; the environment gives rewards or penalties according to the effect of the action, and the agent uses the rewards to update the decision strategy, iteratively optimizes until it can accurately grasp and obtain the optimal strategy; the state information includes images and the position of the robot arm; the optimal state value function under the optimal strategy satisfies the Bellman optimal equation: (1) The optimal action value function under the optimal strategy satisfies the Bellman optimality equation: (2) in, , are the optimal state value function and the optimal action value function, respectively. Represents the discount factor.

[0067] After training is completed, the intelligent agent is deployed in the actual environment. The intelligent agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information.

[0068] Specifically, a reinforcement learning framework is used to build an intelligent agent, which learns how to detect and grasp targets by interacting with target objects and robotic arms. During training, the agent receives state information including images and robotic arm positions, and selects the optimal action. The environment gives rewards or penalties based on the effect of the action. Successful grasping receives positive rewards, while failure or collision results in penalties. The agent uses rewards to update its decision-making strategy and iterates and optimizes until it can grasp accurately. After training, the agent is deployed in the actual environment, selects and executes the optimal action based on the real-time status, and continuously monitors environmental feedback to adjust the strategy to adapt to changes. The schematic diagram of the interaction between the agent and the environment is shown in the figure. Figure 3 shown.

[0069] In a Markov decision process (MDP), the entity that makes decisions is called an agent. An MDP can be described as a set of four elements: S is the state space. A is the action space. is the transition function, which defines the Take Action Then, transfer to the new state probability. is a reward function that maps a state-action pair to a real value, representing Execute an action Instant rewards received. Represents the discount factor, which is used to calculate the cumulative return for a certain strategy , its state value function Indicates from the status Start and follow the strategy It can satisfy the Bellman equation: (3) Its action value function It means in the state Take Action And follow the strategy It can satisfy the Bellman equation: (4) In the above two equations, Represents the agent in its current state Execute an action A new state that may be achieved later. In the new state The action chosen by the agent. It’s a strategy Next, in the state Execute an action The conditional probability of .

[0070] Strategy The expected return in all states is greater than or equal to The expected return of this strategy is Defined as being better than or equal to the strategy For all states , ≥ If and only if MDP guarantees that there is at least one strategy whose performance is no less than any other strategy, and this strategy is called the optimal strategy. The optimal strategy shares a value function, called the optimal state value function and the optimal action value function, which are expressed as and Optimal strategy The optimal state value function under the condition satisfies the Bellman optimal equation shown in formula (1), and the optimal strategy The optimal action-value function satisfies the Bellman optimality equation as shown in formula (2).

[0071] The fixed point in the equation shown in formula (2) corresponds to the optimal value function. Under the condition that all state transition equations and reward functions are known, the optimal value function can be directly approximated by iteratively solving the corresponding fixed point equations. Once the optimal value function is obtained, it can be used to derive the optimal strategy in each state.

[0072] In one embodiment, step 206 includes: in the target grasping process, the manipulator control system converts the information to the manipulator base coordinate system to ensure accurate positioning, and then performs path planning. The manipulator moves according to the planned path, activates the end effector to prepare for grasping, and provides real-time feedback and adjustments during the grasping process. After successful grasping, the manipulator places the object at the designated location.

[0073] Specifically, in the target grasping process, the information provided by the algorithm model is first converted to the robot arm base coordinate system to ensure accurate positioning. Then path planning is performed to find a collision-free path. The robot arm moves according to the planned path and activates the end effector to prepare for grasping. Real-time feedback and adjustment are made during the grasping process to ensure stability and accuracy. After successful grasping, the robot arm places the object at the specified location, evaluates the task execution, and continuously optimizes the algorithm to improve grasping efficiency and accuracy.

[0074] In one embodiment, step 204 includes: after the robot arrives at the destination, the navigation system obtains the real-time status information of the intelligent body detection, and obtains the target detected by the target detection algorithm, and then calculates the three-dimensional coordinates of the target, and corrects it according to the position of the camera and the manipulator; the navigation system sends the path planning results, current position information and environmental parameters to the manipulator control system; the manipulator control system receives the path planning results, current position information and environmental parameters and performs real-time analysis, and generates corresponding control instructions based on the analysis results; the control instructions include parameters describing the motion trajectory, speed and strength of the manipulator; the manipulator control system sends the control instructions to the manipulator to guide the manipulator to perform specific actions; during the execution process, the manipulator will feedback the execution status and results to the manipulator control system in real time; when encountering obstacles or environmental changes, the manipulator control system re-evaluates the current situation and adjusts the control instructions and parameters based on the feedback from the manipulator and the new input of the detection algorithm.

[0075] Specifically, after the navigation system reaches the destination, it obtains the target detected by the YOLOv8 detection algorithm for target detection, then calculates its target three-dimensional coordinates, and performs position correction according to the relative position of the camera and the manipulator. The YOLOv8 algorithm uses a convolutional neural network to extract feature information from the image, and then matches these features with the preset detection box. Through gridded prediction methods and bounding box regression technology, the algorithm can accurately locate the target object. At the same time, it also classifies the target and evaluates its confidence to ensure the accuracy and stability of the detection results. The navigation system is responsible for planning the optimal path from the starting point to the end point, and perceives environmental changes in real time, such as obstacle locations, road conditions, etc. This information is processed by high-precision sensors and algorithms to form a comprehensive understanding of the environment. At the same time, the detection algorithm focuses on real-time monitoring of dynamic changes in the environment to ensure that the navigation system's decisions can quickly adapt to new situations. Subsequently, this information needs to be integrated and analyzed to provide accurate control instructions for the manipulator. In this process, a stable communication channel needs to be established between the navigation system and the manipulator, and a unified communication protocol and interface need to be adopted to ensure accurate data transmission. Through the designed interface, the navigation system sends key data such as path planning results, current position information, and environmental parameters to the manipulator control system. After receiving this information, the manipulator control system will immediately parse it, generate corresponding control instructions based on it, and grab the target. These instructions describe in detail how the manipulator should move, including parameters such as motion trajectory, speed, and force. Subsequently, these instructions are sent to the manipulator through the communication protocol and interface to guide it to perform specific actions. During the execution process, the manipulator will feedback the execution status and results to the control system in real time. This information is crucial for the dynamic adjustment of the control system. When encountering obstacles or environmental changes, the control system re-evaluates the current situation based on the feedback from the manipulator and the new input of the detection algorithm, and may adjust the control instructions and parameters. This closed-loop feedback mechanism ensures that the manipulator can flexibly respond to various complex situations and ensure the smooth completion of the task.

[0076] A unified communication protocol and interface are used for data transmission between the navigation system and the manipulator.

[0077] In one embodiment, the target detection algorithm is the YOLOv8 algorithm.

[0078] Specifically, the model architecture of the YOLOv8 algorithm is as follows Figure 4 As shown, Figure 4 The C2F module is composed of the outputs of different modules and the original feature map. The C2F module can better aggregate multi-scale information, and the CBS module is used to extract and transform features. The convolution operation superimposes a small-sized filter on the image to extract local features in the image. The calculation formula is as follows: (5) (6) Among them, the size of the input image is (H, W, C) (H represents the height of the input, W represents the width of the input), represents the height of the convolution kernel, Represents the width of the convolution kernel, and there is padding P and stride S.

[0079] It should be understood that although Figure 1 , Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0080] In a specific embodiment, Figure 5 As shown, the specific steps of the robot object picking and delivering method include: Step T1: User input. The user uses text to input content according to personal needs, such as "deliver documents to the office", "deliver snacks to the lobby", and other behaviors.

[0081] Step T2: large model call: The system calls the large model module in this module.

[0082] Step T3: Large model instruction parsing. The dense feature word vector generated in the semantic understanding module and the sparse features generated using the user input text are input into the entity extraction and intent recognition joint model to obtain the entities and related intents of the user input text.

[0083] Step T4: Navigation system. By integrating image matching technology and semantic parsing technology, the system acquires visual information with the help of cameras and uses feature matching algorithms to identify and track key points in the environment, accurately self-positions, intelligently plans routes of action, deftly avoids obstacles, and identifies important navigation signs.

[0084] Step T5: 3D mapping. The 3D mapping module performs denoising, filtering, and compression on the point cloud data generated by sensors such as lidar, and extracts key features such as edges, corners, and planes from the processed data. The extracted features are combined with algorithms to reconstruct a 3D model of the surrounding environment to achieve accurate obstacle recognition and path planning.

[0085] Step T6: Motion control. Combine the camera and the position of the manipulator for correction. Through stable communication, the navigation system sends the path, position and environment data to the manipulator control system. The control system parses the data and generates control instructions to guide the manipulator to grasp the target.

[0086] Step T7: Reinforcement learning. The agent will continuously monitor the real-time status information from sensors and the environment, and quickly select and execute the optimal grasping action based on this information.

[0087] Step T8: Target grasping. The algorithm model output information is converted to the basic coordinate system of the robot arm for fine path planning. The robot arm moves according to the planned path and activates the end effector to prepare for grasping. During grasping, real-time feedback adjustment ensures stability and accuracy. After success, the robot arm smoothly transports the object to the specified position, maintaining high stability and accuracy. After the task is completed, the system evaluates the grasping success rate, path efficiency, and motion smoothness.

[0088] Step T9: Detection algorithm. The detection algorithm obtains the image frame to be detected through the camera for preprocessing operations, and uses the convolutional neural network to extract features. The detection algorithm will identify and locate the objects in the image, then calculate the three-dimensional coordinates of the detected object and output the detection results to the robotic arm.

[0089] In one embodiment, a robot is provided, which includes a navigation system, a three-dimensional mapping module, a manipulator control system, and a manipulator arm. The robot uses any of the above-mentioned object picking and delivery methods to grab objects.

[0090] Specifically, the robot proposed in this application has improved the independence and interchangeability of each module through modular design and loosely coupled system architecture, reducing the overall complexity of the system. This design makes it easier to add new functions and fix errors, while also reducing development and maintenance costs. This is of great significance for promoting the popularization and application of robot technology.

[0091] The robot semantic navigation and object grasping technology based on large model control has broad application prospects and important research value. It can not only improve the intelligence level of robots in the fields of home services, logistics distribution, medical care, etc., but also provide strong support for the further development of artificial intelligence technology. Through the implementation of the present invention, the shortcomings of traditional robots in semantic understanding and autonomous navigation can be effectively solved, and the robot technology can be promoted to develop in a more intelligent, autonomous and flexible direction.

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A semantic navigation method based on a large model, characterized in that: The method comprises: Get the user's input text; Using a large model to semantically understand and parse the input text, generating corresponding task instructions according to the obtained semantic parsing results, and controlling the robot to execute the task instructions; The navigation system acquires images of the robot's surroundings and uses feature matching methods to identify and track key points in the environment, achieve self-positioning, intelligently plan routes, avoid obstacles, and identify important navigation signs; The navigation system reconstructs a three-dimensional model of the surrounding environment through three-dimensional mapping, understands the complex surrounding environment based on the three-dimensional model, and achieves accurate positioning and navigation.

2. The semantic navigation method based on a large model according to claim 1, characterized in that: The large model includes: an encoder and a decoder; The input text is semantically understood and parsed using a large model, and corresponding task instructions are generated according to the obtained semantic parsing results, and the robot is controlled to execute the task instructions, including: Encoding the input text into a vector sequence using an encoder; the vector sequence includes semantic information of the text; Using a decoder to combine context information, grammatical rules and prior knowledge on the vector sequence, the context is understood and parsed to obtain a semantic parsing result; Generate corresponding task instructions according to the semantic analysis results, and control the robot to execute the task instructions.

3. The semantic navigation method based on a large model according to claim 1, characterized in that: The feature matching method is the SuperGlub algorithm.

4. The semantic navigation method based on a large model according to claim 1, characterized in that: The navigation system reconstructs a 3D model of the surrounding environment through 3D mapping, understands the complex surrounding environment based on the 3D model, and achieves accurate positioning and navigation, including: The navigation system removes noise, filters, and compresses the point cloud data generated by the LiDAR sensor, and extracts key features from the processed data; the key features include edges, corners, and planes; Reconstructing a three-dimensional model of the surrounding environment using a preset reconstruction algorithm according to the key features; The complex surrounding environment is understood based on the three-dimensional model to achieve accurate positioning and navigation.

5. A method for picking up and delivering an item, characterized in that: The method comprises: An intelligent agent is constructed using a reinforcement learning framework. The intelligent agent continuously monitors the real-time status information from the sensors and environment carried by the robot, and selects and executes the optimal grasping action based on the real-time status information. The intelligent agent learns the optimal strategy for detecting and grasping the target by interacting with the target object and the robotic arm. The robot's navigation system navigates to a destination by using the semantic navigation method based on a large model as described in any one of claims 1 to 4; After the robot reaches the destination, the navigation system obtains the real-time status information of the intelligent body detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator. The navigation system sends the output information to the manipulator control system, which parses the received output information and generates control instructions to guide the manipulator to grasp the target; the output information includes the path planning result, the current position information and the environmental parameters; The robot control system executes the control instructions to complete the object picking and delivering action.

6. The article picking and delivering method according to claim 5, characterized in that: The intelligent agent is constructed using a reinforcement learning framework. The intelligent agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information, including: The process of the robot's mechanical arm grasping an object is modeled as a Markov decision process; the Markov decision process is a set of four elements , where S is the state space and A is the action space. is the transfer function, is the reward function, Represents the current state, For action, Represents the agent in its current state Execute an action the new state that may be achieved later; Adopt reinforcement learning framework to build intelligent agents; During the training process of the agent, the agent receives state information and selects the optimal action; the environment gives rewards or penalties according to the effect of the action, and the agent uses the rewards to update the decision strategy, iteratively optimizes until it can accurately grasp and obtain the optimal strategy; the state information includes images and robot arm positions; the optimal state value function under the optimal strategy satisfies the Bellman optimal equation: ； The optimal action value function under the optimal strategy satisfies the Bellman optimal equation: ； in, , are the optimal state value function and the optimal action value function, respectively. represents the discount factor; After the training is completed, the intelligent agent is deployed in the actual environment. The intelligent agent continuously monitors the real-time status information from the sensors carried by the robot and the environment, and selects and executes the optimal grasping action based on the real-time status information.

7. The article picking and delivering method according to claim 5, characterized in that: The robot control system executes the control instructions to complete the object picking and delivery action, including: In the target grasping process, the robot control system converts the information to the robot arm base coordinate system to ensure accurate positioning, and then performs path planning. The robot arm moves according to the planned path and activates the end effector to prepare for grasping. Real-time feedback and adjustments are made during the grasping process. After a successful grasp, the robot arm places the object at the designated location.

8. The article picking and delivering method according to claim 5, characterized in that: After the robot reaches the destination, the navigation system obtains the real-time status information of the intelligent body detection and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects it according to the position of the camera and the manipulator. The navigation system sends the output information to the manipulator control system, which parses the received output information and generates control instructions to guide the manipulator to grasp the target, including: After the robot reaches the destination, the navigation system obtains the real-time status information detected by the intelligent agent and the target detected by the target detection algorithm, then calculates the three-dimensional coordinates of the target and corrects them according to the position of the camera and the manipulator; The navigation system sends the path planning results, current position information and environmental parameters to the manipulator control system; The manipulator control system receives the path planning results, current position information and environmental parameters, performs real-time analysis, and generates corresponding control instructions according to the analysis results; the control instructions include parameters describing the movement trajectory, speed and strength of the manipulator; The manipulator control system sends the control instruction to the manipulator to instruct the manipulator to perform specific actions; During the execution process, the robot will feedback the execution status and results to the robot control system in real time; When encountering obstacles or environmental changes, the manipulator control system re-evaluates the current situation and adjusts control instructions and parameters based on the manipulator's feedback and new inputs from the detection algorithm.

9. The article picking and delivering method according to claim 5, characterized in that: The target detection algorithm is the YOLOv8 algorithm.

10. A robot, characterized in that: The robot comprises a navigation system, a three-dimensional mapping module, a manipulator control system and a manipulator arm, and the robot adopts the object picking and delivering method according to any one of claims 5 to 9 to grasp objects.

Images