Robot movement and manipulation control method based on large model and deep reinforcement learning

Abstract

The invention discloses a robot movement and manipulation control method based on a large model and deep reinforcement learning, and belongs to the technical field of artificial intelligence and intelligent robot systems. The problems that an existing method is difficult to process complex natural language instructions, and in a dynamic environment, the cooperation efficiency of a robot moving task and a mechanical arm control task is low are solved. According to the method, the input complex natural language instruction is processed by adopting the large language model to obtain the subtask sequence, so that the understanding of the robot on the natural language instruction can be realized; through a hierarchical reinforcement learning framework, environment semantic information is deeply fused, an interpretable risk assessment mechanism is introduced, and the system is endowed with continuous learning and generalization ability; the cooperative efficiency of autonomous navigation and manipulation tasks of the mobile robot in complex, dynamic and unknown environments can be remarkably improved, and the safety and the intelligent level of the robot are guaranteed. The method can be applied to the field of robot movement and manipulation control.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and intelligent robot system technology, specifically relating to a robot movement and manipulation control method based on large models and deep reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence and the widespread application of robotics, intelligent robot systems are playing an increasingly important role in fields such as industrial automation, intelligent warehousing, service robots, and exploratory tasks. Especially in scenarios requiring complex interactions with the environment and the execution of multi-step tasks, higher demands are placed on the autonomy, intelligence, and flexibility of robots.

[0003] Existing mobile robot systems typically possess some navigation and obstacle avoidance capabilities, such as using LiDAR or visual SLAM technology to construct a geometric map of the environment, and combining it with... , Path planning algorithms are used to move within a map. Meanwhile, robotic arms on fixed or mobile robots have been used to perform various tasks, such as grasping, placing, and assembling, typically relying on fixed sequence decisions.

[0004] However, existing technologies still face many challenges when dealing with multi-step tasks in complex, unstructured environments:

[0005] 1. Currently, robot task issuance typically relies on programming, graphical interfaces, or simple instruction sets, making it difficult to handle complex task requirements expressed in natural language. Automatically breaking down advanced, ambiguous human instructions into robot-executable, time-sequential sub-task sequences is a key bottleneck in achieving truly intelligent interaction. Traditional rule-based or simple script-based methods struggle to cover diverse task instructions and lack robustness and generalization capabilities.

[0006] 2. A mobile robot system capable of performing multi-task manipulation requires a high degree of coordination between the mobile platform (such as an omnidirectional mobile vehicle), the perception system (such as cameras and LiDAR), the decision-making system (task planning), and the execution system (such as a robotic arm). How to integrate information from different modules (such as target detection results, map information, and the robot's own status) in real time and efficiently, and coordinate movement, perception, planning, and operation to ensure the successful completion of tasks, is a complex systems engineering problem. Especially in dynamic environments, path planning, target recognition and tracking, and the precise operation of the robotic arm require close cooperation.

[0007] In summary, existing methods still struggle to handle complex natural language instructions, and the coordination efficiency between robot movement and robotic arm task execution remains low in dynamic environments. Therefore, proposing a control method that can understand natural language instructions, intelligently plan and decompose complex tasks, and efficiently coordinate robot movement and robotic arm manipulation tasks is an urgent need to meet people's daily needs. Summary of the Invention

[0008] The purpose of this invention is to address the problems of existing methods' difficulty in handling complex natural language commands and the low coordination efficiency of robot movement tasks and robotic arm manipulation tasks in dynamic environments. It proposes a robot movement and manipulation control method based on large models and deep reinforcement learning. Users only need to input natural language commands to generate a robot task sequence and complete a series of tasks through automatic planning.

[0009] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a robot movement and manipulation control method based on large models and deep reinforcement learning, the method specifically including the following steps:

[0010] Step 1: The user inputs natural language commands into the large language model, and the large language model outputs a subtask sequence S based on the input natural language commands;

[0011] Step 2: Initialize subtask s=1;

[0012] Step 3: Select the next operation based on the type of the s-th subtask in the subtask sequence S, specifically:

[0013] If the s-th subtask in the subtask sequence S is a path planning task, then proceed to step four.

[0014] If the s-th subtask in the subtask sequence S is a robotic arm manipulation task, then proceed to step five.

[0015] Step 4: Use the first-layer reinforcement learning network and the second-layer reinforcement learning network to perform path planning. The robot moves to the target position according to the path planning results; then proceed to step 6.

[0016] Step 5: Use the target detection model deployed on the robot to identify the position of target objects in the environment, and calculate the target angle of each joint of the robot's robotic arm based on the position of the target objects.

[0017] Based on the target angles of each joint of the robotic arm, perform a grasping action on the target object; then proceed to step six.

[0018] Step 6: Have all the subtasks in the subtask sequence S been completed?

[0019] If there are still subtasks that have not been completed, then set s = s + 1 and return to step three.

[0020] The process ends when all subtasks have been completed.

[0021] The beneficial effects of this invention are:

[0022] This invention proposes an intelligent path planning method based on hierarchical reinforcement learning and interpretable semantic risk perception. It uses a large language model to process complex natural language input commands to obtain a sequence of sub-tasks, enabling the robot to understand natural language commands. Furthermore, by using a hierarchical reinforcement learning framework, deeply integrating environmental semantic information, introducing an interpretable risk assessment mechanism, and endowing the system with continuous learning and generalization capabilities, it can significantly improve the collaborative efficiency of mobile robots in autonomous navigation and manipulation tasks in complex, dynamic, and unknown environments, while ensuring the robot's safety and intelligence level. Attached Figure Description

[0023] Figure 1 This is a flowchart of a robot movement and manipulation control method based on large model and deep reinforcement learning according to the present invention;

[0024] Figure 2 This is an architecture diagram of a multi-task manipulating robot system. Detailed Implementation

[0025] Specific implementation method one: Combining Figure 1 This embodiment describes a robot movement and manipulation control method based on large models and deep reinforcement learning. The method specifically includes the following steps:

[0026] Step 1: The user inputs natural language commands into the large language model, and the large language model outputs a subtask sequence S based on the input natural language commands;

[0027] Step 2: Initialize subtask s=1;

[0028] Step 3: Based on the type of the s-th subtask in the subtask sequence S (subtask types only include path planning tasks and manipulation tasks), select the next operation, specifically:

[0029] If the s-th subtask in the subtask sequence S is a path planning task, then proceed to step four.

[0030] If the s-th subtask in the subtask sequence S is a robotic arm manipulation task, then proceed to step five.

[0031] Step 4: Use the first-layer reinforcement learning network and the second-layer reinforcement learning network to perform path planning. The robot moves to the target position according to the path planning results; then proceed to step 6.

[0032] Step 5: Use the target detection model deployed on the robot to identify the position of target objects in the environment, and calculate the target angle of each joint of the robot's robotic arm based on the position of the target objects.

[0033] Based on the target angles of each joint of the robotic arm, perform a grasping action on the target object; then proceed to step six.

[0034] Step 6: Have all the subtasks in the subtask sequence S been completed?

[0035] If there are still subtasks that have not been completed, then set s = s + 1 and return to step three.

[0036] The process ends when all subtasks have been completed.

[0037] Based on the characteristics of omnidirectional mobile platforms and multi-degree-of-freedom robotic arms, a mobile manipulation robot system equipped with a deep learning model is constructed. The multi-task manipulation robot system includes a vehicle platform, a robotic arm platform, and two computing devices, which communicate and collaborate to complete tasks via the ROS framework. The overall system hardware structure and communication process are as follows: Figure 2 As shown.

[0038] The core of the robotic system lies in the high degree of coordination between the mobile platform, perception system, task planning system, and robotic arm grasping. The robot can perform environmental perception and task decision-making, achieving complex multi-task manipulation functions based on natural language commands. Once the task sequence is generated, message passing and service calls between modules are implemented using the ROS framework.

[0039] (1) Motion and navigation execution: Based on the navigation sub-tasks in the task sequence, the system relies on a pre-built environmental semantic map and combines multi-sensor fusion technology (such as data from lidar, IMU, wheel encoders, etc.) to achieve accurate positioning and navigation. It can quickly generate an initial path from the current location to the target area. The first layer of reinforcement learning network will perform global planning based on the initial path, and the second layer of reinforcement learning network will then fine-tune the path based on the global path planning results.

[0040] (2) Visual guidance and target localization: After the robot reaches the predetermined target area, it uses a camera and depth sensor to collect environmental image data and applies a lightweight YOLOv5 target detection model based on deep learning to quickly identify the category and location of target objects in the image. Then, by combining the camera's intrinsic and extrinsic parameters, it accurately calculates the three-dimensional position and orientation of the target object in the coordinate system of the robot arm base, providing accurate spatial information support for the robot arm's grasping.

[0041] (3) Robotic arm grasping / placing: Based on the target pose information provided by the vision system, the robotic arm control system uses its kinematic model inverse kinematics to calculate the target angle of each joint and drives the robotic arm to accurately perform grasping and other operations.

[0042] (4) System Status Feedback and Integration: During the entire task execution process, each functional module can display status information (such as navigation completion status, target detection results, start and end information of capture execution, etc.) through the ROS message mechanism. The system can comprehensively judge the execution effect of each sub-task (success, failure, or need to be retried, etc.) based on the status information, dynamically adjust the task status, and trigger subsequent sub-tasks in sequence according to the preset task sequence logic, so as to ensure the continuity and efficiency of the entire multi-task operation process, and ensure the stable operation of the overall system and the safety of personnel and equipment.

[0043] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the large language model is a GLM, and the input of the large language model also includes an environmental semantic map of the target region.

[0044] The other steps and parameters are the same as in Specific Implementation Method 1.

[0045] To enable robots to understand human natural language commands and execute complex multi-step tasks, this invention introduces a domestically developed open-source large-scale model, GLM, for task understanding and planning. Users issue high-level commands (u) to the system using natural language. Based on its understanding of the commands and its knowledge of the semantic map of the current environment, the GLM model automatically decomposes the complex high-level commands (u) into a series of ordered, robot-executable basic sub-task sequences (S). For example, with the input "Please pick up the red book on the table and put it on the bookshelf," the large-scale language model parses the command (u) to identify key information such as the intent, action (pick up, place), target object (book, bookshelf), and its attributes (red, on the table, on the bookshelf). The output task sequence S is [move, classroom][operate, red book].

[0046] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the specific process of step four is as follows:

[0047] Step 41, adopt The algorithm plans a path from the robot's current position to the target location of the s-th subtask. The first layer of the reinforcement learning network is based on... Global path planning is performed using the algorithm's planning results and current environment information;

[0048] Step 4.2: The second-layer reinforcement learning network fine-tunes the global path planning result based on the global path planning result and the perceived local environment information. The robot then moves towards the target location according to the fine-tuned path.

[0049] Other steps and parameters are the same as in specific implementation method one or two.

[0050] This invention proposes an intelligent path planning method based on hierarchical reinforcement learning and interpretable semantic risk perception. Specifically, it includes a high-level policy network (HLP, i.e., the first-layer reinforcement learning network) and a low-level policy network (LLP, i.e., the second-layer reinforcement learning network). The HLP is responsible for making long-term abstract decisions based on global task and macro-environmental information, selecting navigation strategy modes or setting phased sub-goals. The LLP is responsible for short-term real-time motion control based on instructions issued by the HLP and local fine-grained environmental perception.

[0051] After actual deployment, the robot continuously collects data. The experience tuples are stored in the experience replay pool. Using this real data, the parameters of the pre-trained first-layer and second-layer reinforcement learning models can be periodically or under specific triggering conditions (such as task failure). Fine-tuning is then performed. The update process of the policy network (Actor) can be represented as:

[0052]

[0053] The update process of the value network (Critic) can be represented as follows:

[0054]

[0055] in, It is an experience replay pool.

[0056] Exploration Strategy: Combining Greedy and UCB (Upper Confidence Bound) encourage robots to explore unknown states and semantic configurations in the environment safely in order to collect more valuable training data and accelerate their adaptation to specific work environments.

[0057] Policy network input: These semantic embeddings are used as part of the state input to the policy network (especially the first-layer reinforcement learning network). The first-layer reinforcement learning network learns not isolated semantic labels, but the semantic concepts represented by these embedding vectors.

[0058] Generalization mechanism: When encountering new objects or regions that were not explicitly shown in the training, if their semantic embeddings are similar to the embeddings of known high-risk / low-risk objects or regions in the vector space, the model can generalize the learned navigation strategy based on this similarity. For example, if the robot learns to be cautious about "fragile item A", it can adopt a similar strategy when encountering a new "fragile item B" (whose embedding is similar to A).

[0059] This invention overcomes the limitations of traditional path planning methods in complex dynamic environments, improving the robot's autonomous navigation capabilities, safety, and environmental adaptability in unknown and uncertain situations. It offers the following advantages:

[0060] (1) Hierarchical decision-making framework: simulates the human navigation mode of "strategy first, then execution", decomposes complex navigation tasks into high-level strategy selection and low-level action execution, improves planning efficiency and the ability to cope with complex scenarios.

[0061] (2) Deep semantic fusion: Not only does it use semantic information to identify objects, but it also deeply binds them to navigation risks and behavioral constraints, enabling robots to "understand" the potential dangers and specific requirements of the environment.

[0062] (3) Explainable risk perception: By explicitly modeling semantic risks in the reinforcement learning reward function, the robot's path selection behavior becomes more explainable and can take proactive avoidance or caution strategies based on risk assessment.

[0063] (4) Active adaptation and continuous learning: enable robots to continuously optimize navigation strategies through interaction with the environment and generalize to new environments and new semantics.

[0064] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the agent state space of the first-layer reinforcement learning network is defined as... ;

[0065] in, It is the target location of the s-th sub-task The semantic category (the type of the target location needs to be mapped to a numerical code, for example, when the target location is an office area, the corresponding numerical code is 1, and when the target location is a staircase, the corresponding numerical code is 2. The numerical code represents the semantic category of the location) and the target location. A vector composed of the coordinates of the target location (i.e., the entrance coordinates of the target location);

[0066] Indicates in The algorithm's planning results show the distance from the robot's current position to the target location. A semantic category sequence consisting of all regions that need to be traversed; it should be noted that the present invention uses the robot's centroid coordinates as the robot's current position;

[0067] It is a vector composed of the robot's current position coordinates and the robot's current attitude angle; in this invention, the robot attitude angle is defined as the angle between the robot's forward direction and the X-axis of the global coordinate system;

[0068] This indicates the historical average travel time required for the robot to traverse the current area;

[0069] The action space of the agent in the first layer of the reinforcement learning network is defined as follows: ;

[0070] in, This indicates the sequence of the robot's actions in the current step, specifically the 1st, 2nd, ..., 3rd elements in the environmental semantic map that it needs to traverse sequentially. Each region (i.e., all the regions that the robot needs to pass through in sequence from its current location to the target location).

[0071] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0072] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the reward function used during the training of the first-layer reinforcement learning network is... for:

[0073]

[0074] in: This indicates an indicator function. If the s-th subtask is successfully completed, the indicator function value is 1; otherwise, the indicator function value is 0.

[0075] This represents the navigation efficiency of planning a path from the starting point to the target location based on the actions output by the agent.

[0076] The navigation efficiency can be obtained by dividing the straight-line distance from the starting point to the target position by the time required for the robot to move along the planned path from the starting point to the target position.

[0077] This represents the semantic risk assessment value output by the agent along the path from the starting point to the target location;

[0078] This represents the time required for the robot to move from the starting point to the target location along the planned path of the agent;

[0079] , , and These are the weighting coefficients for each item.

[0080] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0081] The reward function designed in this embodiment is used to guide the first-layer reinforcement learning network to make optimal macro-global decisions.

[0082] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One through Five in that the state space of the second-layer reinforcement learning network is defined as... ;

[0083] in, It is a vector composed of the channel entrance position coordinates of a sub-target in the global path planning result and the semantic category of the sub-target; In this invention, each sub-target in the output action of the first layer reinforcement learning network needs to be sent to the second layer reinforcement learning network in sequence to be added to the state of the agent of the second layer reinforcement learning network at each step;

[0084] It is a vector composed of the position coordinates and velocity of dynamic obstacles obtained from local environment perception information (through LiDAR or depth camera);

[0085] It is a vector composed of the robot's current position coordinates (with the robot's center of mass coordinates as the robot's position coordinates), linear velocity, angular velocity, and the wheel velocities of each Mecanum wheel of the robot.

[0086] Linear velocity includes velocity in the forward / backward direction. Lateral translation speed For small to medium-sized indoor robots, the maximum speed is usually... arrive Between. Therefore, a reasonable range could be Due to the characteristics of Mecanum wheels, lateral movement may be slightly less than forward movement, or its maximum value may be limited for stability reasons. For example... Robots typically possess good in-situ rotation capabilities, with angular velocities ranging from [value missing]. radians per second;

[0087] It is a vector consisting of the labels of objects around the robot (the labels are obtained by mapping objects to specific numerical values ​​according to their categories) and the confidence scores of the objects around the robot.

[0088] The robot uses a depth camera to acquire images of the robot's surroundings, and then uses a lightweight object detection network to detect surrounding objects, while outputting the confidence scores of the objects.

[0089] The actions output by the agent in the second layer of the reinforcement learning network are the robot's desired linear velocity and desired angular velocity.

[0090] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0091] The action space of the second-layer reinforcement learning network is: , This directly corresponds to the robot's three main degrees of freedom in its body coordinate system. The unique structure of the Mecanum wheel allows the robot to independently control these three velocity components, enabling forward, backward, lateral, diagonal, and stationary rotation. The second-layer reinforcement learning network can learn how to combine these motion primitives to complete the navigation task most efficiently and safely under different local environments and first-layer instructions. For example, in a narrow passage, the second-layer reinforcement learning network might learn to use lateral movement more frequently. ) rather than a major shift ( The second-layer reinforcement learning network adjusts its position accordingly. When it needs to quickly avoid lateral dynamic obstacles, it can rapidly output larger values. instruction.

[0092] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the reward function used during the training of the second-layer reinforcement learning network is... for:

[0093]

[0094] in: Indicates navigation efficiency;

[0095] Calculate the distance between the robot's position and the target position before executing the current step action output by the agent. Then calculate the distance between the robot's position and the target position after executing the current step action output by the agent. Subtract the calculated distances Divide the difference by the initial distance between the starting point of the current subtask and the target point. The calculation result is the navigation efficiency. .Right now:

[0096]

[0097] The distance between the robot's position and the target position after executing the current step is normalized to a ratio relative to the initial distance, facilitating reward consistency across tasks of different scales. When the robot moves towards the target, , When positive; when far from the target, It is negative. Calculate once at the start of each subtask and keep it constant until the subtask completes or switches. If the subtask target point is very close to the starting point, it will be too small. May affect navigation efficiency The value fluctuates greatly, when When the value is less than the set threshold, it is necessary to... The value is adjusted to a threshold to avoid large fluctuations in navigation efficiency.

[0098] This indicates an indicator function. If a collision occurs after the agent executes the action output in the current step, the indicator function value is 1; otherwise, the indicator function value is 0.

[0099] This represents the local semantic risk assessment value of the robot executing the action output by the agent in the current step;

[0100] This represents the shortest vertical distance from the robot's current position to the path planned by the first layer of the reinforcement learning network.

[0101] These are the weighting coefficients for each item.

[0102] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0103] The reward function designed in this embodiment is used to guide the second-layer reinforcement learning network to generate smooth, safe, and efficient local trajectories.

[0104] Specific Implementation Method Eight: This implementation method differs from one of Specific Implementation Methods One to Seven in that the semantic risk assessment value... and The calculation method is as follows:

[0105]

[0106] in: This represents the global path output by the agent in the first layer of the reinforcement learning network;

[0107] Representing a path The first The semantic category of each sub-target region;

[0108] express The corresponding basic risk value; when there is a glass door in a sub-target area to be passed through, basic risk values ​​can be set for the open and closed states of the glass door respectively. For other types of sub-target areas, a fixed basic risk value can be set for each.

[0109] express The confidence level (i.e., the corresponding confidence level can be obtained when using a lightweight object detection network to identify the semantic map of the environment);

[0110]

[0111] in: This represents the path required to execute the current step action output by the agent;

[0112] Indicates the robot's current position;

[0113] Indicated by position Centered on, with A circular region with radius [missing information]; The value of is related to the robot's speed and reaction time. This is the product of the robot's speed and reaction time.

[0114] Representing a path Sub-regions within the upper and circular areas;

[0115] Subregion The corresponding basic risk value;

[0116] Subregion The confidence level can be obtained by using a lightweight object detection network to identify the semantic map of the environment.

[0117] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0118] By and By introducing a reward function as a negative reward (penalty), reinforcement learning agents will naturally learn to avoid high semantic risk areas or take more cautious actions in high-risk areas (e.g., lower-level policies output at a lower speed) in the process of maximizing cumulative rewards.

[0119] Because semantic risk is explicitly modeled, when the robot chooses a path that is not optimal in the traditional sense (such as a non-shortest path), the interpretability of the path planning results can be enhanced by analyzing the reward composition in the decision-making process of the first-layer reinforcement learning network to determine whether the choice was made to avoid a high-semantic-risk area.

[0120] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One to Eight in that the target detection model in step five is a lightweight YOLOv5 target detection model. The lightweight YOLOv5 target detection model is the YOLOv5 network obtained by replacing the YOLOv5 backbone network with the MobileNetV3 model.

[0121] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0122] This implementation uses deep learning-based object detection technology to provide the robotic arm with accurate target pose information. To further optimize model performance, the original backbone network of the YOLOv5 object detection model is replaced with a MobileNetV3 network structure, resulting in a lightweight YOLOv5 object detection model. This lightweight YOLOv5 object detection model can maintain high detection accuracy while reducing model size and computational load, enabling faster inference speed and lower computational resource requirements, making it suitable for deployment on mobile robot embedded platforms. The robot uses its onboard depth camera to acquire environmental images in real time. These images are input into the lightweight YOLOv5 model for processing, which identifies the categories of objects in the environment and their positions in the images. Combined with the robot's real-time localization information (which can be provided by technologies such as SLAM) and camera intrinsic and extrinsic parameters, this provides support for task planning and robotic arm grasping.

[0123] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One to Nine in that the specific process of step five is as follows:

[0124] Step 51: Use the depth camera deployed on the robot to acquire color images (img) and depth information in real time. The color images (img) are then processed by a lightweight YOLOv5 object detection model to obtain the two-dimensional bounding box containing the target location. ;

[0125] Based on the two-dimensional bounding box where the target location is located By combining the depth information with the intrinsic parameters of the depth camera, the coordinates of the target position in three-dimensional space can be obtained. ;

[0126] Step 52: Set the target position in three-dimensional space coordinates As expected crawling point Based on the inverse algorithm and the desired capture point Calculate the target angle vectors of each joint of the robotic arm. , These represent the target angles of the 1st, 2nd, ..., Mth joints of the robotic arm, respectively, where M represents the total number of joints in the robotic arm.

[0127] Step 53: Perform the grasping action according to the target angle of each joint of the robotic arm.

[0128] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0129] This implementation effectively integrates visual recognition and spatial positioning, significantly improving the accuracy and robustness of the system in grasping targets in complex scenarios, and also laying the foundation for multi-task manipulators to effectively execute task sequences.

[0130] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A robot movement and manipulation control method based on large models and deep reinforcement learning, characterized in that, The method specifically comprises the following steps: Step 1: The user inputs natural language commands into the large language model, and the large language model outputs a subtask sequence S based on the input natural language commands; Step 2: Initialize subtask s=1; Step 3: Select the next operation based on the type of the s-th subtask in the subtask sequence S, specifically: If the s-th subtask in the subtask sequence S is a path planning task, then proceed to step four. If the s-th subtask in the subtask sequence S is a robotic arm manipulation task, then proceed to step five. Step 4: Use the first-layer reinforcement learning network and the second-layer reinforcement learning network to perform path planning. The robot moves to the target position according to the path planning results; then proceed to step 6. Step 5: Use the target detection model deployed on the robot to identify the position of target objects in the environment, and calculate the target angle of each joint of the robot's robotic arm based on the position of the target objects. Based on the target angles of each joint of the robotic arm, perform a grasping action on the target object; then proceed to step six. Step 6: Have all the subtasks in the subtask sequence S been completed? If there are still subtasks that have not been completed, then set s = s + 1 and return to step three. The process ends when all subtasks have been completed.

2. The robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 1, characterized in that, The large language model is a GLM, and the input of the large language model also includes an environmental semantic map of the target region.

3. The robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 2, characterized in that, The specific process of step four is as follows: Step 41, adopt The algorithm plans a path from the robot's current position to the target location of the s-th subtask. The first layer of the reinforcement learning network is based on... Global path planning is performed using the algorithm's planning results and current environment information; Step 4.2: The second-layer reinforcement learning network fine-tunes the global path planning result based on the global path planning result and the perceived local environment information. The robot then moves towards the target location according to the fine-tuned path.

4. The robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 3, characterized in that, The agent state space of the first layer reinforcement learning network is defined as follows: ; in, It is the target location of the s-th sub-task semantic category and target location A vector composed of the coordinates; Indicates in The algorithm's planning results show the distance from the robot's current position to the target location. A semantic category sequence consisting of all regions that need to be traversed; It is a vector composed of the robot's current position coordinates and the robot's current attitude angle; This indicates the historical average travel time required for the robot to traverse the current area; The action space of the agent in the first layer of the reinforcement learning network is defined as follows: ; in, This indicates the sequence of the robot's actions in the current step, specifically the 1st, 2nd, ..., 3rd elements in the environmental semantic map that it needs to traverse sequentially. Each region.

5. The robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 4, characterized in that, The reward function used during the training of the first layer reinforcement learning network for: in: This indicates an indicator function. If the s-th subtask is successfully completed, the indicator function value is 1; otherwise, the indicator function value is 0. This represents the navigation efficiency of planning a path from the starting point to the target location based on the actions output by the agent. This represents the semantic risk assessment value output by the agent along the path from the starting point to the target location; This represents the time required for the robot to move from the starting point to the target location along the planned path of the agent; , , and These are the weighting coefficients for each item.

6. The robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 5, characterized in that, The state space of the second-layer reinforcement learning network is defined as follows: ; in, It is a vector composed of the portal coordinates of a sub-objective in the global path planning result and the semantic category of the sub-objective; It is a vector composed of the position coordinates and velocity of dynamic obstacles obtained from local environmental perception information; It is a vector composed of the robot's current position coordinates, linear velocity, angular velocity, and the wheel velocities of each Mecanum wheel of the robot; It is a vector consisting of the labels of objects around the robot and the confidence levels of those objects. The actions output by the agent in the second layer of the reinforcement learning network are the robot's desired linear velocity and desired angular velocity.

7. The robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 6, characterized in that, The reward function used during the training of the second-layer reinforcement learning network for: in: Indicates navigation efficiency; This indicates an indicator function. If a collision occurs after the agent executes the action output in the current step, the indicator function value is 1; otherwise, the indicator function value is 0. This represents the local semantic risk assessment value of the robot executing the action output by the agent in the current step; This represents the shortest vertical distance from the robot's current position to the path planned by the first layer of the reinforcement learning network. These are the weighting coefficients for each item.

8. The robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 7, characterized in that, The semantic risk assessment value and The calculation method is as follows: in: This represents the global path output by the agent in the first layer of the reinforcement learning network; Representing a path The first The semantic category of each sub-target region; express The corresponding basic risk value; express Confidence level; in: This represents the path required to execute the current step action output by the agent; Indicates the robot's current position; Indicated by position Centered on, with A circular region with radius [missing information]; Representing a path Sub-regions within the upper and circular areas; Subregion The corresponding basic risk value; Subregion The confidence level.

9. A robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 8, characterized in that, The target detection model in step five is a lightweight YOLOv5 target detection model, which is the YOLOv5 network obtained by replacing the YOLOv5 backbone network with the MobileNetV3 model.

10. A robot movement and manipulation control method based on large model and deep reinforcement learning according to claim 9, characterized in that, The specific process of step five is as follows: Step 51: Use the depth camera deployed on the robot to acquire color images (img) and depth information in real time. The color images (img) are then processed by a lightweight YOLOv5 object detection model to obtain the two-dimensional bounding box containing the target location. ; Based on the two-dimensional bounding box where the target location is located By combining the depth information with the intrinsic parameters of the depth camera, the coordinates of the target position in three-dimensional space can be obtained. ; Step 52: Set the target position in three-dimensional space coordinates As expected crawling point Based on the inverse algorithm and the desired capture point Calculate the target angle vectors of each joint of the robotic arm. , These represent the target angles of the 1st, 2nd, ..., Mth joints of the robotic arm, respectively, where M represents the total number of joints in the robotic arm. Step 53: Perform the grasping action according to the target angle of each joint of the robotic arm.

Images