Robot control method and device, equipment and computer readable storage medium

Abstract

The invention discloses a robot control method and device, equipment and a computer readable storage medium. The method comprises the steps that a user control instruction for a robot is acquired; the user control instruction is analyzed through a large model, a task planning strategy is generated based on an analysis result, the task planning strategy comprises N tasks to be executed by the robot, and N is a positive integer; for each task in the N tasks, calling an atomic skill model corresponding to the task through the large model to control the robot to execute the task according to a first execution strategy; and monitoring an execution result of the task in the task execution process, if the execution result is failure, determining a task failure type, and recalling the atomic skill model corresponding to the task according to the task failure type to control the robot to execute the task according to a second execution strategy.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a robot control method and apparatus, equipment, and computer-readable storage medium. Background Art

[0002] With the rapid development of intelligent manufacturing and automation technologies, robots are increasingly being used in industrial production. However, existing robotic operating systems rely on predefined fixed strategies or limited feedback mechanisms, leaving many unresolved issues, particularly regarding uncertainty and failure handling during execution. Summary of the Invention

[0003] To solve the above technical problems, the present application provides a robot control method and device, equipment, and computer-readable storage medium.

[0004] The robot control method provided by the present application comprises:

[0005] Get user control instructions for the robot;

[0006] The user control instructions are parsed through the large model, and a task planning strategy is generated based on the parsing results. The task planning strategy includes N tasks to be performed by the robot, where N is a positive integer;

[0007] For each of the N tasks, the atomic skill model corresponding to the task is called by the large model to control the robot to perform the task according to the first execution strategy; during the task execution process, the execution result of the task is monitored. If the execution result is a failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to the second execution strategy.

[0008] The robot control device provided by the present application comprises:

[0009] An acquisition unit, used for acquiring user control instructions for the robot;

[0010] The processing unit is used to parse the user control instructions through the large model, and generate a task planning strategy based on the parsing results. The task planning strategy includes N tasks to be performed by the robot, where N is a positive integer; for each of the N tasks, the large model is used to call the atomic skill model corresponding to the task to control the robot to perform the task according to the first execution strategy; during the task execution process, the execution result of the task is monitored. If the execution result is failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to the second execution strategy.

[0011] The robot control device provided in the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the above-mentioned robot control method.

[0012] The computer-readable storage medium provided in the present application is used to store a computer program, and the computer program enables a computer to execute the above-mentioned robot control method.

[0013] In the technical solution of the present application, user control instructions for the robot are obtained; the user control instructions are parsed through a large model, and a task planning strategy is generated based on the parsing results. The task planning strategy includes N tasks to be performed by the robot, where N is a positive integer; for each of the N tasks, the atomic skill model corresponding to the task is called by the large model to control the robot to perform the task according to the first execution strategy; during the task execution process, the execution result of the task is monitored. If the execution result is a failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to the second execution strategy. In this way, the task execution strategy can be accurately obtained by parsing the user control instructions through the large model, and then the task is executed according to the second execution strategy for the task with a failed execution result, which can ensure the continuity of task execution and improve the success rate of the entire task completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0015] Figure 1 1 is a flow chart of a robot control method provided in an embodiment of the present application;

[0016] Figure 2 Schematic diagram of the structure of the robot control device provided in an embodiment of the present application;

[0017] Figure 3 This is a schematic structural diagram of a robot control device provided in an embodiment of the present application;

[0018] Figure 4 It is a schematic structural diagram of the chip of an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here. The term "and / or" herein is merely a description of an association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " herein generally indicates that the objects associated before and after are in an "or" relationship. It should also be understood that the "indication" mentioned in the embodiments of the present application may be a direct indication, an indirect indication, or an indication of an association relationship. For example, "A indicates B" can mean that A directly indicates B, for example, B can be obtained through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also mean that there is an association relationship between A and B. It should also be understood that the "correspondence" mentioned in the embodiments of this application can mean that there is a direct or indirect correspondence relationship between the two, it can also mean that there is an association relationship between the two, and it can also mean a relationship between indicating and being indicated, configuring and being configured, etc.

[0021] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following relevant technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all fall within the protection scope of the embodiments of the present application.

[0022] With the rapid development of intelligent manufacturing and automation technologies, robots are increasingly being used in industrial production. However, existing robotic operating systems rely on predefined fixed strategies or limited feedback mechanisms, leaving many unresolved issues, particularly regarding uncertainty and failure handling during execution. In practical applications, robots face the following challenges when performing tasks:

[0023] Inconsistency between task planning and execution: Task planning generated by a large multimodal model may encounter problems such as dynamic changes in the environment and changes in object positions during execution, leading to execution failure.

[0024] Complex reasons for execution failure: During the execution of a task, the robot may face various failure situations such as grasping failure, inaccessible workspace, placement failure or obstacle blocking, which have a negative impact on the successful completion of the task.

[0025] Inadequate failure handling mechanisms: Existing systems often lack effective failure diagnosis and correction mechanisms, resulting in the entire process being interrupted and inefficiency once a task fails.

[0026] Uncertainty in task execution: Even a high-precision motion execution system cannot guarantee that every task can be completed efficiently and successfully when faced with complex scenarios or changing operational requirements.

[0027] Therefore, how to accurately and efficiently enable the robot to complete the corresponding task becomes a problem that needs to be considered. To this end, the following technical solutions of the embodiments of the present application are proposed.

[0028] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0029] It should be noted that in the embodiments of the present application, the "N tasks" may be performed by one or more mechanical components of the robot. In some embodiments, one mechanical component of the robot performs the N tasks. In other embodiments, multiple mechanical components of the robot perform the N tasks. Here, the mechanical components include but are not limited to robotic arms.

[0030] Figure 1 is a flow chart of the robot control method provided in the embodiment of the present application, such as Figure 1 As shown, the robot control method includes the following steps:

[0031] Step 101: Obtain user control instructions for the robot.

[0032] In some embodiments, a user control instruction for the robot is received. The user control instruction can be a control instruction input by the user's voice or a control instruction input by the user through text. This application does not make specific limitations on this.

[0033] In some embodiments, when a user inputs a control instruction using voice, the voice needs to be converted into text information. For example, the voice is converted into text using an automatic speech recognition (ASR) algorithm.

[0034] For example, the user control instruction is "grab the red cup on the table, rotate it and place it on the right shelf."

[0035] Step 102: parse the user control instructions through the large model, and generate a task planning strategy based on the parsing results. The task planning strategy includes N tasks to be performed by the robot, where N is a positive integer.

[0036] In some embodiments, the large model is a multimodal large model (Visual-Language Model, VLM). The VLM can integrate vision, language, and motion modalities to understand user instructions, recognize and analyze scene information, generate task plans, and control robot execution.

[0037] In some embodiments, generating a task planning strategy based on the analysis results includes: generating a task planning strategy based on the analysis results and visual information in the environment where the robot is located.

[0038] In some embodiments, after obtaining the user control instruction, the user control instruction is parsed using the large model, and a specific task plan, ie, a task planning strategy, is generated in combination with the visual information in the environment in which the robot is located.

[0039] In some embodiments, visual information of the robot's environment is obtained through cameras and sensors.

[0040] In some embodiments, visual information is acquired through two cameras, one of which can be deployed on the robot. For example, when a robotic arm is used to perform N tasks, the camera can be deployed at the end of the robotic arm to provide a local view of the area in front of the robotic arm's gripper. The other camera can be placed in the environment where the robot performs the task, providing a global view of the robot's operation. The specific number and location of cameras can be determined based on actual circumstances and are not specifically limited in this application.

[0041] In some embodiments, during the task planning stage, the large model can identify the position and state of objects in the workspace based on visual information, and decompose N tasks to be performed by the robot based on user needs, where N is a positive integer. For example, when receiving a user control instruction of "grab the red cup on the table, rotate it and place it on the right shelf", the multimodal large model combines the environmental information of the robot and the user control instruction to generate a task planning strategy, that is, decompose the user control instruction into three tasks, such as "grab the red cup", "rotate the cup", and "place it on the shelf".

[0042] Step 103: For each of the N tasks, the atomic skill model corresponding to the task is called by the large model to control the robot to perform the task according to the first execution strategy; during the task execution process, the execution result of the task is monitored. If the execution result is a failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to the second execution strategy.

[0043] In some embodiments, the atomic skill model includes basic action units for the robot to perform tasks. For example, the atomic skill model includes basic action units of the robotic arm, such as grasping, placing, and selecting, etc. Each atomic skill model interacts with the interface of the hardware platform to control the robotic arm to perform specific actions.

[0044] Exemplarily, the implementation of the atomic skill model in the embodiment of the present application is based on reinforcement learning, designing different reward functions for different atomic skills, and training different skill models (i.e., atomic skill models), including "grasping", "releasing", "translating" and other skill models.

[0045] In some embodiments, calling the atomic skill model corresponding to the task to control the robot to perform the task according to the first execution strategy includes: calling the atomic skill model corresponding to the task to generate mechanical control instructions corresponding to the task, and controlling the robot to execute the mechanical control instructions through the hardware interface between the atomic skill model and the robot, wherein the mechanical control instructions correspond to the first execution strategy of the task.

[0046] In some implementations, when different tasks invoke corresponding atomic skill models, the atomic skill models output control instructions for the robot. The robot is then controlled to execute the mechanical control instructions via a hardware interface between the atomic skill models and the robot. For example, when different tasks invoke corresponding atomic skill models, the atomic skill models output control instructions for the robotic arm. The motion control algorithm RMP-FLOW receives the control instructions, generates an operational end-of-gripper motion trajectory, and executes the corresponding task.

[0047] In some embodiments, the control instruction refers to the underlying control instruction of the robot. For example, when the robotic arm performs a grasping task, the control instruction can be the 6Dof grasping posture of the target object. This grasping posture will eventually be converted into the joint angle of the robot as a control signal. For a robotic arm with N degrees of freedom, the joint control signal is an N-dimensional vector.

[0048] In some embodiments, monitoring the execution result of the task includes: obtaining sensor information corresponding to the robot executing the task collected by a sensor, and determining the execution result of the task based on the sensor information.

[0049] In some embodiments, during the execution of each task, the execution result of the task can be determined by acquiring sensor information corresponding to the robot performing the task collected by the sensor.

[0050] For example, when performing a grasping task, the robot's pressure sensor data can be used to determine whether it is clamped by the gripper.

[0051] In some embodiments, when the acquired sensor information is sensor data, if the sensor data is less than a first threshold, the task execution result is considered a failure, and if the sensor data is greater than or equal to the first threshold, the task execution result is considered a success. The specific setting of the first threshold can be determined based on actual circumstances and is not specifically limited in this application.

[0052] For example, when the robotic arm performs a grasping task, it can determine whether the object is successfully grasped by detecting whether the data corresponding to the pressure sensor of the gripper exceeds a certain threshold; when performing a rotation and placement task, it can determine whether the object is correctly moved and placed through position information and sensor data.

[0053] In some embodiments, after determining an execution result based on sensory information corresponding to the robot executing a task collected by sensors, if the execution result is a failure, the task failure type is determined, and based on the task failure type, the atomic skill model corresponding to the task is re-invoked to control the robot to execute the task according to the second execution strategy. Specifically, during the execution of the task, if the robot's sensory information determines that the execution result has failed, the atomic skill model corresponding to the task is re-invoked, the mechanical control instructions corresponding to the task are re-generated, and the robot is controlled to execute the mechanical control instructions through a hardware interface between the atomic skill model and the robot, where the mechanical control instructions correspond to the second execution strategy for the task.

[0054] For example, when the task execution result is failure, the failed task is retried, and different grasping postures, placement positions, or path planning and other control instructions are used to control the robot to re-execute the task.

[0055] In some implementations, monitoring the execution result of a task includes: parsing the execution video information and text information of the task through a large model to obtain the execution result of the task.

[0056] In some embodiments, monitoring the execution results of the task also includes determining the execution results of the task through a large model. Specifically, the execution video information of the task and the text information of the task are input into the large model, and the large model parses the video information and text information to obtain the execution results of the task.

[0057] In some implementations, the task execution video information, the task execution sensor information, and the task text information are input into the big model, and the big model analyzes them to obtain the task execution result.

[0058] In some embodiments, the video information may be captured by a global camera, and the text information is text information for the task in the task planning strategy generated by the large model after parsing the user control instruction.

[0059] In some embodiments, after determining the task execution result, the task failure type can also be determined. Specifically, the task failure type can be determined based on the sensor information corresponding to the robot executing the task collected by the sensor, or the task execution video information and the task text information can be parsed through a large model to obtain the task execution result and the task failure type.

[0060] Exemplary mission failure types determined through sensor information or large models include:

[0061] 1) Grasping failure: The gripper fails to grasp the object, causing the object to fall off;

[0062] 2) Unreachable workspace: The robot cannot reach the target location, possibly due to a narrow space or path obstacles;

[0063] 3) Placement failure: The object cannot be placed stably, which may be caused by uneven placement surface or other reasons.

[0064] 4) Obstacle blocking: The robot encounters an obstacle during execution and cannot complete the task.

[0065] In some embodiments, the method further includes: when the number of times the robot is controlled to perform a task reaches a maximum number of attempts, parsing the task failure type and the task execution scenario information through a large model, and regenerating the task planning strategy based on the parsing results, wherein the regenerated task planning strategy includes different tasks or a different task execution order or a different task execution strategy compared to the previously generated task planning strategy.

[0066] In some embodiments, when the atomic skill model corresponding to the task is re-called based on the task failure type to control the robot to execute the task according to the second execution strategy and the maximum number of attempts is reached, at this time, if the task execution result is still a failure, the task failure type and the task execution scenario information are parsed by the large model, and the task planning strategy is regenerated based on the parsed result, wherein the regenerated task planning strategy includes different tasks or a different task execution order or a different task execution strategy compared to the previously generated task planning strategy. The maximum number of attempts can be determined based on actual conditions and is not specifically limited in this application.

[0067] In some embodiments, the regenerated task planning strategy includes M tasks to be performed by the robot, where M is a positive integer. Based on the regenerated task planning strategy, for each of the M tasks, the corresponding atomic skill model is called to perform the corresponding task.

[0068] For example, after the multimodal model parses and obtains the text descriptions of "failure" and "reason for failure," it replans the current task based on the description of the failure type, obtaining a new task sequence, that is, regenerating the task planning strategy. For example, the original task is "grab the notebook," and the execution result of this task fails because "the notebook was pressed by a cup, and grabbing the notebook caused the cup to overturn." Therefore, after the multimodal model replans, the new task planning strategy is "grab the cup," "place the cup in another location on the table," and "grab the notebook." The multimodal model then calls the corresponding atomic skill model to execute the corresponding task.

[0069] In some embodiments, the method further includes: collecting execution data of tasks with failed execution results, and updating parameters of the large model based on the execution data.

[0070] In some implementations, the multimodal macromodel optimizes the mission planning algorithm by analyzing failure data, creating a closed-loop control system. After each mission execution, the system feeds this data back to the multimodal macromodel for continuous optimization, continuously improving mission success rates. Failure data includes scenario descriptions and mission history information.

[0071] The technical solution of the embodiment of the present application is to obtain user control instructions for the robot; parse the user control instructions through the large model, and generate a task planning strategy based on the parsing results. The task planning strategy includes N tasks to be performed by the robot, where N is a positive integer; for each of the N tasks, the large model calls the atomic skill model corresponding to the task to control the robot to perform the task according to the first execution strategy; during the task execution process, the execution result of the task is monitored. If the execution result is a failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to the second execution strategy. In this way, the task execution strategy can be accurately obtained by parsing the user control instructions through the large model, and then the task is executed according to the second execution strategy for the task with a failed execution result, which can ensure the continuity of task execution and improve the success rate of the entire task completion.

[0072] The following is an example of the technical solution of the embodiment of the present application, which is described by taking a robotic arm performing N tasks as an example.

[0073] With the rapid development of intelligent manufacturing and automation technology, the application of robotic arms in industrial production is becoming more and more extensive. The task execution of robotic arms usually involves complex operations, such as grasping, rotating, placing and other tasks. These tasks often require efficient task planning, precise action execution and real-time feedback mechanisms to complete. In recent years, with the development of deep learning and multimodal large model technology, more and more research is devoted to combining vision, language and action to achieve more intelligent task planning and execution. The multimodal large model (VLM, Visual-Language Model) can understand user instructions while identifying and analyzing scene information, generating task plans and controlling the execution of the robotic arm by fusing vision, language and action modalities. However, existing robotic arm operating systems rely on predefined fixed strategies or limited feedback mechanisms, and there are many problems to be solved, especially in terms of uncertainty and failure handling during execution. In practical applications, robotic arms face the following problems when performing tasks:

[0074] 1) Inconsistency between task planning and execution: The task planning generated by the multimodal large model may encounter problems such as dynamic changes in the environment and changes in the position of objects during execution, resulting in execution failure.

[0075] 2) Complex reasons for execution failure: During the execution of a task, the robot arm may face various failure situations such as grasping failure, inaccessible workspace, placement failure, or obstacle blocking, which have a negative impact on the successful completion of the task.

[0076] 3) Inadequate failure handling mechanism: Existing systems usually lack effective failure diagnosis and correction mechanisms, resulting in the interruption of the entire process and low efficiency once a task fails.

[0077] 4) Uncertainty in task execution: Even a high-precision motion execution system cannot guarantee that every task can be completed efficiently and successfully when faced with complex scenarios or changing operational requirements.

[0078] Therefore, how to detect failures in real time during task execution and adjust and optimize task planning based on the type of failure has become a pressing issue. This application embodiment combines task planning, execution feedback, and failure diagnosis to propose a new solution. When a task fails, a multimodal large model can be used to "reflect" and adjust the task, ultimately completing the entire task.

[0079] Based on this, the present application proposes a closed-loop correction strategy that combines failure detection feedback. When a specific task fails during task planning, a multimodal large model can be used for "reflection" and adjustment, ultimately completing the entire task. Specific content includes:

[0080] 1. User Command Parsing and Task Planning

[0081] First, the multimodal large model receives the user's natural language instructions, such as: "Grab the red cup on the table, rotate it and place it on the right shelf." The model parses the user's instructions through the visual-language understanding module (VLM), and combines the visual information of objects in the environment or other sensors to generate a specific task plan. Among them, visual information can be obtained through cameras, a total of two cameras, one deployed at the end of the robotic arm, providing a local view in front of the gripper; the other camera is mounted on the table to provide a global view of the robotic arm's operation. Exemplarily, the camera is an RGBD camera

[0082] During the task planning stage, the model identifies the position and state of objects in the workspace based on visual information, and decomposes a series of subtasks based on user needs, that is, determines the N tasks to be performed by the robotic arm according to user control instructions. In the embodiment of the present application, N = 3, such as "grab the red cup", "rotate the cup", "place it on the shelf", etc.

[0083] 2. Task Execution and Atomic Skill Model Calling

[0084] For each task, the multimodal large model calls the downstream atomic skill model (also called atomic skill module) corresponding to each of the N tasks to execute. The atomic skill model includes the basic action units of the robot arm, such as grasping, placing, and rotating. Each atomic skill model interacts with the hardware platform interface to control the robot arm to perform specific actions.

[0085] Exemplarily, the atomic skills in the embodiments of the present application are based on reinforcement learning, different reward functions are designed for different atomic skills, and different skill models (i.e., atomic skill models) are trained, including: "grab", "release", "translation", etc.

[0086] During implementation, the motion control algorithm uses an atomic skill model to precisely calculate the control commands for the robotic arm, generating the pose and trajectory plan for each action. Grasping actions require precise contact between the robotic arm and the object, while placement actions require precise control of the target position. The motion control algorithm also enables obstacle avoidance during movement.

[0087] For example, in combination with the aforementioned user instruction parsing, task planning, task execution, and atomic skill model calling, the specific steps may be:

[0088] 1) User inputs voice, and the ASR algorithm converts the voice into text;

[0089] 2) The multimodal large model receives sensory information and user control instructions in text form and decomposes the task, that is, determines the task planning strategy. The task planning strategy includes N tasks to be performed by the robot arm, where N is a positive integer;

[0090] 3) Among the N tasks executed, different tasks call the corresponding atomic skill model, and the atomic skill model outputs control instructions for the robotic arm. The control instructions are the underlying control instructions of the robot. For example, they can be the 6-degree-of-freedom grasping posture of the target object. This grasping posture is ultimately converted into the robot's joint angles as control signals. For a robotic arm with N degrees of freedom, the joint control signals are N-dimensional vectors.

[0091] 4) The motion control algorithm (RMP-FLOW) receives control instructions, generates an operational gripper end motion trajectory, and performs the corresponding task.

[0092] It should be noted that the atomic skill model and motion control algorithm are both deployed in the ORIN development board connected to the robotic arm.

[0093] 3. Task Execution Feedback and Failure Identification

[0094] During task execution, the system monitors the results of each task in real time using sensors built into the simulator. For example, the grasping module determines whether an object has been successfully grasped by monitoring the gripper's pressure sensors. The rotation and placement modules use position information and sensor data to determine whether the object has been correctly moved and placed. The simulator and the real-world environment are two separate systems. The simulator is deployed on a server, where the six-dimensional force of the fingers can be easily acquired. In the real world, this information can be obtained through force sensors on the robot's gripper. The grasping module, rotation module, and placement module represent the corresponding atomic skill models described above.

[0095] If an execution failure is detected, such as a grasping failure or a placement failure, the failure diagnosis mechanism will be triggered. By analyzing the sensor feedback information, the type of failure can be determined as follows:

[0096] 1) Grasping failure: The gripper fails to grasp the object successfully, causing the object to fall off.

[0097] 2) Unreachable workspace: The robot arm cannot reach the target location, possibly due to a narrow space or path obstacles.

[0098] 3) Placement failure: The object cannot be placed stably, which may be caused by uneven placement surface or other reasons.

[0099] 4) Obstacle blocking: The robotic arm encounters an obstacle during execution and cannot complete the task.

[0100] It is understandable that the failure types listed in the embodiments of the present application are not all failure types.

[0101] Specifically, there are two solutions for determining the failure type:

[0102] Solution 1) Use information captured by sensors, such as the pressure sensor in this article, to determine whether the object is grasped by the gripper, etc.

[0103] Solution 2) uses a multimodal large model based on video input. By inputting video clips of the task execution process (the video clips are from the global camera) and the text description of the failed task into the multimodal large model, the multimodal large model determines whether the task is completed. If it fails, it provides the specific reason for failure. The specific failure reason can be found in the detailed description of the failure type mentioned above. The text description of the task is the text description corresponding to the N tasks after the multimodal large model is split.

[0104] It is understood that when executing each task, upon the first failure or failure within the maximum number of attempts, solution 1) is used to monitor the execution results in real time, obtain the execution results and the task failure type, and determine the control instructions for retrying the task based on the task failure type. If the maximum number of attempts is exceeded, solution 2) is used to analyze the task failure type and reason based on the video information and text description of the task execution. The task is then replanned based on the failure type and reason, and the task is then executed again.

[0105] 4. Adaptive Correction of Execution Failure

[0106] For the detected failure type, the following processing is performed:

[0107] 1) Retry: First, the failed task will be retried, using different grasping postures, placement positions or path planning, until the execution is successful or the maximum number of attempts is reached.

[0108] Specifically, when the information captured by the sensor determines that the task execution result is a failure and the type of task failure is determined, the corresponding atomic skill model will be re-called to update the output control instructions for the robotic arm. The motion control algorithm RMP-FLOW receives the control instructions, generates an operable motion trajectory of the gripper end, and executes the task.

[0109] For example, a pressure sensor can determine whether the robotic arm has grasped an object. If it is determined through the aforementioned solution 1) that a problem has occurred in the task execution, the atomic skill model will be re-executed, such as re-grasping the object.

[0110] 2) Task Replanning: If the task still fails after multiple attempts, the failure type and scenario description (including object location, robot arm posture, obstacle information, etc.) will be collected and fed back to the multimodal large model for error analysis. Based on the failure information, the model will replan the task, adjust the task sequence or execution strategy, and provide new operational recommendations.

[0111] Exemplarily, the specific correction method is: after obtaining the text description of "failure" and "reason for failure", the multimodal multi-model re-plans the task according to the description of the failure type to obtain a new task sequence, that is, a new task planning strategy; for example: the original task is "grab the notebook", when it is determined that the execution result of the task is "failure, the reason for failure is that the notebook was pressed by the water cup, and grabbing the notebook caused the water cup to overturn", the large model re-plans based on the reason for failure, and the execution strategy after re-planning can be: "grab the water cup", "put the water cup in another place on the table", "grab the notebook".

[0112] It is understandable that when the re-execution attempt is successful or the execution is successful after task re-planning, the next task is continued to be executed according to the original task planning strategy.

[0113] 3) Closed-Loop Optimization: The multimodal large model optimizes the task planning algorithm by analyzing failure data (including scenario descriptions and task history), forming a closed-loop control. After each task is executed, the system feeds this data back to the model for continuous optimization, continuously improving the task success rate. In other words, by collecting information about task failures, the multimodal large model is trained and iterated.

[0114] 4) Task Completion: Through the above steps, we can ensure that in the face of execution failure, we can adjust the task plan and optimize the execution strategy in real time, ultimately successfully completing the task sequence. The entire process forms a closed loop, ensuring the successful completion of the task through continuous optimization.

[0115] The technical solution of the embodiment of the present application addresses the problem of task planning execution failure and proposes a closed-loop correction strategy combined with failure detection feedback. The main problems it solves include: Identification and diagnosis of failures during task execution: For situations such as grasping failure, inaccessible workspace, placement failure, and obstacle obstruction that may occur when the robot arm performs an operation task, the real-time feedback mechanism can automatically detect the execution failure and identify its specific cause. Adaptive adjustment of the failure feedback mechanism: A task re-planning method based on the cause of failure is proposed, which can automatically adjust the task plan according to different failure types and improve the success rate of task execution. Closed-loop optimization of multimodal large models: By feeding back task execution feedback (including failure information) to the multimodal large model for retraining or adjustment, adaptive optimization of task execution is achieved.

[0116] Based on this, the technical solution of the embodiment of the present application can generate a task plan based on vision and language, combined with visual sensors and user natural language instructions, and convert it into a specific task that the robotic arm can perform. The technical solution of the embodiment of the present application can also provide real-time feedback and judgment during the execution process, using the sensor data of the robotic arm and the simulator to monitor the execution status of each task in real time and determine whether the task is successfully completed. The technical solution of the embodiment of the present application can also classify and analyze the types of execution failures, build a classification system for task execution failures, and analyze the reasons based on the task failure types to generate task adjustment strategies. The technical solution of the embodiment of the present application can also perform task replanning and adaptive adjustment: after identifying the cause of failure, the failure information is input into the multimodal large model, task replanning is performed, and the execution strategy is optimized. In this way, when faced with execution failure, the task plan can be adjusted in real time and the execution strategy can be optimized, and the task sequence can be successfully completed. The whole process forms a closed loop, ensuring that the task is ultimately successfully completed in the process of continuous optimization.

[0117] Figure 2 This is a schematic diagram of the structure of the robot control device provided in the embodiment of the present application. Figure 2 As shown, the robot control device includes:

[0118] An acquisition unit 201 is used to acquire user control instructions for the robot;

[0119] The processing unit 202 is used to parse the user control instructions through the large model, and generate a task planning strategy based on the parsing results. The task planning strategy includes N tasks to be performed by the robot, where N is a positive integer; for each of the N tasks, the large model calls the atomic skill model corresponding to the task to control the robot to perform the task according to the first execution strategy; during the task execution process, the execution result of the task is monitored. If the execution result is a failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to the second execution strategy.

[0120] In some embodiments, the processing unit 202 is used to parse the task failure type and task execution scenario information through a large model when the number of times the robot is controlled to perform a task reaches a maximum number of attempts, and regenerate the task planning strategy based on the parsing results, wherein the regenerated task planning strategy includes different tasks or a different task execution order or a different task execution strategy compared to the previously generated task planning strategy.

[0121] In some embodiments, the processing unit 202 is configured to obtain sensor information corresponding to the robot performing a task collected by a sensor, and determine a task execution result based on the sensor information.

[0122] In some implementations, the processing unit 202 is configured to parse the task execution video information and the task text information using a large model to obtain the task execution result.

[0123] In some embodiments, the processing unit 202 is used to call the atomic skill model corresponding to the task to generate mechanical control instructions corresponding to the task, and control the robot to execute the mechanical control instructions through the hardware interface between the atomic skill model and the robot, wherein the mechanical control instructions correspond to the first execution strategy of the task.

[0124] In some embodiments, the processing unit 202 is configured to generate a task planning strategy based on the analysis results and visual information in the robot's environment.

[0125] In some implementations, the processing unit 202 is configured to collect execution data of tasks with failed execution results, and update parameters of the large model based on the execution data.

[0126] It should be understood by those skilled in the art that Figure 2 The functions implemented by each unit in the robot control device shown can be understood by referring to the relevant description of the aforementioned method. Figure 2 The functions of the various units in the robot control device shown can be implemented by a program running on a processor or by a specific logic circuit.

[0127] Figure 3 It is a schematic structural diagram of a robot control device 300 provided in an embodiment of the present application. Figure 3 The robot control device 300 shown includes a processor 310, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0128] Alternatively, as Figure 3 As shown, the robot control device 300 may further include a memory 320. The processor 310 may call and run a computer program from the memory 320 to implement the method in the embodiment of the present application.

[0129] The memory 320 may be a separate device independent of the processor 310 , or may be integrated into the processor 310 .

[0130] Alternatively, as Figure 3 As shown, the robot control device 300 may further include a transceiver 330 , and the processor 310 may control the transceiver 330 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0131] The transceiver 330 may include a transmitter and a receiver. The transceiver 330 may further include an antenna, and the number of antennas may be one or more.

[0132] The robot control device 300 can implement the corresponding processes implemented by the robot control device in each method of the embodiments of the present application. For the sake of brevity, they are not described here.

[0133] Figure 4 It is a schematic structural diagram of the chip of an embodiment of the present application. Figure 4 The chip 400 shown includes a processor 410, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0134] Alternatively, as Figure 4 As shown, the chip 400 may further include a memory 420. The processor 410 may call and execute a computer program from the memory 420 to implement the method in the embodiment of the present application.

[0135] The memory 420 may be a separate device independent of the processor 410 , or may be integrated into the processor 410 .

[0136] Optionally, the chip 400 may further include an input interface 430. The processor 410 may control the input interface 430 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0137] Optionally, the chip 400 may further include an output interface 440. The processor 410 may control the output interface 440 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0138] This chip can implement the corresponding processes implemented by the robot control device in each method of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0139] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0140] It should be understood that the processor of the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented as a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0141] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] It should be understood that the above-mentioned memories are exemplary but not restrictive. For example, the memories in the embodiments of the present application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM RAM (DR RAM), etc. In other words, the memories in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0143] An embodiment of the present application also provides a computer program product, including a computer program.

[0144] When executed by the processor, the computer program implements the corresponding processes implemented by the robot control device in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here in detail.

[0145] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0146] The computer program enables the computer to execute the corresponding processes implemented by the robot control device in each method of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0148] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0153] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A robot control method, characterized in that: The method comprises: Get user control instructions for the robot; Parsing the user control instructions through a large model, and generating a task planning strategy based on the parsing result, wherein the task planning strategy includes N tasks to be performed by the robot, where N is a positive integer; For each of the N tasks, the atomic skill model corresponding to the task is called by the large model to control the robot to perform the task according to the first execution strategy; during the execution of the task, the execution result of the task is monitored. If the execution result is failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to the second execution strategy.

2. The method according to claim 1, characterized in that The method further comprises: When the number of times the robot is controlled to perform the task reaches the maximum number of attempts, the task failure type and the execution scenario information of the task are analyzed through the large model, and the task planning strategy is regenerated based on the analysis results, wherein the regenerated task planning strategy contains different tasks or a different task execution order or a different task execution strategy compared to the previously generated task planning strategy.

3. The method according to claim 1, characterized in that The monitoring of the execution result of the task includes: Acquire sensor information corresponding to the robot performing the task, collected by a sensor, and determine the execution result of the task based on the sensor information.

4. The method according to claim 1, wherein The monitoring of the execution result of the task includes: The execution video information of the task and the text information of the task are parsed by the large model to obtain the execution result of the task.

5. The method according to any one of claims 1 to 4, characterized in that The calling of the atomic skill model corresponding to the task to control the robot to perform the task according to the first execution strategy includes: The atomic skill model corresponding to the task is called to generate the mechanical control instructions corresponding to the task, and the robot is controlled to execute the mechanical control instructions through the hardware interface between the atomic skill model and the robot, wherein the mechanical control instructions correspond to the first execution strategy of the task.

6. The method according to any one of claims 1 to 4, characterized in that Generating a task planning strategy based on the analysis results includes: A task planning strategy is generated based on the analysis results and visual information in the environment where the robot is located.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Execution data of tasks with failed execution results are collected, and parameters of the large model are updated based on the execution data.

8. A robot control device, characterized in that: The device comprises: An acquisition unit, used for acquiring user control instructions for the robot; A processing unit is used to parse the user control instructions through a large model, and generate a task planning strategy based on the parsing result, wherein the task planning strategy includes N tasks to be performed by the robot, where N is a positive integer; for each of the N tasks, the atomic skill model corresponding to the task is called through the large model to control the robot to perform the task according to a first execution strategy; during the execution of the task, the execution result of the task is monitored, and if the execution result is a failure, the task failure type is determined, and the atomic skill model corresponding to the task is re-called according to the task failure type to control the robot to perform the task according to a second execution strategy.

9. A robot control device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

Images