Control method of mechanical arm, mechanical arm and computer readable storage medium

Abstract

The invention discloses a control method of a mechanical arm, the mechanical arm and a computer readable storage medium. The method comprises the steps of receiving a task instruction and an environment image; wherein the environment image has depth information; inputting the task instruction and the environment image into a large language model, so that the large language model outputs natural language instructions corresponding to the sub-tasks in sequence; wherein the large language model does not have code generation capability; wherein each subtask is predicted when the execution of the previous subtask is completed; and the natural language instructions are sequentially input into the mechanical arm control model according to the sequence, so that the mechanical arm control model controls the mechanical arm to execute corresponding actions based on the natural language instructions until all the subtasks are completed. By means of the mode, the problem that in the related technology, understanding of action semantics is insufficient can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of robot arm control, and in particular to a robot arm control method, a robot arm and a computer-readable storage medium. Background Art

[0002] Multitasking manipulation of robotic arms is an important and rapidly developing area in industrial automation and robotics. It has wide applications in many fields such as manufacturing, healthcare, logistics, agriculture, and services, and can significantly improve production efficiency, reduce costs, and increase system flexibility. In the field of 3D object manipulation, collecting expert data for end-to-end imitation learning is a standard approach. This approach is able to learn complex tasks directly from demonstrations of human experts, capture subtle manipulation skills, and does not require explicit programming of each action, thereby reducing development difficulty and adapting to various operating environments. However, despite the success of this approach, previous studies have ignored the guiding role of language in action execution, resulting in poor performance. Related methods often focus too much on object semantics and ignore the importance of action semantics, resulting in a vague understanding of action semantics.

[0003] To solve the above problems, relevant technologies utilize the outstanding capabilities of large language models in code writing, human language understanding, and tool calling, and introduce them into robotic arm operation tasks to improve the efficiency and flexibility of task execution.

[0004] However, related technologies still have some limitations: they usually require humans to pre-define action APIs, or write multiple prompts to guide large models to perform different task operations, or rely on large models with code generation capabilities. Summary of the invention

[0005] In order to solve the above problems, the control method of the robotic arm, the robotic arm and the computer-readable storage medium provided in the present application can solve the problem of insufficient understanding of action semantics in the related art.

[0006] To solve the above technical problems, a technical solution adopted in the present application is: to provide a method for controlling a robotic arm, the method comprising: receiving task instructions and an environmental image; wherein the environmental image has depth information; inputting the task instructions and the environmental image into a large language model, so that the large language model sequentially outputs natural language instructions corresponding to multiple subtasks; wherein the large language model does not have code generation capabilities; wherein each subtask is predicted when the previous subtask is completed; inputting the natural language instructions into the robotic arm control model in sequence, so that the robotic arm control model controls the robotic arm to perform corresponding actions based on the natural language instructions until all subtasks are completed.

[0007] Among them, multiple subtasks include at least a first subtask and a second subtask; inputting task instructions and environmental images into a large language model so that the large language model outputs natural language instructions corresponding to multiple subtasks in sequence, including: inputting task instructions and environmental images into a large language model so that the large language model outputs a first natural language instruction corresponding to the first subtask; when it is detected that the first subtask is completed, using the large language model to predict the second subtask, and outputting a second natural language instruction corresponding to the second subtask.

[0008] Among them, in the process of controlling the robotic arm to perform the corresponding action, the large language model is used to recognize the current environment image; determine whether the subtask has been completed; if not, continue to input the natural language instructions corresponding to the subtask into the robotic arm control model; if the execution is completed, input the natural language instructions corresponding to the next subtask into the robotic arm control model.

[0009] Among them, using the large language model to recognize the current environment image and judging whether the subtask is completed includes: using the large language model to recognize the position of the robot arm gripper in the current environment image; judging whether the subtask is completed based on the gripper position.

[0010] The large language model is trained in the following way: obtaining the trajectory stream and environmental data corresponding to the robot arm completing the target task; decomposing the trajectory stream into multiple sub-trajectory streams based on key frames; wherein each trajectory stream is annotated with corresponding action language instructions; and using the task instructions corresponding to the target task, multiple sub-trajectory streams, environmental data and the position of the robot arm at the key frame to train the large language model, so that the large language model can learn the task decomposition capability.

[0011] The method of decomposing the trajectory stream into multiple sub-trajectory streams based on key frames includes: determining a trajectory frame in which the gripper state in the trajectory stream changes or the robot arm speed is 0 within a preset number of consecutive frames as a key frame; and decomposing the trajectory stream into multiple sub-trajectory streams based on the key frames.

[0012] Among them, the robotic arm control model is trained in the following way: the robotic arm control model is trained using multiple sub-trajectory streams, environmental data and the position of the robotic arm at key frames.

[0013] Among them, the robot control model is trained using multiple sub-trajectory streams, environmental data and the position of the robot arm at the key frame, including: inputting multiple sub-trajectory streams, environmental data and the position of the robot arm at the key frame into the robot control model, so that the robot arm controls the robot arm for imitation learning; after the imitation learning of each sub-trajectory stream is completed, the actual position, actual rotation angle, and actual state of the gripper on the robot arm are obtained; the first loss is determined using the final position and actual position of the robot arm in each sub-trajectory stream in the environmental data; the second loss is determined using the final rotation angle and actual rotation angle of the robot arm in each sub-trajectory stream; the third loss is determined using the final state and actual state of the gripper on the robot arm in each sub-trajectory stream; and the model parameters of the robot control model are adjusted in combination with the first loss, the second loss and the third loss.

[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide a robotic arm, which includes: a processor and a memory and an actuator coupled to the processor; wherein the memory is used to store computer programs, and the processor is used to execute computer programs to implement a method provided by any of the above technical solutions.

[0015] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the method provided by any of the above technical solutions.

[0016] The beneficial effects of the embodiments of the present application are as follows: different from the prior art, the control method of the robotic arm, the robotic arm and the computer-readable storage medium provided by the present application, by utilizing a large language model that does not have the ability to generate code in combination with environmental images to decompose task instructions, and output natural language instructions with action decisions corresponding to each subtask, so that the robotic arm control model controls the robotic arm to perform corresponding actions based on the natural language instructions until all subtasks are completed. This method can fully utilize the language understanding and reasoning capabilities of the large language model that does not have the ability to generate code, and can provide the robotic arm with rich action semantic information, thereby solving the problem of insufficient understanding of action semantics in related technologies.

[0017] Furthermore, the use of large language models without code generation capabilities in robot arm control significantly reduces the computing resource requirements for the robot arm and improves the reproducibility of the technology and the feasibility of practical applications.

[0018] Furthermore, each subtask output by the large language model is predicted when the previous subtask is completed, which can ensure that each subtask is in the best state to execute the next subtask, and accurately predict the next subtask when the robotic arm is in the ideal position, ensuring the continuity and accuracy of the robotic arm's tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0020] Figure 1 It is a flow chart of an embodiment of a control method of a robotic arm provided by the present application;

[0021] Figure 2 is a flow chart of another embodiment of the control method of the robotic arm provided by the present application;

[0022] Figure 3 It is a flowchart of an embodiment of the execution completion status of the action corresponding to the subtask provided by the present application;

[0023] Figure 4 It is a flowchart of an embodiment of a training method for a large language model provided by the present application;

[0024] Figure 5 It is a flowchart of an embodiment of a training method for a robotic arm control model provided by the present application;

[0025] Figure 6 This is a schematic diagram of an application scenario of a control method for a robotic arm provided in this application;

[0026] Figure 7 It is a structural schematic diagram of an embodiment of a mechanical arm provided by the present application;

[0027] Figure 8 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] Multitasking manipulation of robotic arms is an important and rapidly developing area in industrial automation and robotics. It has wide applications in many fields such as manufacturing, healthcare, logistics, agriculture, and services, and can significantly improve production efficiency, reduce costs, and increase system flexibility. In the field of 3D object manipulation, collecting expert data for end-to-end imitation learning is a standard approach. This approach is able to learn complex tasks directly from demonstrations of human experts, capture subtle manipulation skills, and does not require explicit programming of each action, thereby reducing development difficulty and adapting to various operating environments. However, despite the success of this approach, previous studies have ignored the guiding role of language in action execution, resulting in poor performance. Related methods often focus too much on object semantics and ignore the importance of action semantics, resulting in a vague understanding of action semantics.

[0031] To solve the above problems, relevant technologies utilize the outstanding capabilities of large language models in code writing, human language understanding, and tool calling, and introduce them into robotic arm operation tasks to improve the efficiency and flexibility of task execution.

[0032] However, related technologies still have some limitations: they usually require humans to pre-define action APIs, or write multiple prompts to guide large models to perform different task operations, or rely on large models with code generation capabilities.

[0033] Based on this, the present application utilizes a large language model that does not have the ability to generate code in combination with environmental images to decompose task instructions, and outputs natural language instructions with action decisions corresponding to each subtask, so that the robotic arm control model controls the robotic arm to perform corresponding actions based on the natural language instructions until all subtasks are completed. This method can fully utilize the language understanding and reasoning capabilities of the large language model that does not have the ability to generate code, and can provide the robotic arm with rich action semantic information, thereby solving the problem of insufficient understanding of action semantics in related technologies.

[0034] Furthermore, the use of large language models without code generation capabilities in robot arm control significantly reduces the computing resource requirements for the robot arm and improves the reproducibility of the technology and the feasibility of practical applications.

[0035] Furthermore, each subtask output by the large language model is predicted when the previous subtask is completed, which can ensure that each subtask is in the best state to execute the next subtask, and accurately predict the next subtask when the robot arm is in the ideal position, ensuring the continuity and accuracy of the robot arm task. For details, please refer to any of the following embodiments.

[0036] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a control method for a robotic arm provided by the present application. The method comprises:

[0037] Step 11: Receive a task instruction and an environment image, wherein the environment image has depth information.

[0038] In some embodiments, the task instruction may be a voice instruction, for example, a microphone is provided to collect voices and generate voice instructions.

[0039] In some embodiments, the task instruction may be a text instruction, such as a text instruction that can be input to the corresponding mechanical arm using an input device.

[0040] In some embodiments, an image acquisition component is provided on the robot arm. The image acquisition component can be used to acquire the environment image in real time. Since the environment image has depth information, the spatial information of the corresponding object in the environment image can be acquired, such as the position of the object in the world coordinate system.

[0041] Step 12: Input the task instructions and environment images into the large language model, so that the large language model sequentially outputs natural language instructions corresponding to multiple subtasks; wherein the large language model does not have code generation capability.

[0042] Each subtask is predicted when the previous subtask is completed.

[0043] In some embodiments, the large language model has the ability to decompose tasks through training. After the task instruction is input into the large language model, the large language model can decompose the task based on the task instruction. Exemplarily, the action corresponding to each subtask is a single action. For example, a single translation action, a single rotation action, or a gripper operation after translation. That is, the large language model can decompose complex operation tasks into multiple single actions, and rely on the robot arm to continuously perform multiple single actions to achieve the operation task.

[0044] The robotic arm is composed of multiple joints and grippers. Therefore, when the robotic arm is used to perform a task, the joints are controlled so that the joints cooperate with each other, and the grippers are used to clamp the object and then move it to the corresponding position. Because multiple joints are involved, the robotic arm actually needs to perform multiple subtasks when performing a task. Based on this, when the large language model of the present application receives a task instruction, it first predicts the first subtask, and after predicting the first subtask, generates a corresponding natural language instruction based on the first subtask, and then executes step 13. After determining that the first subtask has been completed, continue to predict the next subtask. After predicting the next subtask, generate a corresponding natural language instruction based on the next subtask, and then execute step 13. Repeat this cycle until the last subtask is predicted, and the corresponding natural language instruction is generated based on the last subtask, and then execute step 13.

[0045] In some embodiments, the large language model can be constructed based on at least one of BERT (Bidirectional Encoder Representations from Transformers), Wenxinyiyan, BLOOM, XLNet, and / or RoBERTa.

[0046] Step 13: Input the natural language instructions into the robot control model in sequence, so that the robot control model controls the robot to perform corresponding actions based on the natural language instructions until all subtasks are completed.

[0047] In some embodiments, the robot control model can be constructed based on a corresponding neural network model, such as a perceptron, a linear neural network, a BP (back propagation) network, a RBF (radial basis function) network, a Hopfield network, a self-organizing feature map (SOM) network, a convolutional neural network (CNN), a long short-term memory (LSTM) network, a VGG network and / or a graph neural network (GNN).

[0048] In some embodiments, after receiving the natural language instruction, the robot control model will identify the current environment image. Based on the current environment image, it determines the corresponding actions that the robot needs to perform to execute the natural language instruction, as well as the robot parameters that need to be set to achieve these actions, such as the movement distance and / or rotation angle of each joint of the robot.

[0049] In an application scenario, task instruction A includes subtask A1, subtask A2 and subtask A3. Task instruction A and the environment image are input into the large language model so that the large language model outputs the natural language instruction corresponding to subtask A1. The natural language instruction corresponding to subtask A1 is input into the robotic arm control model so that the robotic arm control model controls the robotic arm to perform the action corresponding to subtask A1 based on the natural language instruction. When subtask A1 is completed, the large language model predicts the output subtask A2 based on subtask A1. And the natural language instruction corresponding to subtask A2 is output. The natural language instruction corresponding to subtask A2 is input into the robotic arm control model so that the robotic arm control model controls the robotic arm to perform the action corresponding to subtask A2 based on the natural language instruction. When subtask A2 is completed, the large language model predicts the output subtask A3 based on subtask A2. And the natural language instruction corresponding to subtask A3 is output. The natural language instructions corresponding to subtask A3 are input into the robotic arm control model, so that the robotic arm control model controls the robotic arm to perform the action corresponding to subtask A3 based on the natural language instructions. At this point, the robotic arm completes the operation task corresponding to task instruction A.

[0050] In this embodiment, the task instructions are decomposed by utilizing a large language model that does not have the ability to generate code in combination with environmental images, and natural language instructions with action decisions corresponding to each subtask are output, so that the robot arm control model controls the robot arm to perform corresponding actions based on the natural language instructions until all subtasks are completed. This method can fully utilize the language understanding and reasoning capabilities of the large language model that does not have the ability to generate code, and can provide the robot arm with rich action semantic information, thereby solving the problem of insufficient understanding of action semantics in related technologies.

[0051] Action decision refers to the process of selecting one or more options from multiple alternatives, which are action plans proposed to achieve a certain goal. Decision-making involves not only the evaluation and selection of existing options, but also the formulation of new plans when necessary. Decision-making is a complex process, which includes the collection and processing of information, as well as the analysis and evaluation of various possible action plans, with the ultimate goal of selecting the most appropriate action plan. In robotic arm technology, action decision (Behavior Decision) is one of the decision-making planning and control parts in a narrow sense. It involves the formulation of robotic arm behavior decisions, that is, according to each subtask, deciding the action or series of actions that the robotic arm should take.

[0052] Furthermore, the use of large language models without code generation capabilities in robot arm control significantly reduces the computing resource requirements for the robot arm and improves the reproducibility of the technology and the feasibility of practical applications.

[0053] Furthermore, each subtask output by the large language model is predicted when the previous subtask is completed. That is, the large language model can predict the next subtask when each subtask is completed, synchronously using the current state of executing the subtask. It can be known that the state when the subtask is completed should be the optimal state, so as to ensure that each subtask is in the best state to execute the next subtask when it is completed, and accurately predict the next subtask when the robot arm is in the ideal position, thereby ensuring the continuity and accuracy of the robot arm's tasks.

[0054] See also Figure 2 , Figure 2 It is a flow chart of another embodiment of the control method of the robotic arm provided in the present application.

[0055] The method includes:

[0056] Step 21: Receive a task instruction and an environment image, wherein the environment image has depth information.

[0057] Step 22: Input the task instruction and the environment image into the large language model so that the large language model outputs a first natural language instruction corresponding to the first subtask.

[0058] In this embodiment, it is illustrated that the subtasks include at least a first subtask and a second subtask.

[0059] The robot arm action corresponding to the first subtask is the first action of the robot arm to execute the task instruction.

[0060] Step 23: Input the first natural language instruction into the robotic arm control model, so that the robotic arm control model controls the robotic arm to perform the corresponding first action based on the first natural language instruction.

[0061] Step 24: When it is detected that the first action is completed, the second subtask is predicted using the large language model, and a second natural language instruction corresponding to the second subtask is output.

[0062] If it is detected that the first action has not been completed, continue to input the first natural language instruction to the robot arm control model, so that the robot arm control model controls the robot arm to perform the corresponding first action again based on the first natural language instruction until the robot arm completes the first action.

[0063] In some embodiments, in order to avoid the robot arm being unable to complete the action due to a fault, resulting in meaningless repeated issuance of natural language instructions, the number of repeated issuances of the corresponding natural language instructions can be set. When the number of repeated issuances of the same natural language instruction to the robot arm control model exceeds the number of repeated issuances, a robot arm abnormality reminder can be directly issued.

[0064] Step 25: Input the second natural language instruction into the robot arm control model, so that the robot arm control model controls the robot arm to perform the corresponding second action based on the second natural language instruction.

[0065] In other embodiments, if there are other subtasks after the second subtask, when the second action is detected to be completed, the large language model is used to predict the next subtask, and the next natural language instruction corresponding to the next subtask is output. The next natural language instruction is input into the robot control model, so that the robot control model controls the robot to perform the corresponding action based on the next natural language instruction.

[0066] If it is detected that the second action has not been completed, continue to input the second natural language instruction to the robot arm control model, so that the robot arm control model controls the robot arm to perform the corresponding second action again based on the second natural language instruction until the robot arm completes the second action.

[0067] This cycle continues until the robotic arm control model controls the robotic arm to perform corresponding actions based on natural language instructions and completes all subtasks.

[0068] In some embodiments, see Figure 3 , describe the execution completion status of the actions corresponding to the subtasks:

[0069] Step 31: In the process of controlling the robotic arm to perform the corresponding action, the large language model is used to recognize the current environment image to determine whether the action corresponding to the subtask has been completed.

[0070] Among them, the process of controlling the robotic arm to execute the corresponding action can be equivalent to the process of controlling the robotic arm to execute the action corresponding to the subtask.

[0071] In some embodiments, the current environment image is recognized using a large language model, and a target object and a spatial position of the target object can be recognized from the current environment image.

[0072] In some embodiments, the recognition result includes the spatial position of the target object. The action corresponding to the subtask is that the robotic arm moves above the target object. At this time, when the robotic arm performs the action, the corresponding environmental image can be collected in real time. The relative position relationship between the target object and the robotic arm is determined from the environmental image to determine whether the action corresponding to the subtask is completed.

[0073] In some embodiments, a large language model can be used to identify the position of the manipulator's gripper in the current environment image; based on the gripper's position, it is determined whether the subtask has been completed. Because the manipulator usually uses a gripper to perform operations such as gripping the target object, the current environment image may include information about the manipulator's gripper. In addition, the position of the manipulator's gripper can be used to determine whether the subtask has been completed. For example, if the subtask is to move the gripper to directly above the object, and the large language model is used to identify that the gripper's position in the current environment image is not directly above the object, it means that the action corresponding to the subtask has not been completed. If the large language model is used to identify that the gripper's position in the current environment image is directly above the object, it means that the action corresponding to the subtask has been completed.

[0074] In some embodiments, when the speed of each joint of the robotic arm is 0, it can be known that the robotic arm has stopped at this time, and the large language model can be used to identify the current environment image to determine whether the action corresponding to the subtask has been completed. That is, it is determined whether the robotic arm has completed the action corresponding to the subtask at this time. For example, the environmental image corresponding to the robotic arm at this time is identified to determine whether the relative position relationship between the target object and the robotic arm meets the requirements of the subtask. If so, the action corresponding to the subtask is completed, if not, the action corresponding to the subtask is not completed.

[0075] The environment image refers to the image corresponding to the environment where the target object is located. The large language model can use the corresponding target recognition technology to identify the position of the target object from the environment image. For example, the large language model can be used to identify the position of the manipulator gripper in the current environment image; and the subtask can be judged whether it is completed according to the gripper position. Because the manipulator usually uses the gripper to clamp the target object, the current environment image can include information about the manipulator gripper. Then, the relative position relationship between the position of the manipulator gripper and the position of the target object can be combined to judge whether it meets the requirements of the subtask. For example, the subtask is to move the gripper to the top of the object, and the large language model is used to identify that the gripper position in the current environment image and the position of the target object do not overlap in the vertical direction, indicating that the manipulator gripper is not located directly above the object, indicating that the action corresponding to the subtask has not been completed, and the requirements of the subtask are not met at this time. If the large language model is used to identify that the gripper position in the current environment image and the position of the target object do not overlap in the vertical direction, indicating that the manipulator gripper is located directly above the object, indicating that the action corresponding to the subtask has been completed, and the requirements of the subtask are met at this time.

[0076] When it is determined that the action corresponding to the subtask has not been completed, step 32 is executed. When it is determined that the action corresponding to the subtask has been completed, step 33 is executed.

[0077] Step 32: Continue to input natural language instructions corresponding to the subtasks into the robotic arm control model.

[0078] In step 32, the natural language instructions corresponding to the subtask are continuously input to the robot arm control model, so that the robot arm control model controls the robot arm to perform the action corresponding to the subtask again based on the natural language instructions. For example, during the execution of the action corresponding to the current subtask, the large language model is used to recognize the current environment image to determine whether the action corresponding to the subtask has been completed. When it is determined that the action corresponding to the current subtask has not been completed, the natural language instructions corresponding to the current subtask are continuously input to the robot arm control model, so that the robot arm control model controls the robot arm to perform the action corresponding to the current subtask again based on the natural language instructions.

[0079] In this embodiment, the large language model has the ability to continuously correct errors. When the robot arm does not perform the action corresponding to the natural language instruction, the large language model repeatedly inputs the natural language instruction corresponding to the subtask to the robot arm control model so that the robot arm completes the corresponding action. Then the large language model combines the environmental image to determine that the robot arm has completed the corresponding action, and predicts the next subtask. And input the natural language instruction corresponding to the next subtask to the robot arm control model.

[0080] Step 33: Input the natural language instructions corresponding to the next subtask into the robot arm control model.

[0081] In step 33, a natural language instruction corresponding to the next subtask is input into the robot arm control model, so that the robot arm control model controls the robot arm to perform a corresponding action based on the next natural language instruction.

[0082] In this embodiment, the task instructions are decomposed by utilizing a large language model that does not have the ability to generate code in combination with environmental images, and natural language instructions with action decisions corresponding to each subtask are output, so that the robot arm control model controls the robot arm to perform corresponding actions based on the natural language instructions until all subtasks are completed. This method can fully utilize the language understanding and reasoning capabilities of the large language model that does not have the ability to generate code, and can provide the robot arm with rich action semantic information, thereby solving the problem of insufficient understanding of action semantics in related technologies.

[0083] Furthermore, the use of large language models without code generation capabilities in robot arm control significantly reduces the computing resource requirements for the robot arm and improves the reproducibility of the technology and the feasibility of practical applications.

[0084] Furthermore, each subtask output by the large language model is predicted when the previous subtask is completed, which can ensure that each subtask is in the best state to execute the next subtask, and accurately predict the next subtask when the robotic arm is in the ideal position, ensuring the continuity and accuracy of the robotic arm's tasks.

[0085] Furthermore, the large language model has the ability to correct motion errors and make continuous predictions, which effectively reduces the cumulative errors in robotic arm operation tasks.

[0086] The following introduces the training methods of the above-mentioned large language model and robotic arm control model respectively.

[0087] In some embodiments, see Figure 4 , the large language model is trained in the following way:

[0088] Step 41: Obtain the trajectory flow and environmental data corresponding to the robot arm completing the target task.

[0089] In some embodiments, the trajectory flow involved in the robot arm completing the target task can be obtained by teaching. For example, the robot arm is controlled by an expert to complete the target task, and the corresponding trajectory flow and environmental data are collected during the process of the robot arm completing the target task. The environmental data can be an environmental image of the target object in the process of the robot arm completing the target task. The environmental image has depth information.

[0090] In some embodiments, the trajectory stream may include continuous position information of the robot arm, such as the posture of the robot arm, the state of the gripper on the robot arm, etc.

[0091] Step 42: Decompose the track stream into multiple sub-track streams based on the key frames; wherein each track stream is annotated with a corresponding action language instruction.

[0092] In some embodiments, a trajectory frame in which the gripper state changes or the velocity of each joint of the robot arm is 0 within a preset number of consecutive frames is determined as a key frame; and the trajectory stream is decomposed into multiple sub-trajectory streams based on the key frame.

[0093] In some embodiments, the action language instruction may indicate the action of the robotic arm from the state of the current key frame to the state of the next key frame.

[0094] Step 43: Train the large language model using the task instructions corresponding to the target task, multiple sub-trajectory streams, environmental data, and the position of the robot arm at the key frame, so that the large language model can learn the task decomposition capability.

[0095] In some embodiments, the trained large language model can be applied to any embodiment of the present application to output the natural language instructions corresponding to each subtask and predict the next subtask based on the current subtask.

[0096] In some embodiments, the robotic arm control model is trained in the following manner: the robotic arm control model is trained using a plurality of sub-trajectory streams, environmental data, and positions of the robotic arm at key frames.

[0097] For example, see Figure 5 , the robotic arm control model can be trained in the following ways:

[0098] Step 51: Input multiple sub-trajectory streams, environmental data, and the position of the robot arm at the key frame into the robot arm control model, so that the robot arm control model controls the robot arm to perform imitation learning.

[0099] In some embodiments, the robot control model generates corresponding control instructions according to each sub-trajectory stream, environmental data, and the position of the robot at the key frame, and uses these control instructions to control the robot to perform imitation learning.

[0100] Step 52: After the imitation learning of each sub-trajectory flow is completed, the actual position, actual rotation angle, and actual state of the gripper on the robot arm are obtained.

[0101] In some embodiments, the actual position of the robot arm, the actual rotation angle, and the actual state of the gripper on the robot arm can be acquired using corresponding sensors disposed on the robot arm. For example, an angle sensor is disposed at the joint of the robot arm, and the actual rotation angle is acquired in real time using the angle sensor. A state sensor is disposed at the gripper of the robot arm, and the actual state of the gripper is acquired in real time using the state sensor.

[0102] Step 53: Determine the first loss using the final position and the actual position of the robot arm in each sub-trajectory stream in the environmental data.

[0103] In some embodiments, due to imitation learning, there is an imitation error, and therefore, the first loss is determined using the final position (real position) and the actual position (imitation position) of the robot arm in each sub-trajectory stream in the environment data.

[0104] In some embodiments, the divergence between the final position (real position) and the actual position (simulated position) of the robot arm in each sub-trajectory stream may be calculated, and the divergence value may be used as the first loss.

[0105] Step 54: Determine the second loss using the final rotation angle and the actual rotation angle of the robot arm in each sub-trajectory stream.

[0106] In some embodiments, due to imitation learning, there is an imitation error, and therefore, the second loss is determined using the final rotation angle (real rotation angle) and the actual rotation angle (imitation rotation angle) of the robot arm in each sub-trajectory stream in the environment data.

[0107] In some embodiments, the divergence between the final rotation angle (real rotation angle) and the actual rotation angle (simulated rotation angle) of the robot arm in each sub-trajectory stream may be calculated, and the divergence value may be used as the second loss.

[0108] Step 55: Determine the third loss using the final state and the actual state of the gripper on the robot arm in each sub-trajectory flow.

[0109] In some embodiments, due to imitation learning, there is an imitation error, and therefore, the third loss is determined using the final state (real gripper state) and actual state (simulated gripper state) of the gripper on the robot arm in each sub-trajectory stream in the environment data.

[0110] In some embodiments, the divergence between the final state and the actual state of the gripper on the robot arm in each sub-trajectory stream may be calculated, and the divergence value may be used as the third loss.

[0111] In other embodiments, the first loss, the second loss and the third loss mentioned above can be calculated using a traditional supervised loss function or an imitative loss function.

[0112] Step 56: Adjust the model parameters of the robot arm control model in combination with the first loss, the second loss and the third loss.

[0113] In some embodiments, corresponding weights may be set for the first loss, the second loss, and the third loss, and then a weighted sum is performed to obtain a total loss.

[0114] Then, the model parameters of the robot control model are adjusted according to the total loss. In some embodiments, the model parameters of the robot control model can be adjusted using a gradient descent method.

[0115] During the entire training process, the entire operation process is no longer supervised by a single instruction, but supervised training is performed using the corresponding action description for each subtask (i.e., using each sub-trajectory stream). This method significantly improves the ability to understand the action semantics in the robot arm operation task, overcoming the problem of insufficient action semantics understanding in the prior art.

[0116] In one application scenario, the above-mentioned large language model and robotic arm control model are trained in the following manner.

[0117] The expert data for completing a task is defined as a continuous trajectory stream. The trajectory stream is decomposed into multiple sub-trajectory streams through key frames, where a key frame is defined as the moment when the state of the robot arm's gripper changes, or the situation where the speed of all joints of the robot arm is 0 within 5 consecutive frames. Each sub-trajectory stream is added with corresponding action descriptions by human experts. The actions performed in these sub-trajectory streams are usually relatively simple and basic, such as a single translation action, a single rotation action, or a gripper operation after translation. The action descriptions annotated by human experts strive to be concise and clear, accurately describing the actions that occur, such as "moving above the light bulb", "gripping the light bulb downward", "screwing the light bulb outward", etc. In this way, the robot arm operation task is decomposed into multiple sub-tasks according to the key frames, and an action description is added to each sub-task to improve the model's understanding of the action semantics.

[0118] Assuming there is expert data in Represents the language instructions corresponding to each task, and each demonstration data ξ contains a series of continuous position information And the corresponding environment information Each a contains the current 6D position and gripper state of the robot arm, a = {a pose , a open}, each environment information contains RGB-D information under each viewing angle. sim ={o front , o left , o right , o wrist In each demonstration data ξ, locate the key frame and take the corresponding a as k to represent the key position of the robot arm when the key frame is reached. Each key position to the next key position corresponds to a human-labeled action language instruction I, which describes the action from the state of the current key frame to the state of the next key frame. Each demonstration data ξ is intercepted every 10 frames to construct the final data set {o i , I i , k i , g i}, where o i Represents the current environment information, I i Represents the current subtask instruction, k i represents the target position, g i Represents mission instructions.

[0119] Through this keyframe-based task decomposition method, the complex robot arm operation task is finely decomposed. Then the large language model is trained with this data set so that the large language model can have the corresponding task decomposition capability. Then, when the task instruction is obtained, the natural language instruction corresponding to each subtask is output, and the next subtask is predicted based on the current subtask.

[0120] The robotic arm control model can be trained in the following ways:

[0121] From the data {o i , I i , k i , g i}Select {o i , I i , k i} as imitation learning data, where o i Point cloud re-rendering is performed, and the position of the robot gripper in the environment image is marked as As the true value, for the rotation of the end gripper, the Euler angle representation is used, where each angle is discretized to a bin with a resolution of 5, using as a truth value.

[0122] Binary classification training is used for the opening and closing of the gripper. That is, the robot control model involves the calculation of the following loss function.

[0123]

[0124] Among them, L heatmap represents the first loss mentioned above, y ijk Indicates the actual position of the robot gripper. Represents the final position of the robot gripper on the trajectory stream in the dataset.

[0125]

[0126] Among them, L rotation represents the second loss mentioned above, represents the final rotation angle (expressed in Euler angles), r ia represents the actual rotation angle mentioned above (expressed in Euler angles).

[0127]

[0128] Among them, Lgripper represents the third loss mentioned above, represents the final state of the gripper on the robot in each sub-trajectory flow mentioned above, g i Indicates the actual status of the gripper on the above-mentioned robot arm.

[0129] N, H, and W represent the number of corresponding types.

[0130] L=αL heatmap +βL rotation +γL gripper α, β, and γ represent the weight parameters of the corresponding losses.

[0131] After the above-mentioned large language model and robotic arm control model are trained, they are deployed to the corresponding system of the robotic arm for task operations.

[0132] In an application scenario, the large language model receives environmental input and user language instructions, determines which execution stage of the task the current robot arm is in, and outputs the best execution instruction in the current state. For the large language model, the prompt words used can be: "You are a helpful language and visual assistant. You can understand the visual content provided by the user and help the user and use natural language to complete various tasks." In the large language model, a projection scheme is used for visual-text alignment, so that the large language model can understand the picture. Such a projection scheme allows for rapid iteration of data-centric experiments. For the input image X v , consider the pre-trained CLIP visual encoder ViT-L / 14, which provides visual features Z v =g(X v ). Use a trainable projection matrix W to transform Z r Convert to language embedding token H q , H q It has the same dimension as the word embedding space in the language model.

[0133] From the data {o i , I i , k i , g i}Select {o i , g i , I i ,} as the large language model training data, input prompt words and user instructions g i , enter the environment information i , output the next action instruction I i , and use the cross entropy function to train the large language model. After the upstream command model and the downstream action model are trained, the large model can command the downstream tasks, thereby driving the robot arm operation task to continue until it is completed.

[0134] In one application scenario, combined with Figure 6 To explain:

[0135] Input a task instruction to the robot arm, the content of which is "move object C to area B". After receiving the task instruction, the robot arm synchronously acquires an environmental image. The environmental image can be as follows Figure 6 As shown. The robot arm uses the large language model combined with the environment image to disassemble the task instruction and output the first subtask. The large language model is used to output the natural language instruction corresponding to the first subtask to the robot arm control model. The natural language instruction corresponding to the first subtask is used to instruct the robot arm to move directly above the object C. The robot arm control model of the robot arm controls the robot arm to perform the corresponding action based on the natural language instruction and move it directly above the object C.

[0136] The large language model determines that the robot arm moves to the top of object C based on the real-time environment image, and when the speed of each joint of the robot arm is 0, the second subtask is predicted based on the first subtask. The large language model is used to output the natural language instructions corresponding to the second subtask to the robot arm control model. The natural language instructions corresponding to the second subtask are used to instruct the robot arm to move to the position of object C. The robot arm control model of the robot arm controls the robot arm to perform the corresponding action based on the natural language instructions and moves to the position of object C.

[0137] The large language model determines the position of the robot arm moving to object C based on the real-time environment image, and when the speed of each joint of the robot arm is 0, the third subtask is predicted based on the second subtask. The large language model is used to output the natural language instructions corresponding to the third subtask to the robot arm control model. The natural language instructions corresponding to the third subtask are used to instruct the robot arm to clamp object C. The robot arm control model of the robot arm controls the robot arm to perform the corresponding action based on the natural language instructions to clamp object C.

[0138] The large language model determines the position of the robot arm moving to object C based on the real-time environmental image, and when the speed of each joint of the robot arm is 0 and the gripper state of the robot arm is closed, the fourth subtask is predicted based on the third subtask. The large language model is used to output the natural language instructions corresponding to the fourth subtask to the robot arm control model. The natural language instructions corresponding to the fourth subtask are used to instruct the robot arm to lift up. The robot arm control model of the robot arm controls the robot arm to perform the corresponding action based on the natural language instructions to lift the robot arm. It can be understood that because the robot arm clamped object C before it was lifted, the object C was lifted up synchronously when the robot arm was lifted up.

[0139] The large language model determines that the robot arm is lifted up based on the real-time environmental image, and the speed of each joint of the robot arm is 0. When the gripper state of the robot arm is closed, the fifth subtask is predicted based on the fourth subtask. The large language model is used to output the natural language instructions corresponding to the fifth subtask to the robot arm control model. The natural language instructions corresponding to the fifth subtask are used to instruct the robot arm to move directly above area B. The robot arm control model of the robot arm controls the robot arm to perform the corresponding action based on the natural language instructions and moves to directly above area B. It can be understood that because the robot arm clamped object C before it was lifted up, the robot arm simultaneously drove object C to directly above area B when it moved to directly above area B.

[0140] Based on the real-time environmental image, the large language model determines that the robot arm moves to the top of area B, and the speed of each joint of the robot arm is 0, and the gripper state of the robot arm is closed, and predicts the sixth subtask based on the fifth subtask. And use the large language model to output the natural language instructions corresponding to the sixth subtask to the robot arm control model. The natural language instructions corresponding to the sixth subtask are used to instruct the robot arm to move down to area B. The robot arm control model of the robot arm controls the robot arm to perform the corresponding action based on the natural language instructions and move down to area B. It can be understood that when the robot arm moves down to area B, it simultaneously drives object C to move down to area B.

[0141] The large language model determines that the robot arm moves down to area B based on the real-time environmental image, and the speed of each joint of the robot arm is 0. When the gripper state of the robot arm is closed, the seventh subtask is predicted based on the sixth subtask. The large language model is used to output the natural language instructions corresponding to the seventh subtask to the robot arm control model. The natural language instructions corresponding to the seventh subtask are used to instruct the robot arm to open the gripper. The robot arm control model of the robot arm controls the robot arm to perform the corresponding action and open the gripper based on the natural language instructions. It can be understood that after the gripper is opened, object C is placed in area B. At this point, the robot arm moves object C to area B according to the input task instructions.

[0142] See also Figure 7 , Figure 7 is a schematic diagram of the structure of an embodiment of a robotic arm provided by the present application. The robotic arm 70 includes a processor 71, a memory 72 coupled to the processor 71, and an actuator 73; wherein the memory 72 is used to store a computer program, and the processor 71 is used to execute the computer program to implement the following method:

[0143] Receive task instructions and an environment image; wherein the environment image has depth information; input the task instructions and the environment image into a large language model, so that the large language model sequentially outputs natural language instructions corresponding to a plurality of subtasks; wherein the large language model does not have code generation capability; wherein each subtask is predicted when the previous subtask is completed; input the natural language instructions sequentially into a robot arm control model, so that the robot arm control model controls the robot arm to perform corresponding actions based on the natural language instructions until all subtasks are completed.

[0144] It can be understood that the processor 71 is also used to execute computer programs to implement the method of any of the above embodiments, which will not be elaborated here.

[0145] In some embodiments, the actuator 73 may be a joint assembly and a gripper on the robot arm 70 .

[0146] In other embodiments, the actuator 73 further includes an image acquisition component. The image acquisition component can be disposed in a corresponding area around the clamping jaw, and is used to acquire an environmental image in real time when the joint component in the actuator 73 drives the clamping jaw to move.

[0147] See also Figure 8 , Figure 8 1 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 80 is used to store a computer program 81. When the computer program 81 is executed by a processor, it is used to implement the following method:

[0148] Receive task instructions and an environment image; wherein the environment image has depth information; input the task instructions and the environment image into a large language model, so that the large language model sequentially outputs natural language instructions corresponding to a plurality of subtasks; wherein the large language model does not have code generation capability; wherein each subtask is predicted when the previous subtask is completed; input the natural language instructions sequentially into a robot arm control model, so that the robot arm control model controls the robot arm to perform corresponding actions based on the natural language instructions until all subtasks are completed.

[0149] It can be understood that when the computer program 81 is executed by the processor, it is also used to implement the method of any of the above embodiments, which will not be elaborated here.

[0150] To summarize, the control method of the robotic arm, the robotic arm, and the computer-readable storage medium provided in the present application utilize a large language model that does not have the ability to generate code in combination with environmental images to decompose task instructions, and output natural language instructions with action decisions corresponding to each subtask, so that the robotic arm control model controls the robotic arm to perform corresponding actions based on the natural language instructions until all subtasks are completed. This method can fully utilize the language understanding and reasoning capabilities of the large language model that does not have the ability to generate code, and can provide the robotic arm with rich action semantic information, thereby solving the problem of insufficient understanding of action semantics in related technologies.

[0151] Furthermore, the use of large language models without code generation capabilities in robot arm control significantly reduces the computing resource requirements for the robot arm and improves the reproducibility of the technology and the feasibility of practical applications.

[0152] Furthermore, each subtask output by the large language model is predicted when the previous subtask is completed, which can ensure that each subtask is in the best state to execute the next subtask, and accurately predict the next subtask when the robotic arm is in the ideal position, ensuring the continuity and accuracy of the robotic arm's tasks.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the circuit or unit is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the description and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for controlling a robotic arm, characterized in that: The method comprises: Receiving a task instruction and an environment image; wherein the environment image has depth information; Inputting the task instruction and the environment image into a large language model, so that the large language model sequentially outputs natural language instructions corresponding to a plurality of subtasks; wherein the large language model does not have code generation capability; wherein each subtask is predicted when the previous subtask is completed; The natural language instructions are input into the robot arm control model in sequence, so that the robot arm control model controls the robot arm to perform corresponding actions based on the natural language instructions until all subtasks are completed.

2. The method according to claim 1, characterized in that The multiple subtasks include at least a first subtask and a second subtask; the inputting the task instruction and the environment image into the large language model so that the large language model sequentially outputs natural language instructions corresponding to the multiple subtasks includes: Inputting the task instruction and the environment image into a large language model so that the large language model outputs a first natural language instruction corresponding to the first subtask; When it is detected that the first subtask is completed, the second subtask is predicted using the large language model, and a second natural language instruction corresponding to the second subtask is output.

3. The method according to claim 1, characterized in that: In the process of controlling the robot arm to perform the corresponding action, the large language model is used to recognize the current environment image to determine whether the subtask is completed; If the execution is not completed, continue to input the natural language instruction corresponding to the subtask into the robot arm control model; If the execution is completed, the natural language instructions corresponding to the next subtask are input into the robot arm control model.

4. The method according to claim 3, characterized in that The step of using the large language model to identify the current environment image and determining whether the subtask is completed includes: Using the large language model to identify the position of the robot arm gripper in the current environment image; Whether the subtask is completed is determined based on the position of the clamping jaw.

5. The method according to claim 1, characterized in that The large language model is trained in the following way: Obtain the trajectory flow and environmental data corresponding to the robot arm completing the target task; Decomposing the track stream into a plurality of sub-track streams based on key frames; wherein each track stream is annotated with a corresponding action language instruction; The large language model is trained using the task instructions corresponding to the target task, the multiple sub-trajectory streams, the environmental data, and the position of the robotic arm at the key frame, so that the large language model learns the task decomposition capability.

6. The method according to claim 5, characterized in that The step of decomposing the track stream into a plurality of sub-track streams based on key frames includes: Determine the trajectory frame in which the gripper state changes or the robot arm speed is 0 within a preset number of consecutive frames as the key frame; The track stream is decomposed into a plurality of sub-track streams based on the key frames.

7. The method according to claim 5, characterized in that The robot control model is trained in the following way: The robotic arm control model is trained using the multiple sub-trajectory streams, environmental data, and the position of the robotic arm at the key frame.

8. The method according to claim 7, characterized in that The step of training the robotic arm control model by using the plurality of sub-trajectory streams, environmental data, and the position of the robotic arm at the key frame comprises: inputting the plurality of sub-trajectory streams, the environmental data and the position of the robot arm at the key frame into the robot arm control model so that the robot arm controls the robot arm for imitation learning; After the imitation learning of each sub-trajectory flow is completed, the actual position, actual rotation angle, and actual state of the gripper on the robotic arm are obtained; Determine a first loss using the final position of the robot arm and the actual position in each sub-trajectory stream in the environmental data; Determine a second loss using the final rotation angle of the robot arm and the actual rotation angle in each sub-trajectory stream; Determine a third loss using the final state of the gripper on the robot arm and the actual state in each sub-trajectory flow; The model parameters of the robot arm control model are adjusted in combination with the first loss, the second loss and the third loss.

9. A robotic arm, characterized in that: The robotic arm comprises: a processor, a memory coupled to the processor, and an execution mechanism; wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the computer program is used to implement the method according to any one of claims 1 to 8.

Images