A dual-arm manipulation task learning method and system based on a large model
Through visual language big model decomposition of tasks and design reward functions, combining reinforcement learning and imitation learning, the robustness problem of complex two-arm task learning is solved, and a strategy that is consistent with human operation habits is realized, which is suitable for multiple two-arm operation tasks. .
Patent Information
- Application Number
- CN202410096440.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-23
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-01-23
AI Technical Summary
The prior art has robustness problems in learning complex two-arm tasks and close to human operation strategies, and a single task learning method cannot be applied to multiple tasks, requires a large amount of manual design, and lacks universality.
By receiving task environment diagram and task description information, using the visual language big model to decompose it into subtasks, designing reward functions based on large language models, and training subtask models based on reinforcement learning models, combining imitation learning algorithms to build a two-arm operation strategy to achieve The connection between task decomposition and subtasks.
It reduces the difficulty of learning complex two-arm tasks, designs a robust operation strategy that is more in line with human habits, realizes learning of any two-arm operation tasks, and improves the safety and efficiency of task execution.
Smart Images
Figure CN117697763B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of neural network learning and robotics technology, and in particular to a dual-arm operation task learning method and system based on a large model. Background Art
[0002] In recent years, with the rapid development of artificial intelligence, robots have gradually demonstrated excellent capabilities in various tasks and can complete different operation tasks. Traditional technologies mainly focus on single-arm robots, but single-arm robots have great limitations in their use in daily life. Dual-arm robots have higher dexterity and workspace, can achieve more delicate and robust operations, and can better complete different tasks. For example, in order to smoothly lift a heavy bucket, two robotic arms are required to grasp the upper edges of the bucket on both sides and move in the same direction; and in order to rotate a display counterclockwise, one robotic arm is required to support the display downward, and the other robotic arm applies force to rotate it. Although the existing technology has different designs for different tasks and can achieve good performance in different dual-arm operation tasks (such as the operation of articulated objects), there are many tasks in daily life. The learning method for a single task cannot be applied to all tasks, and it takes a lot of time for humans to design methods, network structures, etc., which is not universal.
[0003] The existing technology uses the self-evolution of a large language model to design the reward function required for reinforcement learning, which promotes the learning of different robot tasks. However, the reward function design guided by the task goal alone cannot enable the robot to learn human behavior. Taking the example of lifting a pot with two hands, the result learned by the existing algorithm is that the pot is lifted up with two hands, rather than grabbing the handles on both sides of the pot and lifting it up. Therefore, the traditional strategy is not robust and has safety issues in real-world execution. In addition, the existing algorithm still cannot learn to get good results on more complex tasks. Summary of the invention
[0004] In order to solve the problem of learning complex dual-arm tasks and human-like operation strategies in the prior art, the present invention provides the following technical solutions.
[0005] In a first aspect, the present invention provides a dual-arm manipulation task learning method based on a large model, comprising:
[0006] Receiving a task environment diagram and task description information for dual-arm operation, inputting the task environment diagram and task description information into a visual language macromodel, and decomposing them into a plurality of subtasks;
[0007] Determine a reward function according to the task description information, task perception space information and action space information corresponding to each subtask, and perform training based on a reinforcement learning model to construct a corresponding subtask model, wherein the subtask model is used to complete each subtask respectively;
[0008] According to the operation sequence of multiple subtasks, multiple subtask models are called in sequence to perform dual-arm operation, and a dual-arm trajectory data set is collected. The dual-arm trajectory data set is learned through an imitation learning algorithm to obtain a dual-arm operation strategy.
[0009] Preferably, the task perception space information includes robot joint angles, robot joint angular velocities and / or object postures, etc., and the action space information includes motion range and / or dimensions, etc.
[0010] Preferably, the determining of the reward function further comprises:
[0011] The task description information, task perception space information and action space information corresponding to each subtask are input into a reward function design framework based on a large language model, and the framework is used to create a reward function corresponding to the subtask.
[0012] Preferably, the constructing of the corresponding subtask model further comprises:
[0013] After the training of the current subtask is completed, the end state of the current subtask is collected as the initial state of the next subtask, and the next subtask is trained according to the same steps, and iterated in sequence until all subtasks are trained.
[0014] In a second aspect, the present invention provides a dual-arm manipulation task learning system based on a large model, comprising:
[0015] A task decomposition module, used for receiving a task environment diagram and task description information for dual-arm operation, inputting the task environment diagram and task description information into a visual language macromodel, and decomposing them into a plurality of subtasks;
[0016] A subtask training module is used to determine a reward function according to the task description information, task perception space information and action space information corresponding to each subtask, and to perform training based on a reinforcement learning model to construct a corresponding subtask model, wherein the subtask model is used to complete each subtask;
[0017] The subtask coupling module is used to call multiple subtask models in sequence according to the operation order of multiple subtasks to perform dual-arm operation, collect dual-arm trajectory data sets, learn the dual-arm trajectory data sets through the imitation learning algorithm, and obtain the dual-arm operation strategy.
[0018] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the method of the first aspect.
[0019] A fourth aspect of the present invention provides a storage medium, wherein the storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor to execute the method of the first aspect.
[0020] The beneficial effects of the present invention are:
[0021] The dual-arm operation task learning method based on a large model of the present invention decomposes complex or long-range dual-arm tasks through the ability of the visual language large model, reduces the difficulty of directly learning such problems, realizes the learning of any dual-arm operation task, and provides the same description of operation habits as when humans perform corresponding tasks, so that the reward function can design a more human-friendly and more robust operation strategy. By constructing a successful trajectory data set and using the imitation learning method, the connection between different subtasks is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 The diagram shows the architecture of the large-model-based dual-arm manipulation task learning system described in the present invention.
[0023] Figure 2 A flow chart of the large model-based dual-arm manipulation task learning method described in the present invention is shown.
[0024] Figure 3 An environment diagram used by a specific example of the present invention is shown. DETAILED DESCRIPTION
[0025] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0026] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.
[0027] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.
[0028] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.
[0029] The display screen is used to display the user interface of each application.
[0030] In addition, those skilled in the art will appreciate that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.
[0031] Compared with a single-arm robot, the difficulty of dual-arm collaborative tasks increases significantly, because the degrees of freedom of the action space increase exponentially, and the complexity of the problem increases exponentially. In order to achieve the learning of arbitrary complex dual-arm tasks and human-like operation strategies, the present invention proposes the following technical solutions. First, the target of any task and the image related to the environment are input into the language vision model as prompt words to obtain a series of subtasks. By decomposing complex tasks, the learning difficulty of the tasks is reduced. At the same time, the language vision model is used to obtain task descriptions that conform to human habits for different subtasks from the prior knowledge of human operation of objects.
[0032] Secondly, for each subtask, the subtask and the task description that conforms to human habits are used as input. Through the reward function design algorithm based on the large language model, an effective reward function is designed for each subtask for reinforcement learning training, so as to obtain a model that can complete each subtask. After that, in order to solve the connection problem between different subtasks, different subtasks are executed in sequence to collect a trajectory data set that successfully completes the entire task. Finally, through imitation learning, the successful trajectory data set is distilled to a model with real perception, which is used to complete specific dual-arm operation tasks.
[0033] The first aspect of the present invention provides a dual-arm manipulation task learning system based on a large model, which includes a task decomposition module, a subtask training module and a subtask coupling module.
[0034] The task decomposition module is used to receive a task environment diagram and task description information for dual-arm operation, input the task environment diagram and task description information into a visual language large model, and decompose them into multiple subtasks.
[0035] Specifically, the task decomposition module receives the task environment diagram and the overall task description input by the user, and decomposes the total task through a large visual language model (such as LLaVA, GPT4V, etc.) to obtain n subtasks required to complete the total task, and each subtask contains detailed description information.
[0036] The task description is text information, for example, "This task corresponds to Block Stack. This environment involves two hands and two blocks, and we need to stack the blocks into a tower."
[0037] The subtask training module is used to determine the reward function according to the task description information, task perception space information and action space information corresponding to each subtask, and to perform training based on the reinforcement learning model to construct a corresponding subtask model, which is used to complete each subtask respectively.
[0038] Specifically, the subtask training module receives each subtask 1 to n obtained by the task decomposition module. Assuming that the first subtask is subtask 1, the task perception space information can be the robot joint angle, robot joint angular velocity or object posture, etc., and the action space information can be the range of motion or dimension, etc. Based on the information contained in the user-specified task perception space information and action space information, the task description corresponding to subtask 1 is input into the reward function design framework based on the large language model (such as Eureka) to perform autonomous reward function design, and is trained based on a reinforcement learning model (such as PPO) to obtain a model that can complete the subtask. Collect the end state of the task as the initial state of the next subtask (subtask 2), and continue to train subtask 2 according to the above steps, iterating in sequence until subtask n is trained.
[0039] The subtask coupling module is used to call multiple subtask models in sequence according to the operation order of multiple subtasks to perform dual-arm operation, collect a dual-arm trajectory data set, and learn the dual-arm trajectory data set through an imitation learning algorithm to obtain a dual-arm operation strategy.
[0040] Specifically, the subtask coupling module is based on the subtask modules obtained in the subtask training module, and in accordance with the subtask order, sequentially calls the aforementioned subtask modules to control the actions of the dual-arm robot, and collects a trajectory data set that can eventually successfully complete the task. Based on the collected trajectory data set, the expert demonstration data is learned using an imitation learning-based method (such as a diffusion policy model), and a model with real perception as input is distilled to obtain the operation strategy of the overall task, thereby completing the coupling between different subtasks.
[0041] like Figure 2As shown, another aspect of the present invention provides a dual-arm manipulation task learning method based on a large model, comprising:
[0042] S101, receiving a task environment diagram and task description information for dual-arm operation, inputting the task environment diagram and task description information into a visual language large model, and decomposing them into a plurality of subtasks.
[0043] First, the task decomposition module receives the task environment diagram and the overall task description input by the user, and decomposes the total task through a large visual language model (such as LLaVA, GPT4V, etc.) to obtain n subtasks required to complete the total task, each of which contains detailed description information.
[0044] S102, determining a reward function according to the task description information, task perception space information and action space information corresponding to each subtask, and performing training based on a reinforcement learning model to construct a corresponding subtask model, wherein the subtask model is used to complete each subtask.
[0045] The subtask training module receives each subtask 1 to n. Assuming that the first subtask is subtask 1, based on the information contained in the perception space information and action space information of the task specified by the user (such as the task perception space information can be the robot joint angle, robot joint angular velocity or object posture, etc., and the action space information can be the range of motion or dimension, etc.), the task description corresponding to subtask 1 is input into the reward function design framework based on the large language model (such as Eureka) to perform autonomous reward function design, and train the reinforcement learning model (such as PPO) based on the reward function to obtain a model that can complete the subtask. Collect the end state of the task as the initial state of the next subtask (subtask 2), and continue to train subtask 2 according to the above steps, iterating in sequence until subtask n is trained.
[0046] S103. According to the operation sequence of the multiple subtasks, call the multiple subtask models in sequence to perform dual-arm operation, collect a dual-arm trajectory data set, learn the dual-arm trajectory data set through an imitation learning algorithm, and obtain a dual-arm operation strategy.
[0047] The subtask coupling module is based on the subtask modules obtained in the subtask training module. In accordance with the subtask order, it calls the aforementioned subtask modules in sequence to control the actions of the dual-arm robot, and collects the trajectory data set that can eventually successfully complete the task. Based on the collected trajectory data set, it uses the imitation learning-based method (such as the diffusion model diffusion policy) to learn the expert demonstration data, thereby learning the operation strategy of the overall task and completing the coupling between different subtasks.
[0048] Combined with the following Figure 3, the method of the present invention is explained in the form of specific examples.
[0049] The input task description is "This task corresponds to stacking blocks. This environment involves two hands and two blocks, and we need to stack the blocks into a tower. Please decompose the subtasks according to the current environment picture and task objectives." The corresponding picture is as follows Figure 3 shown.
[0050] Input the above content into the visual language model to obtain the description of the subtask, such as:
[0051] 1. Grab the first block with your right hand;
[0052] 2. Lift the first block you grab;
[0053] 3. Move the first block to the top of the second block and align them;
[0054] 4. Reduce the height of the second block;
[0055] 5. Release the block.
[0056] For each subtask, the subtask description and perception space (such as robot joint angles, robot joint angular velocities or object postures, etc.), action space (robot movable joints and motion range, etc.) information are input into the reward function design framework Eureka, and the reward function that can complete each subtask is automatically obtained. Based on the reward function, the reinforcement learning model is trained to complete each subtask, such as grasping, moving, and putting down.
[0057] Then, in the environment, the subtasks are executed in sequence using the corresponding models, the trajectories of successfully completing the final task are collected, a trajectory dataset is constructed, and finally imitation learning is used to learn the strategy that uses real perception as input.
[0058] It can be seen that the above-mentioned dual-arm manipulation task learning method based on a large model of the present invention has the following advantages over the prior art:
[0059] Through the ability of the visual language large model, complex or long-range two-arm tasks are decomposed, reducing the difficulty of directly learning such problems, and achieving the learning of any two-arm operation task. It provides the same description of operation habits as when humans perform corresponding tasks, so that the reward function can design a more human-friendly and more robust operation strategy. By constructing a successful trajectory dataset and using the imitation learning method, the connection between different subtasks is achieved.
[0060] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute any of the aforementioned methods. The processor and the memory can be connected via a bus or other means, taking the bus connection as an example. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned chips.
[0061] As a non-transient storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the large model-based dual-arm operation task learning method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, realizing the large model-based dual-arm operation task learning method in the above method embodiment.
[0062] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0063] The present invention also provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions can be loaded and executed by a processor, so that the processor can execute any one of the above-mentioned method embodiments. The storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.
[0064] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A dual-arm manipulation task learning method based on a large model, characterized in that: include: Receiving a task environment diagram and task description information for dual-arm operation, inputting the task environment diagram and task description information into a visual language macromodel, and decomposing them into a plurality of subtasks; Determine a reward function according to the task description information, task perception space information and action space information corresponding to each subtask, and perform training based on a reinforcement learning model to construct a corresponding subtask model, wherein the subtask model is used to complete each subtask respectively; According to the operation sequence of the multiple subtasks, the multiple subtask models are called in sequence to perform the dual-arm operation, and a dual-arm trajectory data set is collected. The dual-arm trajectory data set is learned through an imitation learning algorithm to obtain a dual-arm operation strategy; The determining of the reward function further comprises: Input the task description information, task perception space information and action space information corresponding to each subtask into the reward function design framework based on the large language model, and use the framework to create the reward function corresponding to the subtask; The constructing of the corresponding subtask model further includes: After the training of the current subtask is completed, the end state of the current subtask is collected as the initial state of the next subtask, and the next subtask is trained according to the same steps, and iterates in sequence until all subtasks are trained; The method sequentially calls a plurality of subtask models to perform dual-arm operation, collects a dual-arm trajectory data set, and learns the dual-arm trajectory data set through an imitation learning algorithm to obtain a dual-arm operation strategy, further comprising: Based on the collected trajectory dataset, the expert demonstration data is learned based on the diffusion model algorithm to learn the operation strategy of the overall task, complete the coupling between different sub-tasks, distill the successful trajectory dataset into a model with real perception, and complete the dual-arm operation task.
2. The large model-based dual-arm manipulation task learning method according to claim 1 is characterized in that: The task perception space information is the robot joint angle, robot joint angular velocity and / or object posture, and the action space information is the motion range and / or dimension.
3. A dual-arm manipulation task learning system based on a large model, characterized in that: include: A task decomposition module, used for receiving a task environment diagram and task description information for dual-arm operation, inputting the task environment diagram and task description information into a visual language macromodel, and decomposing them into a plurality of subtasks; A subtask training module is used to determine a reward function according to the task description information, task perception space information and action space information corresponding to each subtask, and to perform training based on a reinforcement learning model to construct a corresponding subtask model, wherein the subtask model is used to complete each subtask; A subtask coupling module is used to call multiple subtask models in sequence to perform dual-arm operation according to the operation order of multiple subtasks, collect a dual-arm trajectory data set, and learn the dual-arm trajectory data set through an imitation learning algorithm to obtain a dual-arm operation strategy; The subtask training module is further used to: Input the task description information, task perception space information and action space information corresponding to each subtask into the reward function design framework based on the large language model, and use the framework to create the reward function corresponding to the subtask; The subtask training module is further used to: After the training of the current subtask is completed, the end state of the current subtask is collected as the initial state of the next subtask, and the next subtask is trained according to the same steps, and iterates in sequence until all subtasks are trained; The subtask coupling module is further used to: Based on the collected trajectory dataset, the expert demonstration data is learned based on the diffusion model algorithm to learn the operation strategy of the overall task, complete the coupling between different sub-tasks, distill the successful trajectory dataset into a model with real perception, and complete the dual-arm operation task.
4. The large model-based dual-arm manipulation task learning system according to claim 3 is characterized in that: The task perception space information is the robot joint angle, robot joint angular velocity and / or object posture, and the action space information is the motion range and / or dimension.
5. An electronic device, characterized in that: It includes a processor and a memory, the memory stores multiple instructions, and the processor is used to read the instructions and execute the large model-based dual-arm manipulation task learning method as described in any one of claims 1-2.
6. A storage medium, characterized in that: The storage medium stores multiple instructions, which can be read and executed by a processor as described in any one of claims 1-2 as a large model-based dual-arm manipulation task learning method.