Robot large model-oriented physical fine-grained controllable animation synthesis method and system

Abstract

The invention relates to the field of robot training and teaching, in particular to a physical fine-grained controllable animation synthesis method and system for a robot large model, and aims at solving the problems that in robot operation video simulation in the prior art, the construction cost of a complex scene is high, and the simulation efficiency is high. And the requirement of a large robot model which develops rapidly at present and faces general tasks is difficult to meet. According to the technical scheme, the robot operation scene layout and the related operation process are automatically generated through text description and are converted into the code snippets, the vivid video scene and the fine-grained scene description are generated through running simulation, and support is provided for robot training and teaching.

Description

Technical Field

[0001] The present disclosure relates to the field of robot training and teaching, and in particular to a physically fine-grained controllable animation synthesis method and system for a large robot model. Background Art

[0002] With the widespread use of robots, their operating scenarios are becoming more and more complex, requiring further enhanced descriptions based on multimodal large language models (LLMs) to synthesize more complex scenarios and support precise robot operations.

[0003] The currently open robot operation video simulation has relatively simple operation tasks and relatively simple operation scenes, but the construction cost of complex scenes is high, which makes it difficult to meet the needs of the rapidly developing large robot models for general tasks. Summary of the invention

[0004] The present disclosure aims to solve the problems of lack of training data for robot precision operations in complex operation scenarios, insufficient realism of physical simulation, and high cost of actual environment construction. The specific technical solution is as follows.

[0005] In a first aspect, the present disclosure proposes a physically fine-grained controllable animation synthesis method for a large robot model, the method comprising the following steps: based on a task description and a preset layout text prompt, obtaining a description of the physical properties of the subject in the scene as current layout information, and generating a scene setting code segment based on the layout information and a preset first code text prompt; based on the task description and the current layout information, obtaining an operating task for robot arm motion control, and generating a robot arm motion configuration code segment based on the operating task for robot arm motion control and a preset second code text prompt; and generating an animation video based on the scene setting code segment and the robot arm motion configuration code segment.

[0006] In one implementation of the above technical solution, the current layout information acquisition step includes: based on the task description and the preset first layout text prompt, using the large language model to obtain the scene subject description in the knowledge base corresponding to the layout text prompt; based on the obtained scene subject and the preset second layout text prompt, using the large language model to obtain the subject relationship description in the knowledge base corresponding to the layout text prompt; based on the obtained subject relationship and the third layout text prompt, using the large language model to obtain the subject physical property description in the knowledge base corresponding to the layout text prompt; wherein the first layout text prompt, the second layout text prompt and the third layout text prompt are layout text prompts; the subject fine-grained attribute library is a knowledge base about the subject, the relationship between subjects and the physical properties of the subject.

[0007] In one implementation of the above technical solution, the task description is described using natural language instructions.

[0008] In one implementation of the above technical solution, based on the layout information and the preset first code text prompt, the code language model CodeLLMLayout is used to perform code retrieval in the code knowledge base to obtain the scene setting code segment closest to the layout information and the preset first code text; based on the operation task of the robot motion control and the preset second code text prompt, the code language model CodeLLMAction is used to perform code retrieval in the code knowledge base to obtain the robot motion configuration code segment closest to the operation task of the robot motion control and the preset second code text prompt. The code retrieval adopts SimCodeLLM.

[0009] In one implementation of the above technical solution, the step of acquiring the operation task of the robot arm motion control includes: based on the task description and the current layout information, retrieving the knowledge closest to the task description and the current layout information from the task knowledge base. The retrieval adopts the SimLLM language model.

[0010] In a second aspect, the present disclosure proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the above methods.

[0011] In a third aspect, the present disclosure proposes a physical fine-grained controllable animation synthesis system for a large robot model, the system comprising a scene setting module, a robot arm motion configuration module, and a video generation module; wherein: the scene setting module is configured to obtain a description of the physical properties of the subject in the scene as current layout information based on a task description and a preset layout text prompt, and to generate a scene setting code segment based on the layout information and a preset first code text prompt; the robot arm motion configuration module is configured to obtain a job task of robot arm motion control based on the task description and the current layout information, and to generate a robot arm motion configuration code segment based on the job task of robot arm motion control and a preset second code text prompt;

[0012] The video generation module is configured to generate an animation video based on the scene setting code segment and the robotic arm motion configuration code segment.

[0013] The beneficial technical effects of the technical solution disclosed in the present invention are as follows: it is capable of generating complex operation scene simulation data in a controllable manner at low cost, meeting the needs of large robot models for general tasks, and providing data support for improving the performance of large robot models in fine operation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 , one A schematic diagram of a method flow chart in an embodiment of the present invention.

[0015] Figure 2 , one A schematic diagram of the dialogue-based generation of an initial description of the action scenario layout in this implementation mode.

[0016] Figure 3 , one Schematic diagram of iterative optimization of job videos in this implementation mode. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, the method proposed in the present disclosure includes the following steps:

[0018] S1. Generate scene setting code segment

[0019] Based on the task description and the preset layout text prompt, a description of the physical properties of the subject in the scene is obtained as the current layout information, and based on the layout information and the preset first code text prompt, a scene setting code segment is generated.

[0020] S2. Generate robot arm motion configuration code segment

[0021] Based on the task description and the current layout information, an operation task of the robot arm motion control is acquired, and based on the operation task of the robot arm motion control and a preset second code text prompt, a robot arm motion configuration code segment is generated.

[0022] S3. Generate animation video

[0023] An animation video is generated based on the scene setting code segment and the robot arm motion configuration code segment.

[0024] The following will provide a clear and complete description of how to implement the technical solution of this case in conjunction with the accompanying drawings. Obviously, the described implementation method is only a part of the implementation method of this case, not all of the implementation methods.

[0025] The virtual simulation environment for implementing the above technical solution is created based on the NVIDIA Omniverse platform, which can improve simulation accuracy and use the multimodal visual language large model to automatically generate complex working scenarios and mechanical action instructions, solving the bottlenecks of scene complexity and insufficient data in the current training of large robot models, and providing support for improving the performance of large robot models in fine working tasks.

[0026] (I) Layout and configuration based on multimodal large language model

[0027] The robot operation task is described by giving natural language instructions. The natural language is relative to the programming language, which is the language used by people to share information with each other. For example, the comments in the program are all natural language to explain the meaning of the corresponding code so that others can understand it. Robot operation refers to the process of using robot technology to complete a specific task.

[0028] See also Figure 2 , a subject relationship library (100) and a subject fine-grained attribute library (200) are constructed in advance. Based on the robot operation task described by a given natural language instruction, the subject relationship library is used to perform "scene subject" reasoning (101) and "subject relationship" reasoning (102) in sequence, and then "subject physical attribute" reasoning (201) is performed in the subject fine-grained attribute library. Figure 2 The subject relationship library includes subjects and subject relationships.

[0029] Using a method similar to the WizardMath logical language model, we obtain reasoning information through prompts in a continuous dialogue to construct the initial layout:

[0030] T i+1 =LogLLM(T i , prompt i , PreSet i ) (1)

[0031] Among them, T i+1 Indicates the i+1th layout information, prompt i Represents the i-th layout information T i The prompts include "Derive scene layout, what are the subjects?", "Derive scene layout, the relationship between subjects?", "Derive scene layout, the fine-grained physical properties of the subject?". i The knowledge base corresponding to the preset prompts, such as the subject library, subject relationship library and subject physical property library, can also be a knowledge base, which includes subjects, subject relationships and subject physical properties. LogLMM is fine-tuned by the open large model WizardMath, and introduces 3 special symbols, which are used to bond the parameters in LogLLM into a sequence, so as to understand each content to generate layout and other information. In this example, it specifically includes various existing part bodies, subject physical property information such as screw A: stainless steel; weight 100g; diameter 10mm, and subject relationship library information such as screw A can be assembled on nut B.

[0032] pass Figure 2 The task description of the scenario is progressively generated in this way.

[0033] Use the given natural language instructions to describe the scene. Take "pick up part A from the pile of non-standard parts on the table in front" as an example, and record the task description as Task, then Task = "generate a scene where two robotic arms pick up from the three objects on the table."

[0034] The task description is used as the initial layout information, and the initial layout information is denoted as T 0 , then T 0 = "Generate a scene with two robotic arms to pick up three objects on the table." In the first layout text prompt 0 = "Derive the scene layout, what are the subjects?" Guided by the first layout information, the first layout information is recorded as T 1 ,but:

[0035] T 1 =LogLLM(T 0 , prompt 0 , PreSet 0 )

[0036] T 1 = "The main body of the scene is a) a table named table, located at the origin (0, 0, 0.0). b) a parts pile named part_heap, located to the left of the table (-0.5, 0, 0.3), containing objects A, B, and C. c) two robotic arms, named arm1 and arm2. They are both ordinary robotic arms that can rotate the second joint on the base.". PreSet 0 It is a knowledge base with subjects and subject relationships.

[0037] Based on the layout information T 1 , under the guidance of the second layout text prompt prompt2 = "Derive scene layout, relationship between subjects?", generate the second layout information, and record the second layout information as T 2 ,but:

[0038] T 2 =LogLLM(T 1 , prompt 1 , PreSet 0 )

[0039] T 2 = "The relationships between the subjects are a) The pile of parts is located on the table in the scene. b) B is stacked on A and C is next to A. c) Two robotic arms are mounted on the table.".

[0040] Based on the layout information T 2, under the guidance of the third layout text prompt prompt3 = "derive scene layout, fine-grained physical properties of the subject?", generate the third layout information, and record the third layout information as T 3 ,but:

[0041] T 3 =LogLLM(T 2 , prompt 2 , PreSet 2 )

[0042] T 3 = "a) table: made of metal; length, width and height are 0.5m, 1m and 0.2m respectively. b) part_heap: A is made of plastic, with length, width and height of 0.1m, 0.3m and 0.5m respectively; B is made of glass, with length, width and height of 0.1m, 0.3m and 0.1m respectively; C is made of metal, with length, width and height of 0.1m, 0.1m and 0.1m respectively. c) arm1 and arm2: length, width and height are approximately 0.5m, 0.2m and 0.1m respectively, and are made of metal." PreSet 2 For PreSet 0 The main physical properties of the subject in the.

[0043] The layout information T obtained above 1 、T 2 、T 3 , the specific contents are shown in Table 1 below.

[0044] Table 1

[0045]

[0046] According to the layout information T 3 , generate the robot arm motion control task:

[0047] J = LogLLM(T 3 ,Task,K) (2)

[0048] Task 3 To generate a natural language description of the task, here we exemplarily represent the task prompt of “pick up part A from the pile of non-standard parts on the table in front”, and K is the task that is closest to the current layout information retrieved from the task knowledge base Ω (T 3 , Task) knowledge:

[0049] K=ArgMax x∈Ω SimLLM(T 3 , Task, x)

[0050] The SimLLM language model is fine-tuned through LogLLM distillation. The training data can be completed by iteratively constructing the task knowledge base Ω, that is, the constructed (T 3 , Task), if necessary, it will be manually evaluated, and if the quality is feasible, it will be stored. x is an element in the task knowledge base Ω.

[0051] (II) Generate code based on layout

[0052] In order to reduce the difficulty and improve the generation quality, this patent divides the code generation into two tasks: layout code segment generation and motion code segment generation. The code segment filling method is used to limit the generated code form and improve its quality.

[0053] Based on the code language model CodeLLMLayout similar to Codeqwen, the code segment is constructed as follows:

[0054] C1=CodeLLMLayout(T 3 , K C1 , CodeTemple)

[0055] C1 is the final scene construction code segment, K C1 is a text prompt, and its exemplary content is shown in Table 3. CodeTemple is the code that is closest to the current task (T 3 , K C1 )

[0056] CodeTemple=ArgMax x∈Ω SimCodeLLM(T 3 , K C1 , x)

[0057] The SimCodeLLM language model is fine-tuned through the CodeLLMLayout instruction, which is similar to the training of the SimLLM language model.

[0058] In addition, if the constructed code segment construction scenario is inconsistent with the expectation, manual modification is performed, and then the manual modification results are introduced into the training data, and the instructions are fine-tuned to correct the CodeLLMLayout model. CodeLLMLayout training is completed using the code segment filling method, see Table 2 for the code segment with blanks, and Table 3 for the code segment generated in the blank place. The example code snippet is generated on the 0mniverse platform.

[0059] Table 2

[0060]

[0061]

[0062] Table 3

[0063]

[0064] In addition, a similar language model CodeLLMAction is constructed to generate a robotic arm motion configuration code segment as shown below:

[0065] C2=CodeLLMAction(J,K C2 , CodeTemple)

[0066] C2 is the final constructed robot arm motion configuration code segment. C2 Similar to K C1 At this time, CodeTemple retrieves the code closest to the current task (J, K C2 ) code.

[0067] Generate realistic video scenes and fine-grained scene descriptions by running simulations to support robot training and teaching.

[0068] (III) Interactive loop refinement of operation animation

[0069] See also Figure 3 ,The interactive loop refinement steps of the job animation include:

[0070] 01. Based on the given task, deduce the subjects, the relationships between subjects, and the physical properties of the subjects in the work scene to generate the current layout information.

[0071] 02. For the current layout information generated in 01, manual intervention can be performed to determine whether the layout is appropriate. If not, the large language model LogLLM, SimLLM, etc. can be fine-tuned through instructions.

[0072] 03. If the current layout information is appropriate, generate the scene setting code segment and the robot arm motion configuration code segment.

[0073] 04. If the code segment does not construct the scene as expected, manual review can be performed to determine whether the motion is appropriate. Manual intervention is performed by using instructions to fine-tune the model SimCodeLLM, CodeLLMLayout, and CodeLLMAction to obtain a code segment that meets expectations.

[0074] 05. If the scene constructed by the code segment is consistent with the expectation, the generated code segment (scene setting code segment and robot arm motion configuration code segment) is integrated into the system and simulated to obtain the generated animation video.

[0075] 06. Determine whether the generated animation video meets the requirements. If not, return to step 01 to regenerate new subjects, inter-subject relationships, and subject physical properties to obtain new current layout information. If only the robot motion configuration does not meet the requirements, return to step 03 to generate new scene setting code segments and robot motion configuration code segments.

[0076] In the above steps, manual intervention can further optimize the operation animation scene. Steps 02 and 04 can be corrected by manual intervention to train data, and the large language model can be fine-tuned in a progressive manner. Manual inspection of the robot large model output is carried out through manual intervention correction to find unreasonable layouts or movements, and make direct corrections to improve the efficiency of video generation in complex scenes, thereby providing support for the performance improvement of the robot large model in fine operation tasks.

[0077] Through the description of the above implementation methods, the technicians in the relevant field can clearly understand that the corresponding system can be realized according to the disclosed method. Exemplarily, a physical fine-grained controllable animation synthesis system for a large robot model, the system includes a scene setting module, a robot motion configuration module, and a video generation module; wherein: the scene setting module is configured to obtain the description of the physical properties of the subject in the scene as the current layout information based on the task description and the preset layout text prompt, and generate a scene setting code segment based on the layout information and the preset first code text prompt; the robot motion configuration module is configured to obtain the operation task of the robot motion control based on the task description and the current layout information, and generate the robot motion configuration code segment based on the operation task of the robot motion control and the preset second code text prompt; the video generation module is configured to generate an animation video based on the scene setting code segment and the robot motion configuration code segment.

[0078] Through the description of the above implementation modes, those skilled in the art can clearly understand that the method and system of the present disclosure can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation mode in most cases.

[0079] Although the embodiments of the present disclosure are described above in conjunction with the accompanying drawings, the present disclosure is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are merely illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present disclosure, all of which belong to the scope of protection of the present disclosure.

Claims

1. A physically fine-grained controllable animation synthesis method for a large robot model, characterized in that: The method comprises the following steps: Based on the task description and the preset layout text prompt, obtain the description of the physical properties of the subject in the scene as the current layout information, and based on the layout information and the preset first code text prompt, generate the scene setting code segment; Based on the task description and the current layout information, an operation task of the robot arm motion control is acquired, and based on the operation task of the robot arm motion control and a preset second code text prompt, a robot arm motion configuration code segment is generated; An animation video is generated based on the scene setting code segment and the robot arm motion configuration code segment.

2. The method according to claim 1, characterized in that The current layout information acquisition steps include: Based on the task description and the preset first layout text prompt, a scene main body description is obtained in the knowledge base corresponding to the layout text prompt using a large language model; Based on the obtained scene subject and the preset second layout text prompt, a description of the relationship between the subjects is obtained in the knowledge base corresponding to the layout text prompt by using a large language model; Based on the obtained inter-subject relationship and the third layout text prompt, a description of the subject's physical attributes is obtained in the knowledge base corresponding to the layout text prompt using a large language model; Among them, the first layout text prompt, the second layout text prompt and the third layout text prompt are layout text prompts; the subject fine-grained attribute library is a knowledge base about subjects, relationships between subjects and physical properties of subjects.

3. The method according to claim 1, characterized in that: The task description is described using natural language instructions.

4. The method according to claim 1, characterized in that: Based on the layout information and the preset first code text prompt, code retrieval is performed in a code knowledge base using a code language model CodeLLMLayout to obtain a scene setting code segment that is closest to the layout information and the preset first code text; Based on the operation task of the robot arm motion control and the preset second code text prompt, the code language model CodeLLMAction is used to perform code retrieval in the code knowledge base to obtain the robot arm motion configuration code segment that is closest to the operation task of the robot arm motion control and the preset second code text prompt.

5. The method according to claim 4, characterized in that: The code retrieval adopts SimCodeLLM.

6. The method according to claim 1, characterized in that The operation task of the robot arm motion control, the acquisition step includes: based on the task description and the current layout information, retrieving the knowledge closest to the task description and the current layout information from the task knowledge base.

7. The method according to claim 6, characterized in that The retrieval adopts the SimLLM language model.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

9. A physically fine-grained controllable animation synthesis system for large robot models, characterized in that: The system includes a scene setting module, a robot arm motion configuration module, and a video generation module; wherein: The scene setting module is configured to obtain a description of the physical properties of the subject in the scene as current layout information based on the task description and a preset layout text prompt, and generate a scene setting code segment based on the layout information and a preset first code text prompt; The robot arm motion configuration module is configured to obtain a work task of the robot arm motion control based on the task description and the current layout information, and generate a robot arm motion configuration code segment based on the work task of the robot arm motion control and a preset second code text prompt; The video generation module is configured to generate an animation video based on the scene setting code segment and the robotic arm motion configuration code segment.

Images