Mechanical arm intelligent control method based on demonstration video imitation learning

Abstract

The invention discloses a mechanical arm intelligent control method based on demonstration video imitation learning, and the method comprises the steps: firstly obtaining a human demonstration video containing a task target, and obtaining a video marked by a key point; the method comprises the following steps: hierarchically extracting semantic information and spatial geometric information by using a multi-modal large language model, decomposing a task into a plurality of sub-task stages, and generating a sub-target constraint function and a path constraint function of each sub-task stage; a task scene similar to a human demonstration video is arranged in a mechanical arm simulation environment, and environment key points are generated through feature extraction and clustering; constructing a mapping function from the video key points to the environment key points; and solving the optimal solution of the sub-target constraint function and the path constraint function of the current working environment of the mechanical arm, and driving the mechanical arm to execute the action until the task is completed. Through fine-grained key point analysis, multi-modal information fusion and cross-scene mapping, the action understanding accuracy, the complex scene adaptability and the task generalization ability of the mechanical arm are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot control and planning, multi-modal video understanding, embodied intelligence, and particularly relates to a robot arm intelligent control method based on demonstration video imitation learning. BACKGROUND

[0002] In the field of robot control and planning, multi-modal understanding, and embodied intelligence, enabling robot arms to imitate human behavior to complete diverse tasks is a key goal to achieve efficient human-robot collaboration and expand the application scenarios of robots. Robot motion imitation aims to analyze human behavior data through intelligent algorithms and convert it into action instructions that can be executed by robot arms, enabling them to accurately and flexibly complete tasks in complex environments, providing efficient and reliable automation solutions for industrial manufacturing, logistics and warehousing, household services, and many other fields.

[0003] With the rapid development of artificial intelligence technology, research on robot motion imitation based on human teaching videos has made some progress. However, current methods still have significant limitations in terms of action understanding accuracy, adaptability to complex scenarios, multi-modal information fusion, and generalization ability. On the one hand, existing technologies often focus only on task-related semantic information in videos, often ignoring key skills or fine-grained operation information that are crucial to task execution success. In actual operation, certain skills are often required to complete specific tasks.

[0004] On the other hand, current robot imitation learning models lack the ability to generalize to different scenarios and tasks. They are usually trained in specific environments and tasks, and once faced with new scene layouts, object shapes, or task requirements, they are difficult to effectively transfer and apply. For example, in industrial production, the layout of different production lines and the placement of workpieces vary greatly, and existing imitation models are difficult to quickly adapt to these changes, requiring a lot of retraining and debugging work, which greatly limits the widespread application of robot arms in actual production. In addition, existing methods also have shortcomings in understanding complex semantics and intentions in human teaching videos, making it difficult to accurately capture the underlying deep goals and strategies behind human operations, leading to errors or inefficiencies in robot task execution. SUMMARY

[0005] The present application addresses the shortcomings of existing technologies and proposes a robot arm intelligent control method based on demonstration video imitation learning, aiming to improve the accuracy, adaptability, and generalization ability of robot motion imitation, more effectively fuse multi-modal information, build a more powerful model to understand complex semantics and intentions in human teaching videos, and enhance the model's transferability in different scenarios and tasks.

[0006] Specifically, the application applies fine-grained key point labels to human demonstration videos and robot working environment observation images to anchor key positions in the environment, and continuously tracks the key point trajectories in the human demonstration. Then, for the tracked key point information, a fine-grained spatial key point constraint model, i.e., a set of numerical constraint functions, behind human behavior is derived through a multimodal large language model (MLLM). Then, the multimodal large language model is used as a mapper to construct a key point mapping model between the human video and the current robot working scene, and the key point constraint information is re-derived according to the mapping relationship. Finally, the optimal solution of the robot end motion under the influence of several key point constraints is calculated by using an optimization method, and the trajectory of the robot end effector is controlled by a general path planning method.

[0007] In order to achieve the above purpose, the technical scheme adopted by the application is as follows:

[0008] A robot intelligent control method based on demonstration video imitation learning, comprising the following steps:

[0009] Step 1, obtaining a human demonstration video containing a task target, extracting key points of objects in the human demonstration video, continuously tracking the key point trajectories, and obtaining a key point labeled video;

[0010] Step 2, taking the video labeled with key points as input, using a multimodal large language model to hierarchically extract semantic information and spatial geometric information, dividing the task into multiple subtask stages, and generating subgoal constraint functions and path constraint functions for each subtask stage;

[0011] Step 3, arranging the same task scene as the human demonstration video in the robot simulation environment, and generating environment key points through feature extraction and clustering;

[0012] Step 4, constructing a mapping function from video key points to environment key points based on a multimodal large language model, and mapping the subgoal constraint functions and path constraint functions to the current robot working environment;

[0013] Step 5, using an optimization solving algorithm to solve the optimal solution of the subgoal constraint functions and path constraint functions of the current robot working environment, respectively, and driving the robot to perform actions until the task is completed.

[0014] As a preferred, in the step 1, the method for extracting fine-grained key points is:

[0015] The semantic attributes and bounding boxes of the objects in the first frame of the video are located using Grounding DINO;

[0016] The object mask in the bounding box is segmented from the defined bounding box through the SAM model.

[0017] The DINOv2 model is used to extract a mask feature vector and generate a set of key points through clustering.

[0018] As a preferred, in step 1, the set of key points and the human demonstration video are input into the CoTracker3 model to track the motion of the key points in the human demonstration video, and a video containing key point labels is generated.

[0019] As a preferred, the step 2 includes the following sub-steps:

[0020] 2.1, task semantic information is extracted from the key point labeled video frame through the thinking chain prompt engineering of the multi-modal large language model, and a sub-task sequence is divided according to the task in the video;

[0021] 2.2, the space geometry analysis prompt engineering is constructed by using the multi-modal large model thinking chain, the objects in the scene are represented as abstract geometric models, and the key point feature information of the abstract geometric models is extracted;

[0022] 2.3, the sub-target constraint function and the path constraint function are constructed by using the multi-modal large model.

[0023] As a preferred, the abstract geometric model includes: point type object, line type object and surface type object.

[0024] As a preferred, the method for extracting the key point feature information of the abstract geometric model is: the key point feature corresponding to the point type object is the center coordinate which is represented as the mean value of multiple key points:

[0025]

[0026] where P is P i The key point coordinates, n is the number of key points, x i , y i , z i represent the x, y, z axis coordinates of P i .

[0027] The key point feature corresponding to the line type object is its direction vector which can be represented as the difference between two key points:

[0028]

[0029] where P2 and P1 are two key points on the line type object;

[0030] The key point feature corresponding to the surface type object is its normal vector which can be represented as two direction vectors and The cross product of the two vectors is:

[0031]

[0032] wherein, and are unit vectors. The key point feature attribute of the corresponding object abstract geometric model is obtained through the above calculation, which can be used for the calculation of the sub-target constraint function and the path constraint function.

[0033] As a preferred, in the step 2.1, the task semantic information includes a main summary of the task, key points and a mapping relationship between objects.

[0034] As a preferred, the step 4 includes the following sub-steps:

[0035] 4.1, input the first frame of the human demonstration video and the current robot working environment image, analyze the scene task association information through the multi-modal large language model;

[0036] 4.2, according to the scene task association information multi-modal large language model, output the mapping function of the video key point to the environment key point;

[0037] 4.3, based on the mapping function, adapt the sub-target constraint function and the path constraint function to the current robot working environment, to obtain the sub-target constraint function and the path constraint function of the current robot working environment.

[0038] As a preferred, the scene task association information includes scene description, scene object description and fine-grained key point position description.

[0039] As a preferred, the optimization solving algorithm method includes:

[0040] Using the simulated annealing algorithm to solve the initial optimal solution of the sub-target constraint function and the path constraint function of the current robot working environment;

[0041] During the execution of the robot, the sequence least square quadratic programming algorithm is used to iteratively update the optimal solution.

[0042] As a preferred, in the step 3, the observation picture under the top view and the mask information of each object are obtained through the simulator API, and the DINOv2 model is used for feature extraction, and the K-Means algorithm is used for clustering.

[0043] The present application has the following characteristics and beneficial effects:

[0044] The key point tracking method of the present application can track the position of the key point in the entire time dimension, and can enhance the perception and understanding of the key point position for the subsequent step 2 which depends on the key point. Meanwhile, the 3D position information of the key point is utilized, which is convenient for subsequent construction of a numerical constraint function model. Compared with the method without key point tracking, this method improves the utilization rate of key point information, so that it can appear on every frame.

[0045] The chain thinking large model prompts engineering from task semantics to geometric information to numerical constraint function. It can enhance the perception of the geometric characteristics of the object by the large model, and combine the geometric characteristics of the object, the distribution characteristics of the key points and the semantic information of the task by the inborn intelligence of the large model, and output the constraint value function in the form of python code. Compared with the method of directly observing the video by the large model, this method can obtain more accurate constraint function.

[0046] The mapping relationship between the video environment and the real environment is constructed by the large model. Compared with the use of some ready-made end-to-end models such as DINO, this method has higher mapping accuracy, and rewriting the constraint function by the large model can also avoid some possible syntax errors. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The flowchart of the present application is shown in the figure;

[0048] Figure 2 The overall framework diagram of the present application is shown in the figure; DETAILED DESCRIPTION

[0049] The present application will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0050] The present application provides a mechanical arm intelligent control method based on demonstration video imitation learning, as shown in Figure 1As shown, the human teaching video is used as the only basis for guiding the operation of the robot arm, and advanced technologies such as multi-modal large models are used to understand the human intention in the video and imitate the fine-grained operation skills. First, the objects in the human teaching video are anchored by key point marking to achieve fine-grained anchoring for detailed understanding of object motion in the subsequent video. At the same time, the key point tracking model is used to continuously track the motion between key points in the video, making full use of the rich information in the video. Then, the multi-modal large model is used to analyze the key point marked video in layers, including task semantic analysis and spatial geometric analysis, to obtain fine-grained task related information. On this basis, the key point related constraint model is used to represent the completion of the subtask, which is used for subsequent calculation of the optimal solution of the subtask completion.

[0051] In another aspect, an observation picture of the current robot arm working environment is captured, and the key point positions of the objects in the observation picture are marked by a feature clustering algorithm. The observation results in the current scene are aligned with the video.

[0052] The first frame of the key point marked human demonstration video and the key point marked current working environment image are input into the multi-modal large model, and key point mapping analysis is performed by the multi-modal large model to obtain a key point mapping function. Further, the key point constraint function in the video is input to obtain the mapping result of the key point constraint function in the current scene. Finally, the key point constraint function in the current scene is sequentially optimized according to the optimal solution of the constraint value function in different subtask stages and performs multiple rounds of execution iterations, and drives the robot arm to achieve the task target.

[0053] The specific implementation steps are as follows: Figure 2 As shown:

[0054] Step 1, this method will take human behavior demonstration video as the basis. The video constructs a typical desktop scene, which contains several objects and sets a task target. The video records the process of a human completing the task target.

[0055] Step 2, the key point tracking module uses Grounding DINO, Segment Anything Model (SAM), DINOv2 and other models to anchor fine-grained object key point information from the human demonstration video, and uses CoTracker3 model to continuously track the key point trajectory during the motion process to obtain the key point marked human demonstration video.

[0056] Specifically, in this embodiment, the following sub-steps are included:

[0057] 2-1. Extract the first frame F1 from the human demonstration video, and use the Grounding DINO model to calculate the semantic attributes {P1, P2, P3…} and corresponding bounding boxes {Bbox1, Rbox2, Bbox3…} of all objects in the first frame.

[0058] 2-2. Use the SAM model to segment the determined mask {M1, M2, M3…} from the defined bounding box.

[0059] 2-3. Use the DINOv2 model to calculate the feature vector of the image mask position, and set the i-th mask M i The corresponding feature vector is Use clustering algorithm to cluster the feature vector set Assume the number of clusters is K, and minimize the objective function:

[0060]

[0061] Where C k represents the k-th cluster, is the center vector of the k-th cluster, and a number of cluster centers are obtained by solving the above objective function, and the corresponding labels are key points S = {p1, p2, …, p n}.

[0062] 2-4. Input the key point sequence and the video into the CoTracker3 model to track the motion of the key points in the video, and generate a video V track containing key point labels; assume that the total number of frames of the video V track is N, and extract frames from the video V track at fixed intervals according to the average cutting rule (m is a preset number of frames), to obtain a frame sequence where t i = i × t, i = 1, 2, …, m.

[0063] Step 3, input the video labeled with key points as input, use a multi-modal large language model to extract semantic information and spatial geometric information in layers, decompose the task into multiple sub-task stages, and generate sub-task constraint functions and path constraint functions for each sub-task stage, including the following sub-steps:

[0064] 3-1, based on the multi-modal large model, construct a video semantic information thinking chain prompt project to extract task-related semantic information from the key point labeled video frame set . Among them, the task-related semantic information includes the main summary of the task, the mapping relationship between the key points and the objects from front to back. And divide the task in the video into a series of sub-task segments that can be executed by the robot arm Right now:

[0065]

[0066] 3-2. Use the multi-modal large model thinking chain to build a spatial geometry analysis prompt project, and then conduct each sub-task stage in turn. Perform fine-grained analysis, perform abstract geometric object recognition and targeted geometric information analysis for each stage, represent the objects in the scene as abstract geometric models, and extract key point feature information of the abstract geometric models, where the abstract geometric models include: point objects, line objects and surface objects, and finally extract the set of geometric elements related to the subtask execution By constructing spatial relationship reasoning, we can summarize the key points that need to be met to complete the subtask from the geometric elements. The key point feature information between them.

[0067] Furthermore, key point feature information specifically refers to:

[0068] The key point feature corresponding to the point object is its center coordinate Expressed as the mean of multiple key points:

[0069]

[0070] Where P i Key point coordinates, n is the number of key points, x i 、y i 、z i Indicates P i The x, y, and z axis coordinates of the .

[0071] The key point feature corresponding to the line object is its direction vector It can be expressed as the difference between two key points:

[0072]

[0073] Among them, P2 and P1 are two key points on the line object;

[0074] The key point feature corresponding to the surface object is its normal vector Can be represented as two direction vectors and The cross product of :

[0075]

[0076] in, and is a unit vector. The above calculations yield the key point feature attributes of the abstract geometric model of the corresponding object, which can be used to calculate the sub-goal constraint function and the path constraint function.

[0077] It can be understood that the key point feature information refers to the dependence between the key points and the geometric characteristics of the object. For example, the center position is more important for point class object operation, so it shows the dependence of point class object and key point center position. Or the dependence of surface class object and key point represented direction vector. The dependence between surface class object and key point represented surface normal vector.

[0078] 3-3, finally use multi-modal large model to construct key point constraint function set Wherein, the sub-target constraint function C sub-goal The constraint condition for describing the task target state, the path constraint function C path Used to limit the constraint condition of the motion trajectory of the robot arm, which can be expressed as:

[0079]

[0080] Specifically, the sub-target key point constraint is used to define the spatial relationship that must be met at the end of each subtask obtained in step 2.1. The sub-target key point constraint function represents the task target in natural language through python code and returns a numerical result, denoted as numerical cost, the jth sub-target constraint function of the ith stage Can be expressed as:

[0081] Wherein

[0082] Where K is the set of key points in the scene, The function output is a numerical cost, when the cost ≤ 0, it indicates that the key point relationship at the end of the stage meets the constraint. The optimization process of the sub-target function can be expressed as:

[0083]

[0084] Wherein, Indicates the end effector at the end of the ith stage Is the sub-target optimization auxiliary function (including obstacle avoidance, accessibility and other constraints), Is the key point set of the ith stage, Is the key point constraint function set of the ith stage. When the function f output ≤ 0, it indicates that the sub-target constraint requirement is met.

[0085] The basis for constructing the sub-goal key point function is that the complex operation task is decomposed into multiple continuous stages in 2.1, and each stage needs to achieve a specific sub-goal, and the sub-goal constraint function is designed to define the spatial relationship of these sub-goals. The function encodes the required spatial relationship by performing arithmetic operations on the key points in the scene. The output of the sub-goal constraint function is a numerical cost, and when the cost ≤ 0, it indicates that the constraint is satisfied. This design is derived from the demand of optimization problem, making the constraint can be incorporated into the hierarchical optimization framework, and the robot action that satisfies the constraint is calculated by the solver.

[0086] The path key point constraint function is used to define the spatial relationship that must be satisfied at all times within each stage of the operation task. It represents the rules required to be satisfied during the execution of the task in natural language using python code, and returns a numerical result, denoted as numerical cost. According to the results of 2.1, when the operation task is decomposed into multiple stages, each stage needs to satisfy the path constraint during the entire execution process in addition to satisfying the sub-goal constraint at the end. The jth path constraint function of the ith stage is denoted as The input is a set of 3D key points in the scene k t , where t represents time, The output of the path key point constraint function is a numerical cost, which can be represented as

[0087]

[0088] where [g i-1 , g i ] represents the time range of the ith stage, and when the cost ≤ 0, it indicates that the constraint is satisfied. Its optimization process can be represented as:

[0089]

[0090] where represents the end effector trajectory from the current time t to the end time g i of stage i, is the auxiliary cost function of path optimization (including obstacle avoidance, path length, etc.), is the key point set at time in stage i, is the path constraint set of the ith stage. The basis for the path constraint function is that when the operation task is decomposed into multiple sub-task stages, the execution process of each stage needs to satisfy certain conditions to ensure the smooth progress of the task. The function encodes the required spatial relationship by performing arithmetic operations on the key points in the scene. The path constraint function needs to be effective at all times within the stage (i.e. the cost ≤ 0 needs to be satisfied at every state from the beginning to the end of the stage), which is derived from the demand for process stability, ensuring that the execution process of the sub-task will not fail due to errors in the intermediate state.

[0091] In actual execution, the video frame V marked with key points is taken as a visual prompt of a multi-modal large model (MLLM), and the sub-task decomposition result S of 2.1 and the key point feature attribute set K of 2.2 are injected into the prompt of the multi-modal large model. The multi-modal large model constructs a sub-target constraint function of each sub-task i according to the sub-task order of 2.1 in turn and a path constraint function

[0092]

[0093] It should be noted that the key point feature attribute is used to guide the multi-modal large language model to describe the interaction between human objects according to the geometric characteristics of the objects.

[0094] Step 4, arrange a similar scene in the Isaac Sim simulator according to the video scene, obtain the observation picture under the top view and the mask information of each object through the simulator API, and perform feature extraction through the DINOv2 model. Cluster analysis is performed using the K-Means algorithm to obtain a plurality of cluster centers as key points in the working environment of the mechanical arm.

[0095] Step 5, based on the multi-modal large model thinking chain reasoning, a key point mapping prompt project is constructed. By inputting the first frame image in the video and the current mechanical arm working environment image into the multi-modal large model, the multi-modal large model is prompted to analyze the scene task association information in a top-down order, which includes scene description, scene object description and fine-grained key point position description, and then the multi-modal large model is prompted to output the key point mapping function from the video environment to the mechanical arm working environment.

[0096] Specifically, the following sub-steps are included:

[0097] 4.1, input the first frame of the human demonstration video and the current mechanical arm working environment image, and analyze the scene task association information through the multi-modal large language model;

[0098] Among them, the analysis of the scene task association information specifically refers to enumerating the objects appearing in the two scenes-the first frame of the human demonstration video and the mechanical arm working environment image. It can be understood that the key points on the objects are enumerated, and attention should be paid to the relative position relationship of the key points on the objects, such as one key point at the upper left corner of the object and another key point at the lower right corner of the object.

[0099] 4.2, according to the scene task association information, the multi-modal large language model outputs the mapping function of the video key points to the environment key points, wherein the mapping function is a JSON format mapping table, which is specifically represented as:

[0100] {

[0101] Point sequence number in the demonstration video: point sequence number in the current environment

[0102] }

[0103] 4.3. Map the sub-target constraint function and the path constraint function to the current robot working environment based on the mapping function to obtain the sub-target constraint function and the path constraint function of the current robot working environment.

[0104] Step 6. Construct a constraint mapping based on a multi-modal large model, input the key point constraint mapping function and the video key point constraint function, and output the key point constraint function in the robot working environment after processing by the multi-modal large model.

[0105] Step 7. Use the simulated annealing algorithm to solve the optimal solution of the sub-target value constraint function C s and the path value constraint function C p respectively and execute a number of steps according to the path planning algorithm. In the subsequent execution process, the optimal solution is updated by iteratively solving a series of quadratic programming sub-problems using the sequential least squares quadratic programming algorithm until the execution is completed.

[0106] In order to verify the effectiveness of the present application, an embodiment constructs a test set containing three types of tasks for comparative evaluation, and sets several types of advanced baseline models as references.

[0107] In the construction of the test set, three types of tasks are used: fruit picking, pencil collection, and frying pan cooking. Videos of human operations in real environments are recorded, and the same scenes as in the real environment are constructed in the Isaac Sim simulation environment for evaluation. The baseline models used for comparison are divided into two categories. The first type of comparison method uses a large model to extract task semantic features from the video. Specifically, the steps of the task are extracted from the human demonstration video. Then the obtained steps are input to a large language model programmer (LMP), which uses a set of the same sub-goal constraint functions and path constraint function framework to write these Python constraint function codes. The first type of comparison method has multiple sub-classes, including the most advanced large language models, GPT-4o and Gemini-1.5-pro. The second type of comparison method uses a large model (GPT-4o) to extract two types of relationship key point constraint functions from the video, and uses Dino to extract similar features and compare similar features to complete key point mapping. Unlike the proposed method, the second type of comparison method attempts to directly extract constraint function models that can be executed in the current new environment from the video without a mapping process. This is denoted as Dino. All experiments are conducted in the Isaac Sim simulation environment. For the three types of tasks, three representative tasks are selected for experimentation, and each task is executed for ten rounds to calculate the task completion situation.

[0108] The proposed method uses three indicators to calculate the task completion situation. Task Success Rate (TSR) is used to strictly evaluate whether the robot executes the task according to the human demonstration video. This evaluation includes the process and result of task execution. Final Success Rate (FSR) only evaluates whether the scene after execution and the scene after human demonstration are the same, and is only used to evaluate the execution result. Subtasks Success Rate (SSR) is used to evaluate the strict completion of the robot in each subtask. The results are shown in the following table.

[0109]

[0110]

[0111] As can be seen from the table, our method has better TSR, FSR, and SSR in most cases for the three types of tasks, indicating the effectiveness of the proposed method for human video imitation tasks.

[0112] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A robotic arm intelligent control method based on demonstration video imitation learning, characterized in that: The following steps are involved: Step 1: Obtain a human demonstration video containing the task target, extract the key points of the objects in the human demonstration video, and continuously track the key point trajectories to obtain a video marked with key points; Step 2: Take the video marked with key points as input, use the multimodal large language model to hierarchically extract semantic information and spatial geometric information, decompose the task into multiple subtask stages, and generate sub-goal constraint functions and path constraint functions for each subtask stage; Step 3: Arrange the same task scene as the human demonstration video in the robot arm simulation environment, and generate key points of the environment through feature extraction and clustering; Step 4: Construct a mapping function from video key points to environment key points based on the multimodal large language model, and map the sub-goal constraint function and the path constraint function to the current robotic arm working environment; Step 5: Use the optimization algorithm to solve the optimal solutions of the sub-goal constraint function and the path constraint function of the current robotic arm working environment, and drive the robotic arm to perform actions until the task is completed.

2. The method according to claim 1, characterized in that In step 1, the method for extracting fine-grained key points is: Use Grounding DINO to locate the semantic attributes and bounding boxes of objects in the first frame of the video; Segment the object mask within the bounding box through the SAM model from the defined bounding box; The DINOv2 model is used to extract mask feature vectors and generate a set of key points through clustering.

3. The method according to claim 2, characterized in that In step 1, the set of key points and the human demonstration video are input into the CoTracker3 model, the movement of the key points in the human demonstration video is tracked, and a video containing key point markers is generated.

4. The method according to claim 1, wherein The step 2 includes the following sub-steps: 2.

1. Through the thought chain prompting project of the multimodal large language model, task semantic information is extracted from key point-labeled video frames, and subtask sequences are divided according to the tasks in the video; 2.

2. Use the multimodal large model thinking chain to build a spatial geometric analysis prompt project, represent the objects in the scene as abstract geometric models, and extract the key point feature information of the abstract geometric models; 2.

3. Use the multimodal large model to construct sub-goal constraint functions and path constraint functions.

5. The method according to claim 4, characterized in that The abstract geometric model includes: point objects, line objects and surface objects.

6. The method according to claim 5, characterized in that The method for extracting key point feature information of the abstract geometric model is: The key point feature corresponding to the point object is its center coordinate Expressed as the mean of multiple key points: Where P i Key point coordinates, n is the number of key points, x i 、y i 、z i Indicates P i The x, y, and z axis coordinates, The key point feature corresponding to the line object is its direction vector It can be expressed as the difference between two key points: Among them, P2 and P1 are two key points on the line object; The key point feature corresponding to the surface object is its normal vector Can be represented as two direction vectors and The cross product of : in, and is a unit vector. The above calculations can be used to obtain the key point feature attributes of the abstract geometric model of the corresponding object. This attribute can be used to calculate the sub-goal constraint function and the path constraint function.

7. The method according to claim 4, characterized in that In step 2.1, the task semantic information includes a main overview of the task, key points, and mapping relationships between objects.

8. The method according to claim 1, characterized in that The step 4 includes the following sub-steps: 4.

1. Input the first frame of the human demonstration video and the image of the current robotic arm working environment, and analyze the scene-task association information through a multimodal large language model; 4.

2. Based on the scene-task correlation information, the multimodal large language model outputs a mapping function from video key points to environment key points; 4.

3. Based on the mapping function, the sub-goal constraint function and the path constraint function are mapped to the current robotic arm working environment to obtain the sub-goal constraint function and the path constraint function of the current robotic arm working environment.

9. The method according to claim 8, characterized in that The scene-task association information includes scene description, scene object description, and fine-grained key point position description.

10. The method according to claim 1, characterized in that The optimization solution algorithm method includes: Use the simulated annealing algorithm to solve the initial optimal solution of the sub-goal constraint function and path constraint function of the current robot arm working environment; During the execution of the robotic arm, the sequential least squares quadratic programming algorithm is used to iteratively update the optimal solution.

11. The method according to claim 1, wherein In step 3, the observation image from the top perspective and the mask information of each object are obtained through the simulator API, and the DINOv2 model is used for feature extraction, and the K-Means algorithm is used for clustering.

Images