A 3C assembly programming-free method based on large model action analysis and automatic programming

Abstract

The application provides a 3C assembly programming-free method based on large model action analysis and automatic programming, comprising the following steps: constructing an intelligent assembly production line, constructing a multi-modal virtual environment based on the intelligent assembly production line; in the multi-modal virtual environment, a perception model containing vision, touch and depth is established to realize the collection of multi-modal teaching video data; action analysis is performed on the multi-modal teaching video data to obtain an action sequence set; a basic task learning model is constructed, the action sequence set is input into the basic task learning model for training, and a task learning model is obtained; the input and output results of the task learning model are defined as an execution sequence, and the learned skill strategy is migrated from the multi-modal virtual environment to the real environment according to the execution sequence. Through the method provided by the application, the intelligent agent can better understand and adapt to different scenes and tasks in the 3C manufacturing field, and the intelligent level, skill migration and generalization ability of the system are improved.

Description

Technical Field

[0001] The invention belongs to the field of intelligent assembly. Background Art

[0002] With the rapid development of information technology, demand for computers, communications, and consumer electronics (3C products) continues to increase. While intelligent assembly of rigid parts in existing 3C assembly has been largely achieved, the assembly of flexible and deformable parts, particularly deformable linear objects (DLOs), still relies solely on manual assembly, making it difficult to switch or migrate between production lines.

[0003] In the 3C assembly industry, it is still necessary to build intelligent flexible assembly production lines to solve the problems of material heterogeneity and easy deformation, complex assembly processes, etc. in flexible assembly, improve assembly accuracy and efficiency, meet the needs of intelligent production, break through the limitations of traditional manual assembly, and adapt to the rapid development of intelligent assembly.

[0004] Skill learning and development are crucial for robots to achieve precise control and delicate manipulation, and knowledge representation is a core element of skill learning. Current knowledge representation for assembly scenarios lacks precision and effectiveness, and robots still face significant challenges in task understanding, motion control, memory, and developing new skills and tasks for intelligent assembly.

[0005] First, intelligent assembly requires extensive skill-based knowledge (geometry, physics, function, process, and data) to support modeling, planning, simulation, control, and optimization. Existing knowledge representation systems primarily rely on static knowledge bases, which lack detailed and flexible descriptions of robot-manipulated objects and behaviors. Furthermore, inefficient system architectures lead to low query efficiency. Currently, there is no professional, readily available knowledge base for operational skills, making data collection and analysis difficult.

[0006] Directly implementing intelligent assembly strategies is often infeasible due to data scarcity, complex environments, and safety risks. A learning and training environment based on the digital twin environment and a training and testing platform for intelligent algorithms need to be built. This framework platform needs to have physical properties consistent with real-world scenarios to transfer trained deep reinforcement learning strategies to physical robots. However, the current virtual-to-real transfer of operational skills from simulation environments to real assembly environments faces challenges such as weak task understanding, poor environmental adaptability, and low execution efficiency. It is necessary to improve robots' ability to understand tasks and learn skills, enhance their adaptability to a variety of complex environments, improve execution efficiency, reduce learning costs, and enhance their generalization capabilities.

[0007] Secondly, unlike humans who can naturally learn from videos and natural language to improve language instructions to complete tasks. Robots cannot link language and visual observations and achieve fine-grained understanding learning. Precise and effective knowledge representation methods not only help improve the operation accuracy of robots facing complex assembly tasks and the memory of previous experience, but also facilitate the robot to achieve automatic programming of the underlying hardware.

[0008] Therefore, how to split different knowledge representation layers to facilitate the robot to understand the task and skill learning, and how to make the robot understand the language instruction and make the correct learning and imitation, are still a big challenge for the robot. SUMMARY

[0009] The present application aims to at least solve one of the technical problems in the related art.

[0010] To this end, the purpose of the present application is to propose a 3C assembly programming-free method based on large model action analysis and automatic programming for improving the intelligence level of the assembly system.

[0011] To achieve the above purpose, the first aspect of the present application proposes a 3C assembly programming-free method based on large model action analysis and automatic programming, comprising:

[0012] Constructing an intelligent assembly production line, based on the intelligent assembly production line, constructing a multi-modal virtual environment; in the multi-modal virtual environment, establishing a perception model containing vision, touch and depth, realizing the collection of multi-modal teaching video data;

[0013] Action analysis is performed on the multi-modal teaching video data to obtain an action sequence set;

[0014] A basic task learning model is constructed, and the action sequence set is input into the basic task learning model for training to obtain a task learning model;

[0015] The input and output results of the task learning model are defined as execution sequences, and the learned skill strategy is migrated from the multi-modal virtual environment to the real environment according to the execution sequences.

[0016] In addition, the 3C assembly programming-free method based on large model action analysis and automatic programming according to the above embodiments of the present application can also have the following additional technical features:

[0017] Further, in an embodiment of the present application, the intelligent assembly production line comprises:

[0018] Circular production line collaborative assembly control unit, circulating guide rail control unit, 3C flexible mobile phone carrier loading and unloading assembly unit, 3C flexible mobile phone soft cable intelligent assembly unit, 3C flexible mobile phone front camera intelligent assembly unit, 3C flexible mobile phone SIM card slot intelligent assembly unit, 3C flexible mobile phone coaxial cable intelligent assembly unit.

[0019] Furthermore, in one embodiment of the present invention, the performing motion analysis on the multimodal teaching video data includes:

[0020] Segmenting the multimodal teaching video data into individual frames, and pairing adjacent frames to create a series of frame combinations;

[0021] Analyzing the frame combination through multimodal information question-answering interaction;

[0022] According to the result of the analysis, appropriate actions are selected from a predefined action skill library for matching to construct a coherent action sequence.

[0023] Furthermore, in one embodiment of the present invention, constructing a basic task learning model and inputting the action sequence set into the basic task learning model for training includes:

[0024] The SAC algorithm is used to learn the action sequence, and the optimization objective function is shown as follows:

[0025] ,

[0026] Based on the allowable distance error as the evaluation index of whether the task is successfully completed, the evaluation index can be expressed as:

[0027] ,

[0028] in Is an indicator function used to evaluate whether the distance difference between the target position and the current position is within the allowable error range;

[0029] During the policy optimization process, a curriculum learning mechanism is used to adjust the allowed distance error according to the following formula:

[0030] ,

[0031] in, represents the maximum number of steps of course learning iteration, which is determined by the success rate of task execution, is the distance error allowed in the initial training phase, Is the minimum distance error allowed in the final training stage. Once the number of iterative training steps exceeds , will maintain the allowed distance error for constant.

[0032] Furthermore, in one embodiment of the present invention, the input and output results of the task learning model are defined as an execution sequence, and the transfer of the learned skill strategy from the multimodal virtual environment to the real environment is achieved according to the execution sequence, including:

[0033] A function body is generated for each function in the execution sequence, using GPT-4 to provide prompts in an explanatory manner, and finally an executable function body is generated through a natural language description.

[0034] To achieve the above objectives, the second embodiment of the present invention proposes a 3C assembly programming-free device based on large model motion analysis and automatic programming, including the following modules:

[0035] An acquisition module is used to build an intelligent assembly production line and, based on the intelligent assembly production line, to construct a multimodal virtual environment; within the multimodal virtual environment, a perception model including vision, touch, and depth is established to realize the acquisition of multimodal teaching video data;

[0036] An analysis module, configured to perform action analysis on the multimodal teaching video data to obtain an action sequence set;

[0037] A training module is used to construct a basic task learning model, input the action sequence set into the basic task learning model for training, and obtain a task learning model;

[0038] A migration module is used to define the input and output results of the task learning model as an execution sequence, and realize the migration of the learned skill strategy from the multimodal virtual environment to the real environment according to the execution sequence.

[0039] Furthermore, in one embodiment of the present invention, the parsing module is further configured to:

[0040] Segmenting the multimodal teaching video data into individual frames, and pairing adjacent frames to create a series of frame combinations;

[0041] Analyzing the frame combination through multimodal information question-answering interaction;

[0042] According to the result of the analysis, appropriate actions are selected from a predefined action skill library for matching to construct a coherent action sequence.

[0043] Furthermore, in one embodiment of the present invention, the migration module is further configured to:

[0044] A function body is generated for each function in the execution sequence, using GPT-4 to provide prompts in an explanatory manner, and finally an executable function body is generated through a natural language description.

[0045] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, which is characterized in that it includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the above-mentioned 3C assembly programming-free method based on large model action analysis and automatic programming.

[0046] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned 3C assembly programming-free method based on large model action analysis and automatic programming.

[0047] The proposed method for programming-free 3C assembly based on large-scale model motion analysis and automatic programming builds an intelligent flexible assembly line and a complex operational skills knowledge base system. This method uses a virtual environment with physical properties to train robots to learn and understand knowledge and experience. A transfer algorithm then transfers this knowledge and experience to new skills, enabling them to cope with complex and changing unknown tasks. This completes the construction of an intelligent assembly system and improves its intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A flowchart of a programming-free 3C assembly method based on large-model motion analysis and automatic programming provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the structure of a 3C assembly programming-free device based on large-model motion analysis and automatic programming provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a virtual scene of an intelligent assembly production line provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of the operation skill analysis and motion teaching data collection provided by an embodiment of the present invention;

[0053] Figure 5 The analyzed flexible flat cable assembly action sequence diagram provided by the embodiment of the present invention;

[0054] Figure 6 A schematic diagram of the overall process of the basic task learning model provided by an embodiment of the present invention;

[0055] Figure 7 Schematic diagram of the virtual-reality migration experiment of the loading and unloading scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0057] The following describes a programming-free 3C assembly method based on large model motion analysis and automatic programming according to an embodiment of the present invention with reference to the accompanying drawings.

[0058] Figure 1 The present invention provides a flowchart of a 3C assembly programming-free method based on large-model motion analysis and automatic programming provided by an embodiment of the present invention.

[0059] like Figure 1 As shown, the 3C assembly programming-free method based on large model motion analysis and automatic programming includes the following steps:

[0060] S101: Build an intelligent assembly production line. Based on the intelligent assembly production line, construct a multimodal virtual environment. In the multimodal virtual environment, establish a perception model that includes vision, touch, and depth to realize the collection of multimodal teaching video data.

[0061] Furthermore, in one embodiment of the present invention, the intelligent assembly production line includes:

[0062] Circular production line collaborative assembly control unit, circulating guide rail control unit, 3C flexible mobile phone carrier loading and unloading assembly unit, 3C flexible mobile phone soft cable intelligent assembly unit, 3C flexible mobile phone front camera intelligent assembly unit, 3C flexible mobile phone SIM card slot intelligent assembly unit, 3C flexible mobile phone coaxial cable intelligent assembly unit.

[0063] In one embodiment of the present application, a virtual scene diagram of an intelligent assembly production line is shown in FIG. Figure 3 shown.

[0064] S102: performing motion analysis on the multimodal teaching video data to obtain a set of motion sequences;

[0065] Furthermore, in one embodiment of the present invention, performing motion analysis on multimodal teaching video data includes:

[0066] Segment the multimodal teaching video data into individual frames, pair adjacent frames, and create a series of frame combinations;

[0067] Analyze frame combinations through multimodal information question-answering interaction;

[0068] According to the analysis results, appropriate actions are selected from the pre-defined action skill library to match and build a coherent action sequence.

[0069] Specifically, during the preprocessing phase, the model first processes the input video. This preprocessing simplifies subsequent processing steps and improves the model's efficiency and accuracy. The video is segmented into individual frames and sampled at predetermined intervals. The model then pairs adjacent frames together to create a series of frame combinations. This preserves the temporal information of the action, laying the foundation for subsequent action recognition and analysis.

[0070] Multimodal Question-Answering Interaction: LLaVA combines vision and language processing capabilities to enable in-depth analysis and understanding of visual content. Using Visual Question Answering (VQA) technology, the model can accept natural language questions and provide answers based on the image content. ChatGPT acts as the questioner, using customized prompts to guide the model's focus on specific parts or features of video frames, such as scene transitions or object movement.

[0071] Generation of action sequences and single-step instructions: Through multiple rounds of dialogue, ChatGPT gradually accumulates an understanding of the video content. Based on this information, ChatGPT selects appropriate actions from a predefined action skill library. This action skill library contains customized robot actions, allowing the model to select the most suitable action to construct a coherent action sequence. Finally, ChatGPT assigns corresponding language descriptions to these actions. These descriptions are not just single action labels, but complete sentences containing contextual information, equivalent to task planning for long-term tasks. They are used as planners in subsequent models to guide the model to complete the task step by step.

[0072] In one embodiment of the present application, the flowchart of operation skill analysis and motion teaching data collection is as follows: Figure 4 shown.

[0073] In one embodiment of the present application, the analyzed flexible cable assembly action sequence diagram is as follows: Figure 5 shown.

[0074] S103: Build a basic task learning model, input the action sequence set into the basic task learning model for training, and obtain a task learning model;

[0075] Furthermore, in one embodiment of the present invention, a basic task learning model is constructed, and the action sequence set is input into the basic task learning model for training, including:

[0076] The SAC algorithm is used to learn the action sequence, and the optimization objective function is shown as follows:

[0077] ,

[0078] Based on the allowable distance error as the evaluation index of whether the task is successfully completed, the evaluation index can be expressed as:

[0079] ,

[0080] in Is an indicator function used to evaluate whether the distance difference between the target position and the current position is within the allowable error range;

[0081] During the policy optimization process, a curriculum learning mechanism is used to adjust the allowed distance error according to the following formula:

[0082] ,

[0083] in, represents the maximum number of steps of course learning iteration, which is determined by the success rate of task execution, is the distance error allowed in the initial training phase, Is the minimum distance error allowed in the final training stage. Once the number of iterative training steps exceeds , will maintain the allowed distance error for constant.

[0084] Specifically, a Transformer-based network architecture is designed to enable robots to learn from videos and complete multi-task learning given instructions. This architecture includes a multimodal fusion module that fuses language and visual inputs to achieve a unified representation. A knowledge-enhanced learning module leverages knowledge from LLM planning to improve skill learning. Finally, an action prediction module uses this fused data to predict the next assembly action.

[0085] Multimodal Fusion: In this module, multimodal fusion is first performed on the input language instructions and visual observation images. An attention mechanism is used to weigh the importance of language instruction features in the context of image features. The language instruction information describes the task information and is efficiently encoded using the Text-To-Text Transfer Transformer model. In image processing, standardization ensures consistency in data format, while data augmentation techniques such as random cropping, rotation, and scaling enhance diversity. The Vision Transformer (ViT) network divides the image into blocks, extracts features, and uses cross-attention to establish a connection between image and language features.

[0086] Knowledge-enhanced learning (KEL) aims to leverage skill knowledge to enhance image features, significantly improving the ability to focus on areas of interest. The KEL module consists of two main modules: an LLM planner, multi-step knowledge, and image attention. The LLM planner uses actions and single-step instructions derived from video action parsing as input to the multi-step knowledge guidance and image attention modules for KEL. Cross-attention is employed between skill knowledge sequences and their corresponding images to guide the image to focus on regions specified in the skill knowledge.

[0087] Action Prediction: The next assembly action is predicted using the Transformer embeddings. These are first mean-pooled and then fed into a UNet network to generate a feature map. Each action consists of the end-effector's 3D position, a quaternion representing its pose, and the switch of the suction cup. Grasping is successful when the object is moved to that position and the suction cup is open. Both the feature map and point cloud information are crucial for subsequent position prediction. First, an attention map is determined for each viewpoint by mean-pooling the feature map. This attention map encapsulates the model's intrinsic probability distribution, highlighting areas that are critical for future actions. Element-wise multiplication is then performed between the point cloud representation and the attention map, and the results are finally combined to produce a 3D position vector. To predict the pose quaternion and the end-effector pickup and release, a direct approach using two linear MLP layers is used. The output is a 5-dimensional vector, with the first four dimensions being the quaternion and the last dimension representing the end-effector pickup and release.

[0088] In one embodiment of the present application, the overall process diagram of the basic task learning model is as follows: Figure 6 shown.

[0089] S104: The input and output results of the task learning model are defined as an execution sequence, and the learned skill strategy is transferred from the multimodal virtual environment to the real environment according to the execution sequence.

[0090] In one embodiment of the present application, the schematic diagram of the virtual-real migration experiment of the loading and unloading scene is as follows: Figure 7 shown.

[0091] Furthermore, in one embodiment of the present invention, the input and output results of the task learning model are defined as an execution sequence, and the transfer of the learned skill strategy from the multimodal virtual environment to the real environment is achieved according to the execution sequence, including:

[0092] Generate the function body for each function in the execution sequence using GPT-4, provide prompts in an explanatory manner, and finally generate an executable function body through natural language description.

[0093] Specifically, we first explain the role of GPT, the loading and unloading environment, function names, and output formats in prompts. We then define examples of executable sequences based on the input and output formats. Next, after obtaining these executable sequences, we generate the function bodies for each function, using GPT-4 to provide prompts using explanations. Finally, we generate the executable function bodies from natural language descriptions. Ultimately, we obtain a complete program that not only performs the defined tasks but can also be adjusted or expanded as needed.

[0094] The proposed method for programming-free 3C assembly based on large-scale model motion analysis and automatic programming builds an intelligent flexible assembly line and a complex operational skills knowledge base system. This method uses a virtual environment with physical properties to train robots to learn and understand knowledge and experience. A transfer algorithm then transfers this knowledge and experience to new skills, enabling them to cope with complex and changing unknown tasks. This completes the construction of an intelligent assembly system and improves its intelligence.

[0095] In order to realize the above-mentioned embodiment, the present invention also proposes a 3C assembly programming-free device based on large-scale model motion analysis and automatic programming.

[0096] Figure 2 This is a structural diagram of a 3C assembly programming-free device based on large-model motion analysis and automatic programming provided by an embodiment of the present invention.

[0097] like Figure 2 As shown, the 3C assembly programming-free device based on large model motion analysis and automatic programming includes: an acquisition module 100, an analysis module 200, a training module 300, and a migration module 400, wherein:

[0098] The acquisition module is used to build an intelligent assembly production line and a multimodal virtual environment based on the intelligent assembly production line. In the multimodal virtual environment, a perception model including vision, touch, and depth is established to realize the acquisition of multimodal teaching video data.

[0099] The parsing module is used to perform action parsing on multimodal teaching video data to obtain a set of action sequences;

[0100] The training module is used to build a basic task learning model, input the action sequence set into the basic task learning model for training, and obtain the task learning model;

[0101] The transfer module is used to define the input and output results of the task learning model as an execution sequence, and realize the transfer of the learned skill strategy from the multimodal virtual environment to the real environment according to the execution sequence.

[0102] Furthermore, in one embodiment of the present invention, the parsing module is further configured to:

[0103] Segment the multimodal teaching video data into individual frames, pair adjacent frames, and create a series of frame combinations;

[0104] Analyze frame combinations through multimodal information question-answering interaction;

[0105] According to the analysis results, appropriate actions are selected from the pre-defined action skill library to match and build a coherent action sequence.

[0106] Furthermore, in one embodiment of the present invention, the migration module is further configured to:

[0107] Generate the function body for each function in the execution sequence using GPT-4, provide prompts in an explanatory manner, and finally generate an executable function body through natural language description.

[0108] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, which is characterized in that it includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the 3C assembly programming-free method based on large model action analysis and automatic programming as described above.

[0109] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the 3C assembly programming-free method based on large model action analysis and automatic programming as described above is implemented.

[0110] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0112] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limiting the present invention. A person skilled in the art may change, modify, replace, and modify the above embodiments within the scope of the present invention.

Claims

1. A 3C assembly programming-free method based on large model motion analysis and automatic programming, characterized in that: The following steps are involved: Constructing an intelligent assembly production line and, based on the intelligent assembly production line, building a multimodal virtual environment; in the multimodal virtual environment, establishing a perception model including vision, touch, and depth to realize the collection of multimodal teaching video data; Performing action analysis on the multimodal teaching video data to obtain an action sequence set; Constructing a basic task learning model, inputting the action sequence set into the basic task learning model for training, and obtaining a task learning model; The input and output results of the task learning model are defined as an execution sequence, and the transfer of the learned skill strategy from the multimodal virtual environment to the real environment is achieved according to the execution sequence; The constructing of the basic task learning model and inputting the action sequence set into the basic task learning model for training includes: Using the SAC algorithm to learn the action sequence; Based on the allowable distance error as the evaluation index of whether the task is successfully completed, the evaluation index can be expressed as: ， in Is an indicator function used to evaluate whether the distance difference between the target position and the current position is within the allowable error range; During the policy optimization process, a curriculum learning mechanism is used to adjust the allowed distance error according to the following formula: ， in, represents the maximum number of steps of course learning iteration, which is determined by the success rate of task execution, is the distance error allowed in the initial training phase, Is the minimum distance error allowed in the final training stage. Once the number of iterative training steps exceeds , will maintain the allowed distance error for constant.

2. The method according to claim 1, characterized in that The intelligent assembly production line includes: Circular production line collaborative assembly control unit, circulating guide rail control unit, 3C flexible mobile phone carrier loading and unloading assembly unit, 3C flexible mobile phone soft cable intelligent assembly unit, 3C flexible mobile phone front camera intelligent assembly unit, 3C flexible mobile phone SIM card slot intelligent assembly unit, 3C flexible mobile phone coaxial cable intelligent assembly unit.

3. The method according to claim 1, characterized in that The performing motion analysis on the multimodal teaching video data includes: Segmenting the multimodal teaching video data into individual frames, and pairing adjacent frames to create a series of frame combinations; Analyzing the frame combination through multimodal information question-answering interaction; According to the result of the analysis, actions are selected from a predefined action skill library for matching to construct a coherent action sequence.

4. The method according to claim 1, wherein The step of defining the input and output results of the task learning model as an execution sequence and implementing the migration of the learned skill strategy from the multimodal virtual environment to the real environment according to the execution sequence includes: A function body is generated for each function in the execution sequence, using GPT-4 to provide prompts in an explanatory manner, and finally an executable function body is generated through a natural language description.

5. A 3C assembly programming-free device based on large model motion analysis and automatic programming, characterized in that: Includes the following modules: An acquisition module is used to build an intelligent assembly production line and, based on the intelligent assembly production line, to construct a multimodal virtual environment; within the multimodal virtual environment, a perception model including vision, touch, and depth is established to realize the acquisition of multimodal teaching video data; An analysis module, configured to perform action analysis on the multimodal teaching video data to obtain an action sequence set; A training module is used to construct a basic task learning model, input the action sequence set into the basic task learning model for training, and obtain a task learning model; A transfer module, configured to define the input and output results of the task learning model as an execution sequence, and implement the transfer of the learned skill strategy from the multimodal virtual environment to the real environment according to the execution sequence; The constructing of the basic task learning model and inputting the action sequence set into the basic task learning model for training includes: Using the SAC algorithm to learn the action sequence; Based on the allowable distance error as the evaluation index of whether the task is successfully completed, the evaluation index can be expressed as: ， in Is an indicator function used to evaluate whether the distance difference between the target position and the current position is within the allowable error range; During the policy optimization process, a curriculum learning mechanism is used to adjust the allowed distance error according to the following formula: ， in, represents the maximum number of steps of course learning iteration, which is determined by the success rate of task execution, is the distance error allowed in the initial training phase, Is the minimum distance error allowed in the final training stage. Once the number of iterative training steps exceeds , will maintain the allowed distance error for constant.

6. The device according to claim 5, characterized in that The parsing module is further used to: Segmenting the multimodal teaching video data into individual frames, and pairing adjacent frames to create a series of frame combinations; Analyzing the frame combination through multimodal information question-answering interaction; According to the result of the analysis, actions are selected from a predefined action skill library for matching to construct a coherent action sequence.

7. The device according to claim 5, characterized in that The migration module is further configured to: A function body is generated for each function in the execution sequence, using GPT-4 to provide prompts in an explanatory manner, and finally an executable function body is generated through a natural language description.

8. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the 3C assembly programming-free method based on large model motion analysis and automatic programming as described in any one of claims 1 to 4 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the 3C assembly programming-free method based on large model motion analysis and automatic programming as described in any one of claims 1 to 4 is implemented.

Images