Method and system for dynamic enhancement of complex skills in production line having uncertainty

Abstract

The present disclosure provides a method and system for dynamic enhancement of complex skills in a production line having uncertainty. The method comprises: acquiring a multi-view video segment, comprising an assembly task, in a virtual scene; inputting the multi-view video segment into a video parsing model to perform multiple rounds of interactive questions and answers between large models so as to generate a task action sequence and action instructions obtained after parsing; inputting the task action sequence and action instructions obtained after parsing and all nodes of a multi-level and multi-grade knowledge graph into a data enhancement network model for assembly knowledge integration to generate an enhanced action sequence; and performing secondary enhancement on the enhanced action sequence by using reinforcement learning in a simulation environment, so as to obtain a final task action enhancement result. The present disclosure aims to solve multiple problems in existing 3C product assembly technology, especially how to dynamically cope with different assembly tasks on a production line, and improve the accuracy and efficiency of assembly operation.

Description

A complex skill dynamic enhancement method and system with uncertain production lines TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and in particular to the field of intelligence such as reinforcement learning and data augmentation. BACKGROUND

[0002] With the rapid development of 3C products (computer, communication, consumer electronics) and the continuous pursuit of product quality and innovation by consumers, the complexity and fineness requirements of 3C product assembly have also increased. Especially in tasks such as front camera installation, SIM card installation, flexible flat cable installation, and coaxial cable installation, not only are there very high requirements for operation precision, but also the uncertainty of assembly tasks brings great challenges to the production line. Traditional assembly techniques and methods are often inefficient in handling these complex tasks and cannot meet the needs of modern 3C product production.

[0003] In existing technologies, assembly tasks mainly rely on pre-programmed robots or automated equipment to execute fixed processes, which performs well in scenarios with determined tasks and high repeatability. However, in the face of task type uncertainty and complex interactions in the assembly process, these methods are not up to the task. For example, the same production line may need to handle different types of assembly tasks, such as precise positioning and installation of front cameras and delicate operations of coaxial cables. These tasks not only require precise position control, but also require operators (or machines) to have the ability to adapt to different task requirements flexibly.

[0004] In addition, existing automation solutions often lack the ability to dynamically enhance assembly skills and are difficult to optimize the assembly process through learning and adaptation. Although some advanced technologies attempt to improve assembly operations through machine learning methods, these attempts are usually focused on optimizing a single task rather than providing a comprehensive solution that can dynamically respond to multiple assembly tasks and skill requirements.

[0005] Therefore, the main problems faced by existing technologies include insufficient skills for high-precision assembly operations, improper handling of assembly task type uncertainty, and a lack of a method that can achieve dynamic enhancement of complex skills. These problems not only affect assembly efficiency and product quality, but also limit the further improvement of production flexibility and automation level.

[0006] There are many problems in the existing 3C product assembly technology, especially how to dynamically respond to different assembly tasks (such as mobile phone front camera installation, SIM card installation, soft wire installation and coaxial line installation, etc.) on the production line, and improve the accuracy and efficiency of assembly operation. The specific technical problems include: how to accurately identify and adapt to uncertain assembly task types, realize the flexibility and adaptability of the assembly process; how to enhance complex assembly skills, including moving, sucking, grabbing, aligning, placing, resetting, etc., to adapt to different assembly requirements; how to optimize the assembly action sequence through technical means, improve the accuracy and efficiency of operation, and reduce the error rate and improve the automation level of the production line.

[0007] SUMMARY

[0008] The present disclosure proposes a complex skill dynamic enhancement method, system and device with uncertain production line.

[0009] According to an aspect of the present disclosure, a complex skill dynamic enhancement method with uncertain production line is proposed, comprising: acquiring a multi-view video segment containing an assembly task in a virtual scene; inputting the multi-view video segment into a video analysis model for multi-round interactive question and answer between large models to generate an analyzed task action sequence and action instructions; inputting the analyzed task action sequence and action instructions and all nodes of a multi-level multi-level knowledge graph into a data enhancement network model for assembly knowledge integration to generate an enhanced action sequence; and using reinforcement learning in a simulation environment to perform secondary enhancement on the enhanced action sequence to obtain a final task action enhancement result.

[0010] According to a second aspect of the present disclosure, a complex skill dynamic enhancement system with uncertain production line is proposed, comprising: a video data acquisition module for acquiring a multi-view video segment containing an assembly task in a virtual scene; a video data analysis module for inputting the multi-view video segment into a video analysis model for multi-round interactive question and answer between large models to generate an analyzed task action sequence and action instructions; a sequence initial enhancement module for inputting the analyzed task action sequence and action instructions and all nodes of a multi-level multi-level knowledge graph into a data enhancement network model for assembly knowledge integration to generate an enhanced action sequence; and a sequence final enhancement module for using reinforcement learning in a simulation environment to perform secondary enhancement on the enhanced action sequence to obtain a final task action enhancement result.

[0011] According to a third aspect of the present disclosure, an end-of-task execution device for proactive assembly is provided, comprising: a housing, and a suction assembly, a correction assembly and a mounting assembly arranged in the housing; the suction assembly comprises a first driving member and a suction disc; the first driving member drives the suction disc to suck a front camera from a set feeding position; the correction assembly comprises a second driving member and a positioning member corresponding to both sides of the suction disc, and the second driving member drives the positioning member to adjust the posture of the front camera on the suction disc; and the mounting assembly comprises a third driving member and a fastening member, and the third driving member drives the fastening member to press the front camera on the suction disc to a design position.

[0012] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0014] FIG. 1 is a flowchart of a complex skill dynamic enhancement method with an uncertain production line according to an embodiment of the present disclosure;

[0015] FIG. 2 is a general architecture diagram of a complex skill dynamic enhancement method with an uncertain production line according to an embodiment of the present disclosure;

[0016] FIG. 3 is a ChatGPT-based question and answer interaction video action analysis framework diagram according to an embodiment of the present disclosure;

[0017] FIG. 4 is a knowledge graph-based action sequence enhancement framework diagram according to an embodiment of the present disclosure;

[0018] FIG. 5 is a proactive assembly action sequence diagram after inference enhancement according to an embodiment of the present disclosure;

[0019] FIG. 6 is a skill enhancement framework diagram based on reinforcement learning in a simulation environment according to an embodiment of the present disclosure;

[0020] FIG. 7 is a structural diagram of a complex skill dynamic enhancement system with an uncertain production line according to an embodiment of the present disclosure;

[0021] FIG. 8 is a structural schematic diagram of an end-of-task execution device according to an embodiment of the present disclosure;

[0022] FIG. 9 is a sectional view of FIG. 8;

[0023] FIG. 10 is a partial enlarged view of FIG. 8.

[0024] In the figure, 1, front camera; 2, industrial camera; 3, sliding table cylinder; 4, finger cylinder; 5, micro cylinder; 6, connecting piece; 7, left positioning piece; 8, right positioning piece; 9, buckling piece; 10, suction cup; 11, mechanical arm connecting piece; 12, camera support; 13, shell; 14, suction cup support. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in their context only. Thus, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and structures are omitted in the following description for the sake of clarity and conciseness.

[0026] Data augmentation is a widely used technique in the field of machine learning and deep learning, aiming to increase the diversity of training data through a series of transformations, thereby improving the generalization ability of the model and reducing the risk of overfitting, especially in cases where the data set is small or a large amount of data is needed to train a complex model. The principle of data augmentation is to generate new, reasonable and meaningful training samples from existing data without manually collecting more data.

[0027] Assembly tasks generally refer to the process of assembling individual parts into a complete product according to specific sequences and requirements in manufacturing. This process is widely used in automobile manufacturing, electronic product production, mechanical equipment assembly, aerospace engineering and other fields. Assembly tasks can be manual operation or highly automated, relying on robotics and advanced manufacturing systems.

[0028] Artificial intelligence (AI) is a discipline that studies making computers simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of human life, including both hardware and software technologies. Artificial intelligence hardware technology generally includes computer vision technology, speech recognition technology, natural language processing technology, as well as learning / deep learning, big data processing technology, knowledge graph technology, etc.

[0029] Pre-programmed robots or automated equipment refers to automated devices that have a series of instructions and action sequences set through programming before use, which can automatically execute tasks according to the predetermined program without human intervention each time. Such robots and automated equipment are widely used in manufacturing, assembly, logistics, services, medical care and other fields.

[0030] FIG. 1 is a flowchart of a complex skill dynamic enhancement method with uncertainty production line according to an embodiment of the present disclosure. As shown in FIG. 1, the method comprises the following steps:

[0031] S1, obtaining a multi-view video segment containing an assembly task in a virtual scene.

[0032] It can be understood that the embodiment of the present disclosure obtains a multi-view video in a virtual scene, which contains multiple assembly tasks, and can be a front-facing assembly task. The video segment of the front-facing assembly task is input into a large model-based video analysis module.

[0033] S2, inputting the multi-view video segment into a video analysis model for multi-round interactive question and answer between large models to generate an analyzed task action sequence and action instruction.

[0034] FIG. 3 is an interactive question and answer video action analysis diagram of a video analysis model according to an embodiment of the present disclosure. As shown in FIG. 3:

[0035] Specifically, the present disclosure analyzes different assembly tasks through a large model-based video analysis module, and automatically identifies the action sequence and action instruction required to be executed for each task. The module takes a multi-view video segment in a virtual scene as input, and finally generates an analyzed task action sequence and action instruction through multi-round interactive question and answer between large models.

[0036] In an embodiment of the present disclosure, (taking a front-facing assembly as an example, the action sequence is moving-picking-up-resetting, and the action instruction is moving to the front-facing position, picking up the front-facing, and returning to the initial position). The module can accurately extract task features and action requirements from video data, providing basic data for subsequent enhancement steps. The model uses deep learning technology to extract features and sequence analysis of the assembly task video. Through training the model, high-precision identification and classification of the assembly task can be achieved.

[0037] S3, inputting the analyzed task action sequence and action instruction and all nodes of the multi-level multi-level knowledge graph into a data enhancement network model to integrate assembly knowledge to generate an enhanced action sequence.

[0038] FIG. 4 is a knowledge graph-based action sequence enhancement framework diagram according to an embodiment of the present disclosure. As shown in FIG. 4:

[0039] Specifically, the embodiment of the present disclosure performs first enhancement on the action sequence obtained by the video analysis module based on the large model in the previous step, and adopts a task-to-action reasoning algorithm enhanced by knowledge. The algorithm combines the Bert module, the GAT module and the Seq2Seq module, takes all nodes of the MLMLKG (Multi-layer Multi-level Knowledge Graph) as input, integrates and assembles knowledge, and generates an enhanced action sequence.

[0040] It can be understood that the Bert module is responsible for processing the task description text and the node description, and the action instruction obtained in the previous step is encoded into a high-dimensional word vector through the pre-trained Bert model. This process uses a pre-trained language model to improve the understanding of the sequence and generates a node embedding vector containing rich semantic information, providing input for the subsequent graph attention network layer.

[0041] It can be understood that the GAT module uses the node embedding vector generated by the Bert module and the graph topology of the task to establish a connection between the task nodes through the graph attention network layer. This module optimizes the relevance and dependency between nodes through the graph attention mechanism, and outputs an optimized node embedding vector to provide more accurate task-related information for action sequence generation.

[0042] It can be understood that the Seq2Seq module receives the optimized node embedding vector from the GAT module and the task description obtained by the large model analysis, and processes the input sequence through the embedding layer, the bidirectional GRU layer and the attention mechanism. This module generates specific action sequences and corresponding action parameters required for the task, ensuring the accuracy and enhancement effect of the action sequence. The Seq2Seq module converts the task description and node embedding into a unified vector representation, and outputs the optimized action sequence through the fully connected layer.

[0043] Further, in order to balance the learning weights between the GAT module and the Seq2Seq module and prevent the algorithm from over-optimizing a single module, a composite loss function (L = aL1 + bL2) is designed. where L1 is the loss function of the GAT module, L2 is the loss function of the Seq2Seq module, and ab is a learnable parameter. This design ensures the balance and efficiency of the overall learning process. Taking proactive assembly as an example, the assembly action sequence after reasoning enhancement is shown in FIG. 5.

[0044] S4, in the simulation environment, the enhanced action sequence is enhanced again by reinforcement learning to obtain the final task action enhancement result.

[0045] FIG. 6 is a skill enhancement framework diagram based on reinforcement learning in a simulation environment according to an embodiment of the present disclosure. As shown in FIG. 6:

[0046] Specifically, after completing the action sequence enhancement, the present disclosure further enhances the inferred action sequence a second time by applying reinforcement learning in a simulation environment. First, the system generates a corresponding robot control strategy based on the enhanced action sequence from the previous step and the current state information, such as the end-of-arm pose, force sensor feedback, and collision detection. Next, the system calculates multiple types of rewards, including safety rewards, distance rewards, action smoothness rewards, efficiency rewards, and success / failure rewards, to generate a comprehensive reward function:

[0047] R total = a1R 安全性 + a2R 距离 + a3R 平滑性 + a4R 效率 + a5R 成功 / 失败

[0048] The present disclosure uses reinforcement learning in a simulation environment to train and optimize the action sequence multiple times, continuously improving the accuracy and robustness of the operation. The enhanced and optimized action sequence is executed through the robot control strategy, ensuring the accuracy and stability of the actions. During execution, the system monitors state information in real time and dynamically adjusts the robot control strategy to ensure effective execution of the action sequence. The goal is to refine and optimize the execution parameters of each action, such as position and attitude, to ensure the accuracy and adaptability of the assembly action. The present disclosure also builds a highly simulated assembly environment that not only replicates the physical layout of the entity assembly line but also simulates each step of the assembly process in detail, providing an accurate and controllable experimental platform.

[0049] The complex skill dynamic enhancement method with uncertainty production line proposed by the present disclosure can accurately control assembly actions such as sucking, moving, and placing, ensuring accurate docking of components during assembly. Through this precise control, assembly error rates are significantly reduced, and product pass rates are significantly improved. At the same time, it supports dynamic identification of different assembly tasks and automatically adjusts assembly strategies and action sequences to effectively deal with the uncertainty of task types on the production line. The solution provided is also easy to expand and update, allowing it to quickly adapt to new assembly tasks and technical requirements. This scalability is particularly important for dealing with the rapidly developing 3C product market and changing technical environment, providing a long-term competitive advantage for businesses.

[0050] Corresponding to the complex skill dynamic enhancement method with an uncertain production line provided in the above several embodiments, one embodiment of the present disclosure also provides a complex skill dynamic enhancement system with an uncertain production line. Since the complex skill dynamic enhancement system with an uncertain production line provided in the embodiment of the present disclosure corresponds to the complex skill dynamic enhancement method with an uncertain production line provided in the above several embodiments, the implementation of the complex skill dynamic enhancement method with an uncertain production line is also applicable to the complex skill dynamic enhancement system with an uncertain production line provided in the embodiment of the present disclosure, which will not be described in detail in the following embodiments.

[0051] FIG. 7 is a structural schematic diagram of a complex skill dynamic enhancement system with an uncertain production line according to an embodiment of the present disclosure. As shown in FIG. 7, the complex skill dynamic enhancement system 10 with an uncertain production line includes a data acquisition module 100, a data processing module 200, a model training module 300, and a loading operation module 400.

[0052] The video data acquisition module 100 is configured to acquire a multi-view video segment containing an assembly task in a virtual scene.

[0053] The video data analysis module 200 is configured to input the multi-view video segment into a video analysis model to perform multi-round interactive question and answer between large models, so as to generate a parsed task action sequence and an action instruction.

[0054] The sequence initial enhancement module 300 is configured to input the parsed task action sequence and the action instruction and all nodes of the multi-level multi-level knowledge graph into a data enhancement network model to perform assembly knowledge integration, so as to generate an enhanced action sequence.

[0055] The sequence final enhancement module 400 is configured to perform secondary enhancement on the enhanced action sequence in a simulation environment by using reinforcement learning, so as to obtain a final task action enhancement result.

[0056] The assembly task in the embodiment of the present disclosure includes a proactive assembly task. The parsed task action sequence and the action instruction correspond to, respectively, moving-picking-resetting, moving to a proactive position, picking the proactive, and returning to an initial position.

[0057] The data enhancement network model in the embodiment of the present disclosure includes a Bert module, a GAT module, and a Seq2Seq module.

[0058] The complex skill dynamic enhancement system with uncertainty production line provided by the present disclosure can accurately control assembly actions such as sucking, moving, and placing, and ensure the accurate docking of each component in the assembly process. Through such accurate control, the assembly error rate is greatly reduced, and the product qualification rate is significantly improved. At the same time, it supports dynamic identification of different assembly tasks and automatically adjusts the assembly strategy and action sequence to effectively cope with the uncertainty of task types on the production line. The solution provided is easy to extend and update, and can quickly adapt to new assembly tasks and technical requirements. This scalability is particularly important for coping with the rapidly developing 3C product market and changing technical environment, providing a long-term competitive advantage for enterprises.

[0059] Further, the present embodiment also provides a structure diagram of the execution end of the proactive assembly task.

[0060] As shown in FIGS. 8-9, the execution end of the proactive assembly task according to the first aspect of the present disclosure includes a housing 13 and a sucking assembly, a correcting assembly, and a mounting assembly arranged in the housing 13. The sucking assembly includes a first driving member and a suction cup 10. The first driving member drives the suction cup 10 to suck the front camera 1 from a set feeding position. The correcting assembly includes a second driving member and positioning members corresponding to both sides of the suction cup 10. The second driving member drives the positioning members to adjust the posture of the front camera 1 on the suction cup 10. The mounting assembly includes a third driving member and a buckling member 9. The third driving member drives the buckling member 9 to press the front camera 1 on the suction cup 10 to a designed position.

[0061] The sucking assembly, the correcting assembly, and the mounting assembly are all arranged in the housing 13. The sucking assembly includes a first driving member and a suction cup 10. The first driving member can be a sliding table cylinder 3 fixed to one side of the inside of the housing 13. The suction cup support 14 and the suction cup 10 are connected through a connecting member 6 fixed to the corresponding position of the sliding table cylinder 3. In the case of mounting the front camera 1, the sliding table cylinder 3 drives the suction cup 10 to suck the front camera 1 from the feeding position, and then the suction cup 10 returns to the inside of the housing 13.

[0062] The correcting assembly in the present embodiment includes a second driving member and positioning members. The second driving member can be a finger cylinder 4 fixed to one side of the inside of the housing 13, with its center opposite to the sliding table cylinder 3. The positioning members correspond to both sides of the suction cup 10, such as a left positioning member and a right positioning member, i.e., a left positioning member 7 and a right positioning member 8 connected to both ends of the finger cylinder 4 as shown in FIG. 10. The finger cylinder 4 drives the left positioning member and the right positioning member to move towards each other, and the space formed by the two members is used to accommodate the front camera 1 on the suction cup 10, so as to adjust the posture of the front camera 1 on the suction cup 10, thereby improving the mounting precision of the camera.

[0063] The mounting assembly in this embodiment includes a third driving member and a fastening member 9, wherein the third driving member is a micro air cylinder 5, the micro air cylinder 5 is also mounted on the connecting member 6, and the fastening member 9 is mounted at the end of the micro air cylinder 5, and the micro air cylinder 5 drives the fastening member 9 to press the front camera 1 on the suction cup 10 to the designed position.

[0064] In some embodiments, a camera is further included, which is arranged on the shell 13 to monitor the state information of the suction assembly, the correction assembly and the mounting assembly in real time.

[0065] The camera is an industrial camera 2, which is fixed on a camera support 12, the other end of the camera support 12 is connected to the shell 13, and the camera is used to monitor the state information of the suction assembly, the correction assembly and the mounting assembly in real time, dynamically adjust the control strategy of the external mechanical arm, and ensure the effective execution of the action sequence.

[0066] In some embodiments, a mechanical arm connecting member 11 is further included, one end of which is arranged on the shell 13, and the other end is connected to the mechanical arm.

[0067] The execution end further includes the mechanical arm connecting member 11, one end of which is arranged on the shell 13, and the other end is connected to the mechanical arm, and the execution end can complete all assembly actions involved in the front camera assembly task, and can be applied to different models of mobile phone templates.

[0068] The execution end of the front camera assembly task according to the embodiments of the present disclosure can accurately control assembly actions such as suction, movement, placement, etc., to ensure the accurate docking of each component in the assembly process. Through such accurate control, the assembly error rate is greatly reduced, and the product qualification rate is significantly improved. At the same time, dynamic identification of different assembly tasks is supported, and the assembly strategy and action sequence are automatically adjusted to effectively cope with the uncertainty of task types on the production line. The solution provided is also easy to extend and update, and can quickly adapt to new assembly tasks and technical requirements.

[0069] It should be understood that the various forms of flow shown above can be reordered, added to, or deleted from. For example, each step described in the present disclosure can be executed in parallel, in sequence, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which is not limited herein.

[0070] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for dynamically enhancing complex skills in production lines with uncertainty, characterized in that, include: Acquire multi-view video clips containing assembly tasks within a virtual scene; The multi-view video segments are input into the video parsing model to perform multi-round interactive question and answer between large models in order to generate parsed task action sequences and action instructions; The parsed task action sequence, action instructions, and all nodes of the multi-level knowledge graph are input into the data augmentation network model for knowledge integration to generate an augmented action sequence. In a simulation environment, reinforcement learning is used to perform secondary enhancement on the enhanced action sequence to obtain the final task action enhancement result.

2. The method according to claim 1, characterized in that, The assembly task includes a front camera assembly task; the parsed task action sequence and action instructions respectively correspond to move-grab-reset, move to the front camera position, grab the front camera, and return to the initial position.

3. The method according to claim 2, characterized in that, The data augmentation network model includes the Bert module, the GAT module, and the Seq2Seq module.

4. The method according to claim 3, characterized in that, The parsed task action sequence and action instructions, along with all nodes of the multi-level knowledge graph, are input into the data augmentation network model for knowledge integration and assembly to generate an augmented action sequence, including: All nodes of the multi-level, multi-layered knowledge graph are input into the Bert module for task description and node... Point descriptions are used to encode the action instructions into high-dimensional word vectors; The word vectors and the graph topology information of the task are input into the GAT module. The graph attention network layer establishes associations between task nodes and optimizes the correlation and dependency between nodes to output optimized node embedding vectors. The parsed task action sequence and the optimized node embedding vector are input into the Seq2Seq module for processing to generate the specific action sequence and corresponding action parameters required for the task, so as to output the optimized action sequence.

5. The method according to claim 4, characterized in that, The method further includes: constructing a composite loss function based on the learning weights between the GAT module and the Seq2Seq module. Where L1 is the loss function of the GAT module, L2 is the loss function of the Seq2Seq module, and ab are the learnable parameters.

6. The method according to claim 5, characterized in that, In a simulation environment, reinforcement learning is used to perform secondary enhancement on the enhanced action sequence to obtain the final task action enhancement result, including: In a simulation environment, a corresponding robotic arm control strategy is generated based on the optimized action sequence and the current state information; wherein, the state information includes the robotic arm end-effector pose, force sensor feedback, and collision detection information; The robotic arm control strategy is used to calculate multiple reward types to generate a comprehensive reward function, wherein the multiple reward types include safety reward, distance reward, motion smoothness reward, efficiency reward and success / failure reward; The optimized action sequence is enhanced based on the comprehensive reward function to obtain the final task action enhancement result.

7. The method according to claim 6, characterized in that, The comprehensive reward function: R total =α1R 安全性 +α2R 距离 +α3R 平滑性 +α4R 效率 +α5R 成功 / 失败 .

8. A complex skill dynamic enhancement system for production lines with uncertainty, characterized in that, include: The video data acquisition module is used to acquire multi-view video segments containing assembly tasks in a virtual scene; The video data parsing module is used to input the multi-view video segments into the video parsing model to perform multi-round interactive question and answer between large models in order to generate parsed task action sequences and action instructions; The sequence initial enhancement module is used to input the parsed task action sequence and action instructions, as well as all nodes of the multi-level knowledge graph, into the data augmentation network model to assemble and integrate knowledge to generate an enhanced action sequence. The sequence final enhancement module is used to perform secondary enhancement on the enhanced action sequence in a simulation environment using reinforcement learning to obtain the final task action enhancement result.

9. An execution terminal for a forward assembly task, wherein, include: The housing, and the suction assembly, the correction assembly, and the mounting assembly disposed within the housing; The suction assembly includes a first driving element and a suction cup; The first driving component drives the suction cup to pick up the front-facing camera from a set loading position; the correction assembly includes a second driving component and positioning components corresponding to both sides of the suction cup, the second driving component drives the positioning components to adjust the posture of the front-facing camera on the suction cup; the mounting assembly includes a third driving component and a fastening component, the third driving component drives... The fastener presses the front-facing camera on the suction cup into the designed position.

10. The execution terminal according to claim 9, wherein, It also includes a camera mounted on the housing to monitor the status information of the suction assembly, the correction assembly, and the mounting assembly in real time.

11. The execution terminal according to claim 9 or 10, wherein, It also includes a connector disposed on the first driving member, through which the suction cup is connected to the suction cup bracket; the third driver is disposed on the connector.

12. The execution terminal according to claim 11, wherein, The first driving member is fixed inside the housing on one side, and the second driving member is disposed on the other side of the housing, with the center of the second driving member opposite to that of the first driving member.

13. The execution terminal according to claim 12, wherein, It also includes a robotic arm connector, one end of which is mounted on the housing and the other end is connected to the robotic arm.

Images