Human demonstration assembly method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide a human demonstration assembly method, device and equipment and a storage medium, and relate to the technical field of robot autonomous operation and intelligent manufacturing. A time sequence structure feature sequence is generated based on frame timing and basic perception information obtained from a video stream, the time sequence structure feature sequence is input into a task time sequence segmentation module for bidirectional time sequence modeling, at least one subtask segment is segmented, a dynamic semantic graph sequence is obtained from the video stream, the subtask segment and the time sequence structure feature sequence, each semantic stage includes at least one key frame selected from the video stream, the dynamic semantic graph sequence, the key frame and the time sequence structure feature sequence are encapsulated as a multi-modal prompt, which is input into a visual language model for assembly plan reasoning, an executable assembly plan obtained through reasoning is acquired, actions demonstrated in human demonstration are converted into task instructions that can be understood and adapted by a robot, logical correctness of an assembly process and physical adaptability of action execution are ensured, and control accuracy is improved.

Description

Technical Field

[0001] This application relates to the technical field of autonomous robot operation and intelligent manufacturing, and in particular to human-taught assembly methods, devices, equipment and storage media. Background Technology

[0002] In modern manufacturing, the assembly process accounts for a significant portion of the overall production cycle and is a key link affecting production efficiency and product quality. Learning from Demonstration (LfD) allows robots to quickly acquire assembly skills by observing human demonstrations. However, compared to simple tasks, complex assembly tasks are characterized by strong continuity, intricate geometric constraints, and implicit operational intentions, making it difficult for robots to directly understand the task structure and core operational logic from teaching videos.

[0003] Related technologies typically rely on teaching videos to guide robots in imitation learning, allowing the robot to learn motion patterns directly from the hand movement trajectory, object movement trajectory, or keyframes taught by humans. However, this method essentially depends on the "surface trajectory" during the teaching process, and there is a significant difference between the movement trajectory of a human hand and the movement space of a robotic arm. Even if the robot successfully replicates the teacher's movement path, it cannot be directly converted into effective control commands for the robotic arm, resulting in low accuracy in robot control during assembly. Summary of the Invention

[0004] The main objective of this application is to propose a human-taught assembly method, apparatus, equipment, and storage medium to improve the control accuracy of robots during the assembly process.

[0005] To achieve the above objectives, a first aspect of this application provides a human-taught assembly method, comprising: The video stream of human teaching is acquired through a visual sensor, basic perceptual information is obtained from the video stream, and a temporal structure feature sequence is generated based on the frame temporal sequence and the basic perceptual information. The temporal structure feature sequence is input into a pre-trained task temporal segmentation module for bidirectional temporal modeling, and the video stream is segmented into at least one sub-task segment. A dynamic semantic graph sequence consisting of at least one semantic graph is obtained based on the video stream, the subtask segments, and the temporal structure feature sequence. The semantic graph is used to indicate the semantic stages of assembly teaching, and each semantic stage includes at least one keyframe selected from the video stream. The dynamic semantic graph sequence, the keyframes, and the temporal structure feature sequence are encapsulated into multimodal cues and used as input to the visual language model for assembly plan reasoning. The executable assembly plan output by the visual language model is obtained. The executable assembly plan and real-time visual perception information are combined to generate specific action parameters that drive the assembly robot to perform the assembly task, and a robot control instruction sequence containing the specific action parameters is generated.

[0006] In some embodiments, the video stream includes multiple video frames, and the basic perception information includes key point information of the teaching hand, detection results of the object to be assembled, and six-degree-of-freedom pose. The generation of a temporal structure feature sequence based on the frame temporal sequence and the basic perception information includes: For each video frame, a teaching hand state feature is generated based on the key point information. The teaching hand state feature includes finger opening and closing features and finger curling features. Based on the key point information and the six-degree-of-freedom pose, generate object contact identification features; Based on the detection results, the active object and the passive object are determined, and relative posture features are generated at least based on the six degrees of freedom pose. The relative posture features include a first posture feature from the teaching hand to the active object and a second posture feature from the active object to the passive object. Based on the teaching hand state features, the relative posture features, the object contact identification features, and the object number of the object to be assembled, the structural features corresponding to the video frame are obtained, and the structural features are sorted according to the frame time sequence to obtain the time sequence structural feature sequence.

[0007] In some embodiments, the step of inputting the temporal structure feature sequence into a pre-trained task temporal segmentation module for data processing to obtain at least one subtask segment includes: The temporal structure feature sequence is input into the task temporal segmentation module for forward and backward modeling to obtain the hidden state; The hidden state is input into the boundary classifier, and the probability value of each video frame as the boundary of the assembly stage is calculated by the activation function. At least one subtask segment is obtained based on the frame sequence and the probability value.

[0008] In some embodiments, obtaining at least one of the subtask segments based on the frame timing and the probability value includes: The probability values ​​are sorted according to the frame time sequence to obtain a probability value sequence; The probability value sequence is post-processed to obtain at least one boundary range, each boundary range corresponding to a subtask segment. The post-processing operation includes at least one of local peak detection, minimum interval constraint, and probabilistic smoothing filtering.

[0009] In some embodiments, obtaining a dynamic semantic graph sequence consisting of at least one semantic graph based on the video stream, the subtask segments, and the temporal structure feature sequence includes: Determine the video segment corresponding to each subtask segment from the video stream, and determine at least one keyframe from the video segment; The interaction relationship between different entities is determined based on the structural features corresponding to the keyframe, and semantic edges are obtained based on the interaction relationship. The entities include the teaching hand, the active object, and the passive object. Using the entities as nodes and the semantic edges as connections, construct the semantic graph corresponding to each subtask fragment, and all the semantic graphs constitute the dynamic semantic graph sequence.

[0010] In some embodiments, determining the interaction relationship between different entities based on the structural features corresponding to the keyframe includes: If, based on the structural features, it is determined that the following conditions are met: the teaching hand is in a closed state, the teaching hand is in contact with the active object or the passive object, and the rate of change of relative pose is lower than a preset stability threshold, then the interaction relationship is determined to be a grasping relationship. If, based on the structural features, it is determined that the speed difference between the teaching hand and the active object is less than a preset speed threshold, then the interaction relationship is determined to be a co-motion relationship. If, based on the structural features, it is determined that the translation and rotation of the active object relative to the passive object are both within a preset precision range, then the interaction relationship is determined to be an alignment relationship. If, based on the structural features, it is determined that the interaction relationship satisfies the following condition: when the normal vectors of the active object and the passive object are aligned and their relative positions are close to the insertion start point, then the interaction relationship is determined to be in the insertion preparation state.

[0011] In some embodiments, the step of obtaining the executable assembly plan output by the visual language model, combining the executable assembly plan with real-time visual perception information to generate specific action parameters for driving the assembly robot to perform assembly tasks, and generating a robot control instruction sequence containing the specific action parameters includes: The multimodal prompts are input into a visual language model for intent reasoning to obtain the semantic description of the subtask corresponding to each semantic stage; Based on the semantic description of the subtask, the corresponding function template is retrieved from the preset action library, and the executable assembly plan of the assembly robot is calculated based on the function template. The executable assembly plan includes at least the target grasping pose, approach direction, assembly path and motion axis constraints.

[0012] Real-time visual perception signals are acquired, and the specific action parameters are obtained based on the real-time visual perception signals and the executable assembly plan, generating a robot control command sequence containing the specific action parameters. To achieve the above objective, a second aspect of this application proposes a human-taught assembly device, comprising: Feature construction module: used to acquire video streams of human teaching through visual sensors, obtain basic perceptual information from the video stream, and generate temporal structure feature sequences based on frame temporal sequence and the basic perceptual information; Task partitioning module: used to input the temporal structure feature sequence into a pre-trained task temporal segmentation module for bidirectional temporal modeling, and to segment the video stream into at least one sub-task segment; Semantic generation module: used to obtain a dynamic semantic graph sequence consisting of at least one semantic graph based on the video stream, the subtask segment and the temporal structure feature sequence, the semantic graph being used to indicate the semantic stage of assembly teaching, each semantic stage including at least one keyframe selected from the video stream; Semantic planning module: used to encapsulate the dynamic semantic graph sequence, the keyframes and the temporal structure feature sequence into multimodal cues, and use them as input to the visual language model for assembly plan reasoning; Parameter generation and execution module: used to obtain the executable assembly plan output by the visual language model, combine the executable assembly plan and real-time visual perception information to generate specific action parameters to drive the assembly robot to perform assembly tasks, and generate a robot control instruction sequence containing specific action parameters.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0015] The human-taught assembly method, apparatus, device, and storage medium proposed in this application acquire a video stream of human teaching through a visual sensor, extract basic perceptual information from the video stream, and generate a temporal structure feature sequence based on frame temporal sequence and basic perceptual information. The temporal structure feature sequence is input into a pre-trained task temporal segmentation module for bidirectional temporal modeling, segmenting the video stream into at least one sub-task segment. A dynamic semantic graph sequence consisting of at least one semantic graph is obtained based on the video stream, sub-task segments, and temporal structure feature sequence. The semantic graph indicates the semantic stage of assembly teaching, and each semantic stage includes at least one keyframe selected from the video stream. The dynamic semantic graph sequence, keyframes, and temporal structure feature sequence are encapsulated into a multimodal cue, which is used as input to a visual language model for assembly plan reasoning. An executable assembly plan output by the visual language model is obtained. The executable assembly plan and real-time visual perceptual information are combined to generate specific action parameters that drive the assembly robot to perform the assembly task, generating a robot control command sequence containing these specific action parameters. This application embodiment, based on basic perceptual information extracted from the video stream and combined with frame temporal relationships to generate a temporal structure feature sequence, captures the temporal correlation and logical progression of actions during teaching. Then, through bidirectional temporal modeling, the historical dependencies and future trends of actions are captured synchronously, accurately identifying the logical boundaries of the assembly task and segmenting the video stream into at least one sub-task segment. This process enables the robot to no longer passively imitate continuous surface action trajectories, but to understand the modular logical structure of the assembly task, breaking down the complex assembly process into executable sub-task units. Subsequently, a semantic graph is constructed to correspond to the semantic stages of assembly teaching, transforming surface visual features into high-level semantic information. Finally, based on matched multimodal cues combined with real-time visual perception information, robot execution instructions are generated, achieving a leap from trajectory description to task understanding. Thus, this embodiment constructs a vision-language-action joint modeling framework, extracting structural representations from visual information, performing semantic reasoning and stage understanding through a language model, and mapping it to action control parameters, realizing a closed-loop reasoning mechanism from human teaching understanding to robot assembly execution. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall equipment structure of the teaching and assembly experimental platform in the embodiments of this application.

[0017] Figure 2 This is an optional flowchart of the human teaching assembly method provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of the overall process of the human teaching assembly method provided in the embodiments of this application.

[0019] Figure 4 This is a flowchart of generating a temporal structure feature sequence based on frame timing and basic sensing information, provided in an embodiment of this application.

[0020] Figure 5 This is another schematic diagram illustrating the construction process of the temporal structure feature sequence provided in the embodiments of this application.

[0021] Figure 6 This is a flowchart provided in this application embodiment, which shows how a temporal structure feature sequence is input into a pre-trained task temporal segmentation module for data processing to obtain at least one subtask segment.

[0022] Figure 7 This is a flowchart provided in an embodiment of the present application for obtaining at least one subtask segment based on frame timing and probability values.

[0023] Figure 8 This is a flowchart illustrating the task timing segmentation provided in the embodiments of this application.

[0024] Figure 9 This is a flowchart of a dynamic semantic graph sequence composed of at least one semantic graph obtained from a video stream, subtask segments, and temporal structure feature sequence, provided in an embodiment of this application.

[0025] Figure 10 This is a flowchart provided in an embodiment of the present application for determining the interaction relationship between different entities based on the structural features corresponding to keyframes.

[0026] Figure 11 This application provides an executable assembly plan output by a visual language model, which is then combined with real-time visual perception information to generate specific action parameters that drive the assembly robot to perform assembly tasks, and a flowchart containing a sequence of robot control instructions with specific action parameters.

[0027] Figure 12 This is a schematic diagram of the robot control command sequence generation process for specific motion parameters provided in the embodiments of this application.

[0028] Figure 13 This is a framework diagram of the human teaching assembly method provided in the embodiments of this application.

[0029] Figure 14 This is a structural block diagram of a human teaching assembly device provided in another embodiment of this application.

[0030] Figure 15 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0034] In modern manufacturing, assembly occupies a central position connecting design, processing, and product delivery, accounting for a significant proportion of the overall production cycle and serving as a crucial link affecting efficiency and quality. While other parts of the manufacturing process are highly automated, assembly still relies heavily on manual labor. With rising labor costs, a decrease in skilled workers, and a shift towards smaller batches and personalization in production models, traditional manual assembly methods are struggling to meet the demands of flexible and high-precision production. Traditional industrial robots excel in repetitive, structured tasks, but in customized or complex assembly scenarios, they require frequent reprogramming and lack adaptability. Therefore, learning from demonstration (LfD) is considered an important approach to achieving flexible robotic assembly, enabling robots to quickly acquire operational skills by observing human examples. However, compared to simple operations, complex assembly tasks are characterized by strong continuity, complex geometric relationships, and implicit operational intentions, making it difficult for robots to directly understand the task structure and operational logic from teaching videos. Specific drawbacks include at least the following three.

[0035] First, the lack of representations of assembly structural relationships prevents models from understanding assembly intentions. Assembly tasks heavily rely on the structural relationships between objects and the semantic interactions between hands and objects. However, existing visual models can only process shallow features such as the RGB appearance or trajectory of objects, failing to represent the crucial structural information required for assembly. For example, models cannot recognize hand states (open, closed, grasping), express the relative posture between the hand and the object, describe the geometric relationships and assembly constraints between objects, determine contact states (alignment, insertion, clamping, etc.), or distinguish between active and passive objects. This lack of structural information makes it difficult for models to infer the assembly intentions implied during teaching, thus hindering accurate and reliable action generation.

[0036] Secondly, the robot lacks the ability to infer the temporal logic and sub-task boundaries of assembly tasks. Assembly tasks typically consist of a series of continuous, interconnected sub-actions, such as grasping, moving, aligning, inserting, and pressing. These stages lack clear boundaries, and changes in action often rely on subtle changes in trajectory, posture, or interaction. The continuous videos of human teaching do not explicitly label step transitions. Without the ability to understand the task's temporal sequence, the robot struggles to determine when teaching enters the next stage, when key actions occur, and when intent changes. Without recognizing the "temporal structure" of assembly, it cannot accurately understand the complete assembly process or transform it into a reproducible strategy.

[0037] Third, high-level assembly semantics are difficult to translate into executable motion parameters for the robot: even if the robot can infer some semantic intents from the teaching video, such as "grabbing parts," "aligning components," and "inserting pins into holes," this semantic information still cannot directly guide the execution of the actual robotic arm. What the robot actually needs are explicit 6DoF poses, relative poses between objects, assembly paths, insertion directions, motion axis constraints, and complete trajectories, while these low-level control parameters cannot be directly derived from the semantic layer. The inherent gap between semantics and control makes it difficult for the robot to infer accurate, executable motion plans that satisfy assembly constraints based on the teaching video.

[0038] The following are some methods for assembly teaching in related technologies.

[0039] The first type is imitation learning based on teaching videos. Its core idea is to allow the robot to learn motion patterns directly from the hand trajectories, object motion trajectories, or keyframes taught by a human. Typical examples include early behavior cloning methods, which use supervised learning to enable the robot to reproduce the instructor's motion path as closely as possible; and interactive sampling to mitigate trajectory deviation and error accumulation problems. In addition, some generative policy learning methods fit the distribution of teaching trajectories using diffusion models, also belonging to this category of trajectory-driven teaching learning methods. Although these methods can achieve good results in simple tasks (such as grasping, moving, and placing), they essentially rely on the "surface trajectory" during the teaching process. Teaching videos cannot directly present key structural information in assembly tasks, such as whether the hand has grasped the object, whether two parts are aligned, or whether the insertion stage is about to begin; trajectory curves cannot express these semantic and structural relationships. Furthermore, the motion trajectory of a human hand differs significantly from the motion space of a robotic arm; even if the instructor's path is successfully reproduced, it cannot be directly converted into control commands for the robotic arm. Therefore, this type of trajectory-based imitation learning method is unable to depict the geometric relationships, multi-step logic, and operational intentions required for assembly tasks, and cannot meet the learning needs of complex assembly scenarios.

[0040] The second approach is teaching comprehension based on visual / language models. The development of visual / language models enables robots to achieve higher-level semantic understanding when observing teaching videos. For example, image / text alignment-based models can identify the relationship between objects and actions in a video, while multimodal large-scale models can generate operation steps, action descriptions, or phased intentions based on the teaching video. The advantage of this type of method is that it can extract linguistic explanations from complex visual signals, allowing the robot to "understand" what the instructor is doing, what step is being taken, and the general semantic structure of the overall task. However, this type of method is primarily adept at outputting conclusions in natural language form, such as "grasp object A," "align part B with C," and "complete the insertion operation," which describe an abstract, high-level task semantic. Assembly tasks are essentially processes with extremely strong geometric constraints and high positional accuracy requirements. Natural language cannot contain the control parameters required for the robot to perform actions, such as 6DoF pose, alignment direction, insertion path, constraint axes, and contact state changes. Furthermore, visual / language models lack the ability to model object structural relationships, mechanical constraints, and assembly geometric logic; their reasoning results cannot be directly used as executable parts in the robot's control flow. Therefore, while this type of method can solve the problem of "how to describe the teaching process in language", it cannot solve the core problem of "how to enable the robot to generate executable actions based on the teaching", and there is still a huge gap between semantics and actions.

[0041] The third type is structural understanding technology for assembly tasks. In researching robotic assembly tasks, structural understanding technology is mainly used to identify spatial relationships and contact logic between objects and between the hand and objects during the teaching process. Related technologies typically infer structural information during assembly based on object detection, pose estimation, key point recognition, or simple geometric constraint analysis. For example, some methods use 6DoF pose estimation to extract the position and orientation of parts; some studies construct scene representations based on graph structures, using hand-designed rules or learned relation classifiers to identify spatial relationships such as "contact," "proximity," and "alignment"; and some work attempts to determine whether assembly has entered a critical stage based on changes in distance or contact events between objects. Although these technologies provide fundamental support for assembly structural understanding, the results are often fragmented and localized. For example, pose estimation can only provide the independent pose of objects and cannot describe the mating relationships during assembly; distance- or contact-based judgment methods can only capture local instantaneous states but cannot extract the structural logic of the entire process; while scene graph representation can describe certain relationships, it is difficult to cover the constantly changing and geometrically constraint-dependent operational intentions during assembly. In human instruction, these structural relationships continuously change with stages such as grasping, moving, aligning, and inserting. However, current technologies cannot provide a unified and dynamically updated structural representation that enables robots to "coherently understand" the assembly process. Due to the lack of a holistic structural representation, robots struggle to determine which assembly stage the instructor is entering and cannot generate correct motion plans based on structural information.

[0042] The fourth type is multimodal perception technology. In teaching and learning, multimodal perception technology is mainly used to acquire information about key entities involved in assembly (such as hands, objects, and tools) and their state changes, including posture, contact, and grasping. Related books typically combine RGB images, depth information, human keypoints, object 6DoF pose estimation, and hand / object interaction recognition models to enhance teaching comprehension. For example, keypoint-based hand tracking technology can provide finger position and movement state, deep learning-based object pose estimation methods can obtain the 3D pose of parts, and hand-object interaction recognition models are used to determine whether contact, grasping, or release has occurred. These technologies provide the necessary information for basic perception in teaching, enabling the robot to "see" the action elements during the teaching process. However, relying solely on multimodal perception is still insufficient to meet the needs of complex assembly tasks. On the one hand, perception results are often independent and discrete signals, such as a set of 3D keypoints, a set of pose matrices, or a contact determination label, lacking a unified organizational method and unable to constitute a complete task representation. On the other hand, these perception results cannot directly infer deep semantics in assembly, such as key intentions like "currently aligning," "about to be inserted," or "already clamped." Furthermore, current hand-object interaction recognition methods largely rely on local information, making it difficult to infer long-term structural changes by combining task context. Therefore, while multimodal perception technology can help robots identify various states during the teaching process, it cannot independently handle structural reasoning and motion planning functions in assembly task understanding. It still needs to be combined with higher-level structural representation and task understanding modules to support the learning and execution of complex assembly tasks.

[0043] This shows that the methods in the relevant technologies do not provide high accuracy in controlling the robot during the assembly process.

[0044] Based on this, embodiments of this application provide a human-taught assembly method, apparatus, device, and storage medium. Based on basic perceptual information extracted from a video stream, a temporal structure feature sequence is generated by combining frame temporal relationships to capture the temporal correlation and logical progression of actions during the teaching process. Then, through bidirectional temporal modeling, the historical dependencies and future trends of actions are captured synchronously, accurately identifying the logical boundaries of the assembly task and segmenting the video stream into at least one sub-task segment. This process allows the robot to no longer passively imitate continuous surface-level action trajectories, but rather understand the modular logical structure of the assembly task, breaking down the complex assembly process into executable sub-task units. Subsequently, a semantic graph is constructed corresponding to the semantic stages of the assembly teaching, transforming surface-level visual features into high-level semantic information. Finally, robot execution instructions are generated based on matched multimodal cues, achieving a leap from trajectory description to task understanding. Thus, the above process separates individual differences and spatial characteristics of human hand movements through semantic-level abstraction, focusing on the essential logic of the assembly task. The robotic arm does not need to reproduce the specific trajectory of the human hand; it only needs to execute corresponding actions according to the goals of the semantic stages, ensuring both the logical correctness of the assembly process and the physical adaptability of the action execution, significantly improving control accuracy and robustness.

[0045] This application provides a human teaching assembly method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the human teaching assembly method in this application embodiment is described.

[0046] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0047] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] The human-taught assembly method provided in this application relates to the technical field of robot autonomous operation and intelligent manufacturing. This method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting human-taught assembly, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. This human-taught assembly method can be executed by the terminal or the server, or by the terminal and the server working together.

[0049] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or a network, etc., and this embodiment does not impose any limitations.

[0050] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0051] First, the teaching and assembly experimental platform of the embodiments of this application is described.

[0052] In one embodiment, reference is made to Figure 1 , Figure 1 This is a schematic diagram of the overall equipment structure of the teaching and assembly experimental platform in this application embodiment. The platform integrates a multimodal vision sensor, a robot actuator, and an integrated display and interactive interface to support teaching data acquisition, scene perception, and assembly operations, and serves as the basic deployment environment for this application embodiment.

[0053] Reference Figure 1 An RGB-D camera is mounted above the platform to capture color images and depth information during the teaching process. This camera covers the entire assembly area from a top-down angle, allowing stable observation of the instructor's hand movements and spatial changes in the assembled parts, providing core visual input for structured representation and task analysis. A six-DOF robotic arm is positioned on one side of the platform as the execution body. A two-finger gripper is installed at the end of the robotic arm for grasping, moving, aligning, and inserting assembly actions. This execution mechanism is responsible for completing specific assembly steps after obtaining the action parameters and is a key component of the "language-to-action execution" stage of this invention. An embedded display screen desktop is used in the central area of ​​the assembly table. This desktop can display the teaching status, assembly guidance information, or system output results, making the teaching process more visual and providing a unified interface for interactive prompts and feedback. The presence of the display screen also provides a stable and uniform background for visual recognition, helping to improve the robustness of the perception module.

[0054] Next, the human teaching and assembly method in the embodiments of this application will be described based on the above-mentioned teaching and assembly experimental platform.

[0055] In one embodiment, Figure 2This is an optional flowchart of the human teaching assembly method provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, steps 110 to 150. It is also understood that this embodiment... Figure 2 The order of steps 110 to 150 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0056] Step 110: Acquire the video stream of human teaching through a visual sensor, obtain basic perceptual information from the video stream, and generate a temporal structure feature sequence based on the frame time sequence and the basic perceptual information.

[0057] In one embodiment, reference is made to Figure 3 , Figure 3 This is a schematic diagram of the overall process of the human teaching assembly method provided in this application embodiment. First, a continuous video stream of human teaching is acquired using an RGB-D vision sensor, such as a depth camera or a binocular vision system. This video stream contains multiple video frames arranged in a time sequence, where each video frame synchronously contains RGB image information and depth information, ensuring that the visual texture features, spatial geometric relationships, and three-dimensional position information of the teaching scene can be completely captured. Next, based on the acquired video stream, at least three types of core basic perception information are extracted, specifically including: key point information of the teaching hand, detection results of the object to be assembled, and six-degree-of-freedom pose.

[0058] The key point information of the teaching hand can be obtained through a pre-trained hand key point detection network, such as a multimodal detection model based on HRNet or HandNet. This network jointly processes the RGB image and depth information in each video frame to extract the coordinates of 21 key points of the demonstrator's hand. Each key point coordinate includes the image pixel coordinates and depth value of that key point. Furthermore, it can be converted into 3D key point coordinates in the world coordinate system using camera intrinsics, achieving accurate quantitative representation of hand posture. The detection results of the objects to be assembled can be obtained using multi-object detection algorithms, such as YOLOv8 or Faster R-CNN combined with an RGB-D feature fusion strategy. This allows for object recognition in the video frames, yielding the detection results of the objects to be assembled. The detection results include the bounding box coordinates of the objects, the object's category confidence score, and category label, which are used to clarify the type, quantity, and spatial distribution of the objects to be assembled during the teaching process. The six-degree-of-freedom pose information can be combined with object detection results and depth information, and the six-degree-of-freedom pose of each object to be assembled can be solved by the PnP (Perspective-n-Point) algorithm or the point cloud-based pose estimation model. The six-degree-of-freedom pose includes the translation vector and rotation vector of the object in the world coordinate system, which can completely represent the spatial position and attitude angle of the object.

[0059] In one embodiment, reference is made to Figure 3 Once basic perceptual information is available, the temporal structure can be constructed to obtain the temporal structure feature sequence. (Refer to...) Figure 4 , Figure 4 This is a flowchart of generating a temporal structure feature sequence based on frame timing and basic sensing information, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 410: For each video frame, generate teaching hand state features based on key point information.

[0060] In one embodiment, firstly, based on the key point information of the taught hand extracted from video frames, core features representing the hand movement state are generated through quantization calculation, providing a basis for semantic parsing. Specifically, the taught hand state features include finger opening and closing features and finger curling features.

[0061] Among them, the finger opening and closing feature is used to quantify the degree of finger opening and closing, and is expressed as:

[0062] in, To demonstrate hand position, pthumb and pindex are the three-dimensional coordinates of the thumb tip and index finger tip in the world coordinate system, respectively, among the key points of the hand. This represents the Euclidean distance between the tips of two fingers. To normalize the scale of the hand, the distance from the palm keypoint to the middle fingertip keypoint can be taken to eliminate the influence of the distance between the hand and the camera on the calculation results, so that the features have scale invariance.

[0063] Next, the finger curling feature is used to quantify the overall curling degree of the five fingers to further determine the finger condition. First, the curling angle of a single finger is calculated, expressed as:

[0064] in, This represents the curl angle of the i-th finger, where i = 1 to 5 correspond to the thumb to the little finger, respectively. and This represents the vectors of adjacent joint segments of the finger, such as the vector from the fingertip to the first joint, or the vector from the first joint to the second joint. The dot (·) indicates the dot product of these vectors. It represents the magnitude of the vector.

[0065] Next, the average curling angle of each finger is calculated to obtain the overall curling degree. , is represented as:

[0066] Step 420: Generate object contact identification features based on key point information and six-degree-of-freedom pose.

[0067] In one embodiment, by calculating the spatial distance between the teaching hand and the object to be assembled, a binary identifier characteristic representing whether there is physical contact between the two can be generated, thus clarifying the hand-object interaction state during the operation. First, the key point set of the teaching hand and the surface point set of each object to be assembled are extracted from the video frame. The key point set contains the three-dimensional coordinates of 21 key points of the hand, and the surface point set is a sampled point cloud of the object surface generated based on the object's three-dimensional model and six-degree-of-freedom pose. Finally, the minimum Euclidean distance between the key point set of the hand and the object surface point set is calculated based on the key point information and the six-degree-of-freedom pose, and is expressed as .

[0068]

[0069] in, This indicates the minimum distance between the demonstrating hand and the object to be assembled. This represents the coordinates of key points in the key point set H. This represents a surface point in the set of surface points O.

[0070] Finally, based on the preset contact threshold τc, the contact identification feature corresponding to the object is generated. , is represented as:

[0071] Understandably, if there are multiple objects to be assembled in a video frame, a multi-dimensional contact identification feature vector {contact1, contact2, ..., contactN} is generated, where N is the number of objects to be assembled, which is used to comprehensively characterize the contact state between the hand and each object.

[0072] Step 430: Determine the active and passive objects based on the detection results, and generate relative pose features based on the six degrees of freedom pose.

[0073] In one embodiment, during the assembly process, it is also necessary to first clarify the roles of the objects in the assembly operation, determine the active object and the passive object based on the detection results, and then quantify the spatial relative relationship as the relative posture feature through six-degree-of-freedom pose calculation. The relative posture feature includes the first posture feature from the teaching hand to the active object and the second posture feature from the active object to the passive object.

[0074] First, objects that directly contact and are actively manipulated by the teaching hand are defined as active objects, while objects used as assembly references or to be assembled are defined as passive objects. The active and passive objects are determined based on the detection results and contact marker features of the objects to be assembled. Therefore, the distinction between active and passive objects clarifies the operation and reference objects in the assembly process, forming the basis for relative posture calculation.

[0075] Next, the matrix corresponding to the six degrees of freedom attitude is used. The pose of the teaching hand, the active object, and the passive object is represented by the following relative posture characteristics:

[0076]

[0077] Where R is a 3×3 rotation matrix, representing the attitude angle; t is a 3×1 translation vector, representing the spatial position; SE(3) is used to describe the rigid body motion in three-dimensional space. This represents the relative pose matrix from object A to object B. It is the inverse of the pose matrix of object A.

[0078] Therefore, the first posture characteristics of the teaching hand to the active object are represented as follows: This characterizes the relative position and posture of the teaching hand and the active object, reflecting the hand's gripping and control posture during operation. The second posture characteristic from active object to passive object is... Expressing the geometric constraint relationship between the active and passive objects (such as alignment direction, insertion direction, fitting angle, etc.) is the geometric semantic feature of assembly operations.

[0079] Step 440: Obtain the structural features corresponding to the video frame based on the teaching hand state features, relative posture features, object contact mark features, and object number of the object to be assembled. Sort the structural features according to the frame time sequence to obtain the temporal structural feature sequence.

[0080] In one embodiment, the extracted teaching hand state features, relative posture features, object contact identification features, and object number of the object to be assembled are uniformly encoded to obtain the frame-level structural features corresponding to each video frame. These features are then arranged in chronological order to obtain a temporal structural feature sequence.

[0081] For the t-th video frame, its structural features Represented as:

[0082] in, Indicates the characteristics of finger opening and closing. This indicates the curling characteristic of the fingers. Indicates the first posture feature, This indicates the second posture feature. Indicates the characteristics of object contact markings, The object number represents the object to be assembled, including active and passive object numbers, used to distinguish different objects during the assembly process. It is evident that structural features provide significantly better structural representation capabilities than pixel-level features, enabling explicit modeling of assembly geometric relationships. These multi-dimensional features collectively constitute a structural representation unit, used to explicitly encode the interactive role relationships and structural state changes in the assembly scene. This structural representation unit serves as the input foundation for subsequent temporal modeling and semantic reasoning.

[0083] Next, the structural features {Token1,Token2,...,Token...} are processed in the temporal order of all video frames from frame 1 to frame T. T The sequences are sorted to obtain the temporal structure feature sequence. It is evident that the temporal structure feature sequence can completely preserve the spatial semantics, operational relationships, and stage logic of the assembly teaching process, realizing the transformation of video streams into structured features. This provides accurate and generalizable input features for subsequent temporal segmentation, semantic graph construction, assembly action planning, and robot action generation.

[0084] In one embodiment, reference is made to Figure 5 , Figure 5 This is another schematic diagram illustrating the construction process of the temporal structure feature sequence provided in this application embodiment. The diagram divides the construction process of the temporal structure feature sequence into two main modules: a perception splitting layer and a computational inference layer. First, the video stream serves as the source data input for the entire feature construction process. Then, the video stream enters the perception splitting layer for preliminary decomposition and extraction of the original visual information, achieving layered splitting processing of the perceptual data. In the perception splitting layer, the visual information of the teaching process is decomposed into two parallel processing branches: one is the key point information extraction branch for the teaching hand, which accurately identifies and outputs the key point information of the teaching hand from the video frames using a visual perception algorithm, converting it into the key point coordinates of the teaching hand. The other is the object detection and pose estimation branch, which simultaneously performs target detection and pose calculation on the object to be assembled in the video frames, outputting the object detection results and the corresponding six-degree-of-freedom pose. After processing by the perception splitting layer, the video stream is transformed into structured basic perceptual data.

[0085] Then, the basic perception information enters the computational inference layer. This layer is responsible for quantifying and logically reasoning the basic data output from the perception triage layer, generating four types of core intermediate features. Specifically, it generates teaching hand state features based on key point information of the teaching hand; it calculates object contact identification features based on the spatial distance between the set of key points on the teaching hand and the set of points on the object surface; and it generates relative posture features based on six-degree-of-freedom pose and the roles of the active and passive objects, including the first posture feature from the teaching hand to the active object and the second posture feature from the active object to the passive object. Simultaneously, it determines object numbers based on object detection results to distinguish different objects to be assembled.

[0086] After generating the above four types of features, the process proceeds to feature splicing and encapsulation. This involves uniformly aligning and encapsulating the teaching hand state features, object contact marker features, relative posture features, and object numbers in a consistent dimension, eliminating heterogeneity between features and forming a unified feature vector. Finally, the encapsulated features are transformed into frame-based structural features, containing spatial semantics, operational relationships, and stage logic from the teaching process.

[0087] Step 120: Input the temporal structure feature sequence into the pre-trained task temporal segmentation module to perform bidirectional temporal modeling and segment the video stream into at least one sub-task segment.

[0088] In one embodiment, reference is made to Figure 3 To obtain the temporal structure feature sequence, structured temporal segmentation needs to be performed using a pre-trained task-specific temporal segmentation module. (Refer to...) Figure 6 , Figure 6 This application provides a flowchart illustrating how a temporal structure feature sequence is input into a pre-trained task temporal segmentation module for data processing to obtain at least one subtask segment. The flowchart specifically includes the following steps: Step 610: Input the temporal structure feature sequence into the task temporal segmentation module for forward and backward modeling to obtain the hidden state.

[0089] In one embodiment, the task time sequence segmentation module is used to perform phased analysis of assembly actions, perform bidirectional time dimension modeling on the input time sequence structure feature sequence, capture the preceding and following dependencies in the assembly task, and generate hidden states that integrate contextual information.

[0090] Specifically, the temporal structure feature sequence is input to the task temporal segmentation module. This module employs a bidirectional long short-term memory (Bi-LSTM) network as the modeling network, simultaneously modeling the dynamic changes of the feature sequence in both forward and backward temporal directions. The forward LSTM processes frame by frame from the start frame to the end frame, capturing forward dependencies of the action sequence, while the backward LSTM processes frame by frame from the end frame to the start frame, capturing backward associations. Therefore, after bidirectional modeling, the task temporal segmentation module outputs the hidden state corresponding to each video frame. This hidden state contains the local features and contextual temporal information of the current frame, providing a high-dimensional semantic representation for subsequent boundary recognition.

[0091] Step 620: Input the hidden state into the boundary classifier, calculate the probability value of each video frame as the boundary of the assembly stage through the activation function, and obtain at least one sub-task segment based on the frame time sequence and probability value.

[0092] In one embodiment, structured subtask fragments can be generated by identifying assembly stage boundaries using a classifier based on hidden states.

[0093] For the t-th video frame, first, the hidden state is... Input boundary classifier, the classifier is transformed by linear transformation Then, the probability value of each video frame as the boundary of the assembly stage is calculated using the sigmoid activation function. , is represented as:

[0094] in, This represents the probability that the t-th video frame is a stage switching point. The closer the value is to 1, the more likely that the frame is to be a turning point in the assembly process.

[0095] Reference Figure 7 , Figure 7 This is a flowchart provided in an embodiment of the present application for obtaining at least one subtask segment based on frame timing and probability values, specifically including the following steps: Step 710: Sort the probability values ​​according to the frame time sequence to obtain the probability value sequence.

[0096] Step 720: Post-process the probability value sequence to obtain at least one boundary range.

[0097] In one embodiment, the probability values ​​of all frames are arranged sequentially according to the time order of the video frames to obtain a probability value sequence. Then, the probability value sequence is post-processed to transform the output original boundary probabilities into accurate, robust, and executable assembly stage boundaries, resulting in at least one boundary range. Each boundary range corresponds to a subtask segment. Here, the post-processing operation includes at least one of local peak detection, minimum interval constraint, and probabilistic smoothing filtering.

[0098] In this embodiment, probabilistic smoothing filtering is used to eliminate noise interference and improve the robustness of boundary recognition. Since the original probability value sequence usually contains random interference introduced by perceptual noise, model uncertainty, or local motion fluctuations, it manifests as local spurious peaks or irregular fluctuations. Therefore, this embodiment uses probabilistic smoothing filtering (such as Gaussian smoothing or moving average filtering) to perform low-pass processing on the probability value sequence, effectively smoothing these high-frequency noises and making the probability curve more gradual. This significantly reduces false detections caused by local motion jitter or detection noise, ensuring the global stability of stage boundary recognition and avoiding misjudging minor motion fluctuations as assembly step switching. Furthermore, a local peak detection algorithm can be used to search for local maxima points where the probability value is significantly higher than that of surrounding frames, thereby accurately capturing significant peaks in the probability sequence. These peaks correspond to key points where semantic changes are most drastic in the teaching action (such as the instant from "grab" to "move"). Compared to direct hard thresholding, peak detection is better able to locate the true stage switching points, ensuring the accuracy and semantic saliency of boundary recognition. In addition, the minimum interval constraint must be met between adjacent subtask segments, that is, the boundaries of two consecutive assembly stages cannot be too close, and a sufficient time span must be maintained to form a complete semantic task. Therefore, the minimum interval constraint can also filter out redundant boundaries that are too close, avoid dividing the same assembly step into too many fragmented sub-segments, ensure that each subtask segment has clear and independent assembly semantics, and at the same time ensure that the transition between adjacent tasks is natural and logically coherent, avoiding the problem of segments being too short or semantic overlap. Accurate assembly stage division can be obtained without frame-level annotation, reducing the burden of manual annotation and making the assembly structure clearer.

[0099] In one embodiment, the task temporal segmentation module is trained using a weakly supervised learning paradigm, relying solely on sparsely labeled stage boundary samples for training. The model parameters are optimized using a binary cross-entropy loss function to achieve structured decomposition and stage division of continuous teaching actions.

[0100] The corresponding loss function is expressed as:

[0101] Where T' represents the total number of video frames during the training process. This represents the probability that the t-th frame predicted by the model is the stage boundary. ∈{0,1} are the sparse annotation labels for this frame, where, =1 indicates that this frame is a stage boundary. =0 indicates a non-boundary frame. This weakly supervised optimization method eliminates the need for dense frame-by-frame annotation; model training can be completed with only sparsely labeled boundary samples, significantly reducing annotation costs. At the same time, the cross-entropy loss constrains the model to accurately learn the switching features of the action phase.

[0102] In one embodiment, reference is made to Figure 8 , Figure 8 This is a flowchart illustrating the task temporal segmentation process provided in this application embodiment. As shown in the diagram, through bidirectional temporal modeling and probabilistic post-processing, a continuous sequence of actions can be segmented into structured subtask segments. First, the task temporal segmentation module receives the input temporal structure feature sequence and simultaneously captures the dependencies between actions using two parallel LSTM units, outputting a hidden state that integrates local features of the current frame with global information from the preceding and following contexts. Next, the hidden state output by the bidirectional LSTM is input to a boundary classifier to calculate the probability value of each frame representing an assembly stage boundary. Based on the probability values, a probability value sequence is obtained. To improve the accuracy and robustness of boundary recognition, the probability value sequence undergoes post-processing to finally obtain a boundary set, which serves as the final stage boundary for subsequent task segmentation and analysis.

[0103] Step 130: Obtain a dynamic semantic graph sequence consisting of at least one semantic graph based on the video stream, subtask segments, and temporal structure feature sequence.

[0104] In one embodiment, a semantic map is used to indicate the semantic stages of assembly teaching, explicitly expressing the interactions in the assembly process, making the task's semantic structure clearer, and facilitating subsequent semantic planning. Simultaneously, subtask segments correspond to semantic stages, and each semantic stage includes at least one keyframe selected from the video stream. (Refer to...) Figure 9 , Figure 9 This is a flowchart of an embodiment of the present application providing a dynamic semantic graph sequence composed of at least one semantic graph obtained from a video stream, subtask segments, and temporal structure feature sequences, specifically including the following steps: Step 910: Determine the video segment corresponding to each subtask segment from the video stream, and determine at least one keyframe from the video segment.

[0105] In one embodiment, reference is made to Figure 3 After obtaining multiple sub-task segments, the semantic planning phase based on the Vision-Language Model (VLM) begins. Based on the temporal segmentation results, the continuous video stream is divided into multiple sub-task segments, each corresponding to the execution process of a single semantic action. For each sub-task segment, the corresponding video segment is determined from the video stream, and its temporal range is represented as follows:

[0106] in, This represents the k-th subtask fragment. The starting frame, Set it as the end frame.

[0107] Then, keyframes are extracted from the video clip, redundant frames are removed to reduce subsequent computational complexity, while the core semantic information of the action sequence is fully preserved. (Refer to...) Figure 3 In this embodiment, each subtask segment contains at least three keyframes: a start frame, a middle frame, and an end frame. The start frame corresponds to the beginning node of the subtask segment and is used to represent the switching of action phases. The middle frame is the frame with the most representative action features within the segment and is used to capture the core state during action execution. The end frame corresponds to the end node of the action phase and provides the final state basis for subsequent relationship determination. By extracting keyframes, the amount of data can be effectively compressed, the efficiency of semantic reasoning can be improved, and key semantic information can be ensured without loss.

[0108] Step 920: Determine the interaction relationship between different entities based on the structural features corresponding to the keyframes, and obtain semantic edges based on the interaction relationship.

[0109] In one embodiment, entities include a teaching hand, an active object, and a passive object. This embodiment uses the structural features corresponding to keyframes as a basis to infer the semantic interaction relationships between different entities through decision conditions, and transforms the decision results into structured semantic edges, providing core connection relationships for semantic graph construction. (Refer to...) Figure 10 , Figure 10 This is a flowchart provided in an embodiment of the present application for determining the interaction relationship between different entities based on the structural features corresponding to keyframes, specifically including the following steps: Step 1010: If the following conditions are met based on the structural characteristics: the teaching hand is in a closed state, the teaching hand is in contact with the active or passive object and the rate of change of relative pose is lower than the preset stable threshold, then the interaction relationship is determined to be a grasping relationship.

[0110] In one embodiment, a grasping relationship is determined when the teaching hand is in a closed state, the teaching hand is in contact with the active or passive object, and the rate of change of relative pose is lower than a preset stability threshold. The state of the teaching hand's grasp of the active / passive object is represented as:

[0111]

[0112] in, The threshold for hand opening and closing. To preset a stable threshold, It represents the rate of change of the relative posture of the teaching hand in contact with the active or passive object.

[0113] Step 1020: If the structural characteristics determine that the speed difference between the teaching hand and the active object is less than the preset speed threshold, the interaction relationship is determined to be a common motion relationship.

[0114] In one embodiment, when the speed difference between the teaching hand and the active object is less than a preset speed threshold, it is determined that they are in a co-motion relationship. This reflects the cooperative motion state between entities and is represented as:

[0115]

[0116] in, Indicates the speed of hand movement during instruction. Indicates the velocity of an active object. This indicates the preset speed threshold.

[0117] Step 1030: If the structural characteristics determine that the translation and rotation of the active object relative to the passive object are both within the preset accuracy range, then the interaction relationship is determined to be an alignment relationship.

[0118] In one embodiment, when both the translation and rotation of the active object relative to the passive object are within a preset accuracy range, an alignment relationship is determined. The geometric alignment state between entities is represented as:

[0119]

[0120] in, This represents the relative pose matrix from the active object to the passive object. Indicates the translation accuracy threshold. This indicates the rotation accuracy threshold. Here, the translation accuracy threshold and the rotation accuracy threshold together constitute the preset accuracy range.

[0121] Step 1040: If the structural characteristics determine that the following conditions are met: when the normal vectors of the active object and the passive object are aligned and their relative positions are close to the insertion start point, the interaction relationship is determined to be in the insertion preparation state.

[0122] In one embodiment, when the normal vectors of the active object and the passive object are aligned and their relative positions are close to the insertion start point, the insertion preparation state is determined. The preparatory stage for assembly is represented as:

[0123]

[0124] in, , These are the normal vectors of the active and passive objects, respectively. This indicates the calculation of the included angle. Indicates relative position, For angle threshold, This is the insertion position threshold.

[0125] Step 930: Construct a semantic graph corresponding to each subtask fragment, using entities as nodes and semantic edges as connections.

[0126] In one embodiment, all semantic graphs constitute a dynamic semantic graph sequence. First, using the entities determined above as nodes and semantic edges as connections, a scene semantic graph corresponding to each subtask segment is constructed. Finally, the semantic graphs of all subtask segments are integrated in chronological order to form a dynamic semantic graph sequence.

[0127] Specifically, a semantic graph is defined as a directed graph, represented as:

[0128]

[0129]

[0130]

[0131] in, This represents a set of nodes, corresponding to three entities: the teaching hand, the active object, and the passive object. Let R represent the semantic edge set, defined as a triple (u,v,r), where u and v are nodes, r is the interaction relationship determined between the connection points, and R represents the set of relations.

[0132] For subtask phase k, The corresponding semantic graphs are automatically generated from semantic relations that satisfy the judgment conditions, for example... It can accurately describe the semantic interactions and geometric constraints between entities within the subtask phase.

[0133] Then, the semantic graphs corresponding to all subtask segments are arranged in temporal order according to the video frames, resulting in a dynamic semantic graph sequence. Each semantic graph in this dynamic semantic graph sequence corresponds to an independent semantic stage in the assembly teaching process, fully preserving the temporal logic and semantic evolution of the assembly actions.

[0134] Step 140: Encapsulate the dynamic semantic graph sequence, keyframes, and temporal structure feature sequence into multimodal cues, and use them as input to the visual language model for assembly plan reasoning.

[0135] In one embodiment, to connect high-level semantic understanding with the robot's actual execution, multimodal information fusion and semantic planning are used to transform the structured features of the teaching process into a sequence of actions that can directly drive the assembly robot. Here, dynamic semantic graph sequences, keyframes, and temporal structure feature sequences are integrated and encapsulated into a multimodal prompt that meets the input requirements of a visual language model. This multimodal prompt includes the semantic logic of entity interactions from the dynamic semantic graph sequence, the intuitive visual information from the keyframes, and the quantified geometric and motion features from the temporal structure feature sequence, thereby providing comprehensive and multi-dimensional input information for subsequent intent reasoning.

[0136] Step 150: Obtain the executable assembly plan output by the visual language model, combine the executable assembly plan and real-time visual perception information to generate specific action parameters to drive the assembly robot to perform assembly tasks, and generate a robot control instruction sequence containing specific action parameters.

[0137] In one embodiment, reference is made to Figure 11 , Figure 11 This application provides an executable assembly plan output by a visual language model. It combines the executable assembly plan with real-time visual perception information to generate specific action parameters that drive the assembly robot to perform assembly tasks. A flowchart is then generated containing a sequence of robot control instructions with these specific action parameters. The process includes the following steps: Step 1110: Input the multimodal prompts into the visual language model for intent reasoning to obtain the semantic description of the subtask corresponding to each semantic stage.

[0138] In one embodiment, the encapsulated multimodal prompts are input into a pre-trained visual language model. Based on the interaction relationships between entities in the dynamic semantic graph sequence, scene visual information in keyframes, and quantized features in the temporal structure feature sequence, the model performs deep reasoning on the operational intent of each semantic stage. Through the cross-modal reasoning capability of the visual language model, complex structured features are transformed into semantic descriptions that are understandable to humans and parsable by machines. The reasoning result is a sub-task semantic description with clear semantic meaning, such as "grasp the active object (lid)," "move the lid above the passive object (box)," "align the lid with the box opening," and "prepare to insert the lid into the box." These semantic descriptions clarify the task objectives and operational constraints, rather than directly generating robot actions. This clearly defines the core task of each stage, providing a unified semantic benchmark for subsequent action template matching, significantly improving the system's understanding accuracy of assembly intent, and enhancing its generalization ability across different assembly scenarios without requiring redesign of semantic parsing rules for specific tasks.

[0139] Step 1120: Based on the semantic description of the subtask, retrieve the corresponding function template from the preset action library, and calculate the executable assembly plan of the assembly robot based on the function template.

[0140] In one embodiment, the system pre-sets an action library containing a variety of basic assembly actions. The action library stores function templates corresponding to assembly actions such as grabbing, approaching, aligning, inserting, and releasing. Each function template defines the execution logic, parameter types, and constraint requirements of the action.

[0141] Therefore, based on the semantic description of the subtask, the function template that best matches the current semantic description is selected from the action library through action template retrieval and matching. For example, the semantic description "grasp the active object" corresponds to "grasp action template," and "align the active object and the passive object" corresponds to "align action template." After matching the action templates, an executable assembly plan is directly generated sequentially. This executable assembly plan includes the target grasping pose, approach direction, assembly path, and motion axis constraints. Specifically, the generation of the executable assembly plan is based on semantic relationship constraints and structural state reasoning results, combined with assembly stage features for mapping calculation, thereby ensuring that the generated actions meet the structural constraints and assembly stability requirements.

[0142] As can be seen, this embodiment simplifies the action generation process by using a preset action library and template matching mechanism, ensures the accuracy and scene adaptability of the executable assembly plan by calculating based on function templates, and makes the parameters more in line with actual assembly needs by combining dynamic adjustments with real-time visual feedback.

[0143] Step 1130: Acquire real-time visual perception signals, and obtain specific motion parameters based on the real-time visual perception signals and the executable assembly plan, and generate a robot control instruction sequence containing the specific motion parameters.

[0144] In one embodiment, real-time visual perception signals are acquired. Here, real-time scene perception data can include the target object's 3D pose, spatial position, and environmental observation information. Then, based on the real-time visual perception signals, the executable assembly plan is generated according to the parameter format requirements of the function template, producing complete and specific action parameters including action type, execution parameters, and constraints. This allows the system to adapt to changes in object position, posture, and external disturbances during actual execution, achieving autonomous assembly without trajectory reproduction. For example, the target grasping pose and approach direction are written into the "grasping action template" to obtain specific action parameters such as "grasping the object at position xx (obtained from the real-time visual perception signal) with a pose (x, y, z, α, β, γ), and approaching along the negative z-axis."

[0145] Next, after obtaining the specific motion parameters, they are directly filled into the execution instructions corresponding to the executable assembly plan to generate a robot control instruction sequence containing the specific motion parameters. These specific motion parameters completely reproduce the motion flow of the teaching process, with each specific motion parameter corresponding to a specific motion execution step. The generated robot control instruction sequence ensures that the robot performs assembly operations according to the correct timing, posture, and path.

[0146] In one embodiment, reference is made to Figure 12 , Figure 12 This is a schematic diagram of the robot control command sequence generation process for specific motion parameters provided in the embodiments of this application. Figure 12 This demonstrates the end-to-end assembly action generation process from high-level semantic planning to robot-level execution. The process sequentially involves input layer fusion, semantic relationship reasoning, multimodal prompt construction, intent reasoning, action template matching, and parameter inference, ultimately generating specific action parameters that can directly drive the robot. This fully realizes the transformation from understanding the teaching video to actual assembly execution.

[0147] First, the input layer acquires the video segments and temporal structure feature sequences corresponding to the subtask segments, and then inputs them into different processing branches. In the video segment branch, keyframe extraction is performed, selecting representative keyframes from the video segments corresponding to the subtask segments. In the temporal structure feature sequence branch, semantic relationship inference is performed to generate a dynamic semantic graph sequence. After extracting the keyframes and dynamic semantic graph sequences, the multimodal prompt construction stage begins. The intuitive visual information of the keyframes, the entity interaction semantic logic of the dynamic semantic graph sequence, and the quantized geometric features in the temporal structure feature sequence are deeply fused and encapsulated into multimodal prompts that meet the input requirements of the visual language model. The constructed multimodal prompts are input into the visual language model for intent reasoning. Based on the entity interaction relationships in the dynamic semantic graph, the visual scene information of the keyframes, and the quantized features of the temporal structure features, the model performs deep reasoning on the operational intent of each semantic stage, ultimately outputting a subtask semantic description with clear semantic meaning, such as "grab the active object" or "align the active object with the passive object." Next, the action template matching and parameter inference stage begins. Based on the semantic description of the subtask, the corresponding function template is retrieved and matched from the preset action library. Then, the executable assembly plan of the assembly robot is calculated based on the retrieved function template, including the target grasping pose, approach direction, assembly path, motion axis constraints, etc., to ensure the accuracy of parameters and scene adaptability.

[0148] Finally, the executable assembly plan is inferred from real-time visual perception signals, and the calculated specific action parameters are written into a matching function template to generate a robot control instruction sequence containing these parameters. This instruction sequence completely reproduces the action flow of the teaching process, driving the assembly robot to perform assembly tasks according to the correct timing, posture, and path constraints. This achieves a direct transformation from high-level semantic planning to low-level control execution, significantly improving the efficiency and accuracy of assembly action generation and providing reliable instruction support for autonomous robot assembly.

[0149] In one embodiment, reference is made to Figure 3 The system integrates keyframe visual information, dynamic semantic maps, and temporal structural feature sequences into a multimodal prompt, which is then input into a visual language model for intent inference. The prompt explicitly requires users to "infer the corresponding robot operation based on the visual features and structured information of this stage." After inference by the visual language model, it outputs a semantically clear subtask description, presented in the form of structured steps, such as "Step 1: move_to(lid)", "Step 2: grab(lid)", and "Step 3: move_to(box)", transforming the abstract assembly intent into parsable task steps.

[0150] Next, the semantic description of the subtask is transformed into low-level control code that the robot can directly execute, realizing the implementation from semantic goals to specific action parameters. First, the system matches the semantic description with a predefined action library. Through action template matching, the corresponding action type and its parameterization form are determined, including basic assembly actions such as grasping, approaching, aligning, inserting, and releasing, forming an executable assembly plan. Then, combined with real-time geometric data from scene awareness, specific action parameters are generated, filling in the parameters required for each instruction in the executable assembly plan. Finally, a sequence of robot control instructions containing specific action parameters is generated to drive the assembly robot to complete the assembly task. For example, specific action parameters include the target pose of the end effector (pose=[0.35,0.12,0.05]), gripper opening and closing control (close_gripper()), and movement path constraints. These instructions precisely quantify the execution posture, position, and timing of each assembly action, directly mapping the high-level semantic plan to the robot's low-level motion control instructions, ensuring the accuracy and standardization of action execution.

[0151] Finally, the sequence of robot control commands, containing specific motion parameters, is transmitted to the assembly robot, driving the robotic arm and end effector to perform the assembly operation as planned. For example... Figure 3The video displays continuous motion frames during the robot's execution process, sequentially completing actions such as "moving to the target object (lid)," "grabbing the target object," and "moving to the assembly target (packaging box)." The entire execution process follows the parameter constraints of the executable assembly plan, and the system can simultaneously receive real-time visual feedback to make online corrections to the motion parameters, forming a closed-loop vision-driven execution mechanism. This ensures that the robot can accurately and stably replicate the taught actions and complete the autonomous assembly task.

[0152] In one embodiment, reference is made to Figure 13 , Figure 13 This is a framework diagram of the human teaching and assembly method provided in this application embodiment. The framework diagram revolves around two main stages: "visual language understanding" and "language action execution," constructing a closed-loop chain of "perception-modeling-reasoning-execution-feedback." The basic deployment environment of the framework is a teaching and assembly experimental platform, which integrates an RGB-D camera, a six-degree-of-freedom robotic arm, a two-finger gripper, and an embedded display desktop. The RGB-D camera captures color images and depth information of the teaching process from a top-down angle, the robotic arm acts as the execution subject to complete various assembly actions, and the display desktop provides visual interaction and a stable background, providing hardware support for the entire process.

[0153] In the visual language understanding stage, the system first acquires the video stream of human teaching through an RGB-D vision sensor, extracting basic perceptual information such as key points of the teaching hand, detection results of the object to be assembled, and six-degree-of-freedom pose. Then, based on this information, it generates teaching hand state features, object contact marker features, and relative posture features, which are combined with object numbers and encapsulated into frame-level structural features. These features are then sorted sequentially by frame to obtain a temporal structural feature sequence. Next, this sequence is input into a pre-trained task temporal segmentation module, where bidirectional LSTM is used for forward and backward modeling. After boundary classifier calculation and post-processing, the video stream is segmented into multiple sub-tasks. The process begins with extracting keyframes from the video clips corresponding to each subtask segment. Based on the structural features of the keyframes, the interaction relationships (grasping, joint movement, alignment, insertion preparation, etc.) between the teaching hand, active object, and passive object are determined. Using entities as nodes and interaction relationships as semantic edges, a semantic graph corresponding to each subtask segment is constructed and integrated temporally to form a dynamic semantic graph sequence. Finally, the dynamic semantic graph sequence, keyframes, and temporal structural feature sequence are encapsulated into a multimodal cue and input into a visual language model for intent reasoning. This yields the semantic description of each subtask corresponding to each semantic stage, completing the abstraction from visual structure to high-level assembly semantics.

[0154] During the language-driven action execution phase, the system utilizes a vision-driven action parameter generator to retrieve corresponding function templates from a pre-set action library based on the semantic description of the subtask. Based on the matching results of these function templates, it calculates executable assembly plans, including target grasping pose, approach direction, assembly path, and motion axis constraints. Following real-time visual perception feedback, the system calculates and writes the specific action parameters of the executable assembly plan into the function template, generating a sequence of robot control instructions containing these parameters. Finally, this sequence of control instructions is transmitted to the six-DOF robotic arm, driving it to execute assembly actions such as grasping, moving, aligning, and inserting in sequence. Simultaneously, the system continuously receives real-time visual perception signals for online correction, forming a closed-loop vision-driven execution mechanism that realizes the transition from semantic understanding to physical assembly behavior. This entire framework, through the deep integration of structured perception, semantic understanding, and vision-driven execution, enables the robot to accurately replicate human-taught actions and complete autonomous assembly tasks without manual programming or trajectory teaching.

[0155] The technical solution provided in this application acquires a video stream of human teaching through a visual sensor, obtains basic perceptual information from the video stream, and generates a temporal structure feature sequence based on frame temporal sequence and basic perceptual information. The temporal structure feature sequence is input into a pre-trained task temporal segmentation module for bidirectional temporal modeling, segmenting the video stream into at least one sub-task segment. Based on the video stream, sub-task segments, and temporal structure feature sequence, a dynamic semantic graph sequence consisting of at least one semantic graph is obtained. The semantic graph is used to indicate the semantic stages of assembly teaching. Each semantic stage includes at least one keyframe selected from the video stream. The dynamic semantic graph sequence, keyframes, and temporal structure feature sequence are encapsulated into a multimodal cue, which is used as input to a visual language model for assembly plan reasoning. An executable assembly plan output by the visual language model is obtained. The executable assembly plan and real-time visual perceptual information are combined to generate specific action parameters that drive the assembly robot to perform assembly tasks, generating a robot control command sequence containing these specific action parameters. This application embodiment, based on basic perceptual information extracted from the video stream and combined with frame temporal relationships to generate a temporal structure feature sequence, captures the temporal correlation and logical progression of actions during teaching. Then, through bidirectional temporal modeling, the historical dependencies and future trends of actions are captured synchronously, accurately identifying the logical boundaries of the assembly task and segmenting the video stream into at least one sub-task segment. This process enables the robot to no longer passively imitate continuous surface-level action trajectories, but to understand the modular logical structure of the assembly task, breaking down the complex assembly process into executable sub-task units. Subsequently, a semantic graph is constructed to correspond to the semantic stage of assembly teaching, transforming surface-level visual features into high-level semantic information. Finally, an executable assembly plan is generated based on matching multimodal prompts, achieving a leap from trajectory description to task understanding. As can be seen, the above process separates the individual differences and spatial characteristics of human hand movements through semantic-level abstraction, focusing on the essential logic of the assembly task. The robotic arm does not need to reproduce the specific trajectory of the human hand; it only needs to execute the corresponding actions according to the goals of the semantic stage. This ensures both the logical correctness of the assembly process and the physical adaptability of the action execution, significantly improving control accuracy and robustness.

[0156] This application also provides a human teaching assembly device that can implement the above-described human teaching assembly method, referring to... Figure 14 The device includes: Feature construction module 1410: used to acquire video streams of human teaching through visual sensors, obtain basic perceptual information from the video stream, and generate temporal structure feature sequences based on frame temporal sequence and basic perceptual information.

[0157] Task partitioning module 1420: Used to input the temporal structure feature sequence into the pre-trained task temporal segmentation module for bidirectional temporal modeling, and to segment the video stream into at least one sub-task segment.

[0158] Semantic generation module 1430: used to obtain a dynamic semantic graph sequence consisting of at least one semantic graph based on the video stream, subtask segments and temporal structure feature sequence. The semantic graph is used to indicate the semantic stage of assembly teaching, and each semantic stage includes at least one keyframe selected from the video stream.

[0159] Semantic planning module 1440: used to encapsulate dynamic semantic graph sequences, keyframes and temporal structure feature sequences into multimodal cues, and use them as input to the visual language model for assembly plan reasoning.

[0160] Parameter generation and execution module 1450: used to obtain the executable assembly plan output by the visual language model, combine the executable assembly plan and real-time visual perception information to generate specific action parameters to drive the assembly robot to perform assembly tasks, and generate a sequence of robot control instructions containing specific action parameters.

[0161] The specific implementation of the human teaching assembly device in this embodiment is basically the same as the specific implementation of the human teaching assembly method described above, and will not be repeated here.

[0162] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the human teaching assembly method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0163] Please see Figure 15 , Figure 15 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1501 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501 using the human teaching assembly method of the embodiments of this application. The input / output interface 1503 is used to implement information input and output; The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504); The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0164] This application embodiment also provides a storage medium that stores a computer program, which, when executed by a processor, implements the above-described human teaching assembly method.

[0165] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] The human-taught assembly method, apparatus, device, and storage medium proposed in this application acquire a video stream of human teaching through a visual sensor, extract basic perceptual information from the video stream, and generate a temporal structure feature sequence based on frame temporal sequence and basic perceptual information. The temporal structure feature sequence is input into a pre-trained task temporal segmentation module for bidirectional temporal modeling, segmenting the video stream into at least one sub-task segment. A dynamic semantic graph sequence consisting of at least one semantic graph is obtained based on the video stream, sub-task segments, and temporal structure feature sequence. The semantic graph indicates the semantic stage of assembly teaching, and each semantic stage includes at least one keyframe selected from the video stream. The dynamic semantic graph sequence, keyframes, and temporal structure feature sequence are encapsulated into a multimodal cue, which is used as input to a visual language model for assembly plan reasoning. An executable assembly plan output by the visual language model is obtained. The executable assembly plan and real-time visual perceptual information are combined to generate specific action parameters that drive the assembly robot to perform the assembly task, generating a robot control command sequence containing these specific action parameters. This application embodiment, based on basic perceptual information extracted from the video stream and combined with frame temporal relationships to generate a temporal structure feature sequence, captures the temporal correlation and logical progression of actions during teaching. Then, through bidirectional temporal modeling, the historical dependencies and future trends of actions are captured synchronously, accurately identifying the logical boundaries of the assembly task and segmenting the video stream into at least one sub-task segment. This process enables the robot to no longer passively imitate continuous surface action trajectories, but to understand the modular logical structure of the assembly task, breaking down the complex assembly process into executable sub-task units. Subsequently, a semantic graph is constructed to correspond to the semantic stages of assembly teaching, transforming surface visual features into high-level semantic information. Finally, based on matched multimodal cues combined with real-time visual perception information, robot execution instructions are generated, achieving a leap from trajectory description to task understanding. Thus, this embodiment constructs a vision-language-action joint modeling framework, extracting structural representations from visual information, performing semantic reasoning and stage understanding through a language model, and mapping it to action control parameters, realizing a closed-loop reasoning mechanism from human teaching understanding to robot assembly execution.

[0167] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0168] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0171] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0172] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0174] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A human-taught assembly method, characterized in that, include: The video stream of human teaching is acquired through a visual sensor, basic perceptual information is obtained from the video stream, and a temporal structure feature sequence is generated based on the frame temporal sequence and the basic perceptual information. The temporal structure feature sequence is input into a pre-trained task temporal segmentation module for bidirectional temporal modeling, and the video stream is segmented into at least one sub-task segment. A dynamic semantic graph sequence consisting of at least one semantic graph is obtained based on the video stream, the subtask segments, and the temporal structure feature sequence. The semantic graph is used to indicate the semantic stages of assembly teaching, and each semantic stage includes at least one keyframe selected from the video stream. The dynamic semantic graph sequence, the keyframes, and the temporal structure feature sequence are encapsulated into multimodal cues and used as input to the visual language model for assembly plan reasoning. The executable assembly plan output by the visual language model is obtained. The executable assembly plan and real-time visual perception information are combined to generate specific action parameters that drive the assembly robot to perform the assembly task, and a robot control instruction sequence containing the specific action parameters is generated.

2. The human teaching assembly method according to claim 1, characterized in that, The video stream includes multiple video frames, and the basic perception information includes key point information of the teaching hand, detection results of the object to be assembled, and six-degree-of-freedom pose. The generation of a temporal structure feature sequence based on the frame temporal sequence and the basic perception information includes: For each video frame, a teaching hand state feature is generated based on the key point information. The teaching hand state feature includes finger opening and closing features and finger curling features. Based on the key point information and the six-degree-of-freedom pose, generate object contact identification features; Based on the detection results, active and passive objects are determined, and relative posture features are generated based on the six degrees of freedom pose. The relative posture features include a first posture feature from the teaching hand to the active object and a second posture feature from the active object to the passive object. Based on the teaching hand state features, the relative posture features, the object contact identification features, and the object number of the object to be assembled, the structural features corresponding to the video frame are obtained, and the structural features are sorted according to the frame time sequence to obtain the time sequence structural feature sequence.

3. The human teaching assembly method according to claim 1, characterized in that, The step of inputting the temporal structure feature sequence into a pre-trained task temporal segmentation module for data processing to obtain at least one subtask segment includes: The temporal structure feature sequence is input into the task temporal segmentation module for forward and backward modeling to obtain the hidden state; The hidden state is input into the boundary classifier, and the probability value of each video frame as the boundary of the assembly stage is calculated by the activation function. At least one subtask segment is obtained based on the frame sequence and the probability value.

4. The human teaching assembly method according to claim 3, characterized in that, The step of obtaining at least one of the sub-task segments based on the frame timing and the probability value includes: The probability values ​​are sorted according to the frame time sequence to obtain a probability value sequence; The probability value sequence is post-processed to obtain at least one boundary range, each boundary range corresponding to a subtask segment. The post-processing operation includes at least one of local peak detection, minimum interval constraint, and probabilistic smoothing filtering.

5. The human teaching assembly method according to claim 2, characterized in that, The step of obtaining a dynamic semantic graph sequence consisting of at least one semantic graph based on the video stream, the subtask segments, and the temporal structure feature sequence includes: Determine the video segment corresponding to each subtask segment from the video stream, and determine at least one keyframe from the video segment; The interaction relationship between different entities is determined based on the structural features corresponding to the keyframe, and semantic edges are obtained based on the interaction relationship. The entities include the teaching hand, the active object, and the passive object. Using the entities as nodes and the semantic edges as connections, construct the semantic graph corresponding to each subtask fragment, and all the semantic graphs constitute the dynamic semantic graph sequence.

6. The human teaching assembly method according to claim 5, characterized in that, Determining the interaction relationship between different entities based on the structural features corresponding to the keyframe includes: If, based on the structural features, it is determined that the following conditions are met: the teaching hand is in a closed state, the teaching hand is in contact with the active object or the passive object, and the rate of change of relative pose is lower than a preset stability threshold, then the interaction relationship is determined to be a grasping relationship. If, based on the structural features, it is determined that the speed difference between the teaching hand and the active object is less than a preset speed threshold, then the interaction relationship is determined to be a co-motion relationship. If, based on the structural features, it is determined that the translation and rotation of the active object relative to the passive object are both within a preset precision range, then the interaction relationship is determined to be an alignment relationship. If, based on the structural features, it is determined that the interaction relationship satisfies the following condition: when the normal vectors of the active object and the passive object are aligned and their relative positions are close to the insertion start point, then the interaction relationship is determined to be in the insertion preparation state.

7. The human teaching assembly method according to claim 1, characterized in that, The executable assembly plan output by the visual language model is obtained, and the specific action parameters for driving the assembly robot to perform the assembly task are generated by combining the executable assembly plan and real-time visual perception information. A sequence of robot control instructions containing these specific action parameters is then generated, including: The multimodal prompts are input into a visual language model for intent reasoning to obtain the semantic description of the subtask corresponding to each semantic stage; Based on the semantic description of the subtask, the corresponding function template is retrieved from the preset action library, and the executable assembly plan of the assembly robot is calculated based on the function template. The executable assembly plan includes at least the target grasping pose, approach direction, assembly path and motion axis constraints. The robot acquires real-time visual perception signals and obtains the specific action parameters based on the real-time visual perception signals and the executable assembly plan, thereby generating a robot control instruction sequence containing the specific action parameters.

8. A human teaching assembly device, characterized in that, include: Feature construction module: used to acquire video streams of human teaching through visual sensors, obtain basic perceptual information from the video stream, and generate temporal structure feature sequences based on frame temporal sequence and the basic perceptual information; Task partitioning module: used to input the temporal structure feature sequence into a pre-trained task temporal segmentation module for bidirectional temporal modeling, and to segment the video stream into at least one sub-task segment; Semantic generation module: used to obtain a dynamic semantic graph sequence consisting of at least one semantic graph based on the video stream, the subtask segment and the temporal structure feature sequence, the semantic graph being used to indicate the semantic stage of assembly teaching, each semantic stage including at least one keyframe selected from the video stream; Semantic planning module: used to encapsulate the dynamic semantic graph sequence, the keyframes and the temporal structure feature sequence into multimodal cues, and use them as input to the visual language model for assembly plan reasoning; Parameter generation and execution module: used to obtain the executable assembly plan output by the visual language model, combine the executable assembly plan and real-time visual perception information to generate specific action parameters to drive the assembly robot to perform assembly tasks, and generate a robot control instruction sequence containing specific action parameters.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the human teaching assembly method according to any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the human teaching assembly method as described in any one of claims 1 to 7.

Images