Robotic automated assembly method, system, and storage medium

Abstract

This application discloses a robot automatic assembly method, system, and storage medium, relating to the fields of robot assembly and artificial intelligence. The robot automatic assembly method includes: performing reasoning analysis on an assembly drawing to determine the assembly reasoning result; acquiring a workbench parts placement scene image and performing image recognition on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled; decomposing the assembly task based on the assembly reasoning result to determine the current part to be assembled and the current assembly target; determining assembly control parameters based on the part category corresponding to the current part to be assembled, the pose information corresponding to the current part to be assembled, and the current assembly target; and controlling the robot to assemble the current part to be assembled based on the assembly control parameters.

Description

[0001] Related Explanation This invention claims priority to Chinese invention patent with patent application number 202512035499.0, application date December 31, 2025, entitled "An Intelligent Assembly System for a Dual-Arm Coordination Industrial Robot". Technical Field

[0002] This application relates to the fields of robot assembly, artificial intelligence and other technologies, and in particular to a robot automatic assembly method, system and storage medium. Background Technology

[0003] Industrial robots are being used more and more widely in the field of intelligent manufacturing. However, current industrial robot assembly technology still faces many technical bottlenecks and is difficult to meet the actual needs of flexible and intelligent production.

[0004] First, existing industrial robots generally lack self-assembly capabilities. In actual production scenarios, the deployment and debugging of robot systems usually require manual intervention, with motion trajectories pre-set through instructional programming or offline simulation, resulting in low efficiency.

[0005] Secondly, traditional end effectors lack adaptability. Most existing assembly systems use pneumatic grippers or simple suction cups as actuators. These rigid clamps have high requirements for the shape of the parts. When encountering irregularly shaped parts, flexible materials, or parts with easily damaged surfaces, traditional clamps struggle to achieve stable gripping and precise operation, easily causing parts to slip or suffer surface damage. Especially in scenarios requiring fine assembly, they struggle to complete complex processes requiring smooth control, such as shaft-hole assembly and thread tightening.

[0006] Furthermore, the current level of automation in assembly is low. Although some production lines have achieved semi-automated assembly, key processes still rely on skilled workers for manual intervention and adjustments. This human-machine hybrid operation mode not only increases labor costs but also makes it difficult to guarantee consistent assembly quality.

[0007] Therefore, how to improve the autonomous reasoning ability, environmental adaptability and precision operation level of industrial robots, and achieve fully autonomous intelligent assembly, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] The embodiments of this application aim to at least partially solve one of the technical problems in the related art. Therefore, the first objective of the embodiments of this application is to provide a robotic automated assembly method, system, and storage medium.

[0009] This application provides an automated robot assembly method, which includes: performing reasoning analysis on an assembly drawing to determine an assembly reasoning result; acquiring a workbench parts placement scene image and performing image recognition on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled; decomposing the assembly task based on the assembly reasoning result to determine the current parts to be assembled and the current assembly target; determining assembly control parameters based on the part category, pose information, and assembly target of the current parts to be assembled; and controlling the robot to assemble the current parts to be assembled based on the assembly control parameters.

[0010] In other embodiments, the assembly reasoning result includes the parts required for assembly, the connection relationships between parts, and the assembly sequence; based on the assembly reasoning result, the assembly task is decomposed to determine the current parts to be assembled and the assembly target information, including: based on the parts required for assembly, the connection relationships between parts, and the assembly sequence, multiple assembly processes are reasoned to determine the parts to be assembled corresponding to each assembly process and the assembly target corresponding to each assembly process; the current assembly process is determined from the multiple assembly processes, and the current parts to be assembled and the current assembly target corresponding to the current assembly process are determined.

[0011] In other embodiments, the robot includes a robotic arm and a dexterous hand, and assembly control parameters instruct the robotic arm and dexterous hand to assemble along a collision-free path; based on the assembly control parameters, controlling the robot to assemble the currently assembled part includes: based on the assembly control parameters, controlling the robotic arm and dexterous hand to cooperate with each other and grasp, move and assemble the currently assembled part along a collision-free path.

[0012] In other embodiments, the method further includes: during the movement of the currently assembled part, acquiring the real-time pose of the currently assembled part and acquiring a workbench operation space scene map; performing image recognition on the workbench operation space scene map, and determining, based on the real-time pose, whether the subsequent pose of the currently assembled part will collide or interfere with other objects in the workbench operation space scene map; if so, updating the assembly control parameters to update the subsequent movement planning path; if not, continuing to move the currently assembled part according to the assembly control parameters until the currently assembled part is manipulated to the target assembly pose.

[0013] In other embodiments, assembling the parts to be assembled includes: acquiring tactile sensing data of the dexterous hand, joint motion sensing data of the dexterous hand, motion sensing data of the robotic arm, and multi-dimensional force sensing data of the dexterous hand and the robotic arm; fusing the tactile sensing data of the dexterous hand, joint motion sensing data of the dexterous hand, motion sensing data of the robotic arm, and multi-dimensional force sensing data to obtain multimodal joint characterization data; and controlling the robotic arm and the dexterous hand to assemble the parts to be assembled based on the multimodal joint characterization data.

[0014] In other embodiments, image recognition is performed on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled, including: obtaining part description text, matching the workbench parts placement scene image with the part description text to determine the part category and part region in the workbench parts placement scene image; and recognizing pose information based on the part region.

[0015] In other embodiments, the method further includes: acquiring real-time motion data of the robot during the process of assembling the parts to be assembled by controlling the robot based on assembly control parameters; evaluating the dynamic coordination degree of the robot based on the real-time motion data; determining the assembly risk level based on the dynamic coordination degree; and activating a compensation control process when the assembly risk level is greater than a preset risk level.

[0016] In other implementations, evaluating the robot's dynamic coordination based on real-time motion data includes: inputting real-time motion data into a coordination evaluation model for evaluation to obtain dynamic coordination; wherein the coordination evaluation model is trained by: collecting the robot's raw motion data; performing feature enhancement on the raw motion data based on the robot's historical coordinated operation data to generate enhanced motion data samples; obtaining coordination labels corresponding to the enhanced motion data samples; training and evaluating the coordination evaluation model based on the enhanced motion data samples to obtain coordination training results; and adjusting the model parameters of the coordination evaluation model based on the loss value between the coordination training results and the coordination labels.

[0017] In other implementations, based on the robot's historical coordinated operation data, feature enhancement is performed on the original motion data to generate enhanced motion data samples, including: extracting associated coordinated operation data from the historical coordinated operation data based on the similarity between the historical coordinated operation data and the original motion data; extracting coordination pattern features from the associated coordinated operation data; and fusing the coordination pattern features with the original motion data to generate enhanced motion data samples.

[0018] In other embodiments, the coordination evaluation model is trained and evaluated based on the enhanced motion data samples to obtain coordination training results, including: extracting key motion features from the enhanced motion data samples, wherein the key motion features include at least one of trajectory smoothness features, velocity consistency features, and force interaction features; and inputting the key motion features into the coordination evaluation model for training and evaluation to obtain coordination training results.

[0019] This application provides an automated robot assembly system, comprising: an assembly reasoning module for reasoning and parsing an assembly drawing to determine the assembly reasoning result; an assembly part recognition and perception module for acquiring an image of a workbench part placement scene and performing image recognition on the workbench part placement scene image to determine the part category and pose information of the part to be assembled; and a motion planning module for decomposing the assembly task based on the assembly reasoning result, determining the current part to be assembled and the current assembly target; determining assembly control parameters based on the part category, pose information, and assembly target of the current part to be assembled; and controlling the robot to assemble the current part to be assembled based on the assembly control parameters.

[0020] This application provides a robot including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0021] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0022] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the above embodiments.

[0023] This application provides a computer program product that includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of an automated robot assembly method provided for an embodiment of this application.

[0025] Figure 2 This is a schematic diagram illustrating the relationship between coordination level and assembly risk level in an embodiment of this application.

[0026] Figure 3 A schematic diagram of the training samples provided for the implementation of this application.

[0027] Figure 4 A schematic diagram of a robot automated assembly system provided in an embodiment of this application.

[0028] Figure 5 A block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0030] Figure 1 This is a schematic diagram of an automated robot assembly method provided for an embodiment of this application.

[0031] like Figure 1 As shown, the robot automatic assembly method provided in this application includes steps S110-S150. This robot automatic assembly method can be used to realize robot self-assembly (the robot assembles itself) or robot assembly of other products. This method can be applied to a dual-arm collaborative industrial robot system, which includes an assembly reasoning module, an assembly part recognition and perception module, a motion planning module, and an actuator composed of a robotic arm, a dexterous hand, a 3D vision module, and a tactile perception module. The 3D vision module includes at least one device capable of visually recognizing object depth information and determining object pose; for example, it can employ an RGBD (Red-Green-Blue-Depth) camera, a binocular camera, a ToF (Time of Flight) depth sensor, etc. The tactile perception module includes tactile sensors placed at the fingertips and palm of each finger. The high-degree-of-freedom dexterous hand includes a 20-joint bionic structure integrating distributed tactile sensors. Taking robot self-assembly as an example, the robot currently performing the assembly task analyzes the CAD (Computer-Aided Design) drawings of the robotic arm to be assembled based on a large-scale language model. The vision system locates the parts and reconstructs their 3D pose. If the robotic arm to be assembled belongs to another robot, the robot currently performing the assembly task uses its left hand to fix the base of the other robot, while its right hand completes the precision assembly of the motor, reducer, and end effector in the robotic arm to be assembled. If the robotic arm to be assembled is the same robotic arm as the robot currently performing the assembly task's left hand, the robot currently performing the assembly task uses its right hand to complete the precision assembly of the motor, reducer, and end effector in the robotic arm to be assembled.

[0032] Step S110: Perform reasoning analysis on the assembly drawing to determine the assembly reasoning result.

[0033] In this embodiment, a two-dimensional or three-dimensional assembly drawing file containing part information and assembly details of the items to be assembled is input into the assembly reasoning module. This module integrates a large-scale language model and has been fine-tuned for the assembly task. The large-scale language model automatically analyzes the part composition in the drawing using visual understanding technology, infers the connection relationships and assembly sequence between parts, and generates an assembly reasoning result.

[0034] In the fields of natural language processing and computer vision, large-scale language models are typically pre-trained on massive amounts of general-purpose data (such as internet text and image-text pairs) to equip them with broad language understanding and visual recognition capabilities. However, when dealing with specialized mechanical assembly tasks, these general-purpose models lack specialized knowledge of assembly drawings, part terminology, and assembly processes. Therefore, they need to be fine-tuned to become "assembly experts."

[0035] In the model fine-tuning phase, the prepared assembly drawing samples are used to retrain the pre-trained large model; this process is called fine-tuning. A large-scale language model pre-trained on massive amounts of general-purpose data is loaded, and the assembly drawing samples are input into the model. The model generates assembly reasoning results (predicted parts, relationships, and sequences) based on its existing knowledge. The model's predicted assembly reasoning results are compared with the manually labeled real results, and the loss function value is calculated. The loss function measures the gap between the model's prediction and the real labels. The gradient is calculated using the backpropagation algorithm, and the model's weight parameters are updated slightly. This process is repeated, training with a batch of data each time, gradually adjusting the model parameters to improve its performance on assembly tasks.

[0036] The assembly drawing samples used for fine-tuning include: mechanical assembly drawings (including 2D drawing images, 3D CAD model screenshots, or engineering drawing PDFs (Portable Document Format) files), and may also include relevant parts lists and assembly instruction text; the expected output (labels) include: manually annotated assembly reasoning results, including which parts are included in the drawing, the connection relationships between parts (such as threaded connections, shaft-hole fits), and the reasonable assembly sequence. For example, for an assembly drawing sample of a reducer, parts such as "input shaft," "output shaft," "bearing," and "end cover" are labeled, as are the relationships such as "bearing is installed on the input shaft" and "end cover is bolted to the housing," and the order of "install the input shaft first, then the bearing, and finally the end cover" is indicated.

[0037] Step S120: Obtain a scene image of the workbench parts placement, and perform image recognition on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled.

[0038] A 3D vision module (such as an RGB-D camera) acquires real-time scene images of the workbench surface to obtain scene images of the workbench parts placement. These images are then input into the assembly parts recognition and perception module for identification, which identifies the category (such as the type and specification of bearings, bolts, etc.) and pose of the parts placed on the workbench, thus obtaining the part category and pose information of the parts to be assembled.

[0039] Step S130: Based on the assembly reasoning results, decompose the assembly task to determine the current parts to be assembled and the current assembly target.

[0040] The planning motion module adopts a dynamic task decomposition algorithm based on meta-learning. Based on the assembly reasoning results, the overall assembly task is broken down into multiple executable assembly processes. The current assembly process is identified from the multiple assembly processes, and the parts that need to be operated in the current assembly process and the target state that the parts need to achieve are determined.

[0041] Meta-learning allows algorithms or models to be trained not on a single task, but on thousands of different tasks. Its goal is not to learn a specific task, but to learn how to quickly adapt to a new task. The algorithm "learns" from thousands of different assembly cases, where it learns different part shapes, different connection relationships (threads, snaps, shaft holes), and different assembly sequences. After training, the algorithm no longer memorizes "part A is assembled on part B," but summarizes high-level task decomposition logic. For example: "For any part with a base, the base should be fixed first," "For shaft-type parts, they should be inserted horizontally," and "If the visual recognition indicates a confined space, the gripping posture needs to be adjusted."

[0042] Step S140: Determine assembly control parameters based on the part category corresponding to the current part to be assembled, the pose information corresponding to the current part to be assembled, and the current assembly target.

[0043] The current assembly target can include the target pose that the part needs to achieve. The motion planning module plans the grasping posture, motion path and assembly action parameters according to the spatial transformation relationship between the actual pose of the part and the target pose, and generates assembly control parameters that include control instructions for the robotic arm and dexterous hand.

[0044] Step S150: Based on the assembly control parameters, control the robot to assemble the parts to be assembled.

[0045] The robotic arm and dexterous hand perform grasping, moving, and assembly operations according to the planned path, and make real-time adjustments through visual servo and force feedback during the execution until the parts are accurately installed in place.

[0046] The embodiments of this application integrate visual perception, tactile perception, dual-arm collaboration, and multi-finger dexterity operation. By organically combining assembly drawing reasoning and analysis, scene image recognition, task decomposition, and parameter generation, a complete autonomous robot assembly process is constructed. Compared to traditional assembly methods that rely on instructional programming, the embodiments of this application enable the robot to have end-to-end autonomous operation capabilities from understanding drawings to executing actions, significantly reducing reliance on manual programming and debugging, improving production levels and efficiency, adapting to the handling of irregularly shaped parts, and requiring no manual intervention during the assembly process. By acquiring part pose information through visual recognition, the system can adapt to the actual working conditions of scattered parts, enhancing the system's environmental adaptability. In addition, the solution of the embodiments of this application enables the robot to achieve breakthrough self-assembly capabilities, realizing the autonomous assembly of robot body components, with a repeatability error ≤0.01mm. A single robot self-assembly can replace multiple skilled technicians.

[0047] In another example, step S120 involves performing image recognition on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled. This includes: obtaining part description text and matching the workbench parts placement scene image with the part description text to determine the part category and part region in the workbench parts placement scene image; and recognizing pose information based on the part region.

[0048] When determining the part category and pose information of the parts to be assembled, the part description text is first obtained. This part description text can be a pre-input natural language description (such as "M6×20 hexagon socket screw" or "motor with an output shaft diameter of 8mm"), or it can be the part name or specification parameters parsed from the assembly drawing. Then, the workbench part placement scene image and the part description text are input into the CLIP (Contrastive Language-Image Pre-training) encoder for cross-modal matching processing. Through contrastive learning, the CLIP encoder maps image features and text features to the same semantic space, calculates the similarity between image regions and text descriptions, thereby determining the part category in the workbench part placement scene image that matches the description text, and generates the corresponding mask region for that type of part, marking the specific position and contour of the part in the image.

[0049] In addition to the CLIP encoder, a Graph Attention Network (GAT) can be used to determine the part categories and regions in a workbench parts placement scene image. The CLIP encoder is responsible for "image-text matching," while the GAT network is responsible for "relationship reasoning." The CLIP encoder initially matches the probabilities of part categories and part regions. However, when looking at a single candidate region in the workbench parts placement scene image, the CLIP encoder may not be able to distinguish whether "this metal cylinder" is a "motor" or a "bearing housing," or it may be even more difficult to judge when the part is occluded. The GAT network believes that objects in the image are not isolated, so it establishes connections (i.e., edges) between nodes (part regions) based on the spatial positional relationships (such as distance, whether they are adjacent, or whether they overlap) or visual similarity. Nodes are candidate parts to be identified, and edges are the spatial or visual relationships between nodes. The contextual information contained in the graph connections is used to correct the recognition result of each node. The GAT network uses an attention mechanism to make each part region pay attention to the part regions around it. The part categories and part regions corrected based on contextual relationships are more accurate.

[0050] Finally, based on the identified part region, a pose estimation algorithm is used to calculate the six-dimensional pose information of the part in the current worktable coordinate system (including three-dimensional position coordinates and three-dimensional rotation angle). For example, after inputting the part description text "base part", the CLIP encoder (which can also be combined with a GAT network) accurately locates the part region where the base is located in the worktable scene image. The pose estimation algorithm further calculates the actual placement position and orientation angle of the base for subsequent grasping.

[0051] Pose estimation algorithms include YOLOX-6D-Pose (You Only Look Once X - 6 Degrees of Freedom Pose, YOLOX-6 Degrees of Freedom Pose) and FoundationPose (basic pose model). YOLOX-6D-Pose is a single-stage, end-to-end 6D pose estimation algorithm based on the YOLOX object detection framework. FoundationPose can handle pose estimation based on CAD models and model-less pose tracking tasks without fine-tuning for new objects.

[0052] This implementation introduces a CLIP encoder for cross-modal matching of images and text, achieving semantic-based part recognition. This significantly improves the accuracy and flexibility of recognition, enabling accurate identification even for new part models, requiring only a text description. This drastically reduces model training costs and deployment cycles. Furthermore, combining recognition with text descriptions effectively avoids confusion between parts with similar appearances, improving the reliability of part category judgment. This semantic-based recognition method provides high-quality input for subsequent pose estimation and grasping planning, making it a key element in enhancing the overall intelligence level of the assembly system.

[0053] In another example, the assembly reasoning result includes the parts required for assembly, the connection relationships between parts, and the assembly sequence; in step S130, the assembly task is decomposed based on the assembly reasoning result to determine the current parts to be assembled and the assembly target information, including: based on the parts required for assembly, the connection relationships between parts, and the assembly sequence, multiple assembly processes are reasoned to determine the parts to be assembled corresponding to each assembly process and the assembly target corresponding to each assembly process; the current assembly process is determined from the multiple assembly processes, and the current parts to be assembled and the current assembly target corresponding to the current assembly process are determined.

[0054] For example, the assembly reasoning results specifically include a list of parts required for assembly, the connection relationships between parts, and the assembly sequence. When decomposing the assembly task based on this information, the planning motion module first determines multiple assembly processes, the parts to be assembled for each process, and the assembly target for each process. For instance, in a robot self-assembly scenario, parts such as the base, motor, reducer, and end effector are analyzed from CAD drawings, and the relationships between the base and motor, the motor and reducer, and the reducer and end effector are inferred, thus determining the first process as "base-motor assembly," the second process as "motor-reducer assembly," and the third process as "reducer-end effector assembly." Then, the system dynamically matches and determines the currently executable process from multiple assembly processes based on the actual placement of parts on the workbench, and clarifies the current parts to be assembled and the current assembly target for that process. For example, when it is detected that both the base and motor are placed on the workbench and are reachable, the system determines "base-motor assembly" as the current process, the current part to be assembled as "motor," and the current assembly target as "installing the motor at the specified position on the base."

[0055] This implementation refines the assembly reasoning results into a parts list, connection relationships, and assembly sequence. Based on this, it performs multi-process decomposition and dynamic process selection, achieving hierarchical breakdown and adaptive scheduling of complex assembly tasks. The system no longer simply executes a preset fixed sequence but can flexibly select the optimal currently executable process based on the actual perceived parts distribution, improving the fault tolerance and parallel operation capabilities of the assembly process. This dynamic task decomposition mechanism enables the robot to cope with unexpected situations on-site, such as changes in parts supply order and localized material shortages, enhancing the system's robustness and intelligence.

[0056] In another example, the robot includes a robotic arm and a dexterous hand, and the assembly control parameters instruct the robotic arm and dexterous hand to assemble along a collision-free path; in step S150, based on the assembly control parameters, the robot is controlled to assemble the current part to be assembled, including: based on the assembly control parameters, the robotic arm and dexterous hand are controlled to cooperate with each other and grasp, move and assemble the current part to be assembled along a collision-free path.

[0057] For example, the robot includes a robotic arm and a dexterous hand (multi-fingered dexterous hand). Assembly control parameters specifically instruct the robotic arm and dexterous hand to assemble along a collision-free path. During assembly, the robotic arm and dexterous hand cooperate to perform grasping, moving, and assembling operations on the part to be assembled along the collision-free path. Specifically, the assembly control parameters include path planning information, which is generated by a collision detection algorithm to ensure that the movement trajectory of the robotic arm, the grasping posture of the dexterous hand, and the envelope space during part movement do not interfere with other parts, fixtures, or environmental obstacles on the worktable. For example, in a dual-arm collaboration scenario, the left arm carries the dexterous hand to grasp the base and keep it fixed, while the right arm carries the dexterous hand to grasp the motor. The movement paths of the two arms are collaboratively planned to ensure a safe distance between the two arms during movement, while avoiding collisions with scattered parts on the worktable. During grasping, the dexterous hand adjusts the finger bending angle and grasping force according to the geometric features of the part; during movement, the robotic arm moves smoothly according to the planned path; during assembly, the two arms cooperate to complete alignment and insertion actions.

[0058] This implementation significantly expands the robot's operational capabilities and working range by combining a robotic arm with a multi-fingered dexterous hand and enabling them to work collaboratively along a collision-free path. Compared to traditional pneumatic grippers, the dexterous hand can adapt to the grasping needs of irregularly shaped parts, flexible materials, and fragile components, solving the problem of insufficient adaptability of traditional fixtures. Collision-free path planning ensures the safety of the operation process, avoiding equipment damage or workpiece scrapping caused by collisions. The collaborative cooperation between the robotic arm and the dexterous hand enables the robot to simulate the two-arm operation of a human technician, completing complex assembly actions and providing an execution foundation for achieving high-precision, highly flexible automated assembly.

[0059] In another example, the method further includes: during the movement of the current part to be assembled, obtaining the real-time pose of the current part to be assembled and obtaining the workbench operation space scene map; performing image recognition on the workbench operation space scene map, and determining whether the subsequent pose of the current part to be assembled has collision interference with other objects in the workbench operation space scene map based on the real-time pose; if so, updating the assembly control parameters to update the subsequent movement planning path; if not, continuing to move the current part to be assembled according to the assembly control parameters until the current part to be assembled is manipulated to the target assembly pose.

[0060] The embodiments of this application add a real-time collision detection and path update mechanism during the movement process. While moving the currently assembled part according to the assembly control parameters, the system continuously acquires the real-time pose of the part, and simultaneously, the 3D vision module acquires scene images of the workbench operation space in real time, generating a workbench operation space scene map containing the position information of other parts, fixtures, and other objects. Then, the system performs image recognition on the workbench operation space scene map and, combined with the real-time pose of the currently assembled part, predicts whether its subsequent movement pose will collide with other objects in the workbench operation space scene map. The specific prediction method can employ a collision detection algorithm or a motion prediction model based on deep learning. If the prediction indicates a risk of interference, the system immediately triggers path replanning, updates the subsequent movement planning path in the assembly control parameters, and generates a new collision-free path; if the prediction indicates no interference, the system continues to move the currently assembled part according to the original assembly control parameters until the part is manipulated to the target assembly pose. For example, when the robotic arm moves with the motor, the vision system detects a tool that was not originally present on the workbench being temporarily placed in front of the movement path. The system determines in real time that the original path will collide with the tool, and then dynamically adjusts the path, bypassing the tool before continuing assembly.

[0061] This implementation introduces real-time pose monitoring and collision prediction mechanisms during movement, enabling the robot to possess dynamic environmental perception and real-time obstacle avoidance capabilities. Traditional robots, after pre-planning their paths, cannot cope with environmental changes; once the workbench layout changes or human interference occurs, collisions are highly likely to occur. This method effectively solves this problem through closed-loop feedback and dynamic path updates, significantly improving the robot's operational safety and reliability in unstructured, dynamically changing environments. Simultaneously, the real-time adjustment mechanism reduces reliance on high-precision loading positions and fixed tooling, lowering system deployment costs.

[0062] In another example, assembling the current part to be assembled includes: acquiring tactile sensing data of the dexterous hand, joint motion sensing data of the dexterous hand, motion sensing data of the robotic arm, and multi-dimensional force sensing data of the dexterous hand and the robotic arm; fusing the tactile sensing data of the dexterous hand, joint motion sensing data of the dexterous hand, motion sensing data of the robotic arm, and multi-dimensional force sensing data to obtain multimodal joint characterization data; and controlling the robotic arm and dexterous hand to assemble the current part to be assembled based on the multimodal joint characterization data.

[0063] When assembling the parts to be assembled, the system collects multi-source sensor data, including: tactile sensor data (such as contact force distribution and sliding sensation) collected by the distributed tactile sensors on the dexterous hand; joint motion sensor data (such as finger angle and movement speed) collected by the joint encoder of the dexterous hand; motion sensor data (such as joint angle and angular velocity) collected by the joint encoder of the robotic arm; and force sensor data (such as six-dimensional force and torque) collected by the multi-dimensional force sensor installed between the dexterous hand and the end effector of the robotic arm.

[0064] Then, these multi-source data are fused: tactile data is encoded into tactile feature vectors by a tactile encoder. For example, the visual-tactile image or tactile dot matrix data of a multi-fingered dexterous hand (which can be regarded as a special kind of "tactile image") is input into a Vision Transformer (ViT) encoder for encoding; joint data and robotic arm joint data are encoded into kinematic feature vectors by a Multilayer Perceptron (MLP); multidimensional force data is encoded into force feature vectors by a force encoder (such as a Multilayer Perceptron MLP or a force-sensing hybrid expert module). These feature vectors from different modalities are fused to form unified multimodal joint representation data. Feature fusion methods include splicing fusion, weighted fusion, attention-based fusion, graph neural network fusion, multimodal Transformer fusion, etc. These fusion methods can be selected or combined according to modal characteristics, computational resources, and task complexity. The concatenation fusion method connects the feature vectors of each modality end-to-end to form a joint representation, which is easy to implement; the weighted fusion method introduces learnable weights or attention mechanisms to dynamically adjust the importance of different modalities, improving the flexibility and robustness of fusion; the attention-based fusion method utilizes multi-head attention or cross-attention to deeply explore the nonlinear correlations between modalities, which is suitable for fine-grained operation scenarios; the gating fusion method uses gating mechanisms to adaptively control the amount of information flowing into each modality, effectively suppressing noise interference; the graph neural network fusion method treats the feature vectors of each modality as nodes in a graph and the relationships between modalities as edges, and uses graph neural networks for information propagation and fusion, which is suitable for modeling structured relationships, such as the motion coordination relationship between a robotic arm and a dexterous hand, and is applicable to scenarios where there are clear physical or structural relationships between multiple modalities; the multimodal Transformer fusion method projects the features of each modality onto a unified space and then performs global interaction through self-attention, with a unified architecture that is easy to extend, and is suitable for use in combination with architectures such as the Vision-Language-Action Model (VLA).

[0065] Finally, the multimodal joint representation data is input into the Vision-Language-Action Model (VLA). The model outputs precise assembly action commands, controlling the robotic arm and dexterous hand to adjust parameters such as gripping force, assembly posture, and insertion speed in real time to complete precision assembly. For example, during shaft-hole assembly, when the tactile sensor detects an abnormally increased insertion resistance, the system determines through multimodal fusion that the shaft-hole is stuck. It then controls the robotic arm to fine-tune the part's posture and reduce the insertion force until it is successfully inserted.

[0066] This implementation method achieves refined control of the precision assembly process by integrating multimodal sensor data such as tactile, joint motion, and multidimensional force, and making real-time control decisions based on a VLA model. Compared to traditional methods that rely solely on vision or position control, multimodal fusion enables the robot to perceive microscopic information such as contact states and force changes, thereby simulating the tactile feedback and force perception capabilities of human technicians. This allows the robot to complete complex processes requiring compliant control and real-time adjustment, such as precision shaft and hole assembly, thread tightening, and flexible component insertion, effectively solving the shortcomings of traditional fixtures in adapting to irregularly shaped parts and requiring manual intervention in the assembly process. The introduction of multimodal joint representation provides the robot with richer environmental interaction information, significantly improving assembly success rate and product quality consistency.

[0067] In another example, the method further includes: acquiring real-time motion data of the robot during the process of assembling the current parts to be assembled by controlling the robot based on assembly control parameters; evaluating the dynamic coordination of the robot based on the real-time motion data; determining the assembly risk level based on the dynamic coordination; and activating a compensation control process when the assembly risk level is greater than a preset risk level.

[0068] For example, this embodiment adds a dynamic coordination assessment and risk control mechanism during the assembly process. During the assembly of the parts to be assembled by the robot based on assembly control parameters, real-time motion data of the robot is acquired. By continuously monitoring the robot's coordination level, dynamic assessment and proactive intervention of assembly risks are achieved.

[0069] Specifically, during the robot's grasping, moving, and assembly operations, real-time motion data is collected by multi-source sensors (including inertial measurement units and force sensors) deployed at the joints of the robot's two arms. This real-time motion data can include joint angle data, angular velocity data, and interaction force data between the two arms, reflecting the robot's actual motion state at the current moment. The real-time motion data is analyzed and processed to derive the dynamic coordination degree. This dynamic coordination degree is a quantitative value used to assess the coordination level of the robot's two arms at the current moment; a higher value indicates better coordination between the arms, while a lower value indicates asynchronous movement or abnormal mechanical coordination.

[0070] Industrial robot assembly risk classification standards can be pre-set. The risk level of industrial robot assembly can be determined based on a comprehensive analysis of potential failure modes, safety requirements, and historical accident data in the assembly process. Risks are divided into low, medium, and high risk levels, and a correspondence between dynamic coordination degree and these risk levels is established. The current assembly risk level is determined by querying this correspondence based on the currently calculated dynamic coordination degree. For example, a dynamic coordination degree higher than 0.8 is considered low risk, between 0.5 and 0.8 is considered medium risk, and lower than 0.5 is considered high risk. When the assembly risk level exceeds the preset risk level (e.g., changing from low to medium risk, or from medium to high risk), the system determines that there is a safety hazard in the current assembly process and immediately activates the compensation control process.

[0071] The compensation control process includes measures such as generating joint torque adjustment commands, adjusting the motion parameters of both arms, and correcting motion trajectories to rectify coordination anomalies and prevent risk escalation. During compensation control execution, the adjusted motion data is continuously monitored, dynamic coordination is reassessed, and compensation parameters are iteratively updated until the assembly risk level returns to a safe range. In one example, the joint torque adjustment command is calculated based on the assembly risk level, the robot's kinematic model, and possible collision avoidance strategies. In specific implementations, the calculation of joint torque adjustment commands can employ an impedance-based control strategy, simulating a virtual spring-damping system to adjust the desired torque of each joint. This allows the robot to exhibit compliance when subjected to unexpected forces or motion incoordination, thereby avoiding rigid collisions. The joint torque adjustment command includes the identifier of the joint to be adjusted and the specific torque adjustment amount. The compensation control process can also include reducing the motion speed, stopping the movement of one of the robotic arms, or moving both arms along a predefined retreat trajectory. The specific compensation control process depends on the severity of the hazardous situation and the task type. In addition, real-time monitoring of the execution effect of joint torque adjustment commands is an important part of the compensation control process. After the joint torque adjustment command is issued, the control execution unit needs to continuously collect the motion data of the robot after adjustment to evaluate the compensation effect. Real-time monitoring is achieved by reading the new position data fed back by the joint encoder and the latest force data fed back by the force sensor. This is used to analyze whether the compensation action has effectively improved the coordination of the two arms and whether the real-time assembly risk level has begun to decrease.

[0072] The dynamic coordination degree is recalculated based on the adjusted motion data to iteratively update the compensation control parameters until the assembly risk level is less than or equal to the preset risk level. This is a typical closed-loop feedback control process. In specific implementation, a new dynamic coordination degree is recalculated based on the adjusted motion data. The assembly risk level is then recalculated based on the new dynamic coordination degree. If the new assembly risk level is still greater than the preset risk level, it indicates that the initial compensation control parameters are insufficient, and iterative updates to the compensation control parameters are needed. For example, the gain coefficient of joint torque adjustment can be increased, the virtual impedance parameter can be changed, or different compensation strategies can be switched. In some embodiments, parameter updates can adopt the idea of ​​gradient descent, fine-tuning the control parameters according to the changing trend of the assembly risk level, or switching between several preset compensation modes can be tried, continuously performing the cycle of "performing compensation - monitoring the effect - re-evaluating - updating parameters" until the latest calculated assembly risk level is less than or equal to the preset risk level. The system state returns to a safe state, at which point the compensation control process ends, and the industrial robot can gradually resume normal operation. Optionally, a maximum number of iterations or a maximum execution time can be set throughout the compensation control process to prevent the system from falling into an infinite loop under extreme conditions. If the risk level cannot be restored to a safe state under the limited conditions, a higher level of safety policy will be executed, such as triggering an emergency system stop.

[0073] Determining assembly risk levels based on dynamic coordination degree can be achieved by establishing a risk probability mapping table, which represents the distribution probability of the assembly risk level corresponding to the dynamic coordination degree. Establishing this risk probability mapping table relies on a large amount of sample data collected from historical operational data or specialized tests. This sample data includes the historical coordination degree output by the model and the risk level label actually observed or determined by experts at that moment. In specific implementation, for each preset risk level, the frequency of historical coordination degree falling within each risk probability value interval is statistically analyzed. The range of historical coordination degree values ​​is discretized into several small intervals, for example, dividing the range [0,1] into 100 equal intervals. Then, the number of samples belonging to the low-risk level whose historical coordination degree falls within each small interval is counted. Similarly, the sample distribution of the medium-risk and high-risk levels is statistically analyzed. In some embodiments, the calculation of the distribution probability can employ parametric or non-parametric methods. Non-parametric methods are directly based on histogram statistics, while parametric methods assume that the coordination degree estimate follows a certain probability distribution at each risk level, such as a Gaussian distribution, and estimate the distribution parameters using sample data. Based on the frequency data obtained from statistics, conditional probabilities can be calculated, which is the probability that historical coordination degree appears in a certain interval under a given risk level, or joint probabilities can be calculated. Finally, this probability information is organized into a risk probability mapping table, with historical coordination degree intervals as rows and risk levels as columns, and each cell in the table filled with the corresponding probability value. Optionally, smoothing techniques, such as Laplace smoothing, can be introduced into the construction of the risk probability mapping table to avoid the zero probability problem and improve the robustness of the mapping.

[0074] Based on a risk probability mapping table, a fuzzy logical relationship is established between historical coordination degree and assembled risk level. This fuzzy logical relationship can handle conceptual ambiguity and uncertainty in judgment, more closely resembling human decision-making. In implementation, establishing this fuzzy logical relationship first requires defining fuzzy sets and their membership functions for the input variable (historical coordination degree) and the output variable (assembled risk level). For example, three fuzzy sets are defined for historical coordination degree: "low coordination degree," "medium coordination degree," and "high coordination degree," with a membership function assigned to each set, such as a triangular or trapezoidal membership function. Similarly, fuzzy sets need to be defined for assembled risk level, such as "low risk," "medium risk," and "high risk," with their membership functions specified. The membership functions for risk levels can be discrete or continuous. The core of the fuzzy logical relationship is a set of fuzzy rules designed based on the statistical regularities revealed in the risk probability mapping table. For example, if the historical coordination degree is high, the risk level is low risk; if it is medium, the risk level is medium risk; and if it is low, the risk level is high risk. In some embodiments, the fuzzy rules can be more refined, considering combinations of multiple fuzzy sets, and each rule is accompanied by a confidence weight, which can be set according to the probability values ​​in the risk probability mapping table. When a new historical consistency degree is input, the fuzzy inference process is activated, calculating the membership degree of the historical consistency degree with respect to each input fuzzy set, and then inferring according to the fuzzy rules to obtain the membership degree of the output variable (risk level) with respect to each output fuzzy set. Finally, through defuzzification methods, such as the centroid method or the maximum membership method, the fuzzy output is transformed into a clear risk level judgment or risk probability distribution, thereby forming an operable correspondence.

[0075] In practice, the process of calculating the probability distribution and generating a risk probability mapping table can be quantitatively described using mathematical formulas. A formula for calculating the probability of a specific risk level occurring within a given historical coordination interval can be expressed as:

[0076] Where: symbol This represents the conditional probability that the risk level is at the i-th level when the historical coordination degree falls within the j-th subinterval; (symbol) This indicates that the risk level in historical data is... And the estimated degree of coordination falls within the interval The number of samples; symbol This indicates that the estimated degree of coordination falls within the interval. The total number of samples is used to quantify the correlation strength between historical consistency and risk level by calculating conditional probabilities. In some embodiments, to avoid zero probabilities in sparse data intervals, a small constant term can be added to both the numerator and denominator for smoothing. The calculated conditional probabilities... The core of the risk probability mapping table consists of all risk levels *i* and all coordination intervals *j*. This table can be further used to calculate posterior probabilities, i.e., inferring the probability of belonging to various assembly risk levels when observing a given historical coordination degree. This can be achieved using Bayes' theorem, but requires prior knowledge of the risk level's probability. The final established correspondence, whether based on explicit probability mapping or fuzzy logic rules, enables the system to map continuously changing dynamic coordination degrees to discrete risk classification criteria, providing a basis for subsequent risk assessment decisions.

[0077] Figure 2 This is a schematic diagram illustrating the relationship between coordination level and assembly risk level in an embodiment of this application.

[0078] See Figure 2 Historical coordination degree includes coordination degree estimates. Figure 2 The horizontal axis represents the coordination degree estimate, and the vertical axis represents the risk probability. The three curves represent the distribution patterns of low-risk, medium-risk, and high-risk probabilities, respectively. By statistically analyzing the distribution probability of the coordination degree estimate under different risk levels in historical data, this risk probability mapping curve is generated. This provides a quantitative basis for establishing fuzzy logic relationships, enabling the accurate mapping of continuous coordination degree estimates to low, medium, and high discrete risk levels. This provides core logical support for dynamic risk assessment and compensation decisions, ultimately achieving accurate risk classification and timely early warning in assembly, and improving the safety and reliability of the system in high-precision assembly scenarios.

[0079] In addition, the assembly risk level can be corrected. This correction can be made by combining the industrial robot's current task mode and environmental parameters to fully consider the impact of the task context. In practice, the industrial robot's current task mode refers to the specific assembly process the robot is performing, such as precision insertion, screw tightening, or component handling. Different task modes have different sensitivities and risk tolerances to the coordination of the two arms. Environmental parameters can include the lighting conditions of the workspace, ambient temperature, and the physical characteristics of the assembled components, such as weight or material. The correction process is achieved by introducing correction factors to the assembly risk level. These factors are set based on the risk weights of the task mode and environmental parameters. The risk weights define the adjustment coefficients for the degree of risk under different task contexts. Correcting the assembly risk level involves performing mathematical operations, such as multiplication or weighted summation, on the assembly risk level and the corresponding correction factor to obtain a comprehensive value, i.e., the corrected assembly risk level. Refer to Table 1, which shows a task mode risk weight lookup table used in the correction process.

[0080] Table 1: Risk Weights of Task Modes

[0081] Similarly, different environmental parameters can be assigned different risk weights. The correction factor can be the product or weighted sum of the risk weights corresponding to the current task mode and the environmental parameters.

[0082] This implementation method achieves closed-loop risk monitoring of the robot assembly process by introducing real-time motion data acquisition, dynamic coordination evaluation, and risk level determination mechanisms. Compared to the traditional method that relies solely on preset programs for assembly, this method enables the system to perceive its own motion state, assess coordination levels, and proactively identify risks. When coordination anomalies or increased risks are detected, the system can promptly activate compensation control procedures to intervene, proactively controlling risks and effectively avoiding collisions, workpiece damage, or equipment failures caused by asynchronous arm movements or excessive torque, significantly improving the safety and reliability of the assembly process. This dynamic response mechanism changes the traditional passive response mode of the system, providing safety assurance for high-precision, high-risk assembly operations.

[0083] In another example, evaluating the robot's dynamic coordination based on real-time motion data involves inputting the real-time motion data into a coordination evaluation model to obtain the dynamic coordination.

[0084] Real-time motion data can come from various sources, such as joint angle data fed back from the encoders of each robot joint. This joint angle data records the rotation angle values ​​of each joint at consecutive time points in sequence. It can also come from force sensor data provided by force sensors installed at the end effector of the robotic arm or the joint drive unit. This force sensor data records the vector information of the interaction forces between the robotic arm and the environment or workpiece. It can be understood that the received real-time motion data can undergo preliminary verification and caching to ensure data integrity and timing accuracy, preparing it for subsequent data processing steps.

[0085] Differentiating the joint angle data yields the angular velocity sequence, while integrating the force sensor data yields the force impulse sequence. This is a crucial step in feature extraction. Joint angle data represents positional information, and differentiation allows us to obtain the velocity characteristics of the motion. In practice, numerical differentiation can be used to calculate the difference between the current and previous joint angle values, which is then divided by the sampling time interval to obtain an approximate instantaneous angular velocity value. Calculating the angular velocities of all joints and arranging them in chronological order forms the angular velocity sequence. Optionally, if high-frequency noise is introduced during differentiation, a low-pass digital filter can be added after the differentiation operation for smoothing. Force sensor data reflects instantaneous force values; integrating this data yields the force impulse sequence, which characterizes the cumulative effect of force over a period of time. In practical implementation, the force sensor data can be integrated using a numerical integration method. The force sensor data values ​​at consecutive time points are accumulated and summed, and then multiplied by the sampling time interval to calculate the force impulse value at each time point, forming a force impulse sequence. The force impulse sequence helps to identify continuous force anomalies or the changing trend of interactive forces.

[0086] The angular velocity sequence and force impulse sequence are input into the coordination evaluation model to calculate the instantaneous coordination value. In specific implementations, the angular velocity sequence and force impulse sequence can be processed by feature organization before being input into the coordination evaluation model. Values ​​representing velocity consistency characteristics, such as the norm of the angular velocity difference between corresponding joints of the left and right arms, are extracted from the angular velocity sequence. Values ​​representing force interaction characteristics, such as the amplitude or rate of change of the resultant impulse at the ends of both arms, are extracted from the force impulse sequence. These feature values ​​are combined with other features that may be calculated simultaneously to form a feature vector. Optionally, this newly constructed feature vector is directly input into the trained coordination evaluation model. The coordination evaluation model calculates the input feature vector based on the mapping relationship learned internally and outputs a scalar value, which is the instantaneous coordination value. The instantaneous coordination value reflects the coordination level of the industrial robot's dual-arm movement within a very short time window.

[0087] Instantaneous coordination degree values ​​can exhibit significant fluctuations due to sensor noise and minute variations in motion. Time-series smoothing can be applied to these values, for example, using an exponentially weighted moving average to generate dynamic coordination degree. In practice, the exponentially weighted moving average is applied to the time series of instantaneous coordination degree values. This method assigns a weight to each data point in the sequence, with the weight decreasing exponentially over time. The most recent instantaneous coordination degree value receives the highest weight, while the weights of past values ​​gradually decrease. The calculation of the exponentially weighted moving average requires defining a decay coefficient, which determines the rate of weight decay and thus controls the degree of smoothing. Through recursive calculation, a smoothed new sequence that better reflects the overall trend of coordination degree changes can be obtained; this new sequence is the dynamic coordination degree. Dynamic coordination degree is a continuously changing and relatively stable signal, making it more suitable for risk assessment and decision-making.

[0088] In another example, the coordination evaluation model is trained as follows: raw motion data of the robot is collected; based on the robot's historical coordinated operation data, feature enhancement is performed on the raw motion data to generate enhanced motion data samples; coordination labels corresponding to the enhanced motion data samples are obtained; the coordination evaluation model is trained and evaluated based on the enhanced motion data samples to obtain the coordination training results; and the model parameters of the coordination evaluation model are adjusted based on the loss value between the coordination training results and the coordination labels.

[0089] During the model training phase, the robot's raw motion data is first collected. For example, by using inertial measurement units and force sensors installed at the joints of the industrial robot's two arms, the angular displacement sequences of the left arm, the angular displacement sequences of the right arm, and the interaction force sequences of the two arms are collected. After time-stamp alignment, noise filtering, and sliding window segmentation, structured raw motion data is formed.

[0090] For example, the raw motion data includes the raw motion data stream. Motion information is collected by inertial measurement units (IMUs) installed at various joints of the left and right arms of the industrial robot. The IMUs continuously record the angular displacement sequences of the left and right arms, while force sensors located at the end effector or joints of the robotic arms collect the interaction force sequence reflecting the mechanical coordination of the two arms. In practice, the collected left arm angular displacement sequence, right arm angular displacement sequence, and interaction force sequence are independent data streams, each with its own timestamp but potentially with slight clock deviations. Therefore, strict timestamp alignment is required to rearrange the data points of the three sequences according to a unified time base, ensuring that each moment contains the corresponding left arm angular displacement, right arm angular displacement, and interaction force data, thus forming a synchronous motion data block that is completely synchronized in time. It is understandable that although the synchronous motion data blocks are time-aligned, they may contain noise signals introduced by sensor noise and electromagnetic interference in the field. Therefore, they need to be noise filtered. Digital filters are used to filter out high-frequency noise components. Then, the sliding window method is applied to divide the filtered continuous data stream. The sliding window method slides along the time axis with a fixed-length time window. Each slide covers a segment of data and outputs it as a data segment. These continuous data segments eventually constitute the original motion data stream required for subsequent processing by the system.

[0091] Then, based on the robot's historical coordinated operation data, feature enhancement is performed on the original motion data to generate enhanced motion data samples. Simultaneously, coordination degree labels corresponding to the enhanced motion data samples are obtained. These labels are derived from domain experts' analysis of historical assembly operation videos or data, and are evaluated according to predefined coordination degree level standards. Each training sample contains an enhanced motion data sample and its corresponding coordination degree label.

[0092] During training, augmented motion data samples are input into an initialized coordination evaluation model (e.g., a support vector machine regression model). The model makes predictions based on the current parameters, yielding the coordination training result. Then, the loss value between the training result and the true coordination label, i.e., the prediction error, is calculated. Based on this loss value, the model parameters of the coordination evaluation model are adjusted through backpropagation and optimization algorithms, such as optimizing the kernel function parameters and penalty factor. The training process undergoes multiple iterations until the model's performance on the validation set stabilizes, resulting in the trained coordination evaluation model. This model possesses the ability to estimate dynamic coordination based on input real-time motion data and can be used for subsequent real-time dynamic coordination evaluation.

[0093] This implementation enhances the original motion data by incorporating historical coordination operation data, enabling the trained coordination evaluation model to fully absorb the experience and knowledge accumulated from past operations. Compared to models trained solely on real-time data, the enhanced motion data samples contain richer dual-arm coordination motion patterns, giving the model stronger generalization ability when facing different assembly scenarios and workpiece types. Through supervised learning, the model learns the nonlinear mapping relationship between motion features and coordination labels, resulting in highly accurate and reliable coordination estimates, providing a reliable quantitative basis for subsequent risk assessment. The parameter optimization mechanism during model training ensures continuous improvement in model performance, enabling the system to adapt to changes in assembly tasks.

[0094] In another example, based on the robot's historical coordinated operation data, feature enhancement is performed on the original motion data to generate enhanced motion data samples. This includes: extracting associated coordinated operation data from the historical coordinated operation data based on the similarity between the historical coordinated operation data and the original motion data; extracting coordination pattern features from the associated coordinated operation data; and fusing the coordination pattern features with the original motion data to generate enhanced motion data samples.

[0095] When performing feature enhancement on the original motion data based on the robot's historical coordinated operation data, the first step is to calculate the similarity between the historical coordinated operation data and the original motion data, and then extract related coordinated operation data from the historical coordinated operation data based on the similarity. Specifically, the original motion data and the historical coordinated operation data are normalized to eliminate the influence of dimensions. Normalization typically employs min-max scaling or Z-score standardization methods to transform features with different physical dimensions and numerical ranges to the [0,1] interval or a distribution with a mean of 0 and a variance of 1, thus eliminating the influence of dimensions and making different features comparable. After normalization, the dynamic time warping algorithm is used to calculate the morphological similarity between the original motion data and each historical coordinated operation data. The original motion data and the historical coordinated operation data are time series data. The dynamic time warping algorithm can effectively handle the scaling and bending of time series on the time axis. By finding the optimal bending path, the minimum cumulative distance between the two sequences is calculated, and the reciprocal of this distance constitutes the similarity. The historical coordinated operation data includes multiple sets. The dynamic time warping algorithm first constructs a distance matrix between the time series corresponding to the original motion data and the time series corresponding to each historical coordinated operation data set. Each element in the distance matrix represents the Euclidean distance or other distance metric between the two sequences at a specific time point. Then, the algorithm uses dynamic programming to find a path that minimizes the cumulative distance. The minimum cumulative distance reflects the overall morphological difference between the two sequences. Morphological similarity is usually defined as the reciprocal of the minimum cumulative distance or mapped to a similarity value through an exponential function. Each historical coordinated operation data set corresponds to a similarity value. Based on the calculated similarity values, one or more historical coordinated operation data sets with high similarity to the original motion data are selected as associated coordinated operation data sets.

[0096] Next, coordination pattern features are extracted from the associated coordination operation data. These coordination pattern features reflect the regularity of the coordinated movement of the two arms in historical successful assembly operations, including but not limited to trajectory matching patterns, speed synchronization patterns, and force interaction patterns. For example, abstract coordination pattern features such as the relative motion relationship between the left and right arms in a specific assembly process, the force distribution ratio, and adjustment strategies when dealing with interference can be extracted from the associated coordination operation data.

[0097] Finally, the extracted coordination pattern features are fused with the original motion data to generate enhanced motion data samples. The feature fusion process uses similarity values ​​as weights, weighting the coordination pattern features from the associated coordination operation data onto the corresponding feature dimensions of the original motion data. This fusion is not a simple data splicing, but rather embeds historical experience knowledge into the current original motion data, ensuring that the generated enhanced motion data samples retain both the specificity of the current original motion data and incorporate the common patterns of historical coordination operations. If the enhanced motion data samples have high dimensionality and information redundancy, principal component analysis (PCA) can be used for dimensionality reduction. PCA projects the original high-dimensional features into a low-dimensional space through linear transformation, resulting in enhanced motion data samples with moderate dimensionality and rich information, which can then be used for subsequent model training.

[0098] This implementation uses a dynamic time warping algorithm to accurately calculate the morphological similarity between historical coordination operation data and original motion data. This effectively filters out truly valuable related coordination operation data, avoiding noise interference from blindly fusing irrelevant historical data. Coordination pattern features are extracted from the related coordination operation data and fused, allowing historical experience knowledge to be injected into the enhanced motion data samples in the form of feature enhancement. This enhances the ability of the enhanced motion data samples to express the essential laws of bi-arm coordination. This similarity-weighted feature fusion method fully utilizes the valuable experience in the historical operation database while maintaining relevance to current actual working conditions. It provides high-quality input data for subsequent model training, helping to improve the accuracy and robustness of the coordination evaluation model.

[0099] In another example, the coordination assessment model is trained and evaluated based on augmented motion data samples to obtain coordination training results. This includes: extracting key motion features from the augmented motion data samples, wherein the key motion features include at least one of trajectory smoothness features, velocity consistency features, and force interaction features; and inputting the key motion features into the coordination assessment model for training and evaluation to obtain coordination training results.

[0100] Before training and evaluating the coordination assessment model using augmented motion data samples, it is necessary to extract key motion features from the augmented motion data samples. Key motion features can comprehensively and effectively characterize the essential characteristics of the coordinated movement of the two arms of an industrial robot. Key motion features may include at least one of the following features: First, the trajectory smoothness feature. This feature quantifies the smoothness of motion by calculating the rate of change of curvature or higher-order derivative of the motion trajectory of the robotic arm's end effector. Higher trajectory smoothness indicates fewer jitters and abrupt changes during motion, and better coordination between the two arms. Starting from the joint angle sequence in the enhanced motion data sample, the end-effector position trajectory is calculated, and then the curvature and deflection changes of the trajectory are calculated to finally obtain a smoothness quantification index.

[0101] Second, velocity consistency characteristics. This characteristic assesses the level of velocity coordination by analyzing the matching degree of linear and angular velocities of the left and right arms at the same time point or motion phase. The norm of the angular velocity difference between corresponding joints of the left and right arms is extracted from the angular velocity sequence, or the correlation coefficient of the end-effector velocity vectors of both arms is calculated to form velocity consistency characteristic values. Higher velocity consistency indicates a stronger ability of the two arms to maintain synchronization during movement.

[0102] Third, force interaction characteristics. These characteristics are obtained by processing raw data collected by force sensors and are used to describe the distribution and resistance of forces when the two arms cooperate to manipulate an object. The amplitude, direction, and rate of change of the resultant force at the ends of both arms are extracted from the force sensor data. The ratio of the forces in both arms or the cross-correlation function of the forces are calculated to form the force interaction characteristics. Reasonable force interaction characteristics indicate that the two arms can apply forces in a coordinated manner, avoiding mutual resistance or uneven load distribution.

[0103] The extracted key motion features are used as input variables and trained on a coordination evaluation model (such as a support vector machine regression model). The model learns the mapping relationship between motion patterns and coordination labels based on these key features, outputting the coordination training results. During training, the model focuses on the correlation between these key features and coordination, optimizing model parameters to make the predicted results as close as possible to the true coordination labels.

[0104] In practical implementation, key motion features extracted from augmented motion data samples may contain outliers or noise, which can be identified and processed using statistical methods. The number of samples at different coordination levels may be unbalanced, which can be adjusted using oversampling or undersampling techniques to ensure the trained model does not produce bias. In some embodiments, the training sample set (augmented motion data samples) is randomly divided into a training set, a validation set, and a test set. The training set is used for learning model parameters, the validation set is used to monitor model performance and adjust hyperparameters during training, and the test set is used to finally evaluate the generalization ability of the trained coordination assessment model. Optionally, a support vector machine (SVM) algorithm is used to initialize the coordination assessment model. The SVM algorithm is suitable for handling small-sample, nonlinear classification and regression problems. During initialization, the type of SVM algorithm, the kernel function form, and the initial values ​​of the relevant hyperparameters need to be set. For example, for continuous-value output problems such as coordination estimation, a support vector regression machine is typically used.

[0105] This implementation extracts trajectory smoothness, velocity consistency, and force interaction features as key motion features, capturing the core dimensions characterizing coordinated arm movements. Trajectory smoothness reflects the fluidity of movement, velocity consistency reflects the synchronization level of the arms, and force interaction features reveal the quality of mechanical coordination. These three features comprehensively cover the key elements of coordination assessment from kinematic and dynamic perspectives. This not only reduces the dimensionality of input data and computational complexity but, more importantly, enables the model to focus on core information highly correlated with coordination, improving the model's learning efficiency and prediction accuracy. The coordination assessment model trained based on these key features can more accurately identify abnormal coordination features in arm movements, providing a more reliable decision-making basis for subsequent dynamic coordination assessment and risk control.

[0106] Figure 3 A schematic diagram of the training samples provided for the implementation of this application.

[0107] See Figure 3 , Figure 3The horizontal axis represents trajectory smoothness, and the vertical axis represents velocity consistency. The color gradient, from dark blue to dark red, represents manually labeled coordination. Each scatter plot corresponds to an augmented motion data sample, visually illustrating the relationship between trajectory smoothness, velocity consistency, and coordination. After selecting trajectory smoothness and velocity consistency as the two key motion features from the augmented motion data samples, the rationality of the feature selection can be verified through this scatter plot. The high coordination samples, with a reddish hue, are mostly concentrated in areas with high trajectory smoothness and velocity consistency, indicating a positive correlation between these two features and coordination. These samples form the training data foundation for the coordination assessment model, enabling the model to accurately learn the nonlinear mapping relationship between feature combinations and coordination, ultimately outputting reliable coordination training results or dynamic coordination. This provides a quantitative basis for subsequent risk mapping and compensation decisions, thereby improving the coordination control accuracy and risk warning capabilities of industrial robot dual arms in tasks such as precision assembly and collaborative handling.

[0108] Figure 4 A schematic diagram of a robot automated assembly system provided in an embodiment of this application.

[0109] This application provides a robot automatic assembly system 400, including: an assembly reasoning module 410, an assembly part identification and perception module 420, and a motion planning module 430.

[0110] Assembly reasoning module 410 is used to perform reasoning analysis on assembly drawings and determine the assembly reasoning results.

[0111] The assembly part recognition and perception module 420 is used to acquire the workbench part placement scene image, and perform image recognition on the workbench part placement scene image to determine the part category and pose information of the part to be assembled.

[0112] The planning motion module 430 is used to decompose the assembly task based on the assembly reasoning results, determine the current part to be assembled and the current assembly target; determine the assembly control parameters based on the part category corresponding to the current part to be assembled, the pose information corresponding to the current part to be assembled and the current assembly target; and control the robot to assemble the current part to be assembled based on the assembly control parameters.

[0113] It is understandable that a detailed description of the robot automatic assembly system 400 can be found in the description of the robot automatic assembly method above, and will not be repeated here.

[0114] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0115] This application provides a robot including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0116] One embodiment of this application provides a computer program product including instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0117] Figure 5 A block diagram of an electronic device provided in an embodiment of this application.

[0118] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method in any of the above embodiments.

[0119] like Figure 5 As shown, for ease of understanding, embodiments of this application illustrate a specific electronic device 500.

[0120] Electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0121] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0122] Multiple components in electronic device 500 are connected to I / O interface 505. These components include: input unit 506, such as a keyboard or mouse; output unit 507, such as various types of displays or speakers; storage unit 508, such as a disk or optical disk; and communication unit 509, such as a network interface card (NIC), modem, or wireless transceiver. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods described above. For example, in some embodiments, any one or more of the methods described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of any one or more of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform any one or more of the methods described above by any other suitable means (e.g., by means of firmware).

[0124] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this application, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0125] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0126] In the description of this application, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0128] Furthermore, the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this application can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this application, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly and specifically defined in the embodiments.

[0129] In this application, unless otherwise explicitly specified or limited in the embodiments, the terms "installation," "connection," "joining," and "fixing" appearing in the embodiments should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral part; it can also be a mechanical connection, an electrical connection, etc. Of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication between two components, or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific implementation.

[0130] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

Claims

1. A robot-automated assembly method, characterized in that, The method includes: The assembly drawing is analyzed and reasoned to determine the assembly reasoning result. Acquire a scene image of the workbench parts placement, and perform image recognition on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled. Based on the assembly reasoning results, the assembly task is decomposed to determine the current parts to be assembled and the current assembly target. Based on the part category corresponding to the current part to be assembled, the pose information corresponding to the current part to be assembled, and the current assembly target, the assembly control parameters are determined; Based on the assembly control parameters, the robot is controlled to assemble the parts to be assembled.

2. The method according to claim 1, characterized in that, The assembly reasoning results include the parts required for assembly, the connection relationships between the parts, and the assembly sequence. The assembly task decomposition based on the assembly reasoning results, determining the current parts to be assembled and the assembly target information, includes: Based on the parts required for assembly, the connection relationships between the parts, and the assembly sequence, multiple assembly processes, the parts to be assembled for each assembly process, and the assembly target for each assembly process are determined by reasoning. The current assembly process is determined from the plurality of assembly processes, and the current part to be assembled and the current assembly target corresponding to the current assembly process are determined.

3. The method according to claim 1, characterized in that, The robot includes a robotic arm and a dexterous hand, and the assembly control parameters instruct the robotic arm and the dexterous hand to assemble along a collision-free path; controlling the robot to assemble the currently assembled part based on the assembly control parameters includes: Based on the assembly control parameters, the robotic arm and the dexterous hand are controlled to cooperate with each other and grasp, move and assemble the current part to be assembled along the collision-free path.

4. The method according to claim 3, characterized in that, The method further includes: During the process of moving the part to be assembled, the real-time pose of the part to be assembled is obtained, as well as the scene diagram of the workbench operation space is obtained. The workbench operation space scene map is subjected to image recognition, and based on the real-time pose, it is determined whether the subsequent pose of the part to be assembled will collide or interfere with other objects in the workbench operation space scene map. If yes, update the assembly control parameters to update the subsequent movement planning path; otherwise, continue moving the current part to be assembled according to the assembly control parameters until the current part to be assembled is manipulated to the target assembly pose.

5. The method according to claim 3, characterized in that, The assembly of the currently assembled parts includes: Acquire tactile sensing data of dexterous hand, joint motion sensing data of dexterous hand, motion sensing data of robotic arm, and multidimensional force sensing data of dexterous hand and robotic arm; The tactile sensing data of the dexterous hand, the joint motion sensing data of the dexterous hand, the motion sensing data of the robotic arm, and the multidimensional force sensing data are fused to obtain multimodal joint characterization data; Based on the multimodal joint characterization data, the robotic arm and the dexterous hand are controlled to assemble the parts to be assembled.

6. The method according to claim 1, characterized in that, The step of performing image recognition on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled includes: Obtain the part description text, and match the workbench part placement scene image with the part description text to determine the part category and part area in the workbench part placement scene image; The pose information is identified based on the part region.

7. The method according to claim 1, characterized in that, The method further includes: During the process of assembling the current part to be assembled by the robot based on the assembly control parameters, real-time motion data of the robot is acquired. Based on the real-time motion data, the dynamic coordination of the robot is evaluated; Based on the aforementioned dynamic coordination degree, the assembly risk level is determined; If the assembly risk level is greater than the preset risk level, the compensation control process is activated.

8. The method according to claim 7, characterized in that, The step of evaluating the robot's dynamic coordination based on the real-time motion data includes: inputting the real-time motion data into a coordination evaluation model for evaluation to obtain the dynamic coordination; wherein, the coordination evaluation model is trained in the following manner: Collect the robot's raw motion data; Based on the robot's historical coordinated operation data, feature enhancement is performed on the original motion data to generate enhanced motion data samples; Obtain the coordination labels corresponding to the enhanced motion data samples; Based on the enhanced motion data samples, the coordination assessment model is trained and evaluated to obtain the coordination training results; Based on the loss value between the coordination training results and the coordination labels, the model parameters of the coordination evaluation model are adjusted.

9. The method according to claim 8, characterized in that, The method involves enhancing the original motion data based on the robot's historical coordinated operation data to generate enhanced motion data samples, including: Based on the similarity between the historical coordinated operation data and the original motion data, related coordinated operation data is extracted from the historical coordinated operation data; Extract coordination pattern features from the associated coordination operation data; The coordinated mode features are fused with the original motion data to generate enhanced motion data samples.

10. The method according to claim 8, characterized in that, The process of training and evaluating the coordination assessment model based on the enhanced motion data samples to obtain coordination training results includes: Key motion features are extracted from the enhanced motion data samples, wherein the key motion features include at least one of trajectory smoothness features, velocity consistency features, and force interaction features; The key motion features are input into the coordination evaluation model for training and evaluation to obtain the coordination training results.

11. A robotic automated assembly system, characterized in that, The system includes: The assembly reasoning module is used to perform reasoning and analysis on the assembly drawing and determine the assembly reasoning result. The assembly part recognition and perception module is used to acquire a scene image of the workbench parts placement, and to perform image recognition on the workbench parts placement scene image to determine the part category and pose information of the parts to be assembled. The planning motion module is used to decompose the assembly task based on the assembly reasoning result, determine the current part to be assembled and the current assembly target; determine the assembly control parameters based on the part category corresponding to the current part to be assembled, the pose information corresponding to the current part to be assembled and the current assembly target; and control the robot to assemble the current part to be assembled based on the assembly control parameters.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-10.

Images