A dual-arm cooperative bolt precise assembly system and method based on digital twinning

Abstract

The application discloses a kind of based on digital twinning's double-arm collaborative bolt precision assembly system and method, belong to industrial automation and digital twinning processing technical field.The method is by constructing the virtual double-arm model consistent with the kinematics of physical double mechanical arm system;Based on the real end pose data and virtual end pose data of main mechanical arm under multiple groups of joint configuration, construct the error prediction model of joint angle vector to virtual end pose error;Obtain the visual data of work area containing target bolt to obtain the target pose information of target bolt;According to target pose information, generate the target pose of main mechanical arm and slave mechanical arm, drive double-arm motion to bolt assembly work position;Real-time supervision is carried out to double-arm pose using error prediction model, based on preset relative pose constraint, by real-time calculation slave mechanical arm target pose and generate control instruction.The application significantly improves the automation level and work reliability of complex assembly task.

Description

Technical Field

[0001] This invention relates to the fields of industrial automation and digital twin technology, and in particular to a dual-arm collaborative bolt precision assembly system and method based on digital twin. Background Technology

[0002] With the continuous improvement of industrial automation, robots are increasingly widely used in high-precision operation scenarios such as assembly manufacturing, rail transit maintenance, and aerospace. Bolt connections, as one of the most common mechanical connection methods, directly affect the safety and reliability of the overall structure due to their assembly quality. In typical scenarios such as railway catenary systems, bolts are densely distributed, space is limited, and the operating environment is complex and variable, placing higher demands on the precision and stability of automated assembly systems. Dual-arm robotic systems, due to their greater flexibility and operational efficiency, are gradually becoming a research hotspot in the field of automated bolt assembly.

[0003] However, in practical applications, dual-arm robotic systems still face several technical challenges: at the visual perception level, existing methods mostly rely on target detection based on two-dimensional images or three-dimensional reconstruction technology based on point clouds, making it difficult to simultaneously achieve both detection accuracy and real-time performance; in terms of collaborative control, existing technologies typically employ master-slave control or peer control strategies, relying on ideal kinematic models and lacking the ability to dynamically compensate for actual errors; and in terms of digital twin applications, existing systems are mostly used for offline simulation or state display, failing to deeply participate in the real-time control process.

[0004] Therefore, there is an urgent need in this field for a dual-arm collaborative bolt precision assembly system and method based on digital twins. Summary of the Invention

[0005] In view of this, the present invention provides a method for precise assembly of bolts by dual-arm collaborative assembly based on digital twins, in order to solve the problem that the existing methods, which rely solely on numerical parameter adjustments for error correction, are insufficient to meet the requirements for precise bolt assembly.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: On one hand, the present invention provides a method for precise assembly of dual-arm collaborative bolts based on digital twins, comprising: S1: Construct a virtual dual-manipulator model that is consistent with the kinematics of the physical dual-manipulator system, and establish a digital twin data interaction channel; wherein, the digital twin data interaction channel is used to realize bidirectional real-time data transmission and state synchronization between the physical manipulator and the virtual manipulator. S2: Based on the real end-effector pose data of the main manipulator under multiple joint configurations and the virtual end-effector pose data under the corresponding joint configurations obtained through the virtual dual manipulator model, an error prediction model for the error between the joint angle vector and the virtual-real end-effector pose is constructed; wherein, the real end-effector pose is the end-effector pose of the physical main manipulator, the virtual end-effector pose is the end-effector pose of the virtual main manipulator, and the virtual-real end-effector pose error is the deviation between the real end-effector pose and the virtual end-effector pose; S3: Obtain visual data of the work area containing the target bolt to obtain the target pose information of the target bolt in the robot base coordinate system; S4: Generate the target poses of the main robotic arm and the slave robotic arm based on the target pose information, and drive the two arms to move to the bolt assembly operation position; S5: The error prediction model is used to monitor the pose of the two arms in real time. Based on the preset relative pose constraints between the end effectors of the master and slave robotic arms, the target pose of the slave robotic arm is calculated in real time and control commands are generated through the digital twin data interaction channel to realize the pose synchronization and motion timing coordination of the two arms during the assembly operation. The preset relative pose constraints are constraints that describe the fixed relative spatial relationship between the end effectors of the master and slave robotic arms.

[0007] Preferably, step S2 further includes: S21: Under multiple joint configurations, acquire the real end-effector pose and the virtual end-effector pose, and calculate the virtual and real end-effector pose error vector; S22: Construct joint space feature vectors based on joint angle vectors; S23: Establish a mapping model between the joint space feature vector and the virtual and real end pose error vector, and use the parameter estimation method to solve the mapping parameters to obtain the virtual and real error mapping function; S24: Based on the error vector obtained from the virtual-real error mapping function, construct the error transformation matrix to obtain the virtual-real mapping model, the expression of which is:

[0008] Where q is the joint angle vector of the robotic arm. This represents the true end-effector pose at joint angle q. The virtual end-effector pose at joint angle q. Let be the error transformation matrix.

[0009] Preferably, the error transformation matrix is ​​determined in the following manner:

[0010] in, Let be the three-dimensional attitude error vector at joint angle q. Let be the three-dimensional position error vector at joint angle q. This is the rotation matrix generated based on the three-dimensional attitude error vector.

[0011] Preferably, step S2 further includes: Based on the virtual dual robotic arm model, the digital twin data interaction channel, and the virtual-real mapping model, the end-effector pose deviation of the master robotic arm is predicted in real time, and the motion state of the slave robotic arm is monitored and compensated in real time.

[0012] Preferably, step S3 further includes: S31: Use two-dimensional images to perform target detection on the work area image, obtain the regional location information of the bolt connection, and drive the main robotic arm to reach the preset observation position to complete the first stage of target preliminary positioning; S32: Obtain three-dimensional point cloud data of the bolt area at the preset observation position, perform point cloud fitting based on the preset geometric model, obtain the accurate pose of the bolt in the sensor coordinate system, and convert it into the target pose information through coordinate transformation to complete the second stage of target accurate pose estimation.

[0013] Preferably, step S5 further includes: S51: Obtain the real-time pose of the end effector of the physical master robotic arm through the digital twin data interaction channel; S52: Based on the preset relative pose constraint and the real-time pose of the end of the physical master robot arm, calculate the target pose of the end of the physical slave robot arm, wherein the preset relative pose constraint remains constant during the assembly operation. S53: Convert the target pose of the physical component at the end of the robotic arm into joint motion commands, drive the physical component to follow the movement of the physical main robotic arm, and realize real-time mirror tracking and synchronous assembly.

[0014] On the other hand, the present invention provides a digital twin-based dual-arm collaborative bolt precision assembly system, the system comprising: A construction module is used to build a virtual dual-manipulator model that is consistent with the kinematics of the physical dual-manipulator system and establish a digital twin data interaction channel; wherein, the digital twin data interaction channel is used to realize bidirectional real-time data transmission and state synchronization between the physical manipulator and the virtual manipulator. The error prediction module is used to construct an error prediction model for the error between the joint angle vector and the virtual end-effector pose based on the real end-effector pose data of the main manipulator under multiple joint configurations and the virtual end-effector pose data under the corresponding joint configurations obtained through the virtual dual manipulator model; wherein, the real end-effector pose is the end-effector pose of the physical main manipulator, the virtual end-effector pose is the end-effector pose of the virtual main manipulator, and the virtual-real end-effector pose error is the deviation between the real end-effector pose and the virtual end-effector pose; The visual positioning module is used to acquire visual data of the work area containing the target bolt, so as to obtain the target pose information of the target bolt in the robot base coordinate system; The drive module is used to generate the target poses of the main robotic arm and the slave robotic arm based on the target pose information, and drive the two arms to move to the bolt assembly operation position. The control module is used to monitor the pose of the two arms in real time using the error prediction model. Based on the preset relative pose constraints between the end effectors of the master and slave robotic arms, it calculates the target pose of the slave robotic arm in real time and generates control commands through the digital twin data interaction channel to achieve pose synchronization and motion timing coordination of the two arms during the assembly operation. The preset relative pose constraints are constraints that describe the fixed relative spatial relationship between the end effectors of the master and slave robotic arms.

[0015] On the other hand, the present invention provides a computer device including a memory and a processor, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory to implement the steps of any of the methods described in this specification.

[0016] On the other hand, the present invention provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in this specification.

[0017] On the other hand, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described in this specification.

[0018] This invention provides a digital twin-based method for precise bolt assembly using a dual-arm collaborative system. First, a virtual dual-arm model with kinematics consistent with the physical dual-arm system is constructed, establishing a digital twin data interaction channel to achieve bidirectional real-time data transmission and state synchronization between the physical and virtual arms. Then, based on the real and virtual end-effector poses of the main arm under multiple joint configurations, an error prediction model is constructed to calculate the error between the joint angle vector and the virtual / real end-effector pose. Furthermore, an error transformation matrix is ​​constructed to obtain a virtual / real mapping model, effectively solving the problem that existing digital twin systems only adjust parameters at the numerical level for error correction and cannot achieve precise pose deviation prediction and geometric compensation. Next, the method obtains data containing the target... The visual data of the bolt's working area is processed in stages. The first stage involves target detection and preliminary localization. The second stage, based on the preliminary localization results, performs point cloud fitting and precise pose estimation to obtain the target bolt's pose information in the robot's base coordinate system. Then, the target poses of the master and slave robotic arms are generated based on the target pose information, driving both arms to move to the bolt assembly position. Finally, an error prediction model is used to monitor the poses of both arms in real time. Based on the preset relative pose constraints between the ends of the master and slave robotic arms, the target pose of the slave robotic arm is calculated in real time and control commands are generated through the digital twin data interaction channel, achieving pose synchronization and motion timing coordination between the two arms during the assembly process. This invention integrates digital twin technology with dual-arm collaborative control throughout the entire process, constructing a complete data processing mechanism covering "virtual-real error mapping modeling—staged visual localization—master-slave constraint synchronous assembly." This fundamentally overcomes the shortcomings of existing technologies, such as the accumulation of virtual-real pose deviations, the difficulty in balancing visual positioning accuracy and real-time performance, and the lack of fixed relative pose constraints in dual-arm collaboration. This significantly improves the automation level and operational reliability of complex assembly tasks. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart of a dual-arm collaborative bolt precision assembly method based on digital twin is provided for an embodiment of the present invention; Figure 2 A schematic diagram of an experimental system provided in an embodiment of the present invention; Figure 3 A schematic diagram of a digital twin system framework provided in an embodiment of the present invention; Figure 4A schematic diagram illustrating the collaborative alignment of dual robotic arms in a virtual and physical environment, provided as an embodiment of the present invention; Figure 5 This is a schematic diagram of the detection result of a connector provided in an embodiment of the present invention; Figure 6 A keyframe diagram of a connector tracking experiment provided in an embodiment of the present invention; Figure 7 A bolt segmentation point cloud diagram provided in an embodiment of the present invention; Figure 8 A bolt six-degree-of-freedom attitude estimation diagram provided in an embodiment of the present invention; Figure 9 This is a hardware architecture diagram of a computer device provided in an embodiment of the present invention; Figure 10 This is a structural diagram of a dual-arm collaborative bolt precision assembly system based on digital twins, provided by an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The embodiments of the present invention are described below with reference to the figures.

[0023] like Figure 1 As shown, this invention provides a method for precise assembly of bolts using a dual-arm collaborative system based on digital twins, comprising: S1: Construct a virtual dual-manipulator model that is consistent with the kinematics of the physical dual-manipulator system, and establish a digital twin data interaction channel; wherein, the digital twin data interaction channel is used to realize bidirectional real-time data transmission and state synchronization between the physical manipulator and the virtual manipulator.

[0024] In this invention, step S1 aims to construct a virtual simulation environment that is highly consistent with the physical dual-arm robotic system and establish a bidirectional real-time data transmission channel between the physical end and the virtual end, providing a basic support platform for the construction of the error prediction model in step S2, the processing of visual positioning data in step S3, and the synchronous control of the dual arms in step S5.

[0025] like Figure 2As shown, the experimental setup system of this embodiment includes two UR12e six-DOF collaborative robots, one Intel RealSense D435i camera, and one Mech-Eye industrial-grade 3D camera. The main robotic arm and the slave robotic arm are respectively mounted on both sides of the work platform, and both robotic arms are equipped with bolt fastening tools at their ends for performing bolt assembly tasks. A vision module is mounted at the end of the main robotic arm to acquire RGB-D image information of the work area.

[0026] The construction of a virtual dual-arm model that matches the kinematics of the physical dual-arm system specifically includes: establishing a high-fidelity model using the DH parameters of the real UR12e robot to ensure kinematic consistency between the virtual and physical systems. Preferably, the CoppeliaSim simulation environment can be used.

[0027] like Figure 3 The diagram shown is a block diagram of a digital twin system according to an embodiment of the present invention. This virtual-physical integration platform is built on the ViSPROS framework and the CoppeliaSim simulation environment, and realizes a comprehensive digital twin architecture through several key components: the ViSP ROS software package directly integrates visual servoing functions into the ROS network; a dedicated ROS class is responsible for core algorithm processing; and the CoppeliaSim error prediction model tracks the virtual reality model error in real time, achieving complete synchronization between the virtual and physical environments.

[0028] The digital twin data interaction channel is used to achieve bidirectional real-time data transmission and status synchronization between the physical and virtual robotic arms. During operation, the virtual controller calculates control commands based on environmental status data. These commands are synchronously transmitted to the simulation environment and the physical robot via the ROS network.

[0029] like Figure 4 The diagram shown illustrates the collaborative alignment of the dual robotic arms in virtual and physical environments according to an embodiment of the present invention. Figure 4 (a) illustrates the virtual environment within the digital twin system. Figure 4 (b) shows the corresponding physical environment, with the two maintaining real-time synchronization, demonstrating the bidirectional data transmission and status synchronization function of the digital twin data interaction channel.

[0030] Through the implementation of step S1 above, this invention constructs a high-fidelity virtual model and a two-way real-time data interaction channel consistent with the kinematics of the physical dual-arm robotic system, forming a complete digital twin infrastructure environment. This environment provides the data foundation and communication guarantee for the acquisition and modeling of virtual and real pose errors in step S2, and for the real-time supervision and dual-arm synchronous control based on digital twins in step S5.

[0031] It should be noted that the specific selection of the simulation environment, the specific model and number of degrees of freedom of the robotic arm, the specific values ​​of the DH parameters, and the specific communication protocol of the data interaction channel can all be adapted by those skilled in the art according to the actual application requirements. As long as the function of constructing a virtual model consistent with the kinematics of the physical dual robotic arm system and establishing a two-way real-time data interaction channel can be achieved, it should fall within the protection scope of this invention.

[0032] S2: Based on the real end-effector pose data of the main manipulator under multiple joint configurations and the virtual end-effector pose data under the corresponding joint configurations obtained through the virtual dual manipulator model, an error prediction model for the error between the joint angle vector and the virtual end-effector pose is constructed; wherein, the real end-effector pose is the end-effector pose of the physical main manipulator, the virtual end-effector pose is the end-effector pose of the virtual main manipulator, and the virtual-real end-effector pose error is the deviation between the real end-effector pose and the virtual end-effector pose.

[0033] In this invention, step S2 is the core step of digital twin virtual-real error correction. It aims to establish a quantitative mapping relationship from joint angle vectors to virtual-real end pose error by collecting end pose data of physical master manipulator and virtual master manipulator under multiple joint configurations, and further construct an error transformation matrix to form a complete virtual-real mapping model, providing a model basis for the digital twin system to predict and compensate for end pose deviation of the manipulator in real time.

[0034] In some implementations, step S2 further includes: S21: Under multiple joint configurations, acquire the real end-effector pose and the virtual end-effector pose, and calculate the virtual and real end-effector pose error vector.

[0035] In a digital twin environment, the virtual dual-arm model is constructed based on real DH parameters, theoretically matching the kinematic characteristics of the physical robotic arm. However, due to factors such as manufacturing errors, joint clearances, sensor noise, and control delays, a deviation inevitably exists between the actual end-effector pose of the physical robotic arm and the virtual end-effector pose of the virtual robotic arm. To establish the mapping relationship between this deviation and the joint configuration, it is first necessary to collect multiple sets of sample data under representative joint configurations.

[0036] Specifically, the physical master robot arm is controlled to move to N different joint configurations. Under the i-th joint configuration, the real end-effector pose is read through the positive kinematics model of the physical master robot arm and the joint angle feedback value. At the same time, the virtual end-effector pose under the corresponding joint configuration is read from the digital twin system, and the calculated error is converted into an error vector.

[0037] S22: Construct joint space feature vectors based on joint angle vectors.

[0038] To establish a functional mapping relationship between joint variables and end-effector pose error, a feature vector that can fully characterize the joint space configuration needs to be constructed. The joint angle vector of the robotic arm is q. The joint space feature vector includes a constant term, first-order terms and second-order terms for each joint angle, as well as inter-joint interaction terms. Its expression is:

[0039] Where the feature vector dimension is m, the feature vector corresponding to the i-th sample is represented as:

[0040] in, Let be the joint angle vector of the i-th sample. By constructing the above polynomial feature vectors, the nonlinear relationship between joint configuration and end-effector pose error can be captured more comprehensively, providing a rich feature foundation for the establishment of subsequent mapping models.

[0041] In one specific implementation, for a six-degree-of-freedom robotic arm, m=6.

[0042] S23: Establish a mapping model between the joint space feature vector and the virtual and real end pose error vector, and use the parameter estimation method to solve the mapping parameters to obtain the virtual and real error mapping function.

[0043] In this invention, it is assumed that the virtual and real pose errors satisfy a linear parametric model with respect to joint variables, and an error matrix is ​​constructed. , characteristic matrix Establish a mapping relationship:

[0044] Where A is the mapping parameter matrix to be solved, the parameter matrix A is solved using the least squares method:

[0045] Thus, the complete virtual-real error mapping function is obtained:

[0046] Furthermore, the error vector e(q) is decomposed into a three-dimensional position error vector and a three-dimensional attitude error vector, expressed as:

[0047] in, This is the three-dimensional position error vector. Given a three-dimensional attitude error vector, this mapping function establishes a quantitative mapping relationship from the joint angle vector q to the virtual and real end-effector pose error vector e.

[0048] S24: Based on the error vector obtained from the virtual-real error mapping function, construct the error transformation matrix to obtain the virtual-real mapping model, the expression of which is:

[0049] Where q is the joint angle vector of the robotic arm. This represents the true end-effector pose at joint angle q. The virtual end-effector pose at joint angle q. Let be the error transformation matrix.

[0050] Specifically, after obtaining the error vector from the virtual-real error mapping function e(q), the error transformation matrix is ​​further constructed. First, an antisymmetric matrix is ​​constructed from the three-dimensional attitude error vector, expressed as:

[0051] The Lie algebra matrix is ​​constructed from the antisymmetric matrix and the three-dimensional position error vector, and its expression is:

[0052] By mapping the Lie algebra matrix to the SE(3) space through exponential mapping, the error transformation matrix is ​​obtained, and its expression is:

[0053] After expansion, the error transformation matrix is ​​obtained, and its expression is:

[0054] in, The rotation matrix is ​​generated based on the three-dimensional attitude error vector. The Jacobian matrix corresponding to the three-dimensional attitude error vector is expressed as:

[0055] in, I represents the minimum rotation angle between the virtual pose and the real pose, and I is the identity matrix.

[0056] Under conditions of extremely small error, we can approximate V(Δr)≈I.

[0057] The error transformation matrix is ​​determined in the following way:

[0058] in, Let be the three-dimensional attitude error vector at joint angle q. Let be the three-dimensional position error vector at joint angle q. This is the rotation matrix generated based on the three-dimensional attitude error vector.

[0059] Based on the above error transformation matrix, a virtual-real mapping model is established, expressed as:

[0060] In this invention, through this error prediction model, the digital twin system can predict the end-effector pose deviation of the real robotic arm in real time based on any given joint configuration, and monitor the motion state of the robotic arm in real time.

[0061] like Figure 3 As shown, the block diagram of the digital twin system in this embodiment of the invention includes the CoppeliaSim error prediction model. This model tracks the error of the virtual reality model in real time, achieving complete synchronization between the virtual and physical environments. It is functionally consistent with the error prediction model constructed in step S2 above.

[0062] S3: Obtain visual data of the work area containing the target bolt to obtain the target pose information of the target bolt in the robot base coordinate system.

[0063] In this invention, step S3 aims to achieve high-precision spatial positioning of the target bolt connector through a phased visual processing strategy, providing accurate target pose input for the generation of the dual-arm target pose in the subsequent step S4. This step, through a two-stage processing approach of first performing target detection and preliminary positioning, and then performing point cloud fitting and precise pose estimation, balances the real-time performance and accuracy of visual positioning.

[0064] In some implementations, step S3 further includes: S31: Use two-dimensional images to perform target detection on the work area image, obtain the regional location information of the bolt connection, and drive the main robotic arm to reach the preset observation position to complete the first stage of target preliminary positioning.

[0065] In this invention, firstly, two-dimensional image data of the working area is collected by a vision sensor installed at the end of the main robotic arm. The bolt connector is initially identified by a YOLO-based target detection network. The network outputs a predicted bounding box, thereby obtaining the center position coordinates and depth parameters of the bolt connector in the image, and establishing the three-dimensional attitude coordinates of the component for visual servo control.

[0066] The position-based visual servoing strategy utilizes the aforementioned preliminary positioning information to control and drive the main robotic arm equipped with a visual sensor to reach a preset observation posture at a preset distance from the connector, ensuring that the bolt connector is located at the center of the visual sensor's field of view and at a suitable distance, thus providing good observation conditions for the second stage of accurate pose estimation.

[0067] like Figure 5As shown, the predicted bounding boxes generated by the YOLO model can accurately locate the component center position and depth parameters of the connector, establishing the three-dimensional pose coordinates of the component for visual servo control.

[0068] like Figure 6 As shown, the visual feedback sequence during the servo control process is illustrated: Figure 6 (a) Presents the initial view when the servo starts; Figure 6 (b) Displays the frame showing the first successful detection of the target's attitude; Figure 6 (c) Capture intermediate states during the servo convergence process; Figure 6 (d) shows the final servo state, where the estimated attitude perfectly matches the target attitude.

[0069] S32: Obtain three-dimensional point cloud data of the bolt area at the preset observation position, perform point cloud fitting based on the preset geometric model, obtain the accurate pose of the bolt in the sensor coordinate system, and convert it into the target pose information through coordinate transformation to complete the second stage of target accurate pose estimation.

[0070] In this invention, at the start of the precise positioning phase, the system switches to an industrial-grade 3D camera to simultaneously acquire grayscale images and corresponding 3D point cloud data during the acquisition cycle. First, a YOLO detector is applied to the grayscale image for initial bolt detection.

[0071] like Figure 7 The image shown is a bolt point cloud diagram according to an embodiment of the present invention. Figure 7 (a) shows the detection results of the YOLO detector for bolt detection in grayscale images. Figure 7 (b) shows the results of model fitting of the segmented point cloud data based on the detection results.

[0072] Subsequently, based on the detection results, a model was fitted to the segmented point cloud data to determine the precise six-degree-of-freedom pose of the bolt.

[0073] like Figure 8 The figure shown is a six-degree-of-freedom attitude estimation diagram of the bolts according to an embodiment of the present invention, which shows the final six-degree-of-freedom attitude fitting results of the three bolts.

[0074] After obtaining the precise pose of the bolt in the camera coordinate system, the six-DOF pose of the bolt in the camera coordinate system is transformed to the robot base coordinate system using a pre-calibrated hand-eye calibration matrix, thereby obtaining the final target pose information. This target pose information will serve as the input basis for generating the target poses of the master and slave robotic arms in step S4.

[0075] S4: Generate the target poses of the main robotic arm and the slave robotic arm based on the target pose information, and drive both arms to move to the bolt assembly operation position.

[0076] In this invention, step S4 aims to utilize the precise pose information of the target bolt in the robot base coordinate system obtained in step S3, combined with the preset assembly posture relationship between the end effectors of the main robot arm and the slave robot arm relative to the bolt axis in the bolt assembly task, to generate the target poses of the physical main robot arm and the physical slave robot arm respectively, and drive the two arms to move simultaneously to the bolt assembly operation position through motion planning, so as to prepare the position and posture for the synchronous fastening operation of the two arms in step S5.

[0077] Specifically, after obtaining the pose information of the target bolt, the direction of the bolt axis and the relative geometric relationship between the main arm and the end effector of the slave arm required for the tightening operation are determined based on the geometric characteristics of the bolt connection and the assembly process requirements. In a typical bolt assembly scenario, the main robotic arm is usually responsible for positioning the bolt head, while the slave robotic arm is responsible for supporting or simultaneously tightening the nut. The ends of both arms are mirror-symmetric or collinear with respect to the bolt axis. Based on this preset assembly posture relationship, the target pose matrix that the end effector of the main robotic arm needs to reach and the target pose matrix that the end effector of the slave robotic arm needs to reach are calculated respectively, so that the end effectors of the two arms can be precisely aligned with both ends of the bolt.

[0078] Subsequently, utilizing the robot's motion planning function, motion commands are generated for both the main and slave robotic arms to move from their current positions to their corresponding target poses. These commands are then used to drive the physical arms smoothly and coordinately to the bolt assembly position via a digital twin data interaction channel or a direct control interface. At this point, the end effectors of both arms are positioned at both ends of the bolt, ready for tightening.

[0079] Through the implementation of step S4 above, the present invention transforms the target spatial information obtained by visual perception into pose commands that can be executed by both arms, realizing the transition from "perception" to "movement", and establishing a precise spatial alignment foundation for the synchronous fastening operation based on real-time supervision of digital twin and master-slave relative pose constraints in the subsequent step S5.

[0080] S5: The error prediction model is used to monitor the pose of the two arms in real time. Based on the preset relative pose constraints between the end effectors of the master and slave arms, the target pose of the slave arm is calculated in real time and control commands are generated to realize the pose synchronization and motion timing coordination of the two arms during the assembly operation. The preset relative pose constraints are constraints that describe the fixed relative spatial relationship between the end effectors of the master and slave arms.

[0081] In this invention, step S5 is based on the completion of bolt visual positioning and initial alignment of the two arms. The error prediction model constructed in step S2 is used for real-time monitoring. Through master-slave mirror tracking and synchronous control, the core control link of high-precision pose synchronization and motion timing coordination of the two arms during the fastening operation is realized.

[0082] In some implementations, step S5 further includes: S51: Obtain the real-time pose of the end effector of the physical master robotic arm through the digital twin data interaction channel.

[0083] In this invention, during the assembly process, the physical main robotic arm may experience slight pose changes due to factors such as force control adjustment, external disturbances, or motion errors. In order to perceive the motion state of the main arm in real time, the real-time pose of its end effector relative to the robot base coordinate system is obtained from the physical main robotic arm controller at a fixed frequency through the digital twin data interaction channel established in step S1. This real-time pose data serves as the reference input for subsequent calculation of the target pose of the robotic arm.

[0084] S52: Based on the preset relative pose constraint and the real-time pose of the physical master robot arm end, calculate the target pose of the physical slave robot arm end, wherein the preset relative pose constraint remains constant during the assembly operation.

[0085] In a digital twin environment, a preset relative pose constraint relationship is established between the coordinate system of the master robotic arm's end effector and the coordinate system of the slave robotic arm's end effector. This constraint describes the fixed relative spatial relationship that the dual-arm end effectors should maintain during the fastening operation. Based on the real-time pose of the physical master robotic arm's end effector obtained in step S51, and in conjunction with the preset relative pose constraint, the target pose of the physical slave robotic arm's end effector is calculated in real time.

[0086] In dual-arm alignment mode, the robot controller reads the pose matrix of the master arm relative to the base coordinate system and the pose matrix of the slave arm relative to the base coordinate system, and establishes closed-loop constraint equations by applying robot kinematic relationships: in, This is the pose matrix of the master robot's end effector coordinate system relative to the master robot's base coordinate system. Let be the pose matrix of the end effector coordinate system relative to the end effector base coordinate system. The constraint transformation matrix between the master robot arm's end-effector coordinate system and the slave robot arm's end-effector coordinate system. Let be the relative attitude matrix of the slave arm base coordinate system relative to the master arm base coordinate system. This closed-loop constraint equation ensures that the relative spatial relationship between the master arm end effector and the slave arm end effector remains constant during the fastening process.

[0087] After calibration, the dual-arm collaborative control process includes symmetrical attitude initialization and real-time mirror tracking. First, the real-time attitude of the master arm is acquired and transmitted to the slave arm, while simultaneously establishing constraint transformations between the end effectors of the master and slave arms. The target pose of the slave arm is obtained through the above closed-loop constraint equations. Subsequently, the master arm control node in the virtual environment transmits the data to the slave arm controller through the digital twin interaction platform.

[0088] S53: Convert the target pose of the physical component at the end of the robotic arm into joint motion commands, drive the physical component to follow the movement of the physical main robotic arm, and realize real-time mirror tracking and synchronous assembly.

[0089] After receiving the target pose from the arm control node, the inverse kinematics solver is used to convert the target pose into the target angles of each joint of the robotic arm, generate joint space commands, drive the physical movement from each joint of the robotic arm to the target angle, so that the pose of the end effector of the arm follows the change of the end effector of the main arm in real time, and keep the preset relative pose constraint between the ends of the two arms constant.

[0090] Through the aforementioned real-time mirror tracking mechanism, during the bolt tightening process, regardless of the cause of any positional change in the main robotic arm, the slave robotic arm can quickly and accurately follow the movement of the main arm through real-time monitoring and constraint calculation by the digital twin system. This ensures that the relative spatial relationship between the end effectors of the two arms remains constant, thereby effectively avoiding thread misalignment, jamming, or workpiece damage caused by the positional deviation of the two arms, and achieving high-precision and high-reliability synchronous tightening control.

[0091] Through the implementation of step S5 above, the present invention utilizes a digital twin error prediction model for real-time monitoring and achieves master-slave mirror tracking and synchronous assembly based on preset relative pose constraints. This effectively overcomes the shortcomings of existing technologies, such as dual-arm collaborative control relying on ideal models, lacking actual error dynamic compensation capabilities, and failing to maintain constant relative pose at the end point by using error convergence to zero as the control objective. This significantly improves the accuracy and reliability of bolt assembly operations.

[0092] like Figure 9 , Figure 10 As shown, this embodiment of the invention provides a dual-arm collaborative bolt precision assembly system based on digital twins. The system embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, as... Figure 9 The diagram shown is a hardware architecture diagram of a computing device for a digital twin-based dual-arm collaborative bolt precision assembly system provided in an embodiment of the present invention. Figure 9 In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device housing the system in this embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 10 As shown, a system in a logical sense is formed by the CPU of the computing device in which it resides reading the corresponding computer program from the non-volatile memory into the main memory for execution.

[0093] Reference Figure 10This invention provides a digital twin-based dual-arm collaborative bolt precision assembly system, comprising: The construction module 100 is used to construct a virtual dual-manipulator model that is consistent with the kinematics of the physical dual-manipulator system and establish a digital twin data interaction channel; wherein, the digital twin data interaction channel is used to realize bidirectional real-time data transmission and state synchronization between the physical manipulator and the virtual manipulator. The error prediction module 200 is used to construct an error prediction model for the error between the joint angle vector and the virtual end-effector pose based on the real end-effector pose data of the main manipulator under multiple joint configurations and the virtual end-effector pose data under the corresponding joint configurations obtained through the virtual dual manipulator model; wherein, the real end-effector pose is the end-effector pose of the physical main manipulator, the virtual end-effector pose is the end-effector pose of the virtual main manipulator, and the virtual-real end-effector pose error is the deviation between the real end-effector pose and the virtual end-effector pose; The visual positioning module 300 is used to acquire visual data of the working area containing the target bolt, so as to obtain the target pose information of the target bolt in the robot base coordinate system; The drive module 400 is used to generate the target poses of the main robotic arm and the slave robotic arm based on the target pose information, and drive the two arms to move to the bolt assembly operation position. The control module 500 is used to monitor the pose of the two arms in real time using the error prediction model. Based on the preset relative pose constraints between the end effectors of the master and slave robotic arms, it calculates the target pose of the slave robotic arm in real time and generates control commands through the digital twin data interaction channel to achieve pose synchronization and motion timing coordination of the two arms during the assembly operation. The preset relative pose constraints are constraints that describe the fixed relative spatial relationship between the end effectors of the master and slave robotic arms.

[0094] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the digital twin-based dual-arm collaborative bolt precision assembly system. In other embodiments of the present invention, the digital twin-based dual-arm collaborative bolt precision assembly system may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0096] This invention also provides a computing device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the method in any embodiment of this invention.

[0097] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the method in any embodiment of this invention.

[0098] Embodiments of this application also provide a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium and executes the computer program, causing the computer device to perform any of the test methods described in the above embodiments.

[0099] Specifically, a system equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system can read and execute the program code stored in the storage medium.

[0100] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0101] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0102] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0103] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0104] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0105] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for precise assembly of bolts using a dual-arm collaborative system based on digital twins, characterized in that, include: S1: Construct a virtual dual-manipulator model that is consistent with the kinematics of the physical dual-manipulator system, and establish a digital twin data interaction channel; wherein, the digital twin data interaction channel is used to realize bidirectional real-time data transmission and state synchronization between the physical manipulator and the virtual manipulator. S2: Based on the real end-effector pose data of the main manipulator under multiple joint configurations and the virtual end-effector pose data under the corresponding joint configurations obtained through the virtual dual manipulator model, an error prediction model for the error between the joint angle vector and the virtual-real end-effector pose is constructed; wherein, the real end-effector pose is the end-effector pose of the physical main manipulator, the virtual end-effector pose is the end-effector pose of the virtual main manipulator, and the virtual-real end-effector pose error is the deviation between the real end-effector pose and the virtual end-effector pose; S3: Obtain visual data of the work area containing the target bolt to obtain the target pose information of the target bolt in the robot base coordinate system; S4: Generate the target poses of the main robotic arm and the slave robotic arm based on the target pose information, and drive the two arms to move to the bolt assembly operation position; S5: The error prediction model is used to monitor the pose of the two arms in real time. Based on the preset relative pose constraints between the end effectors of the master and slave robotic arms, the target pose of the slave robotic arm is calculated in real time and control commands are generated through the digital twin data interaction channel to realize the pose synchronization and motion timing coordination of the two arms during the assembly operation. The preset relative pose constraints are constraints that describe the fixed relative spatial relationship between the end effectors of the master and slave robotic arms.

2. The method according to claim 1, characterized in that, Step S2 further includes: S21: Under multiple joint configurations, acquire the real end-effector pose and the virtual end-effector pose, and calculate the virtual and real end-effector pose error vector; S22: Construct joint space feature vectors based on joint angle vectors; S23: Establish a mapping model between the joint space feature vector and the virtual and real end pose error vector, and use the parameter estimation method to solve the mapping parameters to obtain the virtual and real error mapping function; S24: Based on the error vector obtained from the virtual-real error mapping function, construct the error transformation matrix to obtain the virtual-real mapping model, the expression of which is: Where q is the joint angle vector of the robotic arm. This represents the true end-effector pose at joint angle q. The virtual end-effector pose at joint angle q. Let be the error transformation matrix.

3. The method according to claim 2, characterized in that, The error transformation matrix is ​​determined in the following way: in, Let be the three-dimensional attitude error vector at joint angle q. Let be the three-dimensional position error vector at joint angle q. This is the rotation matrix generated based on the three-dimensional attitude error vector.

4. The method according to claim 2, characterized in that, Step S2 also includes: Based on the virtual dual robotic arm model, the digital twin data interaction channel, and the virtual-real mapping model, the end-effector pose deviation of the master robotic arm is predicted in real time, and the motion state of the slave robotic arm is monitored and compensated in real time.

5. The method according to claim 1, characterized in that, Step S3 further includes: S31: Use two-dimensional images to perform target detection on the work area image, obtain the regional location information of the bolt connection, and drive the main robotic arm to reach the preset observation position to complete the first stage of target preliminary positioning; S32: Obtain three-dimensional point cloud data of the bolt area at the preset observation position, perform point cloud fitting based on the preset geometric model, obtain the accurate pose of the bolt in the sensor coordinate system, and convert it into the target pose information through coordinate transformation to complete the second stage of target accurate pose estimation.

6. The method according to claim 1, characterized in that, Step S5 further includes: S51: Obtain the real-time pose of the end effector of the physical master robotic arm through the digital twin data interaction channel; S52: Based on the preset relative pose constraint and the real-time pose of the end of the physical master robot arm, calculate the target pose of the end of the physical slave robot arm, wherein the preset relative pose constraint remains constant during the assembly operation. S53: The target pose of the physical slave robot arm end is converted into joint motion commands through the digital twin data interaction channel, driving the physical slave robot arm to follow the physical master robot arm, thereby realizing real-time mirror tracking and synchronous assembly.

7. A digital twin-based dual-arm collaborative bolt precision assembly system, characterized in that, include: A construction module is used to build a virtual dual-manipulator model that is consistent with the kinematics of the physical dual-manipulator system and establish a digital twin data interaction channel; wherein, the digital twin data interaction channel is used to realize bidirectional real-time data transmission and state synchronization between the physical manipulator and the virtual manipulator. The error prediction module is used to construct an error prediction model for the error between the joint angle vector and the virtual end-effector pose based on the real end-effector pose data of the main manipulator under multiple joint configurations and the virtual end-effector pose data under the corresponding joint configurations obtained through the virtual dual manipulator model; wherein, the real end-effector pose is the end-effector pose of the physical main manipulator, the virtual end-effector pose is the end-effector pose of the virtual main manipulator, and the virtual-real end-effector pose error is the deviation between the real end-effector pose and the virtual end-effector pose; The visual positioning module is used to acquire visual data of the work area containing the target bolt, so as to obtain the target pose information of the target bolt in the robot base coordinate system; The drive module is used to generate the target poses of the main robotic arm and the slave robotic arm based on the target pose information, and drive the two arms to move to the bolt assembly operation position. The control module is used to monitor the pose of the two arms in real time using the error prediction model. Based on the preset relative pose constraints between the end effectors of the master and slave robotic arms, it calculates the target pose of the slave robotic arm in real time and generates control commands through the digital twin data interaction channel to achieve pose synchronization and motion timing coordination of the two arms during the assembly operation. The preset relative pose constraints are constraints that describe the fixed relative spatial relationship between the end effectors of the master and slave robotic arms.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images