A method for cable fastening in a dual-arm robot based on multimodal perception

Abstract

This invention relates to a method for cable fastening in a dual-arm robot based on multimodal perception, comprising: acquiring images of the work scene; identifying and locating the cables in the images; obtaining the bounding boxes of the cables in the image coordinate system; using these bounding boxes to plan the movement of the fastening tool at the end of the robotic arm to a preparatory position; and performing cable fastening in a semi-autonomous or fully autonomous mode. In the semi-autonomous mode, the method involves observing the image from a miniature camera within the fastening tool and mapping the pose of the robotic arm's end effector using a teleoperated handle to generate control commands for cable fastening. In the fully autonomous mode, the method involves inputting perceptual information into a vision-language-action model to obtain action commands for the robotic arm's end effector for cable fastening. The vision-language-action model is trained on a multimodal dataset. This invention overcomes the problems of reliance on manual labor, low automation, and poor adaptability in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of automated robot assembly technology, and in particular to a method for cable fastening of a dual-arm robot based on multimodal perception. Background Technology

[0002] In the manufacturing process of electronic products, the precise fastening of ribbon cables is a typical and critical assembly step. Currently, this process largely relies on manual operation under a microscope, resulting in low efficiency, poor consistency, and high costs. Although industrial robots have been used for automation in some cases, the following technical bottlenecks exist: First, ribbon cables are typically flexible, easily deformable, and small targets, making traditional pure vision methods insufficient in positioning accuracy under complex lighting and occlusion conditions; second, the fastening process involves delicate contact and force interactions, and simple trajectory reproduction cannot adapt to minute changes in pose and force, leading to low success rates.

[0003] While teleoperated robots can incorporate advanced human skills into operations, they still require human intervention for each task, hindering large-scale autonomous production. Robots relying entirely on pre-programmed code lack adaptability and generalization capabilities; changes in product models or robot hardware necessitate extensive and time-consuming reprogramming and debugging. Therefore, there is an urgent need for a wiring harness assembly solution that combines human expert experience with robot autonomy, enabling rapid adaptation to diverse task scenarios.

[0004] Existing automation solutions mostly employ single-arm robots, which are limited in their capabilities when performing complex assembly tasks such as cable fastening that require simultaneous positioning and pressing of the workpiece. Single-arm robots struggle to simulate the collaborative working mode of human hands—one hand fixing the work while the other applies pressure—leading to instability or low success rates. Therefore, utilizing dual-arm robots to simulate the collaborative work of human hands has become an effective approach to solving such precision assembly challenges. Summary of the Invention

[0005] To overcome the problems of reliance on manual labor, low automation, and poor adaptability in existing technologies, this invention provides a dual-arm robot cable fastening method based on multimodal perception.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for cable fastening in a dual-arm robot based on multimodal perception, comprising:

[0008] The system acquires images of the work scene, identifies and locates the wiring in the images, obtains the bounding box of the wiring in the image coordinate system, and plans the movement of the fastening tool at the end of the robotic arm in the dual-arm robot to the preparatory position. The system then performs wiring fastening in a semi-autonomous or fully autonomous mode.

[0009] Semi-autonomous mode is used for cable fastening: the image of the miniature camera in the fastening tool is observed, and the position of the end of the robotic arm is mapped by the remote control handle to generate control commands for cable fastening;

[0010] The fully autonomous mode is used for cable fastening: the perception information is input into the vision-language-action model to obtain the action commands of the robotic arm end effector for cable fastening; the vision-language-action model is obtained by training on a multimodal dataset.

[0011] Optionally, the dual-arm robot includes a left robotic arm and a right robotic arm, with a depth camera and a two-finger gripper respectively installed at the wrist and end of the left robotic arm and the right robotic arm, and a visual-tactile sensor and a fastening tool installed at the end of the two-finger gripper;

[0012] The fastening tool includes a miniature camera and a cotton swab.

[0013] Optionally, planning the movement of the latching tool at the end of the robotic arm to a preparatory position includes:

[0014] Based on the bounding box, combined with the camera intrinsic parameter matrix and depth information, the three-dimensional pose of the ribbon cable in the camera coordinate system is calculated.

[0015] The three-dimensional pose is converted to the robot's base coordinate system, and the joint angles of the robotic arm are calculated. A collision-free path is planned using the joint angles of the robotic arm, and the latching tool at the end of the robotic arm is moved to the preparatory position based on the collision-free path.

[0016] Optionally, the three-dimensional pose of the cable in the camera coordinate system is calculated:

[0017] ;

[0018] in, For the camera intrinsic parameter matrix, For depth information, The image pixel coordinates of the center of the bounding box. This is a transpose.

[0019] Optionally, generating the control commands includes:

[0020] The pose of the teleoperated handle is converted into the desired pose of the robotic arm's end effector through an adaptive mapping function. The inverse kinematics of the desired pose is then solved to generate the control commands.

[0021] ;

[0022] in, For the Jacobian matrix of the robotic arm, Here is the gain matrix. For the desired pose of the end effector of the dual-arm robot, The damping coefficient is... It is the identity matrix. This represents the current pose of the robotic arm's end effector.

[0023] Optionally, the desired pose of the robotic arm's end effector includes:

[0024] ;

[0025] in, The position of the remote control handle. This is the scaling factor.

[0026] Optionally, training the model using the multimodal dataset includes:

[0027] With the objective function of minimizing the model output and expert demonstration actions, the visual-language-action model to be trained is iteratively trained on the multimodal dataset using behavior cloning or offline reinforcement learning algorithms.

[0028] The multimodal dataset includes: joint encoder data, six-dimensional force data, miniature camera video stream, and tactile image sequence from visual-tactile sensors.

[0029] Optionally, the loss function during iterative training is:

[0030] ;

[0031] in, For parameterized policy models, Demonstrating the movements to experts For mathematical expectation, For the system's observation state, Provide a dataset for experts to demonstrate.

[0032] Optionally, during the cable fastening process in the fully autonomous mode, when the robotic arm comes into contact with the environment, impedance control is used to operate the robotic arm.

[0033] The control law for impedance control is:

[0034] ;

[0035] in, These are the desired pose, velocity, and acceleration. This is the actual value. These are the desired inertia, damping, and stiffness matrices, respectively.

[0036] The beneficial effects of this invention are as follows:

[0037] Flexible mode and improved efficiency: This invention proposes a workflow that combines semi-autonomous and fully autonomous modes. The semi-autonomous mode is used to solve unexpected problems or collect expert data, while the fully autonomous mode is used for efficient batch production, combining flexibility and efficiency.

[0038] Data-driven and highly generalizable: This invention uses a multimodal dataset collected through semi-autonomous operation, which covers multiple dimensions of information such as force, vision, and pose, to provide rich learning samples for the VLA model. This allows the fully autonomous model to quickly adapt to new robotic arms or different types of wiring with only a small amount of new data or even no new data, significantly improving the system's generalization ability.

[0039] Precise operation and high success rate: This invention combines macroscopic visual coarse positioning with microscopic force / visual feedback. The design of the fastening action sequence conforms to the principles of ergonomics, ensuring the smoothness and precision of the fastening process and improving the success rate and reliability of the task.

[0040] Dual-arm collaboration, human-like operation: This invention makes full use of the spatial collaboration capability of dual-arm robots. By designing the timing and force-position coordination of the left and right robotic arms, it accurately simulates the process of human technicians operating with both hands (such as aligning with one hand and pressing with the other), solving the complex assembly problem that single-arm robots cannot complete, and significantly improving the stability and success rate of operation. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a cable fastening method for a dual-arm robot based on multimodal perception, according to an embodiment of the present invention.

[0043] Figure 2 This is a flowchart of the coarse positioning operation of the dual-arm robot system according to an embodiment of the present invention;

[0044] Figure 3 This is a flowchart of the semi-autonomous mode of an embodiment of the present invention;

[0045] Figure 4 This is a flowchart of the fully autonomous mode of an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, this embodiment discloses a method for cable fastening of a dual-arm robot based on multimodal perception, including: acquiring images of the work scene, identifying and locating the cables in the images, obtaining the bounding boxes of the cables in the image coordinate system, which are used to plan the movement of the fastening tool at the end of the robotic arm in the dual-arm robot to a preparatory position, and performing cable fastening in a semi-autonomous or fully autonomous mode; performing cable fastening in a semi-autonomous mode: observing the image from the miniature camera in the fastening tool, and using a remote control to map the pose of the end of the robotic arm to generate control commands for cable fastening; performing cable fastening in a fully autonomous mode: inputting the perception information into a vision-language-action model to obtain the action commands of the end of the robotic arm for cable fastening; the vision-language-action model is obtained by training on a multimodal dataset.

[0049] Specifically, this embodiment discloses a cable fastening method for a dual-arm robot based on multimodal perception, including:

[0050] S1. Perform coarse localization on the dual-arm robot system, where coarse localization includes target recognition based on a depth camera and path planning for the robotic arm.

[0051] S2. After coarse positioning is completed, select the operation mode, which includes semi-autonomous mode and fully autonomous mode;

[0052] S3. Based on the selected mode, perform the wiring and fastening operation. In semi-autonomous mode, multimodal data is collected synchronously, and in fully autonomous mode, autonomous operation is performed based on the pre-trained vision-language-action model.

[0053] Furthermore, the dual-arm robot includes a left robotic arm and a right robotic arm, with a depth camera and a two-finger gripper respectively installed at the wrist and end of the left and right robotic arms, and a visual-tactile sensor and a fastening tool installed at the end of the two-finger gripper; wherein, the fastening tool includes a miniature camera and a cotton swab.

[0054] Specifically, the dual-arm robot includes: left and right robotic arms, a depth camera mounted on the wrist of each robotic arm, a two-finger gripper mounted on the end of each robotic arm, a visual-tactile sensor mounted on the end of the two-finger gripper, and a fastening tool consisting of a miniature camera and a cotton swab. First, the depth camera performs coarse identification, guiding the arms to a coarse positioning position close to the ribbon cable. Then, it can choose to enter semi-autonomous or fully autonomous mode to complete the fastening: In semi-autonomous mode, the operator observes the miniature camera image through a remote control handle, manually controlling the coordinated operation of the left and right robotic arms to complete a fine fastening action sequence, while simultaneously collecting a multimodal dataset; in fully autonomous mode, it invokes a vision-language-action model pre-trained based on the multimodal dataset to autonomously control the two arms to collaboratively execute the fastening action sequence.

[0055] Furthermore, planning the movement of the end-effector's latching tool to the preparatory position includes: calculating the 3D pose of the cable in the camera coordinate system based on the bounding box, camera intrinsic matrix, and depth information; converting the 3D pose to the robot base coordinate system and calculating the robot arm joint angles; planning a collision-free path using the robot arm joint angles; and moving the end-effector's latching tool to the preparatory position based on the collision-free path.

[0056] Specifically, such as Figure 2 As shown, the coarse positioning process specifically includes:

[0057] S11. Connect the depth camera to the host computer through the data interface, configure the image acquisition parameters, and ensure that the camera power supply is stable and the communication is normal. This is the initial state. Record the scene depth data when there is no robotic arm movement.

[0058] S12. Control the robotic arm to move to the predefined observation point and simultaneously acquire RGB-D image data output by the depth camera. Ensure that the target area for the wiring is covered by the field of vision;

[0059] S13. Substitute the collected image data into the deep learning-based target detection model, and calculate the bounding box parameters of the wire drawing target through forward inference. and confidence score, where x min The x-coordinate of the top-left corner of the bounding box is y. min x is the ordinate (pixel position) of the top-left corner of the bounding box. max The x-coordinate (pixel position) of the bottom right corner of the bounding box, y max This is the ordinate (pixel position) of the bottom right corner of the bounding box.

[0060] S14, Based on the center pixel coordinates of the bounding box and the corresponding depth value The three-dimensional coordinates of the cabochon in the camera coordinate system are calculated using the principle of perspective transformation. :

[0061] ;

[0062] in, This is the camera intrinsic parameter matrix;

[0063] S15. Using a hand-eye calibration matrix Will Transform to robot base coordinate system:

[0064] ;

[0065] S16, based on calculations The robot arm joint angles are solved using inverse kinematics algorithms, a collision-free path is planned, and the end effectors of both arms are controlled to move to a preparatory position approximately 10mm directly above the target point. .

[0066] Furthermore, generating control commands includes: converting the pose of the teleoperated handle into the desired pose of the robotic arm end effector through an adaptive mapping function, solving the inverse kinematics of the desired pose, and generating control commands.

[0067] Specifically, such as Figure 3 As shown, the semi-autonomous mode operation process specifically includes:

[0068] S21. The operator wears a virtual reality remote control terminal, the system establishes a communication connection with the controller, and completes spatial positioning calibration;

[0069] S22. Configure motion mapping parameters and set the scaling factor. Damping coefficient Control gain matrix ;

[0070] S23. The operator observes the real-time stereoscopic video stream transmitted by the miniature camera at the end of the robotic arm through a head-mounted display device, and maps the pose of the end of the robotic arm through the natural movements of the arm and wrist.

[0071] S24, Pose of the system acquisition controller (Frequency set to 100Hz), the adaptive mapping function is converted into the desired pose of the robot end effector. :

[0072] ;

[0073] S25. The robot control system generates joint control commands based on the desired pose through inverse kinematics solving:

[0074] ;

[0075] in, For the Jacobian matrix of the robotic arm;

[0076] S26. While executing the locking action sequence, the system records the joint encoder data completely using timestamp synchronization. Information from a six-dimensional force sensor installed at the end of the robotic arm, i.e., six-dimensional force data. Miniature camera video stream tactile image sequence of visual-tactile sensor Construct a multimodal dataset for model training. .

[0077] The joint encoder data recorded during model training With six-dimensional force data Its functions are as follows:

[0078] In semi-autonomous mode, operator telemetry generated an expert demonstration trajectory. Recorded joint encoder data. Differential calculations can yield joint velocities or pose changes of the end effector. These data directly serve as supervision signals for the VLA model during behavior clone training, representing the "expert actions" the model needs to learn. (Six-dimensional force data) This serves as an important supervisory signal for judging the contact state and adjusting the control strategy, especially when training the model to learn contact-rich tasks such as impedance control.

[0079] During training and inference, models typically require historical information over a certain time window to understand the current state and trends. Therefore, historical joint encoder data is essential. With six-dimensional force data It will be combined with the current image, segmentation mask, etc., to form the complete observation of the model at time t.

[0080] The specific steps involved in executing the locking action sequence include the following steps that utilize the coordination of both arms:

[0081] a. Control the left robotic arm so that the cotton swab at its end contacts the working surface and presses down, causing the lower cable to move until it is aligned with the position of the upper cable;

[0082] b. Control the right robotic arm so that the cotton swab at its end contacts the cotton swab of the first robotic arm;

[0083] c. Control the cotton swab of the right robotic arm to press down continuously until its deformation is observed, then control both robotic arms to press down simultaneously and maintain for a period of time;

[0084] d. Raise both arms to check the cable connection status; if the connection fails, repeat steps a to c.

[0085] In fully autonomous mode, when the hardware configuration of the robot system or the type of cable to be fastened changes, the task can be rapidly generalized by loading a pre-trained vision-language-action model corresponding to the current task, or by performing a small number of semi-autonomous operations to collect new data and fine-tune the model.

[0086] Furthermore, training the model using a multimodal dataset includes: using the minimization of the model output and expert demonstration actions as the objective function, iteratively training the visual-language-action model to be trained on the multimodal dataset through behavior cloning or offline reinforcement learning algorithms; the multimodal dataset includes: joint encoder data, six-dimensional force data, miniature camera video stream, and tactile image sequences from visual-tactile sensors; during the cable fastening process in fully autonomous mode, when the robotic arm comes into contact with the environment, impedance control is used to operate the robotic arm.

[0087] Specifically, such as Figure 4 As shown, the fully autonomous mode operation process specifically includes:

[0088] S31. Load the pre-trained visual-language-action model weights, initialize the model parameters. The model is based on the Transformer architecture and is trained on a multimodal dataset using the behavior cloning algorithm.

[0089] S32. Start the real-time sensing data stream for each frame of image. Semantic segmentation networks generate segmentation masks in real time. The network employs an encoder-decoder structure, and its loss function is the cross-entropy loss:

[0090] ;

[0091] in, It is a pixel Category The true label, It is the network's predicted probability. and Here, represents the length and width of the input image, respectively, and C represents the total number of semantic segmentation categories.

[0092] S33, The model receives the current observation. and historical context (where (representing the acquired six-dimensional force / torque vector), recursively outputting the action sequence. The system drives the dual arms to autonomously perform the fastening action, i.e., the system calls a multimodal dataset. The trained vision-language-action model is capable of perceiving information in real time. (such as images from miniature cameras) tactile images (etc.) are inputs, and the output is the motion command of the robotic arm's end effector. Its core decision-making process can be modeled as follows:

[0093] ;

[0094] S34. When the robotic arm comes into contact with the environment, impedance control is used to ensure the smoothness of operation. The control law is as follows:

[0095] ;

[0096] in, These are the desired pose, velocity, and acceleration. This is the actual value. These are the desired inertia, damping, and stiffness matrices, respectively.

[0097] S35. After the action is completed, the system automatically judges based on the segmentation result: if the overlap between the upper and lower ribbon cable masks exceeds 0.9 and the gold contacts of the ribbon cable are not displayed, the connection is considered successful.

[0098] In this embodiment, the multimodal dataset is:

[0099] ;

[0100] in, This represents the total number of samples collected.

[0101] In this embodiment, the training loss function of the vision-language-action model is:

[0102] ;

[0103] in, For parameterized policy models, Demonstrating the movements to experts.

[0104] This embodiment discloses a method for cable fastening in a dual-arm robot based on multimodal perception. Its core lies in constructing a closed-loop system of perception-decision-execution. The specific process is as follows:

[0105] Step 1: Coarse Localization Based on Visual Servo: Acquire RGB-D images of the work scene using a depth camera mounted on the robotic arm's wrist. Employ a deep learning-based object detection algorithm (such as YOLOv5) to identify and locate the ribbon cable, obtaining its bounding box in the image coordinate system. Combined with camera intrinsic parameter matrix and depth information The three-dimensional pose of the cabling in the camera coordinate system is calculated using the principle of perspective transformation. Its coordinate transformation formula is:

[0106] ;

[0107] in, The coordinates of the image pixels at the center of the bounding box are then used. Next, the pose is transformed to the robot's base coordinate system using a hand-eye calibration matrix. Based on this pose, the robotic arm's motion is planned, moving its end effector, carrying the fastening tool, to a preset observation position close to the ribbon cable, ensuring the miniature camera obtains a clear local view of the ribbon cable.

[0108] Step Two: Multi-Mode Activation Operation: This step offers two optional execution modes based on different decision-making mechanisms:

[0109] Semi-autonomous mode (based on human expert decision-making): The operator observes real-time first-person perspective stereoscopic images transmitted from a miniature camera at the end effector of the robotic arm through a head-mounted display device, and directly maps the pose of the robotic arm's end effector using a teleoperation handle. Control commands are generated in real time based on visual feedback and experience.

[0110] Specifically, the system acquires the pose of the controller in real time. (Including position and orientation), and through a scaling or adaptive mapping function. Convert it into the desired pose of the robot's end effector. :

[0111] ;

[0112] in, It is an adjustable scaling factor used to achieve high-precision micro-operations (such as large-range gestures corresponding to the movement of tiny robots). For other constraints (such as robot joint limits and obstacle avoidance), this mapping relationship ensures the intuitiveness and accuracy of the operation. Based on this desired pose, the robot control system generates joint control commands through inverse kinematics solutions or velocity control based on the Jacobian matrix.

[0113] ;

[0114] in, This is the control gain matrix. During this process, the system synchronously records multimodal data. ,in This is a sequence of joint poses. It is a six-dimensional force / torque sequence. For miniature camera video streams, We construct a dataset for subsequent model training from tactile image sequences from visual-tactile sensors.

[0115] Fully Autonomous Mode (VLA Model-Based Decision Making): System calls based on multimodal datasets The trained vision-language-action model. This model perceives information in real time. (such as images from miniature cameras) tactile images (etc.) are inputs, and the output is the motion command of the robotic arm's end effector. Its core decision-making process can be modeled as follows:

[0116] ;

[0117] in, For a parameterized policy model (based on the Transformer architecture), minimize its output and the expert demonstration actions through behavior cloning or offline reinforcement learning algorithms. The difference is used for training, and its loss function is... It can be represented as:

[0118] ;

[0119] In fully autonomous mode, the model needs to access images from a miniature camera. The precise outline of the ribbon cable is segmented from the data. This is achieved through a semantic segmentation network with an encoder-decoder structure, which predicts a category label for each pixel. (e.g., background, top line, bottom line). The network output is a segmentation mask. Its loss function is the cross-entropy loss:

[0120] ;

[0121] in, It is a pixel Category The true label, This is the network's predicted probability. (Segmentation result) With force information Together, they serve as state inputs, guiding the model to generate precise and smooth latching actions.

[0122] The dual-arm collaborative strategy is the core of the execution layer of this invention. During the locking process, impedance control is used to ensure operational smoothness when the robotic arm comes into contact with the environment. Its control law is:

[0123] ;

[0124] in, These are the desired pose, velocity, and acceleration. This is the actual value. These are the desired inertia, damping, and stiffness matrices, respectively. By adjusting... and It can control the stiffness of the robot during contact, with the left arm responsible for precise positioning and alignment (higher stiffness) and the right arm responsible for compliant downward pressure (lower stiffness), thus achieving safe and efficient dual-arm collaborative operation.

[0125] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0126] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for cable fastening in a dual-arm robot based on multimodal perception, characterized in that, include: Acquire images of the work scene, identify and locate the wiring in the images, obtain the bounding box of the wiring in the image coordinate system, and use it to plan the movement of the fastening tool at the end of the robotic arm in the dual-arm robot to the preparatory position. Select the operation mode, which includes: semi-autonomous mode or fully autonomous mode, and use semi-autonomous mode or fully autonomous mode to fasten the wiring. Semi-autonomous mode is used for cable fastening: the image of the miniature camera in the fastening tool is observed, and the position of the end of the robotic arm is mapped by the remote control handle to generate control commands for cable fastening; The fully autonomous mode is used for cable fastening: the perception information is input into the vision-language-action model to obtain the action commands of the robotic arm end effector for cable fastening; the vision-language-action model is obtained by training on a multimodal dataset; Training the model using the multimodal dataset includes: With the objective function of minimizing the model output and expert demonstration actions, the visual-language-action model to be trained is iteratively trained on the multimodal dataset using behavior cloning or offline reinforcement learning algorithms. The multimodal dataset includes: joint encoder data, six-dimensional force data, miniature camera video stream, and tactile image sequence from visual-tactile sensors; The loss function during iterative training is: ； in, For parameterized policy models, Demonstrating the movements to experts For mathematical expectation, For the system's observation state, Provide a dataset for experts to demonstrate.

2. The method for cable fastening of a dual-arm robot based on multimodal perception according to claim 1, characterized in that, The dual-arm robot includes a left robotic arm and a right robotic arm, with a depth camera and a two-finger gripper respectively installed at the wrist and end of the left and right robotic arms, and a visual-tactile sensor and a fastening tool installed at the end of the two-finger gripper. The fastening tool includes a miniature camera and a cotton swab.

3. The method for cable fastening of a dual-arm robot based on multimodal perception according to claim 1, characterized in that, Planning the movement of the latching tool at the end of the robotic arm to the preparatory position includes: Based on the bounding box, combined with the camera intrinsic parameter matrix and depth information, the three-dimensional pose of the ribbon cable in the camera coordinate system is calculated. The three-dimensional pose is converted to the robot's base coordinate system, and the joint angles of the robotic arm are calculated. A collision-free path is planned using the joint angles of the robotic arm, and the latching tool at the end of the robotic arm is moved to the preparatory position based on the collision-free path.

4. The method for cable fastening of a dual-arm robot based on multimodal perception according to claim 3, characterized in that, Calculate the 3D pose of the cable in the camera coordinate system: ； in, For the camera intrinsic parameter matrix, For depth information, The image pixel coordinates of the center of the bounding box. This is a transpose.

5. The method for cable fastening of a dual-arm robot based on multimodal perception according to claim 1, characterized in that, Generating the control commands includes: Based on the pose of the teleoperated handle, the desired pose of the robotic arm's end effector is converted through an adaptive mapping function. The inverse kinematics of the desired pose are then solved to generate the control commands. ； in, For the Jacobian matrix of the robotic arm, Here is the gain matrix. For the desired pose of the end effector of the dual-arm robot, The damping coefficient is... It is the identity matrix. This represents the current pose of the robotic arm's end effector.

6. The method for cable fastening of a dual-arm robot based on multimodal perception according to claim 5, characterized in that, The desired pose of the robotic arm's end effector includes: ； in, The position of the remote control handle. This is the scaling factor.

7. The method for cable fastening of a dual-arm robot based on multimodal perception according to claim 1, characterized in that, During the cable fastening process in the fully autonomous mode, when the robotic arm comes into contact with the environment, impedance control is used to control the operation of the robotic arm. The control law for impedance control is: ； in, These are the desired pose, velocity, and acceleration. This is the actual value. These are the desired inertia, damping, and stiffness matrices, respectively.

Images