A double-arm robot wire arranging and buckling method based on multi-feature point recognition

Abstract

The application relates to a double-arm robot wire arranging and buckling method based on multi-feature point recognition, which comprises the following steps: collecting an RGB-D image of a complete work platform, acquiring reference frame feature points based on the RGB-D image of the complete work platform, and determining the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system; collecting an RGB-D image of a work scene, acquiring current frame feature points through the RGB-D image of the work scene, performing FLANN algorithm matching with the reference frame feature points, and obtaining the three-dimensional pose of the reference frame camera in the current camera coordinate system; applying a preset bias to the three-dimensional pose, acquiring the three-dimensional coordinates of the path points of a mechanical arm in the double-arm robot in the base coordinate system, and combining the current pose of the mechanical arm to perform a wire arranging and buckling operation. Through double-arm collaborative operation and multi-feature point data driving, the application realizes high precision and strong generalization capability of the wire arranging and buckling task.

Description

Technical Field

[0001] This invention relates to the field of automated robot assembly technology, and in particular to a method for cable fastening of a dual-arm robot based on multi-feature point recognition. Background Technology

[0002] In the manufacturing process of electronic products, the precise fastening of ribbon cables is a typical and critical assembly step. Currently, this process relies heavily on manual operation, resulting in low efficiency, poor consistency, and high costs. Although industrial robots have been used for automation in some cases, the following technical bottlenecks exist: First, ribbon cables are typically flexible, easily deformable, and small targets, making traditional pure vision methods insufficient in positioning accuracy under complex lighting and occlusion conditions; second, the fastening process involves delicate contact, and simple trajectory reproduction cannot adapt to minute pose changes, leading to a low success rate.

[0003] Existing automation solutions mostly use single-arm robots, which are limited in their capabilities when performing complex assembly tasks such as cable fastening that require simultaneous positioning and pressing of the workpiece. Single-arm robots cannot simulate the collaborative operation mode of two hands applying pressure to different parts of the body during operation, resulting in unstable processes or low success rates.

[0004] Therefore, there is an urgent need for a dual-arm robot cable fastening method based on multi-feature point recognition, which can use a dual-arm robot to simulate the collaborative operation of human hands and provide a cable fastening solution that can quickly adapt to different task scenarios. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide a dual-arm robot cable fastening method based on multi-feature point recognition. Through dual-arm collaborative operation and multi-feature point data-driven operation, the method achieves high precision and strong generalization ability in cable fastening tasks.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for cable fastening in a dual-arm robot based on multi-feature point recognition, comprising: Acquire RGB-D images of the complete working platform, and based on the RGB-D images of the complete working platform, obtain reference frame feature points to determine the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system; Acquire RGB-D images of the work scene, obtain feature points of the current frame from the RGB-D images of the work scene, and use them to perform FLANN algorithm matching with the feature points of the reference frame to obtain the three-dimensional pose of the reference frame camera in the current camera coordinate system. A preset bias is applied to the three-dimensional pose to obtain the three-dimensional coordinates of the path point of the robotic arm in the coordinate system of the robotic arm end effector. The three-dimensional coordinates of the robotic arm in the base coordinate system are obtained by coordinate transformation in combination with the current pose of the robotic arm, and the wiring and fastening operation is performed.

[0007] Optionally, obtaining the reference frame feature points includes: The depth camera in the dual-arm robot is used to acquire RGB-D images of the complete working platform. The SuperPoint algorithm based on deep learning is used to identify the working scene in the RGB-D images of the complete working platform and obtain the reference frame feature points.

[0008] Optionally, determining the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system includes: A perspective transformation is performed on the feature points of the reference frame, combining the depth information and camera intrinsic parameter matrix of the depth camera, to obtain the three-dimensional coordinates of the feature points in the depth camera coordinate system of the reference frame: ; in, The 3D coordinates of the feature points in the depth camera coordinate system of the reference frame. For depth information, For the camera intrinsic parameter matrix, The image pixel coordinates are the center of the bounding box.

[0009] Optionally, obtaining the current frame feature points includes: The depth camera in the dual-arm robot is used to acquire RGB-D images of the work scene. The SuperPoint algorithm based on deep learning is used to identify the current work scene in the RGB-D images of the work scene and obtain the feature points of the current frame.

[0010] Optionally, obtaining the three-dimensional pose of the reference frame camera in the current camera coordinate system includes: The current frame feature points are matched with the reference frame feature points using the FLANN algorithm to obtain the three-dimensional coordinates of the matching points and the reference frame depth camera coordinate system. The three-dimensional pose of the reference frame camera in the current camera coordinate system is calculated by combining the camera intrinsic parameter matrix and the two-dimensional coordinates of the matching points in the current depth camera image.

[0011] Optionally, obtaining the three-dimensional coordinates of the path points of the robotic arm in the dual-arm robot in the base coordinate system includes: A preset bias is applied to the three-dimensional pose to obtain the three-dimensional coordinates of the path points of the robotic arm in the current camera coordinate system. The three-dimensional coordinates of the path points in the current camera coordinate system are then transformed to the robot base coordinate system using a hand-eye calibration matrix to obtain the three-dimensional coordinates of the path points in the base coordinate system.

[0012] Optionally, obtaining the three-dimensional coordinates of the path points in the base coordinate system includes: ; in, These are the 3D coordinates of the path points in the current camera coordinate system. For hand-eye calibration matrix, For preset bias, The coordinates of the path point are in the base coordinate system.

[0013] Optionally, the dual-arm robot includes: a robotic arm, a depth camera is provided at the wrist of the robotic arm for acquiring RGB-D images, and a cotton swab is provided on the robotic arm for contacting the working surface with the cotton swab and continuously pressing it down during the ribbon cable fastening operation, so that the lower ribbon cable moves until it is aligned with the position of the upper ribbon cable. The robotic arm includes a left robotic arm and a right robotic arm.

[0014] The beneficial effects of this invention are as follows: Data-driven and highly generalizable: This invention uses a multimodal dataset collected by a depth camera, which covers multiple dimensions such as vision and pose, providing rich information for target recognition and localization. This allows the fully autonomous model to quickly adapt when faced with new robotic arms or different types of cabling, simply by re-identifying the reference frame and fine-tuning the position offset, significantly improving the system's generalization ability.

[0015] Precise operation and high success rate: This invention combines multi-feature point recognition, FLANN matching algorithm and PnP algorithm to obtain target pose. The design of the latching action sequence conforms to the principles of human-computer interaction, ensuring the smoothness and precision of the latching process and improving the success rate and reliability of the task.

[0016] Dual-arm collaboration, human-like operation: This invention makes full use of the spatial collaboration capability of dual-arm robots. By designing the timing coordination of the left and right robotic arms, it accurately simulates the process of human technicians operating with both hands, solving the complex assembly problems that single-arm robots cannot complete, and significantly improving the stability and success rate of operation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a dual-arm robot cable fastening method based on multi-feature point recognition according to an embodiment of the present invention; Figure 2 This is a flowchart of the reference frame identification process according to an embodiment of the present invention; Figure 3 This is a flowchart of the positioning operation process according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the dual-arm coordinated fastening operation process according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, this embodiment discloses a method for cable fastening of a dual-arm robot based on multi-feature point recognition, including: acquiring an RGB-D image of the complete working platform; obtaining reference frame feature points based on the RGB-D image of the complete working platform to determine the three-dimensional coordinates of the feature points in the depth camera coordinate system of the reference frame; acquiring an RGB-D image of the working scene; obtaining the current frame feature points through the RGB-D image of the working scene, and performing FLANN algorithm matching with the reference frame feature points to obtain the three-dimensional pose of the reference frame camera in the current camera coordinate system; applying a preset bias to the three-dimensional pose to obtain the three-dimensional coordinates of the path points of the robotic arm in the dual-arm robot in the base coordinate system, and performing cable fastening operation in combination with the current pose of the robotic arm.

[0021] Further, determining the 3D coordinates of feature points in the reference frame depth camera coordinate system includes: acquiring RGB-D images of the complete working platform using the depth camera in the dual-arm robot; using the deep learning-based SuperPoint algorithm to identify the working scene in the RGB-D images of the complete working platform to obtain reference frame feature points; and performing perspective transformation on the reference frame feature points by combining the depth information and camera intrinsic parameter matrix to obtain the 3D coordinates of the feature points in the reference frame depth camera coordinate system.

[0022] Specifically, the left and right robotic arms are moved until the RGB-D image of the work scene acquired by the depth camera mounted on the robotic arm wrist completely covers the entire work platform without any redundant background information. The SuperPoint algorithm, based on deep learning, is used to identify the work scene, obtain feature points from the reference frame, and combine this with the depth information obtained from the depth camera. and camera intrinsic parameter matrix The three-dimensional coordinates of feature points in the depth camera coordinate system of the reference frame are obtained using the principle of perspective transformation. Its coordinate transformation formula is: ; in, The image pixel coordinates are the center of the bounding box. The 2D coordinates of the feature points, the descriptor, and the 3D coordinates of the feature points incorporating depth information are stored.

[0023] Furthermore, obtaining the 3D pose of the reference frame camera in the current camera coordinate system includes: acquiring RGB-D images of the work scene using the depth camera in the dual-arm robot; using the deep learning-based SuperPoint algorithm to identify the current work scene in the RGB-D image and obtain the feature points of the current frame; matching the feature points of the current frame with the feature points of the reference frame using the FLANN algorithm to obtain the matching points and the 3D coordinates of the reference frame depth camera in the coordinate system; and combining the camera intrinsic parameter matrix and the 2D coordinates of the matching points in the current depth camera image to calculate the 3D pose of the reference frame camera in the current camera coordinate system.

[0024] Specifically, RGB-D images of the work scene are acquired using a depth camera mounted on the wrist of the robotic arm. The SuperPoint algorithm, based on deep learning, is used to identify the work scene, obtaining feature points in the current frame. These feature points are then matched with feature points in a reference frame using the FLANN algorithm to obtain the matching points and their 3D coordinates in the depth camera coordinate system of the reference frame. This is then combined with the camera intrinsic parameter matrix. The two-dimensional coordinates of the matching point in the current depth camera image are used to calculate the three-dimensional pose of the reference frame camera in the current camera coordinate system using the PnP algorithm.

[0025] The latching operation involves transforming the three-dimensional pose of the reference frame camera in the current camera coordinate system to obtain the three-dimensional coordinates of the robotic arm path points in the base coordinate system. This allows the left and right robotic arms to move, enabling the latching tools installed at their respective ends to reach the corresponding positioning positions. The left and right robotic arms are then controlled to coordinately execute the latching action sequence.

[0026] Furthermore, obtaining the three-dimensional coordinates of the path points of the robotic arm in the dual-arm robot in the base coordinate system includes: applying a preset bias to the three-dimensional pose, obtaining the three-dimensional coordinates of the path points of the robotic arm in the current camera coordinate system, and transforming the three-dimensional coordinates of the path points in the current camera coordinate system to the robot base coordinate system through the hand-eye calibration matrix to obtain the three-dimensional coordinates of the path points in the base coordinate system.

[0027] Specifically, a preset bias is applied to the 3D pose of the reference frame camera in the current camera coordinate system to obtain the 3D coordinates of the path points of the robotic arm in the current camera coordinate system. Hand-eye calibration matrix This transforms the 3D coordinates of the path points to the robot's base coordinate system. The coordinate transformation formula is: ; Based on this pose, plan the movement of the robotic arms. Move the left and right robotic arms according to their respective poses until they reach the working plane. After both arms have completed their movements, press down the cotton swabs at the ends of both arms until the lower cable moves and aligns with the upper cable. After both arms have completed their movements, raise both arms to the working plane and return them to their predetermined positions.

[0028] Furthermore, the dual-arm robot includes: a robotic arm with a depth camera at its wrist for acquiring RGB-D images, and a cotton swab at its arm for contacting the work surface and continuously pressing down during the ribbon cable fastening operation to move the lower ribbon cable until it aligns with the upper ribbon cable; wherein the robotic arm includes: a left robotic arm and a right robotic arm.

[0029] Specifically, the dual-arm robot is executed by a dual-arm robot system, which includes left and right robotic arms, depth cameras mounted on the wrists of the robotic arms, and fastening tools made of cotton swabs. The method is characterized by first identifying feature points in a reference frame using the depth camera, then locating the arms and guiding them to a position close to the cable; subsequently, the fastening is completed fully autonomously: the system calls a pre-trained SuperPoint model to identify feature points in the current frame and autonomously controls the two arms to collaboratively execute the fastening action sequence. This invention achieves high precision and strong generalization capability in cable fastening tasks through dual-arm collaborative operation and multi-feature point data-driven operation.

[0030] The present invention also discloses that when the hardware configuration of the robot system or the type of the cable to be fastened changes, the task can be rapidly generalized by re-identifying the reference frame or by fine-tuning the positioning offset.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] like Figure 1 As shown, this embodiment discloses a cable fastening method for a dual-arm robot based on multi-feature point recognition, used in a dual-arm robot system. The method includes: S1. Perform reference frame recognition on the dual-arm robot system, wherein reference frame recognition includes feature point recognition and data storage based on the SuperPoint model; S2. After the reference frame is identified, a positioning operation is performed, which includes feature point identification, matching, and coordinate calculation. S3. Based on the calculated three-dimensional coordinates of the path points in the end-effector coordinate system and the current pose of both arms, the three-dimensional coordinates in the base coordinate system are obtained through coordinate transformation, and then the cable fastening operation is performed autonomously.

[0033] like Figure 2 As shown, the reference frame identification process specifically includes: S11. Connect the depth camera to the host computer via the data interface, configure the image acquisition parameters, and ensure that the camera has a stable power supply and normal communication. S12. Control the robotic arm to move to the predefined observation point and simultaneously acquire RGB-D image data output by the depth camera. Ensure that the target area for the wiring is covered by the field of vision; S13. Substitute the collected image data into the SuperPoint model based on deep learning, and calculate the pixel coordinates and descriptors of the feature points of the reference frame through forward inference. S14, Based on the pixel coordinates of the reference frame and the corresponding depth value The three-dimensional coordinates of feature points in the reference frame camera coordinate system are calculated using the principle of perspective transformation. : ; in, This is the camera intrinsic parameter matrix; S15. Store the pixel coordinates, descriptors, and 3D coordinates of the feature points in the reference frame.

[0034] like Figure 3 As shown, the positioning operation process specifically includes: S21. Control the robotic arm to move to the predefined observation point and simultaneously acquire RGB-D image data output by the depth camera; S22. Substitute the collected image data into the SuperPoint model based on deep learning, and calculate the pixel coordinates and descriptors of the feature points of the current frame through forward inference; S23. Match the feature points of the current frame with the feature points of the reference frame using the FLANN algorithm based on the descriptor; S24. Sort the matching points from high to low according to the matching degree, and take out the first 100 matching points; S25. Combining the pixel coordinates, 3D coordinates, and camera intrinsic parameters of 100 matching points, the pose of the reference frame camera in the current frame camera coordinate system is calculated using the PnP algorithm.

[0035] like Figure 4 As shown, the specific steps of the dual-arm coordinated fastening operation include: S31. Apply a preset bias to the pose estimation result of the reference frame camera in the current frame camera coordinate system to obtain the three-dimensional coordinates of the path points of the robotic arm in the current frame coordinate system. ; S32, Combine the path points in the current frame's camera coordinate system Hand-eye calibration matrix Perform a coordinate system transformation to obtain the three-dimensional coordinates of the path points in the base coordinate system: ; S33. Based on the three-dimensional coordinates of the path points in the base coordinate system, move the left and right robotic arms to complete the locking operation.

[0036] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0037] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for cable fastening in a dual-arm robot based on multi-feature point recognition, characterized in that, include: Acquire RGB-D images of the complete working platform, and based on the RGB-D images of the complete working platform, obtain reference frame feature points to determine the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system; Acquire RGB-D images of the work scene, obtain feature points of the current frame from the RGB-D images of the work scene, and use them to perform FLANN algorithm matching with the feature points of the reference frame to obtain the three-dimensional pose of the reference frame camera in the current camera coordinate system. Obtaining the current frame feature points includes: The depth camera in the dual-arm robot is used to acquire RGB-D images of the work scene, and the SuperPoint algorithm based on deep learning is used to identify the current work scene in the RGB-D images of the work scene to obtain the feature points of the current frame. Obtaining the three-dimensional pose of the reference frame camera in the current camera coordinate system includes: The current frame feature points are matched with the reference frame feature points using the FLANN algorithm to obtain the three-dimensional coordinates of the matching points and the reference frame depth camera coordinate system. The three-dimensional pose of the reference frame camera in the current camera coordinate system is calculated by combining the camera intrinsic parameter matrix and the two-dimensional coordinates of the matching points in the current depth camera image. A preset bias is applied to the three-dimensional pose to obtain the three-dimensional coordinates of the path point of the robotic arm in the coordinate system of the robotic arm end effector in the dual-arm robot. The three-dimensional coordinates of the robotic arm in the base coordinate system are obtained by coordinate transformation in combination with the current pose of the robotic arm, and the wiring fastening operation is performed. Obtaining the three-dimensional coordinates of the path points of the robotic arm in the base coordinate system includes: A preset bias is applied to the three-dimensional pose to obtain the three-dimensional coordinates of the path points of the robotic arm in the current camera coordinate system. The three-dimensional coordinates of the path points in the current camera coordinate system are then transformed to the robot base coordinate system using a hand-eye calibration matrix to obtain the three-dimensional coordinates of the path points in the base coordinate system.

2. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Obtaining the feature points of the reference frame includes: The depth camera in the dual-arm robot is used to acquire RGB-D images of the complete working platform. The SuperPoint algorithm based on deep learning is used to identify the working scene in the RGB-D images of the complete working platform and obtain the reference frame feature points.

3. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Determining the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system includes: A perspective transformation is performed on the feature points of the reference frame, combining the depth information and camera intrinsic parameter matrix of the depth camera, to obtain the three-dimensional coordinates of the feature points in the depth camera coordinate system of the reference frame: ； in, The three-dimensional coordinates of the feature points in the depth camera coordinate system of the reference frame. For depth information, For the camera intrinsic parameter matrix, The image pixel coordinates are the center of the bounding box.

4. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Obtaining the three-dimensional coordinates of the path points in the base coordinate system includes: ； in, These are the 3D coordinates of the path points in the current camera coordinate system. For hand-eye calibration matrix, For preset bias, The coordinates of the path point are in the base coordinate system.

5. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, The dual-arm robot includes: a robotic arm, a depth camera at the wrist of the robotic arm for acquiring RGB-D images, and a cotton swab at the robotic arm for contacting the working surface and continuously pressing down during the ribbon cable fastening operation, so that the lower ribbon cable moves until it is aligned with the position of the upper ribbon cable. The robotic arm includes a left robotic arm and a right robotic arm.

Images