Double-arm robot flat cable buckling method based on multi-feature point recognition

Abstract

The invention relates to a double-arm robot flat cable buckling method based on multi-feature-point identification, and the method comprises the steps: collecting an RGB-D image of a complete work platform, and obtaining reference frame feature points based on the RGB-D image of the complete work platform, so as to determine the three-dimensional coordinates of the feature points under a reference frame depth camera coordinate system; acquiring an RGB-D image of the operation scene, and acquiring a current frame feature point through the RGB-D image of the operation scene for performing FLANN algorithm matching with the reference frame feature point to obtain a three-dimensional pose of the reference frame camera in a current camera coordinate system; and preset offset is applied to the three-dimensional pose, a path point three-dimensional coordinate of a mechanical arm in the double-arm robot under the base coordinate system is obtained, and flat cable buckling operation is executed in combination with the current pose of the mechanical arm. According to the invention, through double-arm cooperative operation and multi-feature-point data driving, high precision and strong generalization ability of a flat cable buckling task are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot automation, in particular to a double-arm robot wire arranging and buckling method based on multi-feature point recognition. BACKGROUND

[0002] In the manufacturing process of electronic products, precise buckling of the wire is a typical and key assembly process. At present, this process mostly relies on manual operation, which has problems such as low efficiency, poor consistency, and high cost. Although industrial robots are tried in some cases for automation, the following technical bottlenecks are faced: first, the wire is usually flexible, easily deformed, and small in size, and the positioning accuracy of traditional pure visual methods is insufficient under complex lighting and occlusion; second, the buckling process involves fine contact, and simple trajectory reproduction cannot adapt to small pose changes, resulting in low success rate.

[0003] The existing automation scheme mostly uses a single-arm robot, which is limited in capability when performing complex assembly tasks such as wire buckling that require positioning and pressing of the operation object at the same time. The single-arm robot is difficult to simulate the cooperative operation mode of two hands at different parts of the human hand during operation, resulting in unstable process or low success rate.

[0004] Therefore, there is an urgent need for a double-arm robot wire arranging and buckling method based on multi-feature point recognition, which simulates the cooperative operation of human hands using a double-arm robot, and can quickly adapt to different task scenarios of the wire arranging and buckling solution. SUMMARY

[0005] In order to solve the problems existing in the prior art, the purpose of the present application is to provide a double-arm robot wire arranging and buckling method based on multi-feature point recognition, which realizes high precision and strong generalization ability of the wire arranging and buckling task through double-arm cooperative operation and multi-feature point data driving.

[0006] To achieve the above purpose, the present application provides the following scheme: A double-arm robot wire arranging and buckling method based on multi-feature point recognition, comprising: acquiring a reference frame feature point based on the RGB-D image of the complete work platform, for determining the three-dimensional coordinates of the feature point in the reference frame depth camera coordinate system; acquiring a current frame feature point through the RGB-D image of the work scene, for FLANN algorithm matching with the reference frame feature point, to obtain the three-dimensional pose of the reference frame camera in the current camera coordinate system; A preset bias is applied to the three-dimensional pose, three-dimensional coordinates of a path point of a mechanical arm in the dual-arm robot in a mechanical arm end coordinate system are obtained, and three-dimensional coordinates of the mechanical arm in a base coordinate system are obtained through coordinate transformation in combination with a current pose of the mechanical arm, and a wire arranging and buckling operation is performed.

[0007] Optionally, obtaining the reference frame feature points comprises: An RGB-D image of the complete work platform is collected by using a depth camera in the dual-arm robot, a work scene in the RGB-D image of the complete work platform is identified by using a SuperPoint algorithm based on deep learning, and the reference frame feature points are obtained.

[0008] Optionally, determining the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system comprises: The three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system are obtained through perspective transformation on the reference frame feature points in combination with depth information and a camera intrinsic matrix of the depth camera. ; wherein, is the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system, is the depth information, is the camera intrinsic matrix, is an image pixel coordinate of a center of the bounding box.

[0009] Optionally, obtaining the current frame feature points comprises: An RGB-D image of the work scene is collected by using a depth camera in the dual-arm robot, a current work scene in the RGB-D image of the work scene is identified by using a SuperPoint algorithm based on deep learning, and the current frame feature points are obtained.

[0010] Optionally, obtaining the three-dimensional pose of the reference frame camera in the current camera coordinate system comprises: The current frame feature points are matched with the reference frame feature points by using a FLANN algorithm, matched points and three-dimensional coordinates in a reference frame depth camera coordinate system are obtained, and a three-dimensional pose of the reference frame camera in a current camera coordinate system is calculated in combination with a camera intrinsic matrix and two-dimensional coordinates of the matched points in a current depth camera image.

[0011] Optionally, obtaining the three-dimensional coordinates of the path point of the mechanical arm in the dual-arm robot in the base coordinate system comprises: A preset bias is applied to the three-dimensional pose, path point three-dimensional coordinates of the mechanical arm in the current camera coordinate system are obtained, and the path point three-dimensional coordinates in the current camera coordinate system are converted to a robot base coordinate system through a hand-eye calibration matrix to obtain path point three-dimensional coordinates in the base coordinate system.

[0012] Optionally, obtaining the path point three-dimensional coordinates in the base coordinate system comprises: ; wherein, is the path point three-dimensional coordinates in the current camera coordinate system, is the hand-eye calibration matrix, is the preset bias, is the path point three-dimensional coordinates in the base coordinate system.

[0013] Optionally, the dual-arm robot comprises a mechanical arm, a depth camera is arranged at a wrist of the mechanical arm to collect an RGB-D image, and a cotton swab is arranged on the mechanical arm to contact a work plane and continuously press down when performing a wire clamping operation, so that the lower wire moves until it is aligned with the upper wire. wherein the mechanical arm comprises a left mechanical arm and a right mechanical arm.

[0014] The present application has the following advantages: Data-driven and strong generalization: the multi-modal data set collected by the depth camera covers multi-dimensional information such as vision and pose, providing rich information for target recognition and positioning, so that the fully autonomous model can quickly adapt to new mechanical arms or different types of wires by only re-identifying the reference frame and fine-tuning the position bias, significantly improving the generalization ability of the system.

[0015] Fine operation and high success rate: the present application combines multi-feature point recognition, FLANN matching algorithm and PnP algorithm to obtain the target pose, and the clamping action sequence design conforms to the principle of ergonomics, ensuring the flexibility and accuracy of the clamping process and improving the success rate and reliability of the task.

[0016] Dual-arm cooperation and human-like operation: the present application makes full use of the spatial cooperation capability of the dual-arm robot, accurately simulates the process of human technical personnel operating with both hands by designing the timing coordination action of the left and right mechanical arms, solves the complex assembly problem that cannot be completed by a single-arm robot, and significantly improves the stability and success rate of the operation. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only illustrate some of the embodiments of the present application, and other drawings can be obtained by those of ordinary skill in the art without any creative effort based on these drawings.

[0018] Figure 1 A flow chart of a double-arm robot wire arranging and buckling method based on multi-feature point recognition according to an embodiment of the present application; Figure 2 A flow chart of a reference frame recognition process according to an embodiment of the present application; Figure 3 A flow chart of a positioning operation process according to an embodiment of the present application; Figure 4 A flow chart of a double-arm cooperative buckling operation process according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the scope of the present application.

[0020] As shown in Figure 1 the present embodiment discloses a double-arm robot wire arranging and buckling method based on multi-feature point recognition, comprising: collecting an RGB-D image of a complete work platform, based on the RGB-D image of the complete work platform, obtaining reference frame feature points for determining feature point three-dimensional coordinates in a reference frame depth camera coordinate system; collecting an RGB-D image of a work scene, obtaining current frame feature points through the RGB-D image of the work scene, for FLANN algorithm matching with the reference frame feature points to obtain three-dimensional pose of the reference frame camera in the current camera coordinate system; applying a preset bias to the three-dimensional pose to obtain path point three-dimensional coordinates of a mechanical arm in the double-arm robot in the base coordinate system, and combining the current pose of the mechanical arm to perform wire arranging and buckling operation.

[0021] Further, determining the feature point three-dimensional coordinates in the reference frame depth camera coordinate system comprises: collecting the RGB-D image of the complete work platform by using a depth camera in the double-arm robot, identifying the work scene in the RGB-D image of the complete work platform by using a SuperPoint algorithm based on deep learning to obtain the reference frame feature points. Perspective transformation is performed on the reference frame feature points in combination with depth information of the depth camera and an intrinsic parameter matrix of the camera to obtain the feature point three-dimensional coordinates in the reference frame depth camera coordinate system.

[0022] Specifically, the left and right mechanical arms are moved until the RGB-D image of the work scene acquired by the depth camera installed on the wrist of the mechanical arm can completely cover the entire work platform without redundant irrelevant background information. The work scene is identified by using a SuperPoint algorithm based on deep learning to obtain the feature points of the reference frame, and the depth information obtained by the depth camera is combined with the intrinsic matrix of the camera and the camera intrinsic matrix The three-dimensional coordinates of the feature points in the depth camera coordinate system of the reference frame are obtained by perspective transformation principle The coordinate transformation formula is: ; wherein, is the image pixel coordinate of the center of the bounding box. The two-dimensional coordinates of the feature points, the descriptor, and the three-dimensional coordinates of the feature points combined with the depth information are stored.

[0023] Further, obtaining the three-dimensional pose of the reference frame camera in the current camera coordinate system comprises: collecting the RGB-D image of the work scene by using the depth camera in the dual-arm robot, identifying the current work scene in the RGB-D image of the work scene by using a SuperPoint algorithm based on deep learning to obtain the feature points of the current frame. The feature points of the current frame are matched with the feature points of the reference frame by using the FLANN algorithm to obtain the matching points and the three-dimensional coordinates in the depth camera coordinate system of the reference frame, and the three-dimensional pose of the reference frame camera in the current camera coordinate system is calculated by combining the camera intrinsic matrix and the two-dimensional coordinates of the matching points in the current depth camera image.

[0024] Specifically, the RGB-D image of the work scene is acquired by the depth camera installed on the wrist of the mechanical arm. The work scene is identified by using a SuperPoint algorithm based on deep learning to obtain the feature points of the current frame, and the feature points of the current frame are matched with the feature points of the reference frame by using the FLANN algorithm to obtain the matching points and the three-dimensional coordinates in the depth camera coordinate system of the reference frame, and the three-dimensional pose of the reference frame camera in the current camera coordinate system is calculated by combining the camera intrinsic matrix and the two-dimensional coordinates of the matching points in the current depth camera image by using the PnP algorithm.

[0025] Clamping operation: the three-dimensional pose of the reference frame camera in the current camera coordinate system is converted to obtain the three-dimensional coordinates of the mechanical arm path points in the base coordinate system, the left and right mechanical arms are controlled to move so that the clamping tools respectively installed at the ends thereof reach the corresponding positioning positions, and the left and right mechanical arms are controlled to cooperatively execute the clamping action sequence.

[0026] Further, the method for obtaining the path point three-dimensional coordinates of the mechanical arm in the dual-arm robot in the base coordinate system comprises: applying a preset bias to the three-dimensional pose, obtaining the path point three-dimensional coordinates of the mechanical arm in the current camera coordinate system, and converting the path point three-dimensional coordinates in the current camera coordinate system to the robot base coordinate system through the hand-eye calibration matrix to obtain the path point three-dimensional coordinates in the base coordinate system.

[0027] Specifically, a preset bias is applied to the three-dimensional pose of the reference frame camera in the current camera coordinate system to obtain the path point three-dimensional coordinates of the mechanical arm in the current camera coordinate system , and the path point three-dimensional coordinates are converted to the robot base coordinate system through the hand-eye calibration matrix . The coordinate transformation formula is: ; Based on the pose, the movement of the mechanical arm is planned. The left and right mechanical arms are moved according to the respective obtained poses until the working plane. After the dual arms complete the action, the cotton swabs at the ends of the dual arms are pressed down until the lower row of wires moves until aligned with the position of the upper row of wires. After the dual arms complete the action, the dual arms are lifted to the working plane, and the dual arms return to the predetermined positions.

[0028] Further, the dual-arm robot comprises: a mechanical arm, a depth camera arranged at the wrist of the mechanical arm and used for collecting an RGB-D image, and a cotton swab arranged at the mechanical arm and used for contacting the working plane and continuously pressing down when performing a row of wire buckling operation, so that the lower row of wires moves until aligned with the position of the upper row of wires. The mechanical arm comprises: a left mechanical arm and a right mechanical arm.

[0029] Specifically, the dual-arm robot is executed by a set of dual-arm robot system, which comprises left and right mechanical arms, a depth camera installed at the wrist of the mechanical arm, and a buckling tool composed of a cotton swab. The method is characterized in that: first, the reference frame feature point recognition is performed through the depth camera, then the positioning is performed, and the dual arms are guided to move to the positioning position close to the row of wires; subsequently, the buckling is completed autonomously: the system calls the pre-trained SuperPoint model to perform the current frame feature point recognition, and autonomously controls the dual arms to cooperatively execute the buckling action sequence. Through the dual-arm cooperative operation and the multi-feature point data driving, the high precision and strong generalization ability of the row of wire buckling task are realized.

[0030] The application also discloses that when the hardware configuration of the robot system or the type of the row of wires to be buckled changes, the task is quickly generalized through the re-recognition of the reference frame or through the fine adjustment of the positioning position bias.

[0031] In order to make the above-mentioned purposes, characteristics and advantages of the application more obvious and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0032] As Figure 1 shown, the embodiment discloses a double-arm robot wire arranging and buckling method based on multi-feature point recognition, which is used for a double-arm robot system, and the method comprises the following steps: S1, performing a reference frame recognition operation on the double-arm robot system, wherein the reference frame recognition comprises feature point recognition based on a SuperPoint model and data storage; S2, after the reference frame recognition is completed, performing a positioning operation, wherein the positioning comprises feature point recognition, matching and coordinate calculation; S3, obtaining three-dimensional coordinates in the base coordinate system according to the calculated path point three-dimensional coordinates in the mechanical arm end coordinate system and the current pose of the double arm through coordinate transformation, and then autonomously performing a wire arranging and buckling operation.

[0033] As Figure 2 shown, the reference frame recognition process specifically comprises the following steps: S11, connecting a depth camera to an upper computer through a data interface, configuring image acquisition parameters, and ensuring stable power supply and normal communication of the camera; S12, controlling the mechanical arm to move to a predefined observation point, and synchronously acquiring RGB-D image data output by the depth camera , ensuring that the field of view covers the wire target area; S13, substituting the acquired image data into a SuperPoint model based on deep learning, and calculating pixel coordinates and descriptors of reference frame feature points through forward reasoning; S14, calculating three-dimensional coordinates of the feature points in the reference frame camera coordinate system based on the reference frame pixel coordinates and corresponding depth values through perspective transformation principle: ; wherein, is an internal parameter matrix of the camera; S15, storing the pixel coordinates, descriptors and three-dimensional coordinates of the reference frame feature points.

[0034] As Figure 3 shown, the positioning operation process specifically comprises the following steps: S21, controlling the mechanical arm to move to a predefined observation point, and synchronously acquiring RGB-D image data output by the depth camera; S22, substituting the acquired image data into a SuperPoint model based on deep learning, and calculating pixel coordinates and descriptors of current frame feature points through forward reasoning; S23, matching the current frame feature points and the reference frame feature points through a FLANN algorithm according to the descriptors; S24, the matching points are sorted according to the matching degree from high to low, and the first 100 matching points are taken out; S25, the pixel coordinates, three-dimensional coordinates and camera intrinsic parameters of the 100 matching points are combined, and the pose of the reference frame camera in the current frame camera coordinate system is calculated through the PnP algorithm.

[0035] As shown in Figure 4 , the two-arm cooperative buckling operation process specifically includes: S31, a preset bias is applied to the pose estimation result of the reference frame camera in the current frame camera coordinate system, to obtain the path point three-dimensional coordinates of the mechanical arm in the current frame coordinate system ; S32, combined with the path point and the hand-eye calibration matrix , coordinate system conversion is performed to obtain the path point three-dimensional coordinates in the base coordinate system: ; S33, according to the path point three-dimensional coordinates in the base coordinate system, the left and right mechanical arms are moved to complete the buckling operation.

[0036] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0037] The above-described embodiments are only descriptions of the preferred modes of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A method for cable fastening in a dual-arm robot based on multi-feature point recognition, characterized in that, include: Acquire RGB-D images of the complete working platform, and based on the RGB-D images of the complete working platform, obtain reference frame feature points to determine the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system; Acquire RGB-D images of the work scene, obtain feature points of the current frame from the RGB-D images of the work scene, and use them to perform FLANN algorithm matching with the feature points of the reference frame to obtain the three-dimensional pose of the reference frame camera in the current camera coordinate system. A preset bias is applied to the three-dimensional pose to obtain the three-dimensional coordinates of the path point of the robotic arm in the coordinate system of the robotic arm end effector. The three-dimensional coordinates of the robotic arm in the base coordinate system are obtained by coordinate transformation in combination with the current pose of the robotic arm, and the wiring and fastening operation is performed.

2. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Obtaining the feature points of the reference frame includes: The depth camera in the dual-arm robot is used to acquire RGB-D images of the complete working platform. The SuperPoint algorithm based on deep learning is used to identify the working scene in the RGB-D images of the complete working platform and obtain the reference frame feature points.

3. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Determining the three-dimensional coordinates of the feature points in the reference frame depth camera coordinate system includes: A perspective transformation is performed on the feature points of the reference frame, combining the depth information and camera intrinsic parameter matrix of the depth camera, to obtain the three-dimensional coordinates of the feature points in the depth camera coordinate system of the reference frame: ； in, The 3D coordinates of the feature points in the depth camera coordinate system of the reference frame. For depth information, For the camera intrinsic parameter matrix, The image pixel coordinates are the center of the bounding box.

4. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Obtaining the current frame feature points includes: The depth camera in the dual-arm robot is used to acquire RGB-D images of the work scene. The SuperPoint algorithm based on deep learning is used to identify the current work scene in the RGB-D images of the work scene and obtain the feature points of the current frame.

5. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Obtaining the 3D pose of the reference frame camera in the current camera coordinate system includes: The current frame feature points are matched with the reference frame feature points using the FLANN algorithm to obtain the three-dimensional coordinates of the matching points and the reference frame depth camera coordinate system. The three-dimensional pose of the reference frame camera in the current camera coordinate system is calculated by combining the camera intrinsic parameter matrix and the two-dimensional coordinates of the matching points in the current depth camera image.

6. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, Obtaining the three-dimensional coordinates of the path points of the robotic arm in the base coordinate system includes: A preset bias is applied to the three-dimensional pose to obtain the three-dimensional coordinates of the path points of the robotic arm in the current camera coordinate system. The three-dimensional coordinates of the path points in the current camera coordinate system are then transformed to the robot base coordinate system using a hand-eye calibration matrix to obtain the three-dimensional coordinates of the path points in the base coordinate system.

7. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 6, characterized in that, Obtaining the three-dimensional coordinates of the path points in the base coordinate system includes: ； in, These are the 3D coordinates of the path points in the current camera coordinate system. For hand-eye calibration matrix, For preset bias, The coordinates of the path point are in the base coordinate system.

8. The method for cable fastening of a dual-arm robot based on multi-feature point recognition according to claim 1, characterized in that, The dual-arm robot includes: a robotic arm, a depth camera at the wrist of the robotic arm for acquiring RGB-D images, and a cotton swab at the robotic arm for contacting the working surface and continuously pressing down during the ribbon cable fastening operation, so that the lower ribbon cable moves until it is aligned with the position of the upper ribbon cable. The robotic arm includes a left robotic arm and a right robotic arm.

Images