Collaborative robot disassembly and assembly target accurate positioning method based on depth vision

Abstract

The invention discloses a depth-vision-based precise positioning method for a dismounting target of a collaborative robot, and the method comprises the steps: collecting data through a depth camera module, and carrying out the recognition and tracking of the collaborative robot; based on the depth data of the depth camera module, feeding back the current coordinate of the collaborative robot in real time, and performing target identification and positioning on the to-be-disassembled object by using the depth camera module; based on the distance between the target and the depth camera, the position of the robot is adjusted, and the to-be-disassembled object enters the mechanical arm working space; when the object to be disassembled and assembled is located in the working space of the mechanical arm, the mechanical arm is controlled to move the end executor to the corresponding position based on the coordinates of the object to be disassembled and assembled, target recognition positioning is conducted on the disassembling and assembling point through a common camera module on the end executor, and the accurate position of the disassembling and assembling point is determined; and controlling the collaborative robot to carry out target disassembly and assembly work. According to the method, the accurate positions of the to-be-dismounted object and the dismounting point in the complex environment can be rapidly determined, and a collaborative robot can be conveniently deployed for dismounting work.

Description

Technical Field

[0001] The present invention relates to the field of robot target recognition and positioning, and in particular to a method for accurately positioning a collaborative robot disassembly and assembly target based on depth vision. Background Art

[0002] In complex working environments, the collaborative robot has low positioning accuracy for the disassembly and assembly points of disassembly and assembly targets due to problems such as object occlusion, light changes, and small feature points. Existing single-depth cameras often have problems with low efficiency and poor positioning accuracy when detecting small targets. Although multi-angle binocular cameras can provide higher accuracy, they are difficult to flexibly deploy on collaborative robots, and the registration process of binocular cameras is complex and time-consuming in complex scenes. Therefore, existing technologies cannot meet the needs of collaborative robots to achieve fast, effective and precise positioning of small targets in complex scenes.

[0003] With the rapid development of industrial automation and intelligent manufacturing, the application of collaborative robots in industrial production has gradually increased. Unlike traditional industrial robots, collaborative robots can share the workspace with human workers, have high flexibility, safety and easy operation, and are particularly suitable for complex assembly and disassembly tasks.

[0004] Collaborative robots are usually used in scenarios that require high precision, high flexibility and safety, such as target disassembly and assembly operations in equipment maintenance, precision manufacturing and automobile manufacturing. In these scenarios, robots need to work with workers to complete complex parts disassembly and assembly tasks, and disassembly and assembly tasks require not only the identification and positioning of large structural parts, but also the precise positioning and operation of small parts such as screws and pins.

[0005] Most existing collaborative robot vision systems use RGB cameras or LiDAR for environmental scanning and target recognition. However, these sensors are more efficient in locating large targets, but lack the accuracy to recognize small parts, especially in complex working environments, where they are easily affected by ambient light, reflection and occlusion, resulting in reduced recognition and positioning accuracy.

[0006] Therefore, in order to solve the above problems, there is an urgent need for a multi-level positioning technology that can combine the advantages of multiple sensors in a collaborative robot system to achieve rapid positioning of large targets and precise identification of small targets, so as to improve the efficiency and accuracy of collaborative robots in target disassembly and assembly tasks. Summary of the invention

[0007] Purpose of the invention: In order to improve the problem that collaborative robots are unable to quickly, effectively and accurately locate small target objects in complex scenarios, the present invention proposes a collaborative robot disassembly and assembly target accurate positioning method based on depth vision. By quickly obtaining the precise position information of objects to be disassembled and assembled and disassembly points in a complex environment, it is beneficial to the flexible deployment of collaborative robots to perform disassembly and assembly work, and realizes efficient and accurate positioning of objects to be disassembled and assembled and disassembly points in a complex environment.

[0008] Technical solution: The method for accurately positioning the disassembly and assembly target of a collaborative robot based on depth vision of the present invention comprises the following steps:

[0009] (1) The depth camera is used to identify and locate the object to be disassembled. The process is as follows:

[0010] (1.1) Collecting color image data stream and depth image data stream through a depth camera, wherein the color image is used for visual feature recognition of the target object and the depth image is used for spatial positioning;

[0011] (1.2) Obtain the intrinsic parameter matrix of the depth camera color camera through camera calibration And the intrinsic parameter matrix of the depth camera where f x is the horizontal value of focal length, f y is the vertical value of focal length, c x is the principal point coordinate, c y is the coordinate of the principal point xx. The rotation matrix R from the depth camera to the color camera CD and the translation vector T CD .

[0012] Combined with the calibration parameters, the collected color image and depth image are dedistorted. First, the image pixels are normalized to obtain the image coordinates:

[0013]

[0014] Among them, f x ,f y is the focal length; c x ,c y are the principal point coordinates; u represents the horizontal coordinate of the image pixel, v represents the vertical coordinate of the image pixel, x represents the horizontal coordinate of the image point, and y represents the vertical coordinate of the image point.

[0015] Then perform distortion correction:

[0016] r 2 =x 2 +y 2 (2)

[0017]

[0018] Among them, k1 represents the first-order radial distortion coefficient, k2 represents the second-order radial distortion coefficient, k3 represents the third-order radial distortion coefficient, p1 represents the first-order tangential distortion coefficient, p2 represents the second-order tangential distortion coefficient, x′ represents the horizontal coordinate of the point on the image after distortion correction, and y′ represents the vertical coordinate of the point on the image after distortion correction.

[0019] After substituting the calibration parameters of the color camera and the depth camera, the coordinates of the pixel points after dedistortion of the color image (u c ′,v c ′), and the pixel coordinates after dedistortion of the depth image (u d ′,v d ′):

[0020] u′=x′f x +c x ,v′=y′f y +c y (4)

[0021] The color image and depth image are dedistorted and then the field of view (FOV) is compensated.

[0022] After de-distortion and field of view (FOV) compensation, the images captured by the color camera and the depth camera are ensured to be consistent. Combining the relationship between the pixel coordinate system and the camera coordinate system, as well as the relative position relationship between the color camera and the depth camera, a mathematical model of the mapping relationship between the color image and the depth image pixels is established. The process is as follows:

[0023] Set the position of a point in the world in the color camera coordinate system to P C (X C ,Y C ,Z C ), the position in the depth camera coordinate system is P D (X D ,Y D ,Z D ), whose position in the color image pixel coordinate system is p c =(u c ′,v c ′), the position in the depth image pixel coordinate system is p d =(u d ′,v d ′), the relative relationship is as follows:

[0024] Z C p c =K C P C ,Z D pd =K D P D (7)

[0025] Combined with the rotation matrix R from the depth camera to the color camera CD The translation vector T connects the depth camera to the color camera. The relative relationship is as follows:

[0026] P C =R CD P D +T CD (8)

[0027] Combining equations (6) and (7), a mathematical model of the mapping relationship between color image and depth image pixels is established as follows:

[0028] Z C p c =K C R CD K D -1 Z D p d +K C T CD (9)

[0029] Among them, Z C is the depth value in the color camera coordinate system of the depth camera, p c is the pixel coordinate of the color image, K C is the internal parameter of the depth camera color camera. D is the depth value of the depth camera in the camera coordinate system, p d is the pixel coordinate of the depth image, K D is the internal parameter of the depth camera. CD is the rotation matrix between the color camera and the depth camera, T CD is the translation matrix between the color camera and the depth camera.

[0030] After the mapping mathematical model is applied to the registration, the color image and depth image data are fused, and the corresponding depth value is attached to each color image pixel to generate a registered image data stream containing two-dimensional image coordinates, color information, and depth information.

[0031] (1.3) The image datasets of objects to be disassembled and assembly points and the image datasets of assembly points to be disassembled are generated by using pictures containing the objects to be disassembled and assembly points to be disassembled and the annotation files of the objects to be disassembled and assembly points in the pictures. The image datasets of disassembled objects are input into the deep learning algorithm for training, and the AP50 and mAP in the training results are used as the model evaluation results to obtain the optimal target detection model for objects to be disassembled. Among them, AP50 is the average precision when the intersection-over-union ratio is greater than 0.5, and mAP is the average value of 10 APs with an intersection-over-union ratio of 0.5 to 0.95 and a step size of 0.05. Ensure that the mapping relationship mathematical model recognizes objects to be disassembled under different angles, lighting and complex backgrounds.

[0032] During the detection process, the target detection model recognizes and classifies objects in the color image data stream frame by frame. The model generates multiple candidate regions through feature extraction and region proposal network (RPN), and combines non-maximum suppression (NMS) to filter out the optimal target detection frame, and finally generates a preliminary target detection frame on the color image.

[0033] (1.4) The color image in the initially generated target detection frame is converted into a grayscale image, and the grayscale image is Gaussian blurred using the Canny edge detection algorithm to reduce noise. Then the gradient intensity of the image is calculated, and a clear edge line that conforms to the edge features of the object to be disassembled is generated through double threshold suppression and edge tracking, such as the straight line or curve of the contour and the boundary line of the surface.

[0034] The edge line of the object to be disassembled is fitted into a specific target frame that conforms to the edge features of the object to be disassembled and synchronized to the registration image, and the center point coordinates p of the specific target frame are given. c (u c ′,v c ′). The specific target box is used to lock the actual area of ​​the object for subsequent depth data processing.

[0035] (1.5) Based on the registered image, the depth data in the specific target frame is filled based on the local mean, that is, a neighborhood window of a predetermined size is first selected around the invalid or missing depth values ​​in the specific target frame, the local mean of the valid depth values ​​in the neighborhood is calculated, and the invalid or missing depth values ​​are set as their neighborhood mean.

[0036] Then bilateral filtering is performed, that is, the depth data is denoised based on the spatial proximity between pixels and the similarity of pixel depth values ​​to ensure that the depth data of the target area is complete and continuous.

[0037] (1.6) Perform weighted average processing on the depth data in the frame to obtain the weighted depth value Z P , Z P Set to the depth value in the color camera coordinate system. Set the weight in Calculate the weighted depth value Z P , the formula is as follows:

[0038]

[0039] Among them, D(x i ,y i ) is the depth value of the i-th pixel in the specific target box, and N is the number of valid pixels in the specific target box.

[0040] (1.7) Through coordinate transformation, obtain the position coordinates P of the object to be disassembled in the world coordinate system W (X W ,Y W ,Z W ).

[0041] (2) Determine whether the object to be disassembled is in the working space of the robot arm. If it is not in the working space, adjust the position of the robot based on the distance between the target and the depth camera so that the object to be disassembled enters the working space of the robot arm.

[0042] (3) When the object to be disassembled is in the working space of the robot arm, based on the position of the object to be disassembled, the robot arm is controlled to move the end effector to the vicinity of the object to be disassembled, and the binocular camera module on the end effector is used to identify and locate the disassembly point to obtain the precise position of the disassembly point. The process is as follows:

[0043] (3.1) Based on the world position coordinates P of the object to be disassembled W The robot arm plans the movement trajectory of the robot arm through the motion planning algorithm, and then controls the robot arm to move the end effector along the trajectory to the front of the object to be disassembled and assembled.

[0044] (3.2) Start the binocular camera on the end effector to collect image data of the object to be disassembled and assembled.

[0045] (3.3) Using the captured left and right images, a disparity map based on the left view is generated through a trained deep learning stereo matching algorithm. In the disparity map, each pixel value represents the disparity between the left and right images, which is used to further calculate the depth information.

[0046] (3.4) The c2f module in yolov8 is improved by combining the lightweight backbone network MoblieNetV4, and features are extracted and expressed through the initial deep convolution, expansion layer (1x1 convolution), intermediate deep convolution and projection layer (1x1 convolution). The attention mechanism module DAT is added before the backbone network SPPF module, and the sampling area is dynamically adjusted through offset convolution to focus on features. The detection head is optimized by combining the AFPN network, and the multi-scale feature map generated by the backbone network is layer-by-layer feature fusion to avoid the loss of small target features. Then, the target box of each disassembly point is generated by the detection head, including the position and category of the target box.

[0047] (3.5) By extracting the disparity value D of the pixel in the target frame of the point to be disassembled P , calculate the target depth value in the area. Combine the baseline length B and focal length f of the camera to calculate the depth Z of the point to be disassembled s , the formula is as follows:

[0048]

[0049] Among them, Z S is the depth value of the point to be disassembled, f is the focal length of the binocular camera. B is the baseline length of the binocular camera, D p is the parallax value.

[0050] (3.6) Through coordinate transformation, obtain the position coordinates P of the point to be disassembled and assembled in the world coordinate system SW (X SW ,Y SW ,Z SW ), the conversion formula is as follows:

[0051] Z S p l =K S P SC (17)

[0052] Among them, Z S is the depth of the point to be disassembled; p l is the pixel coordinate of the point to be disassembled and assembled in the left image; K S is the internal reference of the left camera of the binocular camera; P SC (X SC ,Y SC ,Z S ) is the coordinate of the point to be disassembled and assembled in the left camera coordinate system of the binocular camera;

[0053] P SW =R SW P SC +T SW (18)

[0054] Among them, R SWand T SW is the external parameter of the left camera of the binocular camera; P SW (X SW ,Y SW ,Z SW ) is the coordinate of the point to be disassembled and assembled in the world coordinate system; P SC (X SC ,Y SC ,Z S ) is the coordinate of the point to be disassembled and assembled in the left camera coordinate system of the binocular camera.

[0055] Step (4), based on the positioning coordinates of the disassembly and assembly points, the collaborative robot derives the robotic arm disassembly and assembly path according to the planned path, and controls the robotic arm to perform the target disassembly and assembly work.

[0056] In step (1), it is assumed that the horizontal field of view of the color camera is The vertical field of view is The horizontal field of view of the depth camera is The vertical field of view is The cropping range of the image is calculated based on the viewing angle difference. The process is as follows:

[0057] Calculate the horizontal and vertical viewing angle differences:

[0058]

[0059] where r h is the horizontal viewing angle difference, r v is the vertical viewing angle difference, is the horizontal field of view of the color camera, is the vertical field of view of the color camera; is the horizontal field of view of the depth camera, is the vertical field of view of the depth camera.

[0060] Then calculate the cropping range based on the screen resolution:

[0061] cut h =w(1-r h ),cut v =h(1-r v ) (6)

[0062] Among them, cut h is the horizontal cutting range, cut v is the vertical cropping range, w is the width of the image to be cropped, and h is the height of the image to be cropped.

[0063] In step (1.5), the formula for calculating the local mean is as follows:

[0064]

[0065] Among them, D m is the local mean, D i is the i-th valid depth value in the neighborhood window, and N is the number of valid depth values ​​in the neighborhood window.

[0066] In step (1.7), the formula for coordinate transformation is as follows:

[0067] P W =R CW P C +T CW (12)

[0068] Among them, R CW and T CW is the external parameter of the color camera; P W (X W ,Y W ,Z W ) is the coordinate of the object to be disassembled in the world coordinate system; P C (X C ,Y C ,Z P ) is the coordinate of the object to be disassembled in the camera coordinate system.

[0069] The process of step (2) is:

[0070] (2.1) Based on the world position coordinates P of the object to be disassembled W (X W ,Y W ,Z W ), convert it to polar coordinates (r W ,θ W ,Z W ), and combined with the working range of the robot arm (r min ,r max ,θ min ,θ max ,Z min ,Z max ) to determine whether it is within the working range of the collaborative robot disassembly and assembly arm. The conversion formula is as follows:

[0071]

[0072] Among them, r W is the horizontal distance from the object to be disassembled to the base of the robot arm, θ W is the azimuth of the object to be disassembled on the XY plane, X W ,Y W ,Z W is the world coordinate of the object to be disassembled.

[0073] (2.2) If the object to be disassembled is not within the working range of the robot arm, based on the distance difference Δr and angle difference Δθ between the object position and the working range, the collaborative robot is controlled to move slowly until the object to be disassembled enters the working space of the robot arm. The formulas for Δr and Δθ are as follows:

[0074]

[0075] Among them, r W is the horizontal distance from the object to be disassembled to the base of the robot arm, r min ,r max is the minimum and maximum value of the robot's working range. If Δr>0, the robot moves forward, and if Δr<0, the robot moves backward.

[0076]

[0077] Among them, θ W is the azimuth angle of the object to be disassembled on the XY plane, θ min is the minimum value of the robot arm rotation, θ max is the maximum value of the robot arm rotation. If Δθ>0, the robot rotates clockwise, and if Δθ<0, the robot rotates counterclockwise.

[0078] The process of step (4) is:

[0079] (4.1) Accurately locate the coordinates P based on the disassembly and assembly points SW (X SW ,Y SW ,Z SW ), the collaborative robot plans the disassembly and assembly path through the path planning algorithm, and optimizes the disassembly and assembly motion trajectory of the robot arm in combination with the joint space interpolation algorithm, and finally obtains a disassembly and assembly path for the robot arm.

[0080] (4.2) According to the planned path, the collaborative robot controls the position and speed of key points to control the multifunctional assembly and disassembly manipulator on the end effector of the robot arm to reach the assembly and disassembly point and perform the assembly and disassembly work.

[0081] Working principle: The method for precise positioning of collaborative robot disassembly and assembly targets based on depth vision of the present invention comprises the following steps: using a depth camera module to identify and locate the object to be disassembled; adjusting the robot position based on the distance between the target and the depth camera so that the object to be disassembled enters the working space of the robotic arm; when the object to be disassembled is in the working space of the robotic arm, based on the coordinates of the object to be disassembled, controlling the robotic arm to move the end effector to the corresponding position, and using the binocular camera module on the end effector to identify and locate the disassembly point to determine the precise position of the disassembly point; and controlling the collaborative robot to perform target disassembly and assembly. The present invention can quickly obtain the precise position information of the object to be disassembled and the disassembly point in a complex environment, which is conducive to the flexible deployment of collaborative robots for disassembly and assembly.

[0082] Beneficial effects: Compared with the prior art, the present invention has the following advantages: The present invention aims to realize the rapid and accurate positioning of small targets in complex environments by collaborative robots, uses depth cameras and binocular cameras as data acquisition devices, and utilizes target detection models and image processing technology to realize efficient and accurate positioning of objects and disassembly points in complex environments, making it convenient for collaborative robots to flexibly deploy disassembly and assembly work. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Schematic diagram of the structure of a collaborative robot in an embodiment of the present invention;

[0084] Figure 2 Schematic diagram of a method for precise positioning of collaborative robot disassembly and assembly targets based on depth vision in an embodiment of the present invention;

[0085] Figure 3 This is a schematic diagram of the process of obtaining the location of an object to be disassembled in an embodiment of the present invention;

[0086] Figure 4 This is an example diagram of the effect of edge detection in an embodiment of the present invention;

[0087] Figure 5 The figure is a schematic diagram of the process of obtaining the location of the assembly and disassembly point in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In this embodiment, Figure 1 As shown, the method for precise positioning of collaborative robot disassembly and assembly targets based on depth vision of the present invention is applied to collaborative robots to realize human-machine collaborative disassembly and assembly of target objects. The collaborative robot is composed of a camera module, a mechanical arm and a mobile device. The camera module includes a binocular camera 1 and a depth camera 3. The mechanical arm uses a six-degree-of-freedom mechanical arm 2. The mobile device uses an AGV loader 4.

[0089] like Figures 2 to 5 As shown, the method for accurately positioning the disassembly and assembly target of the collaborative robot based on depth vision of the present invention is as follows:

[0090] (1) Use a depth camera to identify and locate the axle box cover.

[0091] (2) Determine whether the axle box cover is in the working space of the robot arm. If it is not in the working space, adjust the robot position based on the distance between the axle box cover and the depth camera so that the axle box cover enters the working space of the robot arm.

[0092] (3) When the axle box cover is in the working space of the robot arm, based on the position information of the axle box cover, the robot arm is controlled to move the end effector to the vicinity of the axle box cover, and the screw is identified and located through the binocular camera module on the end effector to obtain the precise position information of the screw.

[0093] (4) Control the collaborative robot to perform screw disassembly and assembly.

[0094] The following will explain in detail the steps of the collaborative robot disassembly and assembly target precise positioning method based on depth vision.

[0095] In the embodiment of the present invention, Figure 3 As shown, the process of step (1) is:

[0096] The process of step (1) is:

[0097] (1.1) The color image data and depth image data of the collaborative robot's axle box cover disassembly and assembly work scene are collected through a depth camera. The color image is used for visual feature recognition of the axle box cover, and the depth image is used for spatial positioning of the axle box cover.

[0098] (1.2) Obtain the intrinsic parameter matrix of the depth camera color camera through camera calibration And the intrinsic parameter matrix of the depth camera where f x is the horizontal value of focal length, f y is the vertical value of focal length, c x is the principal point coordinate, c y is the coordinate of the principal point xx. The rotation matrix R from the depth camera to the color camera CD and the translation vector T CD , and the distortion coefficients of the color camera and the depth camera and

[0099] Combined with the calibration parameters, the collected color image and depth image are dedistorted. First, the image pixels are normalized to obtain their image coordinates. The specific formula is as follows:

[0100]

[0101] where f x,f y is the focal length; c x ,c y is the principal point coordinate. u represents the horizontal coordinate of the image pixel, v represents the vertical coordinate of the image pixel, x represents the horizontal coordinate of the image point, and y represents the vertical coordinate of the image point.

[0102] Then perform distortion correction, the formula is as follows:

[0103] r 2 =x 2 +y 2 (2)

[0104]

[0105] Where k1, k2, k3, p1, p2 are the distortion coefficients of the camera. k1 represents the first-order radial distortion coefficient, k2 represents the second-order radial distortion coefficient, k3 represents the third-order radial distortion coefficient, p1 represents the first-order tangential distortion coefficient, p2 represents the second-order tangential distortion coefficient, x′ represents the horizontal coordinate of the point on the image after distortion correction, and y′ represents the vertical coordinate of the point on the image after distortion correction.

[0106] After the calibration parameters of the color camera and the depth camera are input, the pixel coordinates (u c ′,v c ′), and the pixel coordinates after dedistortion of the depth image (u d ′,v d ′).

[0107] u′=x′f x +c x ,v′=y′f y +c y (4)

[0108] After dedistorting the color image and depth image, the field of view (FOV) compensation is performed. Assume that the horizontal field of view angle of the color camera is The vertical field of view is The horizontal field of view of the depth camera is The vertical field of view is The cropping range of the image is calculated based on the viewing angle difference. The specific process is as follows:

[0109] Calculate the horizontal viewing angle difference and vertical viewing angle difference respectively, the formula is as follows:

[0110]

[0111] where r h is the horizontal viewing angle difference, r v is the vertical viewing angle difference, is the horizontal field of view of the color camera, is the vertical field of view of the color camera; is the horizontal field of view of the depth camera, is the vertical field of view of the depth camera.

[0112] Then calculate the cropping range based on the image resolution. The formula is as follows:

[0113] cut h =w(1-r h ),cut v =h(1-r v ) (6)

[0114] Among them, cut h is the horizontal cutting range, cut v is the vertical cropping range, w is the width of the image to be cropped, and h is the height of the image to be cropped.

[0115] After de-distortion and field of view (FOV) compensation, the images captured by the color camera and the depth camera are ensured to be consistent. Combining the relationship between the pixel coordinate system and the camera coordinate system, as well as the relative position relationship between the color camera and the depth camera, a mathematical model for mapping between color image and depth image pixels is established. The specific process is as follows:

[0116] Set the position of the axle box cover in the world in the color camera coordinate system to P C (X C ,Y C ,Z C ), the position in the depth camera coordinate system is P D (X D ,Y D ,Z D ), whose position in the color image pixel coordinate system is p c =(u c ′,v c ′), the position in the depth image pixel coordinate system is p d =(u d ′,v d ′). The relative relationship is as follows:

[0117] Z C p c =K C P C ,Z D p d =K D P D (7)

[0118] Combined with the rotation matrix R from the depth camera to the color cameraCD And the translation vector T, connect the depth camera with the color camera, the relative relationship is as follows:

[0119] P C =R CD P D +T CD (8)

[0120] Combining equations (6) and (7), a mathematical model of the mapping relationship between color image and depth image pixels is established as follows:

[0121]

[0122] Among them, Z C is the depth value in the color camera coordinate system of the depth camera, p c is the pixel coordinate of the color image, K C is the internal parameter of the depth camera color camera. D is the depth value of the depth camera in the camera coordinate system, p d is the pixel coordinate of the depth image, K D is the internal parameter of the depth camera. CD is the rotation matrix between the color camera and the depth camera, T CD is the translation matrix between the color camera and the depth camera.

[0123] After the mapping relationship mathematical model is applied to the registration, the color image and depth image data are fused, and the corresponding depth value is attached to each color image pixel to generate a registered image data stream containing two-dimensional image coordinates, color information, and depth information.

[0124] (1.3) An axle box cover image dataset and a screw image dataset are generated by using pictures containing axle box covers and screws and annotation files of the axle box covers and screws in the pictures. The axle box cover image dataset is input into the deep learning algorithm for training to obtain the optimal axle box cover target detection model. The collected color image is transmitted to the trained optimal axle box cover detection model for axle box cover recognition.

[0125] The data set is input into the deep learning algorithm for training, and the AP50 and mAP in the training results are used as the model evaluation results to obtain the optimal axle box cover target detection model. AP50 is the average precision when the intersection-over-union ratio is greater than 0.5, and mAP is the average value of 10 APs with an intersection-over-union ratio of 0.5 to 0.95 and a step size of 0.05, ensuring that the target detection model can recognize the axle box cover under different angles, lighting and complex backgrounds.

[0126] During the detection process, the target detection model recognizes and classifies objects in the color image data stream frame by frame. The model generates multiple candidate regions through feature extraction and region proposal network (RPN), and selects the optimal target detection frame in combination with non-maximum suppression (NMS). Finally, a preliminary target detection frame is generated on the color image, which is used to limit the area that needs further processing.

[0127] (1.4) Figure 4 As shown in the figure, the edge detection algorithm is used to detect the edge of the preliminary identification frame of the axle box cover. When the color image data recognizes the axle box cover through the target detection model, a minimum rectangular frame that frames the axle box cover is generated. At this time, the edge detection is performed on the image in the rectangular frame to generate a hexagonal frame that conforms to the edge contour of the axle box cover. A specific target frame that conforms to the edge features of the axle box cover is generated and the center point coordinates p are given. c (u c ′,v c ′). The target frame is used to lock the actual area of ​​the axle box cover screws that need to be removed for subsequent in-depth data processing.

[0128] (1.5) Based on the registered image, the depth data in the specific target frame is filled based on the local mean, that is, a neighborhood window of a predetermined size is first selected around the invalid or missing depth values ​​in the specific target frame, the local mean of the valid depth values ​​in the neighborhood is calculated, and the invalid or missing depth values ​​are set as their neighborhood mean. The formula for calculating the local mean is as follows:

[0129]

[0130] Among them, D m is the local mean, D i is the i-th valid depth value in the neighborhood window, and N is the number of valid depth values ​​in the neighborhood window.

[0131] Then bilateral filtering is performed, that is, the depth data is denoised based on the spatial proximity between pixels and the similarity of pixel depth values ​​to ensure that the depth data of the target area is complete and continuous.

[0132] (1.6) Perform weighted average processing on the depth data in the frame to obtain the depth value Z P , Z P Set to the depth value in the color camera coordinate system. Set the weight in Calculate the weighted depth value Z P , the formula is as follows:

[0133]

[0134] In formula (9), D(x i ,yi ) is the depth value of the i-th pixel in the specific target box, and N is the number of valid pixels in the specific target box.

[0135] (1.7) Through coordinate transformation, obtain the position coordinates P of the axle box cover in the world coordinate system W (X W ,Y W ,Z W ). The conversion formula is as follows:

[0136] P W =R CW P C +T CW (12)

[0137] Among them, R CW and T CW is the external parameter of the color camera; P W (X W ,Y W ,Z W ) is the coordinate of the axle box cover in the world coordinate system; P C (X C ,Y C ,Z P ) is the coordinate of the axle box cover in the camera coordinate system.

[0138] The process of step (2) is:

[0139] (2.1) Based on the world position coordinates P of the axle box cover W (X W ,Y W ,Z W ), convert it to polar coordinates (r W ,θ W ,Z W ), and combined with the working range of the robot arm (r min ,r max ,θ min ,θ max ,Z min ,Z max ) to determine whether it is within the working range of the collaborative robot disassembly and assembly arm. The conversion formula is as follows:

[0140]

[0141] Among them, r W is the horizontal distance from the axle box cover to the base of the robot arm, θ W is the azimuth of the axle box cover on the XY plane, X W ,Y W ,Z W is the world coordinate of the axle box cover.

[0142] (2.2) If the axle box cover is not within the working range of the robot arm, based on the distance difference Δr and angle difference Δθ between the object position and the working range, the collaborative robot is controlled to move until the axle box cover enters the working space of the robot arm. The formulas for Δr and Δθ are as follows:

[0143]

[0144] Among them, r W is the horizontal distance from the axle box cover to the base of the robot arm, r min is the minimum value of the robot's working range, r max is the maximum value of the robot's working range. If Δr>0, the robot moves forward, and if Δr<0, the robot moves backward.

[0145]

[0146] Among them, θ W is the azimuth angle of the axle box cover on the XY plane, θ min , is the minimum value of the robot arm rotation, θ max is the maximum value of the robot arm rotation. If Δθ>0, the robot rotates clockwise, and if Δθ<0, the robot rotates counterclockwise.

[0147] The process of step (3) is:

[0148] (3.1) Based on the world position coordinates P of the axle box cover W The robot arm plans the movement trajectory of the robot arm through the motion planning algorithm, and then controls the robot arm to move the end effector along the trajectory to the front of the axle box cover.

[0149] (3.2) Start the binocular camera on the end effector to collect the image data of the axle box cover.

[0150] (3.3) The collected left and right images are sent to a trained deep learning stereo matching algorithm to obtain a disparity map based on the left view. Each pixel in the disparity map represents the disparity between the left and right images.

[0151] (3.4) The c2f module in yolov8 is improved by combining the lightweight backbone network MoblieNetV4, and features are extracted and expressed through the initial depth convolution, expansion layer (1x1 convolution), intermediate depth convolution and projection layer (1x1 convolution). The attention mechanism module DAT is added before the backbone network SPPF module, and the sampling area is dynamically adjusted through offset convolution to focus on features. The detection head is optimized by combining the AFPN network, and the multi-scale feature map generated by the backbone network is layer-by-layer feature fusion to avoid the loss of small target features. Then, the target box of each disassembly point is generated by the detection head, including the position and category of the target box.

[0152] (3.5) By extracting the disparity value D of the pixel in the target frame of the point to be disassembled P , calculate the target depth value in the area. Combine the baseline length B and focal length f of the camera to calculate the depth Z of the point to be disassembled s , the formula is as follows:

[0153]

[0154] Among them, Z S is the depth value of the point to be disassembled, f is the focal length of the binocular camera. B is the baseline length of the binocular camera, D p is the parallax value.

[0155] (3.6) Through coordinate transformation, the position coordinates P of the screw in the world coordinate system are obtained. SW (X SW ,Y SW ,Z SW ). The conversion formula is as follows:

[0156] Z S p l =K S P SC (17)

[0157] Among them, Z S is the depth of the screw; p l is the pixel coordinate of the screw in the left image; P SC (X SC ,Y SC ,Z S )K S is the internal reference of the left camera of the binocular camera, and is the coordinate of the screw in the coordinate system of the left camera of the binocular camera.

[0158] P SW =R SW P SC +T SW (18)

[0159] Among them, R SW and T SW is the external parameter of the left camera of the binocular camera; P SW (X SW ,Y SW ,Z SW ) is the coordinate of the screw in the world coordinate system; P SC (X SC ,Y SC ,Z S ) is the coordinate of the screw in the left camera coordinate system of the binocular camera.

[0160] Step (4), controlling the collaborative robot to perform the target disassembly and assembly work, the process is:

[0161] (4.1) Accurately locate the coordinates P based on the disassembly and assembly points SW (X SW ,Y SW ,Z SW ), the collaborative robot plans the disassembly and assembly path through the path planning algorithm, and optimizes the disassembly and assembly motion trajectory of the robot arm in combination with the joint space interpolation algorithm, and finally obtains a disassembly and assembly path for the robot arm.

[0162] (4.2) According to the planned path, the collaborative robot controls the position and speed of key points to control the disassembly and assembly manipulator on the robot arm to move the disassembly and assembly manipulator on the end effector to reach the disassembly and assembly point and perform the disassembly and assembly work.

Claims

1. A method for accurately positioning disassembly and assembly targets of collaborative robots based on depth vision, characterized in that: The following steps are involved: Step (1) uses a depth camera to identify and locate the object to be disassembled. The process is as follows: Step (1.1), collecting a data stream of a color image and a data stream of a depth image through a depth camera; Step (1.2), get the intrinsic parameter matrix of the depth camera color camera And the intrinsic parameter matrix of the depth camera where f x is the horizontal value of focal length, f y is the vertical value of focal length, c x is the principal point coordinate, c y is the coordinate of the principal point xx; Dedistort the collected color image and depth image; first normalize the image pixels to obtain the image coordinates: Among them, f x ,f y is the focal length; c x ,c y is the coordinate of the principal point; u represents the horizontal coordinate of the image pixel, v represents the vertical coordinate of the image pixel, x represents the horizontal coordinate of the point on the image, and y represents the vertical coordinate of the point on the image; Then perform distortion correction: r 2 =x 2 +y 2 (2) x′=x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 ) (3) <h2 style=";text-align:left;direction:ltr">y′ = y(1+k1r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +k2r<h2 style=";text-align:left;direction:ltr"> 4 <h2 style=";text-align:left;direction:ltr"> +k3r<h2 style=";text-align:left;direction:ltr"> 6 <h2 style=";text-align:left;direction:ltr"> )+p1(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2y<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> )+2p2xy Wherein, k1 represents the first-order radial distortion coefficient, k2 represents the second-order radial distortion coefficient, k3 represents the third-order radial distortion coefficient, p1 represents the first-order tangential distortion coefficient, p2 represents the second-order tangential distortion coefficient, x′ represents the horizontal coordinate of the point on the image after distortion correction, and y′ represents the vertical coordinate of the point on the image after distortion correction; Calculate the pixel coordinates of the color image after dedistortion (u c ′,v c ′) and the pixel coordinates after dedistortion of the depth image (u d ′,v d ′): u′=x′f x +c x ,v′=y′f y +c y (4) After dedistorting the color image and depth image, field of view compensation is performed to make the image captured by the color camera and the depth camera consistent; a mathematical model for mapping between color image and depth image pixels is established: Suppose the position of a point in the color camera coordinate system is P C (X C ,Y C ,Z C ), the position in the depth camera coordinate system is P D (X D ,Y D ,Z D ), whose position in the color image pixel coordinate system is p c =(u c ′,v c ′), the position in the depth image pixel coordinate system is p d =(u d ′,v d ′), we get: Z C p c =K C P C ,Z D p d =K D P D (7) The rotation matrix R from the depth camera to the color camera CD The relative relationship with the translation vector T is: P C =R CD P D +T CD (8) Combining equations (6) and (7), a mathematical model of the mapping relationship between color image and depth image pixels is established: Among them, Z C is the depth value in the color camera coordinate system of the depth camera, p c is the pixel coordinate of the color image, K C is the internal parameter of the depth camera color camera; Z D is the depth value of the depth camera in the camera coordinate system, p d is the pixel coordinate of the depth image, K D is the internal parameter of the depth camera; R CD is the rotation matrix between the color camera and the depth camera, T CD is the translation matrix between the color camera and the depth camera; Step (1.3), inputting the image data set of the objects to be disassembled into the deep learning algorithm for training to obtain the optimal target detection model of the objects to be disassembled; Step (1.4), convert the color image in the generated target detection frame into a grayscale image, and perform Gaussian blur on the grayscale image; then calculate the gradient strength of the image, and generate a clear edge line that meets the edge features of the object to be disassembled; fit the edge line of the object to be disassembled into a specific target frame that meets the edge features of the object to be disassembled and synchronized to the registration image, and give the center point coordinates p of the specific target frame c (u c ′,v c ′); Step (1.5), based on the registered image, the depth data within the specific target frame is filled based on the local mean, and then bilateral filtering is performed; Step (1.6): perform weighted average processing on the depth data in the target frame to obtain the weighted depth value Z P , Z P Set to the depth value in the color camera coordinate system; set the weight in Calculate the weighted depth value Z P : Among them, D(x i ,y i ) is the depth value of the i-th pixel in the specific target frame, and N is the number of valid pixels in the specific target frame; Step (1.7), after coordinate conversion, obtain the position coordinates P of the object to be disassembled in the world coordinate system W (X W ,Y W ,Z W ); Step (2), adjusting the position of the robot based on the distance between the target and the depth camera so that the object to be disassembled and assembled enters the working space of the robot arm; Step (3), when the object to be disassembled is in the working space of the robot arm, the robot arm is controlled to move the end effector to the object to be disassembled, and the disassembly point is identified and located by the binocular camera module on the end effector to obtain the precise position of the disassembly point. The process is as follows: (3.1) Based on the world position coordinates P of the object to be disassembled W , the robot arm plans the movement trajectory of the robot arm through the motion planning algorithm, and then controls the robot arm to move the end effector along the trajectory to the front of the object to be disassembled; (3.2) Start the binocular camera on the end effector to collect image data of the object to be disassembled; (3.3) Using the collected left and right images, a disparity map based on the left view is generated through a trained deep learning stereo matching algorithm; (3.4) Combine the lightweight backbone network MoblieNetV4 to improve the c2f module in yolov8 to extract and express features; add the attention mechanism module DAT before the backbone network SPPF module, dynamically adjust the sampling area through offset convolution, and perform feature focusing; combine the AFPN network to optimize the detection head to generate the target box of each disassembly point; (3.5) By extracting the disparity value D of the pixel in the target frame of the point to be disassembled P , calculate the target depth value Z in the area s ; (3.6) Through coordinate transformation, obtain the position coordinates P of the point to be disassembled and assembled in the world coordinate system SW (X SW ,Y SW ,Z SW ): Z S p l =K S P SC (17) Among them, Z S is the depth of the point to be disassembled; p l is the pixel coordinate of the point to be disassembled and assembled in the left image; K S is the internal reference of the left camera of the binocular camera; P SC (X SC ,Y SC ,Z S ) is the coordinate of the point to be disassembled and assembled in the left camera coordinate system of the binocular camera; P SW =R SW P SC +T SW (18) Among them, R SW and T SW is the external parameter of the left camera of the binocular camera; P SW (X SW ,Y SW ,Z SW ) is the coordinate of the point to be disassembled and assembled in the world coordinate system; P SC (X SC ,Y SC ,Z S ) is the coordinate of the point to be disassembled and assembled in the left camera coordinate system of the binocular camera; Step (4), based on the positioning coordinates of the disassembly and assembly points, the collaborative robot derives the robotic arm disassembly and assembly path according to the planned path, and controls the robotic arm to perform the target disassembly and assembly work.

2. The method for accurately positioning a collaborative robot disassembly and assembly target based on depth vision according to claim 1 is characterized in that: In step (1.2), the field of view compensation process is as follows: Assume that the horizontal field of view angle of the color camera is The vertical field of view is The horizontal field of view of the depth camera is The vertical field of view is Calculate the horizontal and vertical viewing angles based on the viewing angle difference: where r h is the horizontal viewing angle difference, r v is the vertical viewing angle difference, is the horizontal field of view of the color camera, is the vertical field of view of the color camera; is the horizontal field of view of the depth camera, is the vertical field of view of the depth camera; Then calculate the cropping range based on the screen resolution: cut h =w(1-r h ),cut v =h(1-r v ) (6) Among them, cut h is the horizontal cutting range, cut v is the vertical cropping range, w is the width of the image to be cropped, and h is the height of the image to be cropped.

3. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: In step (1.3), the images containing the objects to be disassembled and the disassembly points, as well as the annotation files of the objects to be disassembled and the disassembly points in the images, are converted into an image dataset of the objects to be disassembled and an image dataset of the disassembly points.

4. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: Under the local mean in step (1.5): Among them, D m is the local mean, D i is the i-th valid depth value in the neighborhood window, and N is the number of valid depth values ​​in the neighborhood window.

5. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: In step (1.7), the formula for coordinate transformation is as follows: P W =R CW P C +T CW (12) Among them, R CW and T CW is the external parameter of the color camera; P W (X W ,Y W ,Z W ) is the coordinate of the object to be disassembled in the world coordinate system; P C (X C ,Y C ,Z P ) is the coordinate of the object to be disassembled in the camera coordinate system.

6. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: The process of step (2) is: (2.1) The world position coordinates P of the object to be disassembled W (X W ,Y W ,Z W ) is converted to polar coordinates (r W ,θ W ,Z W ), determine whether the object to be disassembled is within the working range of the robot arm (r min ,r max ,θ min ,θ max ,Z min ,Z max ); (2.2) If the object to be disassembled is not within the working range of the robot arm, the collaborative robot is controlled to move until the object to be disassembled enters the working space of the robot arm based on the distance difference Δr and angle difference Δθ between the object position and the working range; the formulas for Δr and Δθ are as follows: Among them, r W is the horizontal distance from the object to be disassembled to the base of the robot arm, r min is the minimum value of the robot's working range, r max is the maximum value of the robot's working range; Among them, θ W is the azimuth angle of the object to be disassembled on the XY plane, θ min is the minimum value of the robot arm rotation, θ max The maximum value of the robot arm rotation.

7. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: In step (2.1), the world position coordinates P of the object to be disassembled are W (X W ,Y W ,Z W ) is converted to polar coordinates (r W ,θ W ,Z W )The process is as follows: Among them, r W is the horizontal distance from the object to be disassembled to the base of the robot arm, θ W is the azimuth of the object to be disassembled on the XY plane, X W ,Y W ,Z W is the world coordinate of the object to be disassembled.

8. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: Step (3.4), combine the lightweight backbone network MoblieNetV4 to improve the c2f module in yolov8, extract and express features through the initial depth convolution, expansion layer, intermediate depth convolution and projection layer in sequence; add the attention mechanism module DAT before the backbone network SPPF module, dynamically adjust the sampling area through offset convolution, and perform feature focusing; Combined with the AFPN network optimization detection head, the multi-scale feature map generated by the backbone network is fused layer by layer, and then the target box of each disassembly and assembly point is generated by the AFPN network optimization detection head.

9. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: In step (3.5), the depth value Z of the point to be disassembled is calculated by combining the baseline length B and focal length f of the camera s : Among them, Z S is the depth value of the point to be disassembled and assembled, f is the focal length of the binocular camera; B is the baseline length of the binocular camera, D p is the parallax value.

10. The method for accurately positioning disassembling and assembling targets of collaborative robots based on depth vision according to claim 1, characterized in that: The process of step (4) is as follows: (4.1) Accurately locate the coordinates P based on the disassembly and assembly points SW (X SW ,Y SW ,Z SW ), the collaborative robot plans the disassembly and assembly path through the path planning algorithm, and optimizes the disassembly and assembly motion trajectory of the robot arm to obtain a disassembly and assembly path for the robot arm; (4.2) According to the planned path, the collaborative robot controls the position and speed of key points to control the disassembly and assembly manipulator on the robot arm to move the end effector for disassembly and assembly.

Images