Robot 2d vision 6dof pose measurement and positioning grasping method for reflective metal objects

Abstract

The present application relates to a robot 2D vision 6DOF pose measurement and positioning grabbing method of a reflective metal object, a virtual graphics dataset is established by a virtual engine through a reflective metal object pattern collected by a monocular camera in a 6DOF system; Unet is used as a generator of a generative adversarial network, and a Transform module is combined with a cycleGAN network to improve the quality of image features; a KLD loss function is combined with a YOLOv7 network to optimize the coordinates and angles output by the YOLOv7 network, and to reduce the angle error; a fixed calibration plate is used to take a picture to calculate the pose of the calibration plate relative to the robot bottom, the pose of the calibration plane in the monocular camera coordinate system is calculated through a transpose matrix, and the conversion from 2D pose to 3D pose is realized in cooperation with the monocular camera internal parameter.The present application improves the YOLOv7 target detection network, realizes the positioning of the YOLOv7 target detection network on a rotating object, and realizes the high-precision calculation of the 2D grabbing pose of the robot grabbing.

Description

Technical Field

[0001] This invention relates to the field of 6DOF robot positioning and grasping, specifically a robot 2D vision 6DOF pose measurement and positioning grasping method for reflective metal objects. Background Technology

[0002] As the main force in replacing human labor with machines, multi-axis robots are playing an increasingly important role in the industrial field. Vision-based robot vision guidance technology, because it can compensate for the shortcomings of robots' limited perception capabilities, has always been a key research direction in robotics. Currently, multi-axis robots combined with vision technology are widely used in common processing fields in industrial production, such as material handling, grinding, and assembly.

[0003] Metal objects are common components in industrial production. Using industrial robots equipped with machine vision to replace manual labor in repetitive processing steps such as handling, grinding, and loading / unloading of metal objects can not only improve production efficiency and ensure product quality, but also better protect the personal safety of workers and help enterprises achieve automation upgrades. However, the complex working environment faced by most manufacturing enterprises restricts the application and promotion of vision technology in metal object processing sites, and the highly reflective nature of some metal objects limits the accurate extraction of their positional information by vision systems.

[0004] Although machine vision technology has made breakthroughs in the integrated application of robot systems, the poor anti-interference capability and high system cost of 6DOF vision systems remain major obstacles to the widespread application of robots in the industrial field. The complex industrial environment makes the imaging and localization problems of vision systems unsolvable by simply enhancing hardware devices such as light sources. Improving the anti-interference capability of vision systems in complex environments and enhancing the recognition accuracy and efficiency of vision algorithms are currently the focus of robot vision technology research. Early image processing methods, such as thresholding, histogram equalization, and frequency domain transformation, have limited effectiveness. In recent years, the development of artificial intelligence has provided many new ideas for solving existing vision problems. Using image generation and image restoration techniques in deep neural networks to enhance low-light images has become a mainstream research approach. While there is considerable research on this topic, work on image enhancement for industrial objects is currently uncommon.

[0005] Currently, the localization and grasping technology of multi-axis robots based on monocular cameras for planar workpieces mainly relies on template matching algorithms. This method extracts single features from the image of the object being measured, and determines the object's position in the target image by sequential detection and calculation of similarity scores. Insufficient single feature information is the main reason affecting the accuracy of template matching algorithms. Furthermore, this method can mostly only detect the offset of the object in the XY and Z rotation axes, and cannot accurately provide 6DOF information. While 3D sensors based on structured light and other methods offer better 6DOF pose measurement accuracy, they also suffer from low measurement efficiency, high cost, and complex operation. Detecting the 6DOF pose of the object using monocular vision remains one of the most economical and efficient measurement methods. The proposed method, combined with deep neural network technology, extracts multi-layer features from the image to determine the object's pose, achieving significant breakthroughs in detection accuracy, speed, and anti-interference capabilities. How to better apply deep learning-based object detection technology to 6DOF localization and measurement of industrial objects, thereby reducing the cost of using vision systems in industrial fields and improving detection accuracy and efficiency, is currently a hot research topic in robot vision.

[0006] To further expand the application scope of industrial robots and reduce the cost of robot vision systems, this paper proposes a 2D robot 6DOF visual positioning system based on image enhancement and target detection technologies, specifically targeting the visual imaging and characteristics of reflective metal objects. Compared to traditional robot vision positioning methods, the proposed method improves the adaptability of the vision system to complex environments by introducing image enhancement technology and solves the problem of high cost of image acquisition for industrial objects, thereby enhancing the robustness of the robot vision system. Furthermore, it proposes a two-step positioning method from coarse to fine, applying a deep learning-based rotational target detection algorithm to target detection of industrial objects, achieving high-precision target positioning of the workpiece. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a robot 2D vision 6DOF pose measurement and localization grasping method for reflective metal objects. The technical problem to be solved by this invention is achieved through the following technical solution:

[0008] A virtual graphics dataset was created using images of reflective metal objects captured by a monocular camera in a 6DOF system. Unet was used as the generator for a generative adversarial network, and the Transformer module was combined with the CycleGAN network to improve the quality of image features.

[0009] The KLD loss function is combined with the YOLOv7 network to optimize the coordinates and angles output by the YOLOv7 network, thereby reducing angle errors.

[0010] The pose of the calibration plate relative to the robot base is calculated by taking pictures of the fixed calibration plate. The pose of the calibration plane in the monocular camera coordinate system is calculated by the transpose matrix. The 2D pose is converted to 3D pose by the monocular camera intrinsic parameters.

[0011] The Transformer module calculates attention weights by performing a dot product operation between the query vector and the key vector, and then dividing by a scaling factor. The attention weights are then multiplied by the value vector, and the products are summed with weights to obtain the final output. The ability and efficiency of feature extraction in the image are changed by adjusting the weights of the query vector, key vector, and value vector.

[0012] The loss function of the YOLOv7 network is:

[0013] Loss = Loss Reg +Loss Cls +Loss Obj ;

[0014] Loss Cls For category loss, Loss Reg For bounding box regression loss, Loss Obj This represents the confidence loss.

[0015] The category loss and confidence loss are binary cross-entropy loss functions, and their expressions are as follows:

[0016]

[0017] Among them, σ(x) i () is the Sigmoid function, which maps x to the interval (0,1). Its expression is:

[0018]

[0019] The bounding box in any direction is determined by five parameters (x, y, w, h, θ), and the angular information dimension of the target is introduced into the YOLOv7 network by θ.

[0020] Where x and y represent the x and y coordinates of the center point of the bounding box, respectively, w and h represent the long side and short side of the bounding box, respectively, and θ is determined by the angle between the positive x-axis and the long side w, with a value range of (-90°, 90°).

[0021] When regressing the θ angle parameter, the KLD loss function transforms the rotated bounding box into a two-dimensional Gaussian distribution, and then calculates the relative entropy between the Gaussian distributions, i.e., the KL divergence, which is expressed as follows:

[0022]

[0023] In the above formula:

[0024]

[0025]

[0026] Where, Δx=x p -x t Δy=y p -y t Δθ=θ p -θ t , which are the center point coordinates and angle difference between the predicted bounding box and the ground truth bounding box, respectively.

[0027] The KLD loss function uses the ln function to smooth the loss function, and its expression is as follows:

[0028]

[0029] The pose expression of the calibration board relative to the robot base is as follows:

[0030] C T Plane = C T U × U T Plane ;

[0031] for C T U There is also a formula:

[0032] C T U = C T CT × CT T U ;

[0033] in C T CT That is CT T C The transpose of the matrix; CT T U for U T CT The transpose matrix; based on the camera's internal participation C T Plane The pose of the 2D pixels of the target workpiece in this coordinate system is calculated, thereby completing the conversion of the 2D pose of the target workpiece to the 3D pose.

[0034] The beneficial effects of this invention are as follows: by transforming the workpiece under test in a complex environment into a virtual image in a structural environment, and in conjunction with a virtual engine, data augmentation of small sample objects is achieved; the YOLOv7 target detection network is improved, reducing the impact of small angle errors on accuracy, enabling high-precision rotation detection, and realizing the positioning of rotating objects by the YOLOv7 target detection network; in conjunction with camera intrinsic parameters, the 2D pose of the camera-acquired image is converted to 3D pose through matrix transformation, and then, in conjunction with robot hand-eye calibration parameters, high-precision calculation of the robot's 2D grasping pose is achieved. Attached Figure Description

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0037] Figure 2 This is a schematic diagram of the improved YOLOv7 network of this invention;

[0038] Figure 3 This is a schematic diagram of the ISE module of the present invention;

[0039] Figure 4 This is a schematic diagram of the CSE module of the present invention;

[0040] Figure 5 This is a schematic diagram of the IFA module of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described more clearly and completely below with reference to the accompanying drawings in the embodiments. Of course, the described embodiments are only a part of the present invention and not all of it. Based on this embodiment, other embodiments obtained by those skilled in the art without creative effort are all within the protection scope of the present invention.

[0042] like Figures 1 to 5 As shown, the robot's 2D vision 6DOF pose measurement and localization grasping method for reflective metal objects includes the following steps:

[0043] A virtual image dataset is created using images of reflective metal objects captured by a monocular camera in a 6DOF system through a virtual engine. Unet is used as the generator for a generative adversarial network, and a Transformer module is combined with a CycleGAN network to improve the quality of image features. To further improve the clarity of images generated by the CycleGAN network and reduce the network's training time, an attention mechanism is introduced into the CycleGAN network, and a self-Attention-CycleGAN network is proposed in conjunction with a newly proposed contour loss function. The main components of the self-Attention-CycleGAN network include two residual modules and a Transformer module. The two residual modules use Unet, which incorporates residual structures, as the generator for the generative adversarial network to improve the generator's efficiency in generating industrial objects.

[0044] The KLD loss function is combined with the YOLOv7 network to optimize the coordinates and angles output by the YOLOv7 network, thereby reducing angle errors.

[0045] The pose of the calibration plate relative to the robot base is calculated by taking pictures of the fixed calibration plate. The pose of the calibration plane in the monocular camera coordinate system is calculated by the transpose matrix. The 2D pose is converted to 3D pose by the monocular camera intrinsic parameters.

[0046] The Transformer module calculates attention weights by performing a dot product operation between the query vector and the key vector, and then dividing by a scaling factor. The attention weights are multiplied by the value vector, and the products are summed with weights to obtain the final output. The ability and efficiency of feature extraction in the image are changed by adjusting the weights of the query vector, key vector, and value vector.

[0047] The Transformer module calculations include the following steps:

[0048] Step 1: Transform the input features into embedding vectors;

[0049] Step 2: Obtain three vectors Q, K, and V based on the embedding vector, where Q, K, and V are the query vector, key vector, and value vector, respectively.

[0050] Step 3: Calculate the score for each vector, score = Q * V;

[0051] Step 4: To stabilize the gradient, normalize the score by dividing by .

[0052] Step 5: Activate the softmax function on the score;

[0053] Step 6: Multiply the activated score vector by v to obtain the weighted input vector ∑V;

[0054] Sum all the weighted vectors v to get the output vector;

[0055] The corresponding formula is: Where d is the dimension of the vector.

[0056] The improved generator combines a two-layer residual generator with a Transformer module. The two-layer residual generator serves as the feature extraction unit for the UNet downsampling stage, while the Transformer module serves as the feature extraction unit for the upsampling stage. Compared to the original CycleGAN network, the improved generator further enhances the network's feature extraction capabilities, better achieving the conversion from the original image to the target image.

[0057] To further improve the execution efficiency of the network discriminator, a discriminator with few-shot learning is used, and the detection performance of the discriminator is improved through pre-training.

[0058] By collecting defective images generated by image generation networks, these images can be classified, primarily including those with blurriness, deformation, noise interference, and failed background restoration. These images are then labeled as negative samples and, together with positive sample images generated by the virtual engine, form a workpiece discrimination dataset. This dataset is then used to pre-train the discriminator network, thereby improving its discrimination accuracy.

[0059] The loss function of the YOLOv7 network is:

[0060] Loss = Loss Reg +Loss Cls +Loss Obj ;

[0061] Loss Cls For category loss, Loss Reg For bounding box regression loss, Loss Obj This represents the confidence loss.

[0062] The category loss and confidence loss are binary cross-entropy loss functions, and their expressions are as follows:

[0063]

[0064] Among them, σ(x) i () is the Sigmoid function, which maps x to the interval (0,1). Its expression is:

[0065]

[0066] The bounding box in any direction is determined by five parameters (x, y, w, h, θ), and the angular information dimension of the target is introduced into the YOLOv7 network by θ.

[0067] Where x and y represent the x and y coordinates of the center point of the bounding box, respectively, w and h represent the long side and short side of the bounding box, respectively, and θ is determined by the angle between the positive x-axis and the long side w, with a value range of (-90°, 90°).

[0068] Conventional target detection methods use horizontal detection, typically representing the horizontal bounding box with four parameters (x, y, w, h), representing the center point coordinates, width, and height of the bounding box, respectively. However, horizontal bounding boxes cannot effectively cover targets in different directions and contain information redundancy, thus having certain limitations. Rotational target detection, in addition to obtaining the target's position information, also obtains predicted information about the rotation angle, enabling the perception of more complete and comprehensive target information.

[0069] When regressing the θ angle parameter, the KLD loss function transforms the rotated bounding box into a two-dimensional Gaussian distribution, and then calculates the relative entropy between the Gaussian distributions, i.e., the KL divergence, which is expressed as follows:

[0070]

[0071] In the above formula:

[0072]

[0073]

[0074] Where, Δx=x p -x t Δy=y p -y t Δθ=θ p -θ t , where are the center point coordinates and angle difference between the predicted bounding box and the ground truth bounding box, respectively. When regressing the added angle parameter, the regression of the rotated bounding box will cause rotation sensitivity error, leading to unstable model training and affecting detection accuracy. Therefore, the KLD loss function is introduced to calculate the bounding box regression loss.

[0075] The KLD loss function uses the ln function to smooth the loss function, and its expression is as follows:

[0076] The KLD loss function calculated above can dynamically adjust the weights according to the target scale, reducing the impact of small angular errors on accuracy and enabling high-precision rotation detection.

[0077] The YOLOv7 network structure consists of an Input layer, a Backbone layer, a Neck layer, and a Head layer connected sequentially. An ISE module is added to the backbone feature extraction part of the improved YOLOv7 model. In workpiece detection tasks, some targets are small, with few pixels in the image, and the real environment contains a lot of background interference. This module helps the network model quickly and effectively acquire the characteristic information of the workpiece in the image region and locate its position in the image. Meanwhile, the backbone feature network uses a large number of convolutions during feature extraction, which can lead to the loss of features for small workpiece targets. ISE utilizes the ability to acquire spatial and channel information of the target in the image to improve the target feature information and reduce feature loss.

[0078] An Integral Feature Enhancement (IFA) module is incorporated into the Head layer. This IFA module precedes the three Replication Modules (REP) modules in the Head layer. Through Convolutional Subtraction Animation (CSA) and convolution, the IFA module addresses the issue of insufficient feature information after the target object reaches the deep network, which negatively impacts feature representation. It allows the network model to absorb more feature information from the target object while reducing interference from existing feature information, thus helping the network achieve higher detection accuracy. Furthermore, it enhances semantic features, providing more accurate recognition.

[0079] The CSA module consists of a concatenated fusion layer of the input and output of the CA and SE modules connected in parallel, followed by the output. While the CA module improves the efficiency and accuracy of the attention module, its utilization of spatial domain information features is still insufficient, failing to capture long-range dependencies. By fusing with the SE module, the network better mitigates the impact of background information on the detection results, thus enabling better detection of workpiece targets.

[0080] The pose expression of the calibration board relative to the robot base is as follows:

[0081] C T Plane = C T U × U T Plane ;

[0082] for C T U There is also a formula:

[0083] C T U = C T CT × CT T U ;

[0084] in C TCT That is CT T C The transpose of the matrix; CT T U for U T CT The transpose matrix; based on the camera's internal participation C T Plane The pose of the 2D pixels of the target workpiece in this coordinate system is calculated, thereby completing the conversion of the 2D pose of the target workpiece to the 3D pose.

[0085] Traditional hand-eye calibration methods determine the robot's grasping pose at different workpiece positions by establishing a correspondence between the robot's grasping of the workpiece and the photographing of the workpiece. When an industrial robot grasps a workpiece, the following applies: in, To capture the pose of industrial robots. T T P The pose of the workpiece in the robot gripper coordinate system.

[0086] When an industrial robot takes a picture of a workpiece, the following applies: in, The robot's pose when the vision system acquires images. CT T C It has already been obtained in hand-eye calibration. C T P Let be the pose of the workpiece relative to the vision system. Using the above formula, the appropriate position can be calculated. U T CT When the robot's photo-taking pose changes, that is, when the robot's pose changes, the robot's new grasping pose. The transformation from 2D pose to 3D pose of camera-acquired images was achieved through matrix transformation. Then, in conjunction with robot hand-eye calibration parameters, high-precision calculation of the robot's 2D grasping pose was realized.

[0087] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely prisms of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A robot 2D vision 6DOF pose measurement and positioning grasping method for reflective metal objects, characterized in that: Includes the following steps: A virtual graphics dataset is created using images of reflective metal objects captured by a monocular camera in a 6DOF system and then processed by a virtual engine. Unet was used as the generator for the generative adversarial network, and the Transformer module was combined with the CycleGAN network to improve the quality of image features. The KLD loss function is combined with the YOLOv7 network to optimize the coordinates and angles output by the YOLOv7 network, thereby reducing angle errors. The pose of the calibration plate relative to the robot base is calculated by taking pictures of the fixed calibration plate. The pose of the calibration plane in the monocular camera coordinate system is calculated by the transpose matrix. The 2D pose is converted to 3D pose by the monocular camera intrinsic parameters. The loss function of the YOLOv7 network is: ； in For category loss, For bounding box regression loss, For confidence loss; pass Five parameters are used to determine the bounding box in any direction, through The angular information dimension of the target is introduced into the YOLOv7 network; in , These represent the x and y coordinates of the center point of the bounding box, respectively. , These represent the long and short sides of the bounding box, respectively. Depend on Positive axis direction and long side The angle between the two points determines the range of values, which is (-90°, 90°). exist When regressing the angle parameters, the KLD loss function transforms the rotated bounding box into a two-dimensional Gaussian distribution, and then calculates the relative entropy between the Gaussian distributions, i.e., the KL divergence, which is expressed as follows: ； In the above formula: ; ； ； in, , , , which are the center point coordinates and angle difference between the predicted bounding box and the ground truth bounding box, respectively; The KLD loss function uses the ln function to smooth the loss function, and its expression is as follows: 。 2. The robot 2D vision 6DOF pose measurement and positioning grasping method for reflective metal objects according to claim 1, characterized in that: The Transformer module calculates attention weights by performing a dot product operation between the query vector and the key vector, and then dividing by a scaling factor. The attention weights are then multiplied by the value vector, and the products are summed with weights to obtain the final output. The ability and efficiency of feature extraction in the image are changed by adjusting the weights of the query vector, key vector, and value vector.

3. The robot 2D vision 6DOF pose measurement and positioning grasping method for reflective metal objects according to claim 1, characterized in that: The category loss and confidence loss are binary cross-entropy loss functions, and their expressions are as follows: ； in The sigmoid function can convert... The expression for mapping to the interval (0, 1) is: 。 4. The robot 2D vision 6DOF pose measurement and positioning grasping method for reflective metal objects according to claim 1, characterized in that: The pose expression of the calibration board relative to the robot base is as follows: ； for The following formulas are given: ； in That is The transpose of the matrix; for The transpose matrix; based on the camera's internal participation The pose of the 2D pixels of the target workpiece in this coordinate system is calculated, thereby completing the conversion of the 2D pose of the target workpiece to the 3D pose.

Images