Visual grasping detection method and system based on self-supervised representation learning
Through self-supervised representation learning and twin networks, the common features of visual grab objects are extracted, combined with the encoder-decoder structure, the problem of expensive visual grab labeling and improved performance is solved, and more efficient visual grabber ability and generalization are achieved performance.
Patent Information
- Application Number
- CN202210530861.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-05-16
AI Technical Summary
In the prior art, visual grab labeling is expensive and the effect of improving visual grab performance is poor, making it difficult to effectively solve the need for robots to capture independently.
Using a self-supervised representation learning method, the common features of similar crawler objects are extracted through twin networks and metric learning losses, combined with the encoder-decoder architecture, reduce the burden of data annotation, improve the generalization ability of model, and utilize the marked data The fine-tuning codec generates the best grab pose.
It realizes more accurate spatial and richer semantic representations, reduces the burden of data annotation, speeds up model training, and improves the generalization ability and grab detection performance of visual grabbing.
Smart Images

Figure CN114820796B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of visual grasping detection, and in particular to a visual grasping detection method and system based on self-supervised representation learning. Background Art
[0002] With the continuous development of robotics technology, robots are gradually used in scenarios such as industry and home, and these application scenarios usually require robots to be able to perform autonomous grasping. At present, most industrial robots still perform repetitive tasks based on fixed-position grasping. In order to adapt to the development of robotics technology, autonomous grasping of robots has emerged. Autonomous grasping involves perception, planning, and control. Robots need to interact with the environment and complete tasks autonomously, that is, they need to know which object to grasp and how to grasp it.
[0003] A complete robot grasping process can be divided into the following three steps: grasp detection, trajectory planning, and motion control. Grasp detection is the process of locating an object and generating the grasping posture of the object. It can be roughly divided into two types of methods: geometric analysis-based methods and data-driven methods. Geometric analysis-based methods usually perform grasp detection by analyzing and calculating the geometry and kinematics of physical models. Data-driven methods are mainly based on machine learning methods and usually require a large number of manually labeled samples.
[0004] Patent document CN109702741A discloses a robotic arm visual grasping system based on a self-supervised learning neural network, including: a depth of field camera, an instance segmentation module, a pose estimation neural network module, a stereo pose acquisition module and a robotic arm control module, wherein the depth of field camera outputs a color image to the instance segmentation module, and outputs a depth of field image to the stereo pose acquisition module; the instance segmentation module inputs at least one of the classification, frame or segmentation information into the pose estimation neural network module; the pose estimation neural network module then outputs a plane pose to the stereo pose acquisition module, which is fused with the depth of field image to obtain a stereo pose; the robotic arm control module obtains the stereo pose, and implements the robotic arm grasping operation according to the stereo pose. However, this method does not effectively solve the technical problems of expensive visual grasping annotation and improving visual grasping performance. Summary of the invention
[0005] In view of the defects in the prior art, the object of the present invention is to provide a visual grasping detection method and system based on self-supervised representation learning.
[0006] A visual grasping detection method based on self-supervised representation learning provided by the present invention comprises:
[0007] Step 1: According to the acquired depth image, obtain a labeled image dataset and an unlabeled image dataset;
[0008] Step 2: Train the unlabeled image dataset through the Siamese network in the encoder, and extract the common features between similar grasped objects through the metric learning loss method;
[0009] Step 3: Fine-tune the decoder using the labeled image dataset to obtain a training model;
[0010] Step 4: Perform visual crawling by training the model.
[0011] Preferably, step 1 comprises:
[0012] Step 101: Acquire a depth image through a 3D camera;
[0013] Step 102: labeling the first part of the depth image to obtain a labeled image data set;
[0014] Step 103: Obtain an unlabeled image dataset based on the remaining parts of the depth image.
[0015] Preferably, step 2 comprises:
[0016] Step 201: After data enhancement processing is performed on the depth image in the unlabeled image dataset, the data is input into the twin network for training to obtain an enhanced image;
[0017] Step 202: Utilize the loss value of metric learning to extract common features between similar grasped objects in the enhanced image.
[0018] Preferably, step 3 comprises:
[0019] Step 301: fine-tune the labeled image dataset using a back-propagation algorithm and a gradient optimization algorithm to obtain a training model.
[0020] Preferably, step 4 comprises:
[0021] Step 401: according to the training model, input a depth image and output a capture quality map;
[0022] Step 402: Obtain grasping quality points according to the grasping quality map and perform visual grasping.
[0023] A visual grasping detection system based on self-supervised representation learning provided by the present invention comprises:
[0024] Module M1: Obtain annotated image dataset and unannotated image dataset based on the acquired depth image;
[0025] Module M2: Train the unlabeled image dataset through the Siamese network in the encoder and extract the common features between similar grasped objects through the metric learning loss method;
[0026] Module M3: Fine-tune the decoder using the labeled image dataset to obtain a training model;
[0027] Module M4: Visual grasping by training models.
[0028] Preferably, module M1 comprises:
[0029] Submodule M101: obtain depth image through 3D camera;
[0030] Submodule M102: annotate the first part of the depth image to obtain an annotated image data set;
[0031] Module M103: Obtain an unlabeled image dataset based on the rest of the depth image.
[0032] Preferably, module M2 comprises:
[0033] Submodule M201: After data enhancement processing is performed on the depth image in the unlabeled image dataset, it is input into the twin network for training to obtain an enhanced image;
[0034] Submodule M202: Utilize the loss value of metric learning to extract common features between similar grasped objects in the enhanced image.
[0035] Preferably, module M3 comprises:
[0036] Submodule M301: Use the back propagation algorithm and gradient optimization algorithm to fine-tune the labeled image dataset to obtain a training model.
[0037] Preferably, module M4 comprises:
[0038] Submodule M401: according to the training model, input the depth image and output the capture quality map;
[0039] Submodule M402: According to the grasping quality map, the grasping quality points are obtained and visual grasping is performed.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] 1. The encoder-decoder architecture in the present invention integrates low-level contour feature information and high-level semantic feature information, achieving a more spatially accurate and semantically richer representation, reducing the burden of data annotation, accelerating model training, and improving the generalization ability during capture and detection.
[0042] 2. The present invention uses a twin network in the encoder to filter out redundant information from the input data. At the same time, through various data enhancement techniques and with the help of metric learning loss, the commonalities between similar grasped objects are extracted, thus improving the generalization ability of the model.
[0043] 3. The present invention uses the labeled data to fine-tune the decoder, infer the best grasping candidate from the grasping quality map and the grasping pose generated by the subsequent grasping plan, and thus improve the visual grasping capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0045] Figure 1 It is a schematic diagram of the process of the present invention;
[0046] Figure 2 It is a schematic diagram of the self-supervised representation learning structure of the present invention;
[0047] Figure 3 It is a schematic diagram of the network structure of the present invention. DETAILED DESCRIPTION
[0048] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0049] The nouns involved in the present invention are explained below:
[0050] Self-supervised learning: refers to a type of unsupervised learning, which is a method of learning a method for expressing common features of downstream tasks. Self-supervised learning mainly uses auxiliary tasks to mine its own supervisory information from large-scale unsupervised data, and trains the network through this constructed supervisory information to learn representations that are valuable for downstream tasks.
[0051] Representation Learning: refers to a data preprocessing method that projects high-dimensional data into a low-dimensional space, making it easier to find a better expression for the original data for subsequent tasks.
[0052] Figure 1 It is a schematic diagram of the process of the present invention, such as Figure 1 As shown, the present invention provides a visual grasping detection method based on self-supervised representation learning, comprising the following steps:
[0053] Step 1: Based on the acquired depth image, obtain a labeled image dataset and an unlabeled image dataset.
[0054] Specifically, step 1 includes: step 101: acquiring a depth image through a 3D camera; step 102: annotating the first part of the depth image to obtain an annotated image data set; step 103: obtaining an unannotated image data set based on the remaining parts of the depth image.
[0055] The 3D camera may be an RGB-D depth camera.
[0056] The present invention does not limit the selection of the first part, and it can be set according to specific circumstances. For example, it can be 30% of all the depth image data.
[0057] In an optional implementation, a large number of depth images may be collected using an RGB-D camera, and part of the depth image data may be manually annotated to produce an annotated image dataset, and the rest may be an unannotated image dataset.
[0058] Specifically, the depth image with a small amount of annotation information is presented in the form of a grab rectangle around the object, which is specifically expressed as formula (1):
[0059] g=(x,y,θ,ω); (1)
[0060] Among them, (x, y) represents the center coordinates of the grasping rectangle in the depth image; θ represents the counterclockwise rotation angle of the gripper along the z-axis; ω represents the predicted width of the grasping rectangle.
[0061] It should be noted that because the grasping rectangle cannot reflect the quality of the generated grasping rectangle, multiple candidates must be generated and the best grasping rectangle must be selected from them. This step is very time-consuming. To solve this problem, the grasping graph G is used as the output of the decoder as a pixel-level method to describe the grasping state, which can be expressed by formula (2):
[0062]
[0063] Where G represents the grasping graph; G includes the grasping quality graph Q, the grasping angle graph θ, and the grasping width graph W. Grasping quality graph is a heat map of the same size as the input image, H and W represent the height and width of the input image respectively. The value of each pixel in Q is a probability value used to evaluate the possibility of successful grasping at that point. All pixel values on the grasp quality map are between 0 and 1, where a larger value indicates a higher probability of successful grasping at that location. Each pixel value in the W and Θ maps represents the corresponding width and angle of the gripper at that grasping location.
[0064] Step 2: In the encoder, the Siamese network is trained on the unlabeled image dataset, and the common features between similar grasped objects are extracted through the metric learning loss method.
[0065] Among them, step 2 includes: step 201: after performing data enhancement processing on the depth image in the unlabeled image dataset, input it into the twin network for training to obtain an enhanced image; step 202: using the loss value of metric learning to extract common features between similar captured objects in the enhanced image.
[0066] Specifically, data augmentation technology is used to scale, translate, and rotate the depth images of unlabeled information in the unlabeled image dataset, and then input them into the twin network for training to obtain enhanced images. The loss value of metric learning is used to extract invariant features, so that it can capture similar characteristics of different objects at different angles.
[0067] In an optional embodiment, the RGB-D image x is generated by different data augmentation methods. and As the positive sample of the original image, it is put into the two branches of the twin network respectively. Figure 2 Schematic diagram of the self-supervised representation learning structure of the present invention, Figure 2 As shown in Figure 1, the observation state after data enhancement technology is input as a positive sample into the two branches of the twin network. The encoder is represented by f, and the multi-layer perceptron prediction projection layer after the encoder is represented by h. The output p1 of a branch after the encoder f and the prediction projection layer h is represented as Matched with the other branch output z2, it is expressed as Metric learning is performed, and the output z2 of the other branch is input into the decoder of the grasp detection process. The distance function D(p1,z2) is expressed as a negative cosine similarity function, which can be expressed by formula (3):
[0068]
[0069] Here, ||·|| represents the l2-norm equation.
[0070] Furthermore, the loss function L is defined as formula (4):
[0071] L=D(p1,z2)+D(p2,z1); (4)
[0072] Among them, p1 and p2 represent the output through the encoder and prediction projection layer; z1 and z2 represent the output through the encoder only. Then, the similarity characteristics of the two branches are maximized by measuring the learning loss.
[0073] In order to avoid mode collapse of the solution obtained in the present invention, the loss function L is modified to formula (5):
[0074] L=D(p1,stopgrad(z2))+D(p2,stopgrad(z1)); (5)
[0075] The stopgrad operation means that no gradient update is performed. After the stopgrad operation, z2 can be regarded as a constant term and is no longer updated by the gradient.
[0076] Step 3: Fine-tune the decoder using the labeled image dataset to obtain a training model.
[0077] Wherein, step 3 includes: step 301: using the back propagation algorithm and the gradient optimization algorithm to fine-tune the labeled image data set to obtain a training model.
[0078] Specifically, the decoder is fine-tuned using the depth image data with annotated information in a small amount of annotated image data sets, and the gradient of the objective function is optimized using the back propagation algorithm and the gradient optimization algorithm to minimize the difference between the detected grasping model and the true value, thus obtaining the training model. The objective function can be expressed by formula (6):
[0079]
[0080] Among them, L represents the difference between the network output and the true value; represents the true value of the grasping map; G = (Θ, W, Q) represents the three heat maps output by the network, Θ represents the grasping angle map; W represents the grasping width map; Q represents the grasping quality map.
[0081] In an optional implementation, a small amount of depth image data with annotated information is used to fine-tune the decoder. Figure 3 It is a schematic diagram of the network structure of the present invention, such as Figure 3As shown in the figure, it includes an RGB-D camera, a workspace, and an overall network. The encoder consists of four blocks, each of which contains a 3*3 convolutional (Conv) layer, a linear rectifier function (ReLU), a batch normalization (BN) layer, and a maximum pooling layer (MaxPooling) with a stride of 2. The decoder consists of a deconvolutional neural network, which is upsampled so that the output gradually recovers to the same size as the input. The output of the encoder is connected to the corresponding layer of the decoder through skip connections. The last layer is a 1*1 convolutional (Conv) layer. The network decodes the data output by the encoder into a capture quality map, a capture angle map, and a capture width map of the same size as the input. The fine-tuning process uses the backpropagation algorithm and the gradient optimization algorithm to minimize the difference between the detected capture model and the true value to obtain the training model.
[0082] Step 4: Perform visual crawling by training the model.
[0083] Among them, step 4 includes: step 401: according to the training model, input the depth image and output the grasping quality map; step 402: according to the grasping quality map, obtain the grasping quality points and perform visual grasping.
[0084] Specifically, according to the training model based on self-supervised representation learning obtained through training, real depth image data is input, and a grasping quality map, a grasping angle map and a grasping width heat map are output. According to the grasping quality map, the grasping quality points are obtained and visual grasping is performed.
[0085] In an optional implementation, according to the trained grasp detection model, i.e., the training model, a real RGB-D image is used as input to output three pixel-level grasp quality maps, grasp angle maps, and grasp width heat maps. The resulting grasp point is the position with the highest quality score in the grasp quality map.
[0086] The present invention can be used to efficiently process image data and improve visual capture capabilities.
[0087] The following is the algorithm corresponding to the visual detection method based on self-supervised representation learning provided by the present invention:
[0088] Specifically, the input is a set of unlabeled images D, labeled images D′, and image transformation distribution 、Encoder f θ ,d θ and the multilayer perceptron predicts the projection layer h θ, pre-training step K, training step N and batch size M; the output is the encoder f θ and decoder d φ .
[0089] First, for self-supervised training, the corresponding algorithm is as follows:
[0090]
[0091] Then, fine-tuning is performed using a small amount of annotated depth data.
[0092]
[0093]
[0094] The present invention also provides a visual grasping detection system based on self-supervised representation learning, comprising:
[0095] Module M1: Based on the acquired depth image, obtain a labeled image dataset and an unlabeled image dataset.
[0096] Among them, module M1 includes: submodule M101: obtaining a depth image through a 3D camera; submodule M102: annotating the first part of the depth image to obtain an annotated image data set; module M103: obtaining an unannotated image data set based on the remaining parts of the depth image.
[0097] Module M2: In the encoder, the unlabeled image dataset is trained through a Siamese network, and the common features between similar grasped objects are extracted through a metric learning loss method.
[0098] Among them, module M2 includes: sub-module M201: after performing data enhancement processing on the depth image in the unlabeled image dataset, it is input into the twin network for training to obtain an enhanced image; sub-module M202: using the loss value of metric learning to extract common features between similar grasped objects in the enhanced image.
[0099] Module M3: Fine-tune the decoder using the labeled image dataset to obtain a training model.
[0100] Among them, module M3 includes: submodule M301: using the back propagation algorithm and the gradient optimization algorithm to fine-tune the labeled image data set to obtain a training model.
[0101] Module M4: Visual grasping by training models.
[0102] Among them, module M4 includes: sub-module M401: according to the training model, input the depth image and output the grasping quality map; sub-module M402: according to the grasping quality map, obtain the grasping quality points and perform visual grasping.
[0103] The technical principle of the present invention is:
[0104] The present invention designs a new encoder-decoder architecture. By introducing self-supervised learning, a Siamese Network is used in the encoder to filter redundant information from the input data. At the same time, various data enhancement techniques are used to extract the commonalities between similar grasped objects with the help of metric learning loss. The decoder is fine-tuned using labeled data, and the best grasping candidate is inferred from the grasping quality map and the grasping pose generated by the subsequent grasping plan, thereby improving the robot's visual grasping ability. Self-supervised learning is divided into two stages: the first stage does not involve any downstream grasping tasks, that is, pre-training is performed using unlabeled grasped images; the second stage involves downstream grasping tasks, that is, fine-tuning on downstream grasping tasks using labeled data.
[0105] Compared with the prior art, the present invention has the following beneficial effects:
[0106] 1. The present invention designs a new encoder-decoder structure, which combines low-level contour feature information and high-level semantic feature information to achieve a more spatially accurate and semantically richer representation.
[0107] 2. The present invention reduces the burden of data annotation by introducing self-supervised learning, solves the problem of expensive visual capture and annotation, and speeds up model training.
[0108] 3. The present invention uses a twin network in the encoder to filter out redundant information from the input data. At the same time, through various data enhancement techniques and with the help of metric learning loss, the commonalities between similar grasped objects are extracted, thereby improving the generalization ability of the model.
[0109] 4. The present invention uses the labeled data to fine-tune the decoder, infer the best grasping candidate from the grasping quality map and the grasping pose generated by the subsequent grasping plan, thereby improving the visual grasping capability.
[0110] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method submodule M. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures within hardware components.
[0111] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.
Claims
1. A visual grasping detection method based on self-supervised representation learning, characterized in that: include: Step 1: According to the acquired depth image, obtain a labeled image dataset and an unlabeled image dataset; Step 2: In the encoder, the unlabeled image dataset is trained by a Siamese network, and common features between similar grasped objects are extracted by a metric learning loss method; Step 3: fine-tune the decoder using the labeled image dataset to obtain a training model; Step 4: Perform visual capture using the training model; The step 1 comprises: Step 101: Acquire the depth image through a 3D camera; Step 102: performing labeling processing on the first part of the depth image to obtain the labeled image data set; Step 103: Obtain the unlabeled image dataset according to the remaining parts of the depth image; The step 2 comprises: Step 201: After performing data enhancement processing on the depth image in the unlabeled image dataset, the data is input into the twin network for training to obtain an enhanced image; Step 202: extracting common features between similar captured objects in the enhanced image using the loss value of metric learning.
2. The visual grasping detection method based on self-supervised representation learning according to claim 1 is characterized in that: The step 3 comprises: Step 301: fine-tune the labeled image dataset using a back-propagation algorithm and a gradient optimization algorithm to obtain the training model.
3. The visual grasping detection method based on self-supervised representation learning according to claim 1, characterized in that: The step 4 comprises: Step 401: according to the training model, input a depth image and output a capture quality map; Step 402: Obtain grasping quality points according to the grasping quality map and perform visual grasping.
4. A visual grasping detection system based on self-supervised representation learning, characterized in that: include: Module M1: Obtain annotated image dataset and unannotated image dataset based on the acquired depth image; Module M2: training the unlabeled image dataset through a Siamese network in an encoder, and extracting common features between similar grasped objects through a metric learning loss method; Module M3: fine-tuning the decoder using the labeled image dataset to obtain a training model; Module M4: visual grasping through the training model; The module M1 comprises: Submodule M101: acquiring the depth image through a 3D camera; Submodule M102: performing labeling processing on the first part of the depth image to obtain the labeled image data set; Module M103: obtaining the unlabeled image dataset according to the remaining parts of the depth image; The module M2 comprises: Submodule M201: After performing data enhancement processing on the depth image in the unlabeled image dataset, the data is input into the twin network for training to obtain an enhanced image; Submodule M202: extracting common features between similar captured objects in the enhanced image using the loss value of metric learning.
5. The visual grasping detection system based on self-supervised representation learning according to claim 4 is characterized in that: The module M3 comprises: Submodule M301: fine-tune the labeled image data set using a back-propagation algorithm and a gradient optimization algorithm to obtain the training model.
6. The visual grasping detection system based on self-supervised representation learning according to claim 4, characterized in that: The module M4 comprises: Submodule M401: according to the training model, input the depth image and output the capture quality map; Submodule M402: Obtain grasping quality points according to the grasping quality map and perform visual grasping.
