Multi-class grasp detection incremental learning method emphasizing task relationship

Abstract

A multi-class grasping detection incremental learning method emphasizing task relationship, in the training stage, a teacher network for grasping detection is constructed and trained using grasping detection dataset in turn, a student network for semantic segmentation and grasping detection is constructed and trained using small semantic segmentation dataset, and the grasping detection function of the student network is supervised and protected by the teacher network based on a task relationship function; in the detection stage, the real-time collected RGB image is input into the trained student network to obtain the grasping detection frame with category information and the semantic segmentation mask corresponding to the RGB image. The present application can efficiently train a multi-class grasping detection network to realize multi-class grasping detection in engineering.

Description

Technical Field

[0001] This invention relates to a technology in the fields of image recognition and robot control, specifically a multi-class grasping and detection incremental learning method that emphasizes task relationships. Background Technology

[0002] Existing multi-class grasping and detection technologies for robots simultaneously determine the category and location of the object to be grasped through multi-task neural networks. However, they require training and validation on multi-task datasets with complex labels. Due to the special characteristics of the target to be grasped in engineering, publicly available multi-task datasets cannot be used directly. Creating complex multi-task datasets is a huge undertaking and cannot make full use of existing single-task dataset resources. Summary of the Invention

[0003] This invention addresses the shortcomings of existing robot grasping technologies for complex multi-class scenarios, which require training and validating networks using multi-task datasets with both object detection and grasping detection labels. It also addresses the issue that the categories of targets to be grasped in real-world engineering are specific, making it impossible to directly use publicly available multi-task datasets, and that creating complex multi-task datasets is extremely labor-intensive. The invention proposes a multi-class grasping detection incremental learning method that emphasizes task relationships, enabling efficient training of multi-class grasping detection networks for use in engineering projects.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a multi-class grasping detection incremental learning method that emphasizes task relationships. In the training phase, a teacher network for grasping detection is constructed and trained using a grasping detection dataset, and a student network for semantic segmentation and grasping detection is constructed and trained using a small semantic segmentation dataset. At the same time, the teacher network supervises and protects the grasping detection function of the student network based on a task relationship function. In the detection phase, real-time acquired RGB images are input into the trained student network to obtain grasping detection boxes and semantic segmentation masks with category information corresponding to the RGB images.

[0006] Technical effect

[0007] This invention divides multi-class grasping detection into two sub-tasks: grasping detection and semantic segmentation. It employs ordered training, using separate grasping detection and semantic segmentation datasets to train the network. During training, a task relation function is used to link the two sub-tasks, emphasizing their task relationship. This method allows the trained multi-task network to simultaneously determine the category and location of the grasped object. Training does not rely on a multi-task dataset with both grasping detection and semantic segmentation labels, and the model achieves high detection accuracy and efficiency. Compared to existing technologies, this invention does not require training on a multi-task dataset with both grasping detection and semantic segmentation labels. This technical solution makes the training process more efficient and achieves higher detection accuracy and efficiency. By sharing a feature extraction network, the model can learn multiple tasks within a single network, improving the overall performance and effectiveness of the model. During incremental learning, by introducing task relations, the semantic segmentation dataset is effectively used to correct and filter pseudo-labels output by the teacher network. This allows the student network with the added semantic segmentation branch to retain or improve its performance in grasping detection accuracy while simultaneously acquiring semantic segmentation capabilities, further assisting the model's performance in grasping and detecting specific object categories. This method can make full use of existing large-scale crawling detection datasets to improve the model's generalization ability and adaptability; at the same time, it can create small-scale semantic segmentation datasets to meet the specific needs of crawling targets in engineering. Compared with creating large-scale multi-task datasets, it requires less work and is more practical. Attached Figure Description

[0008] Figure 1 This is a flowchart of the present invention;

[0009] Figure 2 This is a diagram showing the overall structure of the system of the present invention;

[0010] Figure 3 This is a comparison chart showing the visualization results of whether or not the task relationship function is used. Detailed Implementation

[0011] like Figure 1 As shown in the figure, this embodiment relates to a multi-class incremental learning method for grasping detection that emphasizes task relationships. In the training phase, a teacher network for grasping detection is constructed and trained using a grasping detection dataset, and a student network for semantic segmentation and grasping detection is constructed and trained using a small semantic segmentation dataset. At the same time, the teacher network supervises and protects the grasping detection function of the student network based on the task relationship function. In the detection phase, the real-time acquired RGB images are input into the trained student network to obtain the grasping detection boxes and semantic segmentation masks with category information corresponding to the RGB images.

[0012] The aforementioned crawling detection dataset refers to a crawling detection branch that is robust and capable of crawling any object, based on a publicly available large-scale crawling detection dataset. This dataset includes RGB images and crawling boxes using a five-dimensional crawling representation {x,y,θ,w,h}.

[0013] The teacher network includes a first feature extraction network and a first crawling detection branch connected in sequence, wherein: the first feature extraction network performs calculations and recognition based on RGB images in the crawling detection dataset to obtain high-dimensional features related to crawling detection corresponding to the RGB images; the first crawling detection branch calculates the crawling detection loss based on the high-dimensional features and the true crawling detection labels.

[0014] The training using the crawling detection dataset specifically includes:

[0015] 1) Convert the RGB images of the capture detection dataset into the input format of the feature extraction network, and input them into the feature extraction network for recognition calculation to extract high-dimensional features of the RGB image capture detection information;

[0016] 2) The grasping and detection branch outputs the grasping and detection results of the target object based on high-dimensional features, including multiple grasping detection boxes and recognition confidence scores;

[0017] 3) Using the crawling detection results and the true crawling detection labels in the crawling detection dataset as input, calculate the crawling detection loss L. grasp =L RPN +L box +L rot L RPN To train the region candidate network in the grasp detection branch, the network outputs undirected region candidate boxes {x,y,w,h}, L RPN It consists of classification loss and regression loss. The classification loss uses the cross-entropy loss function to calculate the difference between the target score of the candidate box generated by the region candidate network and the target score of the true label. The regression loss uses a smoothing function. L1 The function calculates the positioning offset; L box To use smooth L1 The function calculates the positioning offset of the capture box {x, y, w, h}; L rot The score for the category of the detection box rotation direction θ and the true label rotation angle is calculated using the cross-entropy function.

[0018] 4) To minimize the capture detection loss, update the parameters of the feature extraction network and the capture detection branch, repeating steps 1) to 3) until the preset number of iterations is reached. The student network includes: a second feature extraction network and a second capture detection branch and a second semantic segmentation branch connected to it, wherein: the second feature extraction network performs calculations and recognition based on the RGB images in the semantic segmentation dataset to obtain high-dimensional features related to capture detection and semantic segmentation corresponding to the RGB images; the second capture detection branch calculates the knowledge distillation loss L based on the capture detection results and capture detection pseudo-labels. d The second semantic segmentation branch generates a semantic segmentation mask based on the high-dimensional features obtained from the second feature extraction network, and calculates the semantic segmentation loss L based on the semantic segmentation results and the true semantic segmentation labels. seg .

[0019] The parameters of the second feature extraction network and the second grasping and detection branch are preferably initialized to the parameters of the corresponding network in the trained teacher network, while the parameters of the second semantic segmentation branch are randomly initialized.

[0020] The pseudo-labels for capture detection are obtained by the teacher network through calculation and recognition of RGB images in the semantic segmentation dataset, and then filtered by the task relation function to obtain the valid capture boxes.

[0021] The task relationship function refers to setting a capture box as a valid capture box when its center point {x,y} is located within the mask of the target object and the number of capture boxes of that category has not reached a threshold, and ultimately obtaining a specific threshold number of valid capture boxes, specifically including:

[0022] i) Preset the total number of capture boxes N and the number of capture boxes for each category N array. cl ={n1,n2,...,n n The semantic segmentation mask and its category array S for the current target object are obtained by acquiring the ground truth labels for semantic segmentation using a small semantic segmentation dataset. cl ={s1,s2,...,s n}, where n is the number of categories;

[0023] ii) Based on the crawling detection results obtained from the first crawling branch of the teacher network, namely the crawling boxes {x,y, θ,w,h} and the crawling confidence, sequentially traverse the crawling boxes {x,y, θ,w,h} arranged in descending order of crawling confidence. When the center point {x,y} of the crawling box is located within the semantic segmentation mask of the current target object and s(x,y)=s i n i When n > 0, the capture box is set as a valid capture box, and n iDecrease by one, the total number of capture boxes N is reduced by one, and valid capture boxes are saved;

[0024] iii) After traversing all the capture detection results obtained from the first capture branch, output all valid capture boxes.

[0025] The aforementioned small semantic segmentation dataset is obtained in the following way: a dataset with the specific characteristics of targets such as parts and retail goods is established based on actual engineering projects. This data includes: RGB images, mask contours of target objects in RGB images and their categories.

[0026] The training using a small semantic segmentation dataset refers to the following: RGB images from the small semantic segmentation dataset are simultaneously input into the trained teacher network and student network. After obtaining the grasping detection results, the teacher network uses a task relation function to filter based on the real label mask in the semantic segmentation dataset to obtain grasping detection pseudo-labels corresponding to the RGB images. The student network, after recognition calculations by the second grasping detection branch and the second semantic segmentation branch, obtains grasping detection results and semantic segmentation results corresponding to the RGB images, respectively. Then, the knowledge distillation loss is calculated based on the grasping detection information and grasping detection pseudo-labels obtained from the second grasping detection branch. Finally, the semantic segmentation loss is calculated based on the semantic segmentation information and the semantic segmentation real labels in the small semantic segmentation dataset. The above processing is iteratively performed with the goal of minimizing the knowledge distillation loss and the semantic segmentation loss to achieve training.

[0027] The knowledge distillation loss L d In form, it is similar to the loss of capture detection. grasp Maintain consistency to monitor and protect the scraping detection capabilities of student networks.

[0028] The semantic segmentation loss L seg The classification loss is calculated for each pixel in the image using a weighted cross-entropy function.

[0029] like Figure 2As shown, this invention relates to a system for implementing the above method, comprising: a teacher network construction and training module, a student network construction and training module, a task relationship module, and a recognition, detection, and matching module, wherein: the teacher network construction and training module constructs and trains a teacher network with grasping and detection functions, consisting of a feature extraction network and a grasping and detection branch cascaded together; the student network construction and training module adds a semantic segmentation branch in parallel with the grasping and detection branch to the teacher network to construct a student network branch, and trains the semantic segmentation branch of the student network using a semantic segmentation dataset to obtain a student network with both grasping and detection functions and semantic segmentation functions; the task relationship module, when the teacher network supervises and protects the student network, performs a filtering operation on the pseudo-labels of the grasping and detection of the teacher network based on the real labels of the semantic segmentation dataset, and supervises and protects the student network according to the grasping and detection branch of the student network provided by the teacher network to prevent catastrophic forgetting; the recognition, detection, and matching module uses an RGB image sensor to acquire scene detection images in real time, inputs the scene detection images into the student network, and the recognition information matching unit performs a matching operation on the grasping and detection results and semantic segmentation results provided by the student network, outputting a grasping detection box and a semantic segmentation mask with category information.

[0030] like Figure 3 As shown in the actual experiments, incremental learning of the network was performed using a crawling detection dataset and a semantic segmentation dataset. The task relation function was used to filter pseudo-labels. Compared with not using the relation function, the recall rate of crawling detection was significantly improved, and more crawling boxes that were not easy to identify were captured. At the same time, it was also more robust to textured backgrounds, which could prevent misidentification of backgrounds with complex textures.

[0031] Compared with existing technologies, this invention uses task relation functions to filter and correct pseudo-labels in the teacher network, making knowledge distillation more reliable, making full use of relation information provided by different task datasets, and improving the accuracy and efficiency of joint tasks.

[0032] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A multi-class grasping and detection incremental learning method emphasizing task relationships, characterized in that, During the training phase, a teacher network for grasp detection is constructed and trained using a grasp detection dataset, and a student network for semantic segmentation and grasp detection is constructed and trained using a small semantic segmentation dataset. Meanwhile, the teacher network supervises and protects the grasp detection function of the student network based on a task relation function. During the detection phase, real-time acquired RGB images are input into the trained student network to obtain grasp detection boxes and semantic segmentation masks with category information corresponding to the RGB images. The aforementioned crawling detection dataset refers to a dataset used to train a robust crawling detection branch capable of crawling any object, based on a publicly available large-scale crawling detection dataset. This dataset includes RGB images and five-dimensional crawling representations. The capture box; The student network includes a second feature extraction network and a second grasping detection branch and a second semantic segmentation branch connected to it. The second feature extraction network performs calculations and recognition based on RGB images in the semantic segmentation dataset to obtain high-dimensional features related to grasping detection and semantic segmentation corresponding to the RGB images. The second grasping detection branch calculates the knowledge distillation loss based on the grasping detection results and grasping detection pseudo-labels. The second semantic segmentation branch generates a semantic segmentation mask based on the high-dimensional features obtained from the second feature extraction network, and calculates the semantic segmentation loss based on the semantic segmentation results and the true semantic segmentation labels. ; The parameters of the second feature extraction network and the second grasping and detection branch are initialized to the parameters of the corresponding network in the trained teacher network, and the parameters of the second semantic segmentation branch are randomly initialized. The aforementioned capture detection pseudo-labels are obtained by the teacher network through calculation and recognition of RGB images in the semantic segmentation dataset, and then filtered by the task relation function to obtain the valid capture boxes; The aforementioned task relationship function refers to: based on the capture box center point If the bounding box of a certain category is located within the mask of the target object and the threshold has not been reached, the bounding box is set as a valid bounding box, and a specific threshold number of valid bounding boxes are finally obtained. The aforementioned task relationship function is obtained in the following way: i) Preset total number of capture boxes and the array of category capture boxes By obtaining the true semantic segmentation labels from a small semantic segmentation dataset, we can obtain the semantic segmentation mask and its class array for the current target object. Where: n is the number of categories; ii) Based on the crawl detection results obtained from the first crawl branch of the teacher network, i.e., the crawl box And crawl confidence, iterate through the crawl boxes sorted in descending order of crawl confidence. When the center point of the capture box Located within the semantic segmentation mask of the current target object and , When this happens, the capture box is set as a valid capture box. Subtract one, the total number of capture boxes Decrease by one, and save the valid capture frame; iii) After traversing all the capture detection results obtained from the first capture branch, output all valid capture boxes.

2. The multi-class grasping and detection incremental learning method emphasizing task relationships as described in claim 1, characterized in that, The teacher network includes a first feature extraction network and a first crawling detection branch connected in sequence, wherein: the first feature extraction network performs calculations and recognition based on RGB images in the crawling detection dataset to obtain high-dimensional features related to crawling detection corresponding to the RGB images; the first crawling detection branch calculates the crawling detection loss based on the high-dimensional features and the true crawling detection labels.

3. The multi-class grasping and detection incremental learning method emphasizing task relationships as described in claim 1, characterized in that, The training using the crawling detection dataset specifically includes: 1) Convert the RGB images of the capture detection dataset into the input format of the feature extraction network, and input them into the feature extraction network for recognition calculation to extract high-dimensional features of the RGB image capture detection information; 2) The grasping and detection branch outputs the grasping and detection results of the target object based on high-dimensional features, including multiple grasping detection boxes and recognition confidence scores; 3) Calculate the crawl detection loss by taking the crawl detection results and the true crawl detection labels in the crawl detection dataset as input. ;in To train the region candidate network in the grasp detection branch, this network outputs unoriented region candidate boxes. , It consists of classification loss and regression loss. The classification loss uses the cross-entropy loss function to calculate the difference between the target score of the candidate box generated by the region candidate network and the target score of the true label. The regression loss uses... The function calculates the positioning offset; For use Function calculation capture box The positioning offset; Calculate the rotation direction of the detection frame The score for the rotation angle category relative to the true label is calculated using the cross-entropy function; 4) With the goal of minimizing the capture detection loss, update the parameters of the feature extraction network and the capture detection branch, and repeat steps 1) to 3) until the preset number of iterations is reached.

4. The multi-class grasping and detection incremental learning method emphasizing task relationships as described in claim 1, characterized in that, The aforementioned small semantic segmentation dataset is obtained in the following way: a dataset with the specific characteristics of parts and retail goods targets is established based on actual engineering projects. This data includes: RGB images, mask contours of target objects in RGB images and their categories; The training using a small semantic segmentation dataset refers to the following: RGB images from the small semantic segmentation dataset are simultaneously input into the trained teacher network and student network. After obtaining the grasping detection results, the teacher network uses a task relation function to filter based on the real label mask in the semantic segmentation dataset to obtain grasping detection pseudo-labels corresponding to the RGB images. The student network, after recognition calculations by the second grasping detection branch and the second semantic segmentation branch, obtains grasping detection results and semantic segmentation results corresponding to the RGB images, respectively. Then, the knowledge distillation loss is calculated based on the grasping detection information and grasping detection pseudo-labels obtained from the second grasping detection branch. Finally, the semantic segmentation loss is calculated based on the semantic segmentation information and the semantic segmentation real labels in the small semantic segmentation dataset. The above processing is iteratively performed with the goal of minimizing the knowledge distillation loss and the semantic segmentation loss to achieve training.

5. A multi-class grasping and detection incremental learning system emphasizing task relationships, implementing the method of any one of claims 1-4, characterized in that, include: The system comprises a teacher network construction and training module, a student network construction and training module, a task relationship module, and a recognition, detection, and matching module. Specifically: the teacher network construction and training module constructs and trains a teacher network with grasping and detection capabilities, consisting of a cascaded feature extraction network and a grasping and detection branch; the student network construction and training module adds a semantic segmentation branch in parallel with the grasping and detection branch to the teacher network, constructing a student network branch; and trains the semantic segmentation branch of the student network using a semantic segmentation dataset, resulting in a student network with both grasping and detection capabilities; the task relationship module, when supervising and protecting the student network, filters pseudo-labels for grasping and detection based on the real labels in the semantic segmentation dataset, and supervises and protects the student network based on the grasping and detection branch of the teacher network to prevent catastrophic forgetting; the recognition, detection, and matching module uses an RGB image sensor to acquire scene detection images in real time, inputs these images into the student network, and the recognition information matching unit performs matching operations on the grasping and detection results and semantic segmentation results provided by the student network, outputting a grasping detection bounding box with category information and a semantic segmentation mask.

Images