Target detection method and autonomous driving method based on incremental learning

Abstract

The present invention discloses a target detection method based on incremental learning, including obtaining an original target detection initial model; pre-training some parameters of the feature extractor of the original target detection initial model to obtain a general target detection feature extractor; using a detection head and a parameter mask to structurally expand the target detection feature extractor to obtain an extended target detection model: using the extended target detection model to perform actual target detection. The present invention also discloses an autonomous driving method including the target detection method based on incremental learning. The target detection method based on incremental learning and the autonomous driving method provided by the present invention innovatively propose a new incremental learning target detection method; through the innovative addition of detection head technology and mask technology, not only incremental learning target detection is achieved, but also high accuracy, high reliability and good practicality.

Description

Technical Field

[0001] The present invention belongs to the field of computer image processing, and specifically relates to a target detection method and an automatic driving method based on incremental learning. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, target detection technology has been widely used in people's production and life, bringing endless convenience to people's production and life.

[0003] Current target detection methods often use deep learning models, whose learning method is mainly to fit the distribution of given data through stochastic gradient descent. This form makes the performance of deep learning models heavily dependent on the number of training samples at this stage, and each training process is overlay rather than incremental. Continuing to train with new data on an already trained model will cause catastrophic forgetting. This means that once the model is trained, it is difficult to change it, and it is not easy for the model to benefit from the knowledge learned in previous training.

[0004] Incremental learning is one way to improve the above problems. In the incremental learning target detection task, how to learn to maintain the model's ability to recognize old targets while recognizing new targets will be the core issue. The current commonly used method is mainly distillation loss. Current research mainly applies the incremental learning method based on knowledge distillation to target detection; when training the model to learn new tasks, it will use losses to let the new model learn the output of the old model at the same time, thereby preventing the model from forgetting the old tasks. However, this approach still cannot avoid the forgetting problem and faces a dilemma - the increased loss will limit the model's ability to learn new tasks. The more new tasks are learned, the more serious the forgetting will be, resulting in a serious decline in model performance. Summary of the invention

[0005] One of the purposes of the present invention is to provide a target detection method based on incremental learning with high accuracy, high reliability and good practicality.

[0006] A second object of the present invention is to provide an autonomous driving method that includes the target detection method based on incremental learning.

[0007] The target detection method based on incremental learning provided by the present invention comprises the following steps:

[0008] S1. Obtain the original target detection initial model;

[0009] S2. Pre-train some parameters of the feature extractor of the original target detection initial model obtained in step S1 to obtain a general target detection feature extractor;

[0010] S3. Use the detection head and parameter mask to structurally expand the object detection feature extractor obtained in step S2 to obtain an extended object detection model:

[0011] S3.1. Add parameter masks and an independent detection head to the pre-trained parameters in the object detection feature extractor obtained in step S2 to obtain an extended detection model;

[0012] S3.2. Perform forward propagation on the extended detection model obtained in step S3.1 on the training data set and calculate the error loss function;

[0013] S3.3. Update the values ​​of the detection head parameters and parameter masks;

[0014] S3.4 Repeat steps S3.2 and S3.3 until the conditions are set, the training is completed and the parameters are saved;

[0015] S3.5. During detection, the detection head performs corresponding target detection on the features filtered by the parameter mask;

[0016] S4. Use the extended target detection model obtained in step S3 to perform actual target detection.

[0017] The original target detection initial model described in step S1 is specifically a neural network-based target detection model that can abstract the structure into the form of P(x)=d(f(.)); wherein f(.) is a feature extractor for mapping an image into a feature tensor of a specific dimension; d(.) is a detection head for decoding features into a set number of target boxes and corresponding categories.

[0018] The original target detection initial model specifically includes a Faster-RCNN model, a YOLO model and a Swin-Transformer model.

[0019] Step S2 pre-trains some parameters of the feature extractor of the original target detection initial model obtained in step S1, specifically, performs unsupervised learning on the original target detection initial model on the target domain dataset to obtain a pre-trained model, or directly trains the original target detection initial model on the Imagenet dataset to obtain a pre-trained model, or directly obtains a public pre-trained model; finally, the public parameters of the pre-trained model are obtained.

[0020] Step S3.1 adds a parameter mask and an independent detection head to the pre-trained parameters in the object detection feature extractor obtained in step S2, specifically including the following steps:

[0021] A. Add the corresponding detection head according to the original target detection initial model obtained in step S1;

[0022] B. Replace the convolutional layer in the target detection feature extractor obtained in step S2 with a masked convolution, and replace the linear layer in the target detection feature extractor obtained in step S2 with a masked linear layer; the masked layer is the i-th parameter w of the j-th layer in the target detection feature extractor obtained in step S2 ji Added a floating point mask switch

[0023] C. Introduce the set hyperparameter θ as the threshold of the mask, and use the following formula to calculate the binary mask:

[0024]

[0025] In the formula is the binary mask;

[0026] D. During forward propagation, the following formula is used to calculate the parameter W' of the replacement model:

[0027]

[0028] Where W is all the parameters replaced by masks in the object detection feature extractor obtained in step S2; is the binary mask corresponding to all parameters replaced by the mask in the target detection feature extractor obtained in step C; ⊙ is the element-by-element dot multiplication operation.

[0029] Step S3.2 performs forward propagation on the training data set to calculate the error loss function, which specifically includes the following steps:

[0030] The image dataset for this task is I = {i1,i2,...,i i ,...,i n} and the corresponding annotation set L = {l1,l2,...,l i ,...,l n}, where i i ∈R 3×H×W , n is the number of images in the image dataset, H is the height of the image, W is the width of the image, l i For picture i i The corresponding annotation results and l i ={(x i ,y i ,w i ,h i ,c i )},x i is the horizontal coordinate of the prediction box, y i is the vertical coordinate of the prediction box, w i is the prediction box width, h i is the predicted box height, c iClassify the prediction box;

[0031] In the forward propagation, the model is trained on image i i Calculate and predict in is the horizontal coordinate of the predicted prediction box, is the vertical coordinate of the predicted prediction box, is the predicted prediction box width, is the predicted box height, Classify the predicted prediction box;

[0032] The boxes matched by unknown labels are treated as negative samples;

[0033] For the matched prediction box, the loss function Loss is calculated using the following formula:

[0034]

[0035] Where L bos () is the box regression loss function, L cls () is the box classification loss function.

[0036] The updating of the detection head parameters and parameter mask values ​​described in step S3.3 specifically includes the following steps:

[0037] Adopting neural network training method to update the parameters of the detection head;

[0038] The sampled mask gradient is used to update the parameter mask; the mask gradient is calculated using the following formula:

[0039]

[0040] In the formula is the mask gradient; L1 is the gradient returned by the loss function; m ji is the mask corresponding to the jth parameter of the i-th layer; δ is the derivative operation; y in is the output of the i-th layer; w ji is the jth parameter of the i-th layer; x in is the input of the i-th layer.

[0041] The present invention also discloses an automatic driving method including the target detection method based on incremental learning, which specifically includes the following steps:

[0042] S1. Real-time acquisition of environmental image information around the vehicle to be controlled;

[0043] S2. Using the above-mentioned target detection method based on incremental learning to perform real-time target detection and incremental training on the environmental image information obtained in step S1;

[0044] S3. According to the real-time target detection results obtained in step S2, the road information is identified in real time, and the vehicle is controlled to perform automatic driving; at the same time, the parameter mask and detection head in the model are updated in real time.

[0045] The incremental learning-based target detection method and autonomous driving method provided by the present invention innovatively propose a new incremental learning target detection method; through the innovative addition of detection head technology and mask technology, not only incremental learning target detection is achieved, but also high accuracy, high reliability and good practicality are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the method flow of the target detection method of the present invention.

[0047] Figure 2 Schematic diagram of the method flow of the automatic driving method of the present invention. DETAILED DESCRIPTION

[0048] like Figure 1 The method flow diagram of the target detection method of the present invention is shown as follows: The target detection method based on incremental learning provided by the present invention comprises the following steps:

[0049] S1. Obtain the original target detection initial model; specifically, a neural network-based target detection model that can abstract the structure into the form of P(x)=d(f(.)); where f(.) is a feature extractor used to map the image into a feature tensor of a specific dimension; d(.) is a detection head used to decode the features into a set number of target boxes and the corresponding categories;

[0050] The original target detection initial models specifically include Faster-RCNN model, YOLO model and Swin-Transformer model;

[0051] S2. Pre-train some parameters of the feature extractor of the original target detection initial model obtained in step S1 to obtain a general target detection feature extractor; specifically, the original target detection initial model is subjected to unsupervised learning on the target domain dataset to obtain a pre-trained model, or the original target detection initial model is directly trained on the Imagenet dataset to obtain a pre-trained model, or a public pre-trained model is directly obtained; finally, the public parameters of the pre-trained model are obtained;

[0052] S3. Use the detection head and parameter mask to structurally expand the object detection feature extractor obtained in step S2 to obtain an extended object detection model:

[0053] S3.1. Add parameter masks and an independent detection head to the pre-trained parameters in the object detection feature extractor obtained in step S2, thereby obtaining an extended detection model; specifically comprising the following steps:

[0054] A. Add the corresponding detection head according to the original target detection initial model obtained in step S1;

[0055] B. Replace the convolutional layer in the target detection feature extractor obtained in step S2 with a masked convolution, and replace the linear layer in the target detection feature extractor obtained in step S2 with a masked linear layer; the masked layer is the i-th parameter w of the j-th layer in the target detection feature extractor obtained in step S2 ji Added a floating point mask switch

[0056] C. Introduce the set hyperparameter θ as the threshold of the mask, and use the following formula to calculate the binary mask:

[0057]

[0058] In the formula is the binary mask;

[0059] D. During forward propagation, the following formula is used to calculate the parameter W' of the replacement model:

[0060]

[0061] Where W is all the parameters replaced by masks in the object detection feature extractor obtained in step S2; is the binary mask corresponding to all parameters replaced by the mask in the target detection feature extractor obtained in step C; ⊙ is the element-by-element dot multiplication operation;

[0062] S3.2. The extended detection model obtained in step S3.1 is forward propagated on the training data set to calculate the error loss function; specifically, the steps include:

[0063] The image dataset for this task is I = {i1,i2,...,i i ,...,i n} and the corresponding annotation set L = {l1,l2,...,l i ,...,l n}, where i i ∈R 3×H×W , n is the number of images in the image dataset, H is the height of the image, W is the width of the image, l i For picture i i The corresponding annotation results and l i ={(x i ,y i,w i ,h i ,c i )},x i is the horizontal coordinate of the prediction box, y i is the vertical coordinate of the prediction box, w i is the prediction box width, h i is the predicted box height, c i Classify the prediction box;

[0064] In the forward propagation, the model is trained on image i i Calculate and predict in is the horizontal coordinate of the predicted prediction box, is the vertical coordinate of the predicted prediction box, is the predicted prediction box width, is the predicted box height, Classify the predicted prediction box;

[0065] The boxes matched by unknown labels are treated as negative samples;

[0066] For the matching prediction boxes (overlapping degree greater than 70%), the loss function Loss is calculated using the following formula:

[0067]

[0068] Where L bos () is the box regression loss function, L cls () is the box classification loss function;

[0069] When the original target detection initial model is the Faster-RCNN model, L bos () Using the two-norm loss, L cls () using cross entropy loss;

[0070] S3.3. Update the values ​​of the detection head parameters and parameter masks; specifically, the steps include:

[0071] Adopting neural network training method to update the parameters of the detection head;

[0072] The sampled mask gradient is used to update the parameter mask; the mask gradient is calculated using the following formula:

[0073]

[0074] In the formula is the mask gradient; L1 is the gradient returned by the loss function; m ji is the mask corresponding to the jth parameter of the i-th layer; δ is the derivative operation; y in is the output of the i-th layer; w jiis the jth parameter of the i-th layer; x in is the input of the i-th layer;

[0075] S3.4 Repeat steps S3.2 and S3.3 until the conditions are set, the training is completed and the parameters are saved;

[0076] S3.5. During detection, the detection head performs corresponding target detection on the features filtered by the parameter mask;

[0077] S4. Use the extended target detection model obtained in step S3 to perform actual target detection.

[0078] When the initial model of the original target detection is the Faster-RCNN model, the corresponding detection head includes the RPN network head, ROI head, classification head and box regression head; the RPN network head is composed of two 1*1 convolution branches, which output the rough target box that may contain the object and the probability of it being a known object for each pixel in the feature map; the ROI head filters the rough target box obtained by RPN, intercepts the features corresponding to all boxes, and compresses them into a roi vector of size 7*7*64 using a pooling layer; the classification head and box regression head are the outputs of the current task classification number and the four-dimensional linear layer respectively, and output the scores and coordinate correction values ​​of all classifications of the roi vector.

[0079] The effects of the method of the present invention are described below in conjunction with the embodiments:

[0080] The performance of this method is evaluated on the standard incremental learning protocol for object detection. The incremental learning dataset for object detection on VOC is divided as shown in Table 1. 19+1, 15+5, and 10+10 are three settings for incremental learning. The 20 categories of images in the VOC dataset are divided into two steps in sequence for learning. The data in each step only contains the annotations of its own category. Even if the image actually contains objects from other training steps, it is unlabeled.

[0081] Table 1 Schematic diagram of data set settings

[0082]

[0083] In order to evaluate the effectiveness of the method of the present invention, experiments were conducted on the Faster-RCNN network implemented based on the deep learning framework PyTorch and the target detection framework Detectron2, and compared with other methods. The learning rate of the mask was set to 0.0001, the binarization threshold θ was set to 0.005, the learning rate of the detection head was set to 0.00001, the Adam optimizer was used for optimization, the batch size was 8, and the training was performed on a 4*2080Ti GPU. The number of training iterations was 18,000 times, and the learning rate attenuation was set twice by dividing by 10 when the number of iterations was 12,000 and 16,000, respectively. The experiments all took AP50 (meanaverage precision at 0.5IOU) as the evaluation index.

[0084] The experimental results are shown in Table 2. Taking 19+1 as an example, Step 1 means training on a dataset containing annotations of categories 1-19. Step 2 means training on a dataset of category 20 based on training on datasets of categories 1-19. The best experimental results are highlighted in bold.

[0085] Table 2 Schematic diagram of the results of incremental learning standard protocol for target detection

[0086]

[0087] It can be found that the performance of the method of the present invention is significantly higher than that of the previous method in most cases, and the forgetting problem is successfully avoided. Parameter isolation reduces the interference between tasks of different incremental steps. As the number of incremental tasks increases, the performance of the method of the present invention will show more advantages.

[0088] In addition, further ablation experiments were conducted for the addition of masks, and the results are shown in Table 3:

[0089] Table 3. Schematic diagram of the results of incremental learning standard protocol for target detection

[0090]

[0091] Since the images of the ImageNet pre-training method are all natural images, they may not be applicable to data sets such as medicine and industrial detection. Sometimes it is necessary to obtain the initial parameters through the pre-training method on the target domain data set. From the results, it can be seen that under such restrictions, the results of the method of the present invention can still show certain advantages. At the same time, the results also show the relationship between the number of added parameter masks and the number of parameters. The results of the method of the present invention are better than training a model separately for each detection task, and it also saves nearly 50% of the parameters. This is partly due to the regularization effect of the parameter mask.

[0092] The present invention also discloses an automatic driving method including the target detection method based on incremental learning, which specifically includes the following steps:

[0093] S1. Real-time acquisition of environmental image information around the vehicle to be controlled;

[0094] S2. Using the above-mentioned target detection method based on incremental learning to perform real-time target detection and incremental training on the environmental image information obtained in step S1;

[0095] S3. According to the real-time target detection results obtained in step S2, the road information is identified in real time, and the vehicle is controlled to perform automatic driving; at the same time, the parameter mask and detection head in the model are updated in real time.

Claims

1. A target detection method based on incremental learning, comprising the following steps: S1. Obtain the original target detection initial model; specifically, a neural network-based target detection model that can abstract the structure into the form of P(x)=d(f(.)); where f(.) is a feature extractor used to map the image into a feature tensor of a specific dimension; d(.) is a detection head used to decode the features into a set number of target boxes and the corresponding categories; S2. Pre-train some parameters of the feature extractor of the original target detection initial model obtained in step S1 to obtain a general target detection feature extractor; Specifically, the original target detection initial model is subjected to unsupervised learning on the target domain dataset to obtain a pre-trained model, or the original target detection initial model is directly trained on the Imagenet dataset to obtain a pre-trained model, or a public pre-trained model is directly obtained; finally, the public parameters of the pre-trained model are obtained; S3. Use the detection head and parameter mask to structurally expand the object detection feature extractor obtained in step S2 to obtain an extended object detection model: S3.

1. Add parameter masks and an independent detection head to the pre-trained parameters in the object detection feature extractor obtained in step S2, thereby obtaining an extended detection model; specifically comprising the following steps: A. Add the corresponding detection head according to the original target detection initial model obtained in step S1; B. Replace the convolutional layer in the target detection feature extractor obtained in step S2 with a masked convolution, and replace the linear layer in the target detection feature extractor obtained in step S2 with a masked linear layer; the masked layer is the i-th parameter w of the j-th layer in the target detection feature extractor obtained in step S2 ji Added a floating point mask switch C. Introduce the set hyperparameter θ as the threshold of the mask, and use the following formula to calculate the binary mask: In the formula is the binary mask; D. During forward propagation, the following formula is used to calculate the parameter W' of the replacement model: Where W is all the parameters replaced by masks in the object detection feature extractor obtained in step S2; is the binary mask corresponding to all parameters replaced by the mask in the target detection feature extractor obtained in step C; ⊙ is the element-by-element dot multiplication operation; S3.

2. Perform forward propagation on the extended detection model obtained in step S3.1 on the training data set and calculate the error loss function; S3.

3. Update the values ​​of the detection head parameters and parameter masks; S3.

4. Repeat steps S3.2 and S3.3 until the conditions are set, the training is completed and the parameters are saved; S3.

5. During detection, the detection head performs corresponding target detection on the features filtered by the parameter mask; S4. Use the extended target detection model obtained in step S3 to perform actual target detection.

2. The target detection method based on incremental learning according to claim 1 is characterized in that The original target detection initial model specifically includes a Faster-RCNN model, a YOLO model and a Swin-Transformer model.

3. The target detection method based on incremental learning according to claim 2 is characterized in that Step S3.2 performs forward propagation on the training data set to calculate the error loss function, which specifically includes the following steps: The image dataset for this task is I = {i1,i2,...,i i ,...,i n } and the corresponding annotation set L = {l1,l2,...,l i ,...,l n }, where i i ∈R 3×H×W , n is the number of images in the image dataset, H is the height of the image, W is the width of the image, l i For picture i i The corresponding annotation results and l i ={(x i ,y i ,w i ,h i ,c i )},x i is the horizontal coordinate of the prediction box, y i is the vertical coordinate of the prediction box, w i is the prediction box width, h i is the predicted box height, c i Classify the prediction box; In the forward propagation, the model is trained on image i i Calculate and predict in is the horizontal coordinate of the predicted prediction box, is the vertical coordinate of the predicted prediction box, is the predicted prediction box width, is the predicted box height, Classify the predicted prediction box; The boxes matched by unknown labels are treated as negative samples; For the matched prediction box, the loss function Loss is calculated using the following formula: Where L bos () is the box regression loss function, L cls () is the box classification loss function.

4. The target detection method based on incremental learning according to claim 3 is characterized in that The updating of the detection head parameters and parameter mask values ​​described in step S3.3 specifically includes the following steps: Adopting neural network training method to update the parameters of the detection head; The sampled mask gradient is used to update the parameter mask; the mask gradient is calculated using the following formula: In the formula is the mask gradient; L1 is the gradient returned by the loss function; m ji is the mask corresponding to the jth parameter of the i-th layer; δ is the derivative operation; y in is the output of the i-th layer; w ji is the jth parameter of the i-th layer; x in is the input of the i-th layer.

5. An autonomous driving method comprising the target detection method based on incremental learning according to any one of claims 1 to 4, characterized in that The specific steps include: S1. Real-time acquisition of environmental image information around the vehicle to be controlled; S2. Using the target detection method based on incremental learning as described in any one of claims 1 to 4, real-time target detection and incremental training are performed on the environmental image obtained in step S1; S3. According to the real-time target detection results obtained in step S2, the road information is identified in real time, and the vehicle is controlled to perform automatic driving; at the same time, the parameter mask and detection head in the model are updated in real time.

Images