Self-optimization method for unmanned self-learning target recognition

Abstract

The application discloses a self-optimization method for unmanned self-learning target identification, relates to the technical field of computer vision, and solves the technical problems that the target identification network training process is not stable enough and the real-time performance and accuracy of the identification result are insufficient in the unmanned technology, and the technical solution points are that data expansion is performed through an improved CycleGAN network, the YOLOv3 network structure and a loss function are improved, a series of improved CutMix data enhancement, simulated annealing and Dropblock algorithms are used to optimize the training mode, and the generalization performance of the model is enhanced; meanwhile, the real-time performance and accuracy of the target identification algorithm are improved, and the problem of low computing power of the vehicle-mounted processor is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a self-optimization method for unmanned self-learning target identification. BACKGROUND

[0002] In recent years, target detection models based on deep learning have been favored by most people in the industry due to their efficient performance. As the most basic link in autonomous driving, target detection collects real-time environmental information for vehicles to ensure safety and provide correct planning decisions. Target detection and semantic segmentation algorithms based on deep learning have shown unique advantages in the field of autonomous driving, and can achieve high detection accuracy with less computing resources, thus becoming an indispensable method in autonomous driving systems.

[0003] In 2012, Krizhevsky et al. proposed AlexNet convolutional neural network (CNN) in the ILSVRC image classification competition, which overcame the problem of high running cost and time complexity of traditional detection methods. Since then, the research in the field of computer vision has focused on deep learning technology. He et al. used pyramid pooling layers to improve the running speed of R-CNN and proposed SPP-Net. Girshick R then proposed Fast R-CNN, Ren et al. proposed Faster R-CNN algorithm, and He et al. proposed Mask R-CNN. Two-stage target detection models have been gradually improved. In 2015, Redmon J et al. proposed the YOLO algorithm, which abandoned the candidate frame extraction and selected the regression method for object classification and prediction frame generation. Its detection speed is ten times that of the advanced Faster R-CNN algorithm in two-stage algorithms.

[0004] As a representative of single-stage detection models, YOLO uses the divide-and-conquer concept to divide the input image into several equal-sized squares and perform prediction in each square, cleverly dividing the target detection problem into two sub-tasks of obtaining target bounding boxes and classes, and changing the prediction task to a regression task, greatly increasing the operation speed. In YOLOv3, the feature pyramid network structure is referenced, and three sizes of features are used to detect large, medium and small objects, greatly improving the detection accuracy of small targets. However, it still has problems such as low object positioning accuracy and poor recall rate.

[0005] How to improve the stability of the target recognition network training process and make the recognition result faster and more accurate is a problem that needs to be solved. SUMMARY

[0006] This application provides a self-optimization method for self-learning target recognition in unmanned driving, the technical purpose of which is to improve the accuracy and real-time performance of target recognition of intelligent driving vehicles on complex roads.

[0007] The above technical objectives of this application are achieved through the following technical solutions:

[0008] A self-optimization method for self-learning target recognition in unmanned driving, comprising:

[0009] S1: Save n frames of images collected by the vehicle-mounted camera corresponding to target detection errors;

[0010] S2: Input the image into the improved CycleGAN network for data expansion to obtain expanded samples;

[0011] S3: Optimize the backbone network and loss function of the YOLOv3 network for target detection to obtain the first YOLOv3 network;

[0012] S4: Optimizing the training method of the first YOLOv3 network through improved CutMix data augmentation, simulated annealing, and Dropblock algorithms to obtain a second YOLOv3 network;

[0013] S5: Input the expanded sample into the second YOLOv3 network for training to obtain an updated third YOLOv3 network, perform target detection through the third YOLOv3 network, and repeat steps S1 to S5 until self-optimization is completed.

[0014] Furthermore, in step S1, obtaining the n-frame images captured by the vehicle-mounted camera corresponding to the target detection error includes: inputting each video segment into the YOLOv3 network for frame-by-frame detection, and when a target detection error occurs, saving the n-frame images captured by the vehicle-mounted camera corresponding to the target detection error.

[0015] Furthermore, the improved CycleGAN network includes a mapping network layer and two pairs of GAN networks, and the loss function of the improved CycleGAN network is expressed as:

[0016]

[0017]

[0018] in, represents the expected value of the real data x; represents the expectation value of the noise z; D(.) represents the discriminator; G(.) represents the generator; a represents the label of the real picture; b represents the label of the generated picture; c represents the value determined by the generator in order to make the discriminator think that the generated picture is real data.

[0019] Further, the mapping network layer comprises a plurality of fully connected layers.

[0020] Further, in step S3, comprising:

[0021] Only the shallow feature layer of the backbone network Darknet53 is used for target detection, the backbone network Darknet53 of the YOLOv3 network is optimized into a backbone network Darknet-25, and a first YOLOv3 network is obtained, and the loss function of the first YOLOv3 network is represented as:

[0022] L(O,o,C,c,t,g)=λ1L conf (o,c)+λ2L cla (O,C)+λ3L loc (t,g);

[0023] Wherein,

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] L conf represents the target confidence loss; L cla represents the target class loss; L loc represents the target class loss; λ1, λ2, λ3 all represent balance coefficients; o i ∈{0,1} represents whether the target exists in the predicted target bounding box i, 0 represents that it does not exist, and 1 represents that it exists; represents the Sigmoid probability of whether the target exists in the predicted target rectangular frame i; O ijdenotes whether the j-th object exists in the prediction target bounding box i, 0 denotes non-existence, and 1 denotes existence; denotes the Sigmoid probability that the j-th object exists in the prediction target bounding box i; x y denotes the predicted offset parameter about the center coordinate; w h denotes the predicted scaling factor about the object width and height; x y denotes the upper left corner coordinate of the corresponding Grid Cell; w denotes the width of the corresponding Anchor template; h denotes the height of the corresponding Anchor template; x y denotes the coordinate of the center point of the GT Box; w denotes the width of the GT Box template; h denotes the height of the GT Box template.

[0033] Further, the loss function of the first YOLOv3 network comprises label smoothing processing, and the label smoothing processing comprises:

[0034]

[0035] wherein, ε = 0.005, that is, the original labels 0 and 1 are adjusted to 0.005 and 0.995.

[0036] Further, the improved CutMix data enhancement comprises:

[0037] randomly cropping a part on a picture A, and filling a picture B in a corresponding position of the cropped area to form a new sample; wherein, when calculating the loss of the new sample, the loss of the picture A and the picture B is obtained according to the size of the cropping frame.

[0038] The four new samples are randomly cropped again, and then the four cropped new samples are spliced into one sample.

[0039] The application has the beneficial effects that: the application firstly performs data expansion through the improved CycleGAN network, then improves the YOLOv3 network structure and the loss function, and optimizes the training method by using a series of improved CutMix data enhancement, simulated annealing and Dropblock algorithm, thereby enhancing the generalization performance of the model; meanwhile, the real-time performance and the accuracy of the object recognition algorithm are improved, and the problem of low computing power of the vehicle-mounted processor is solved. BRIEF DESCRIPTION OF DRAWINGS

[0040] ​​​​Figure 1 Flowchart of the method of the present application. DETAILED DESCRIPTION

[0041] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0042] As shown in the figure, the self-optimization method for unmanned self-learning target recognition of the present application comprises: Figure 1

[0043] S1: saving the pictures collected by the n frames of vehicle-mounted cameras corresponding to the target detection error.

[0044] Specifically, due to the limitations of the deep learning network itself, if there is not enough amount of data set for training, combined with the use in various conditions difficult to predict complex road environment, its detection accuracy cannot meet the requirements of intelligent driving, or it will always encounter detection failure, the network model optimization process is to input the video into the YOLOv3 network, frame by frame to detect whether the image needs to be optimized, when the target detection error occurs, the pictures collected by the n frames of vehicle-mounted cameras corresponding to the target detection error are saved for optimization.

[0045] S2: inputting the pictures into the improved CycleGAN network for data augmentation to obtain augmented samples.

[0046] The traditional CycleGAN network includes two pairs of GAN networks, which are a double generative adversarial network, and the loss function of the double generative adversarial network is represented as:

[0047]

[0048] Among them, represents the adversarial loss function, and:

[0049]

[0050]

[0051] represents the cycle consistency loss function, which is used to solve the problem that it cannot be determined which pair of matching is a meaningful pair, and the adversarial loss can be optimized independently, represented as:

[0052] ​In training, the model is trained using the Pix2Pix paired data, therefore, the improved CycleGAN network adds a mapping network layer before the GAN network, which controls the style of the generated image. The mapping network layer is composed of several full connection layers (FC). The hidden variable input by the ordinary GAN is obtained after several full connection layers, which can better classify and generate data. Then the intermediate variable is input into the new picture generator, and different new pictures can be generated by adding random noise.

[0053] In order to avoid the gradient dispersion caused by the model during back propagation, that is, the false samples at the classification boundary cannot continue iteration, resulting in poor performance of the generator, therefore, the least square LSGAN loss function is used as the loss function of the improved CycleGAN network, which can make the training of the GAN network more stable, which is expressed as:

[0054]

[0055]

[0056] wherein, E(x) represents the expected value of the real data x; E(z) represents the expected value of the noise z; D(.) represents the discriminator; G(.) represents the generator; a represents the label of the real picture; b represents the label of the generated picture; c represents the value determined by the generator in order to make the discriminator think that the generated picture is real data.

[0057] S3: The backbone network and loss function of the YOLOv3 network for target detection are optimized to obtain a first YOLOv3 network.

[0058] The YOLOv3 network learns from the idea of feature pyramid network, extracts three feature layers for target detection, and these three feature layers are located at different positions of Darknet53. Then the three feature layers are up-sampled and stacked for feature fusion, and the fused high-level feature layer is used for predicting larger targets, the middle-level feature layer is used for predicting smaller targets, and the shallow-level feature layer is used for predicting small targets.

[0059] In the present application, in order to speed up the training process, the backbone network Darknet53 is improved, and in the optimization stage, the calculation of three feature maps is omitted to enhance the small target recognition performance, only one layer of features (i.e. shallow feature layer) is used for classification and prediction, so there are 28 less convolutional layers compared to the original backbone network Darknet53, and finally the backbone network is simplified from three 75-channel feature layers to one 21-channel feature layer, 21 represents the conditional probability of each target category in the VOC dataset when the picture contains objects and the probability of whether the picture contains targets, 75 contains 63 target type prediction values and 3 groups of prediction box information; in addition, for the picture format of the VOC dataset, the input vector is set to 416x416; since the improved backbone network has 25 convolutional layers, the backbone network is named Darknet-25.

[0060] In the above training process, the loss function only includes target confidence loss and target category loss, the loss function no longer needs to process three feature layers, only the calculation of the largest feature layer (i.e. shallow feature layer) is needed, and the loss function is represented as:

[0061] L(O,o,C,c,t,g)=λ1L conf (o,c)+λ2L cla (O,C)+λ3L loc (t,g);

[0062] Wherein,

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] L conf represents the target confidence loss; L cla represents the target category loss; L loc represents the target category loss; λ1, λ2, λ3 all represent balance coefficients; o i∈{0,1} indicates whether the target actually exists in the predicted target bounding box i, 0 means no target exists, and 1 means yes target exists; Indicates the Sigmoid probability of predicting whether there is a target in the target rectangle i; O ij ∈{0,1} indicates whether the j-th type of target actually exists in the predicted target bounding box i, 0 means no, 1 means yes; Indicates the Sigmoid probability of predicting the j-th target in the target bounding box i; (t x ,t y ) represents the predicted offset parameter about the center coordinate; (t w ,t h ) represents the predicted scaling factor of the target width and height; (c x ,c y ) represents the coordinates of the upper left corner of the corresponding Grid Cell; p w Indicates the width of the corresponding Anchor template, p h Indicates the height of the corresponding Anchor template; (g x ,g y ) represents the coordinates of the center point of GT Boxes, g w Indicates the width of the GT Boxes template, g h Indicates the height of the GT Boxes template.

[0072] Furthermore, since the optimization process is a small sample learning process and is very prone to overfitting, label smoothing is added to the loss function to improve generalization performance and have a certain degree of noise resistance. The label smoothing process is expressed as:

[0073]

[0074] Here, ε = 0.005, which means that the original labels 0 and 1 are adjusted to 0.005 and 0.995. Setting ε = 0.005 also imposes some penalties on classification accuracy to prevent overfitting.

[0075] S4: Optimize the training method of the first YOLOv3 network through improved CutMix data enhancement, simulated annealing and Dropblock algorithms to obtain a second YOLOv3 network.

[0076] Specifically, general CutMix data augmentation includes: combining two images into one image through a series of processing. The specific operation is to randomly crop a part of image A and then fill the corresponding position of image B into the cropped area to form a new sample; wherein, when calculating the loss of the new sample, the loss of image A and image B obtains corresponding weights according to the size of the cropping box.

[0077] In the present application, in order to further reduce the computing power required by the model, the four new samples processed by CutMix are randomly cropped again and then spliced into one picture as training data. Since the backgrounds of the pictures are all real traffic scenes, the division line of the picture splicing will not affect the target detection. In this way, the BN calculation will process 8 pictures at a time, which also enhances the running speed of single-core processing.

[0078] At the same time, the learning rate simulated annealing algorithm is used in the self-optimization algorithm training process, that is, the learning rate is adjusted after each batch training. Specifically, it increases linearly when increasing, and decreases according to the cosine function when decreasing. This algorithm can avoid falling into local minimum.

[0079] In order to avoid overfitting, a regularization method, Dropblock algorithm, is added during training. Dropblock algorithm refers to randomly deleting some neurons during training, and setting a proportion, that is, the deleted information increases linearly during training, so as to make the network learn more generalizable features.

[0080] S5: input the augmented samples into the second YOLOv3 network for training to obtain an updated third YOLOv3 network, perform target detection through the third YOLOv3 network, and repeat steps S1 to S5 until the self-optimization is completed.

[0081] The above is an exemplary embodiment of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A self-optimization method for self-learning target recognition for unmanned driving, characterized in that: include: S1: Save n frames of images collected by the vehicle-mounted camera corresponding to target detection errors; S2: Input the image into the improved CycleGAN network for data expansion to obtain expanded samples; S3: Optimize the backbone network and loss function of the YOLOv3 network for target detection to obtain the first YOLOv3 network; S4: Optimizing the training method of the first YOLOv3 network through improved CutMix data augmentation, simulated annealing, and Dropblock algorithms to obtain a second YOLOv3 network; S5: Input the expanded sample into the second YOLOv3 network for training to obtain an updated third YOLOv3 network, perform target detection through the third YOLOv3 network, and repeat steps S1 to S5 until self-optimization is completed; Wherein, step S3 includes: Only the shallow feature layer of the backbone network Darknet53 is used for target detection, and the backbone network Darknet53 of the YOLOv3 network is optimized to the backbone network Darknet-25 to obtain the first YOLOv3 network. The loss function of the first YOLOv3 network is expressed as: L(O,o,C,c,t,g)=λ1L conf (o,c)+λ2L cla (O,C)+λ3L loc (t,g); in, L conf represents the target confidence loss; L cla represents the target category loss; L loc represents the target category loss; λ1, λ2, and λ3 all represent balance coefficients; o i ∈{0,1} indicates whether the target actually exists in the predicted target bounding box i, 0 means no target exists, and 1 means yes target exists; Indicates the Sigmoid probability of predicting whether there is a target in the target rectangle i; O ij ∈{0,1} indicates whether the j-th type of target actually exists in the predicted target bounding box i, 0 means no, 1 means yes; Indicates the Sigmoid probability of predicting the j-th target in the target bounding box i; (t x ,t y ) represents the predicted offset parameter about the center coordinate; (t w ,t h ) represents the predicted scaling factor of the target width and height; (c x ,c y ) represents the coordinates of the upper left corner of the corresponding Grid Cell; p w Indicates the width of the corresponding Anchor template, p h Indicates the height of the corresponding Anchor template; (g x ,g y ) represents the coordinates of the center point of GT Boxes, g w Indicates the width of the GTBoxes template, g h Indicates the height of the GT Boxes template.

2. The method according to claim 1, wherein In step S1, obtaining the n frames of images captured by the vehicle-mounted camera corresponding to the target detection error includes: inputting each video segment into the YOLOv3 network for frame-by-frame detection, and when a target detection error occurs, saving the n frames of images captured by the vehicle-mounted camera corresponding to the target detection error.

3. The method according to claim 1, wherein The improved CycleGAN network includes a mapping network layer and two pairs of GAN networks. The loss function of the improved CycleGAN network is expressed as: in, represents the expected value of the real data x; Represents the expected value of noise z; D(.) represents the discriminator; G(.) represents the generator; a represents the label of the real picture; b represents the label of the generated picture; c represents the value set by the generator in order to make the discriminator think that the generated picture is real data.

4. The method according to claim 3, wherein The mapping network layer includes multiple fully connected layers.

5. The method according to claim 1, wherein The loss function of the first YOLOv3 network includes a label smoothing process, which includes: Among them, ε=0.005, that is, the original labels 0 and 1 are adjusted to 0.005 and 0.

995.

6. The method according to claim 1, wherein The improved CutMix data enhancement includes: Randomly crop a portion of image A and fill the cropped area with the corresponding portion of image B to form a new sample. When calculating the loss for the new sample, the losses of image A and image B are weighted according to the size of the cropping box. The four new samples are randomly cropped again, and then the cropped four new samples are spliced ​​into one sample.

Images