A loss feedback-based class and scale balanced target detection training method

Abstract

A kind of class and scale balanced target detection training method based on loss feedback, when iteratively training detection network, respectively carry out class balancing and scale balancing to training sample based on the feedback of loss function, class balancing dynamically adjusts the sampling probability of each class and each sample in iterative training, so that training class and sample data are balanced, obtain training set I', scale balancing dynamically adjusts the scale of training sample of training set I', the target scale of training set I' is made to meet the training requirement by image splicing, and training set I'' after class balancing and scale balancing is used as new iterative training sample for training.The present application makes the model pay more attention to the few-sample class in training, can improve the model generalization, provides training samples of different scales according to the network needs, improves the detection accuracy of small objects while ensuring the accuracy of large objects.The present application only increases a small amount of calculation of sample gradient proportion, does not affect the training and reasoning speed, and improves the training accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning and computer vision, and relates to the application of target detection in vehicle detection, and is a class and scale balanced target detection training method based on loss feedback. BACKGROUND

[0002] Target detection is one of the most important and most challenging branches in the field of computer vision, and it has been widely applied in people's life, such as monitoring safety, automatic driving and the like. As one of the important application occasions of target detection, vehicle detection is the core and basis in the intelligent transportation system, and the basic information of the traffic scene is obtained through vehicle detection, thereby laying a foundation for subsequent vehicle tracking, obtaining traffic flow, vehicle speed, road occupancy, event detection and the like. The process of the traditional vehicle detection method based on images is region selection, feature extraction, classification and post-processing. The region selection adopts a sliding window to generate a candidate frame, and the time complexity is high, and there are too many redundant windows, so that the feature extraction and classification process are time-consuming; the feature extraction uses HOG, SIFT, LBP operators, but the traditional operators have poor robustness to light, shape and the like, and cannot adapt to complex scenes, and have poor generalization, so that the extracted features are not good, and the classification effect is affected; a binary classification classifier is trained for each class, and the feature vector is input into the classifier for judgment, and common classifiers include SVM, Adaboost and the like; the post-processing is non-maximum suppression, and the redundant frame around each target is removed. Since the region selection method based on the sliding window is not targeted, the window is redundant, and the traditional feature extraction operator is difficult to adapt to the diversity of changes, and the series of methods are low in efficiency.

[0003] With the advent of the era of deep learning, the convolutional neural network has been widely used in the field of image. Compared with the traditional feature extraction operator which is designed by human through a large number of prior knowledge, the convolutional neural network feature extraction operator is the trained convolution kernel, and the parameters in the convolution kernel are learned by the neural network, which is more robust and general. In the field of target detection, the target detection based on deep learning has become the mainstream, which can be divided into two categories, one-stage and two-stage detection algorithm. The two-stage algorithm first extracts the candidate frame, extracts the possible foreground frame, and then classifies and positions the candidate foreground frame, and the representative algorithm is RCNN series; The one-stage algorithm does not need to extract the candidate frame, directly classifies and positions the regression, and the representative algorithm is SSD, YOLO, RetinaNet, etc. Class imbalance and small object detection are challenges faced by deep learning-based target detection algorithms. The sample quantity of each class in the real scene dataset is not uniform, the sample quantity of the common class is easy to obtain, and the quantity is sufficient, while the sample of the uncommon class is difficult to obtain, and the quantity is small, so the sample quantity of different classes is not balanced, which increases the training difficulty. The accuracy of small object detection is usually much lower than that of large and medium objects. The pixel information carried by small objects is less, and it may disappear in the feature map after multiple pooling layers. The detection training sample matching method, training method and other factors are also the reasons for the poor detection effect of small objects. SUMMARY

[0004] The problem to be solved by the present application is that in the training of the target detection model, class imbalance and small object detection will affect the training accuracy, and the imbalance of the sample quantity increases the training difficulty.

[0005] The technical scheme of the present application is: a class and scale balanced target detection training method based on loss feedback, when the target detection network is iteratively trained, the loss feedback based on the loss function is used to balance the classes and scales of the training samples respectively, the class balance is dynamically adjusted according to the gradient statistical information of each class sample in the current iteration, the sampling probability of each class and each sample in the iterative training is adjusted, the training classes and sample data are balanced, the training set I' after class balance is obtained, the scale balance is based on the loss feedback of the loss function, and the scale of the training sample of the training set I' is dynamically adjusted according to the gradient statistical information of the training sample of the training set I' in the current iteration, the target scale of the training set I' is adjusted through image splicing to meet the training requirements, and the training set I'' after class balance and scale balance is input into the target detection network as a new iteration training sample.

[0006] Further, the class balance is specifically:

[0007] S1.1) define g(c) as the gradient proportion of the class c training sample in the current iteration, r(c) as the oversampling coefficient of the class c, r(i) as the oversampling coefficient of the sample i, D as the data augmentation operation set, k as the oversampling threshold, and as a hyperparameter;

[0008] S1.2) For each class, count the gradient proportion g(c) of the training sample in the current iteration;

[0009] S1.3) Update the oversampling coefficient of each class and sample for the next iteration according to the gradient proportion g(c) of each class, and the oversampling coefficient of the class c for the next iteration is That is, when g(c)≥k, the class c does not perform oversampling, when g(c)<k, the class c performs oversampling, and the oversampling coefficient is

[0010] S1.4) The oversampling coefficient of the sample i is That is, among the classes in which the sample i appears, the maximum class oversampling coefficient is taken as the oversampling coefficient of the sample i;

[0011] S1.5) Traverse the training sample set of the current iteration to perform steps S1.2)-S1.4), and the oversampling coefficient of each sample is rounded up as the repetition number of the sample in the training sample set of the next iteration;

[0012] S1.6) Return the training set I' after balancing the sample classes based on loss feedback, for the next iteration training.

[0013] Further, the scale balance is specifically:

[0014] S2.1) define s as the sample scale, s1 as the small object determination scale, s≤s1 as the small object, s2 as the extremely small object determination scale, s≤s2<s1 as the extremely small object, r1 as the gradient proportion of the training sample whose scale satisfies s2<s<s1 in the current iteration, r2 as the gradient proportion of the training sample whose scale satisfies s<s2 in the current iteration, I as the training picture splicing strategy set, and t as the gradient proportion threshold for image splicing operation;

[0015] S2.2) Configure the training picture splicing strategy set I according to the training requirements of the target size, and the strategy set includes the regular training picture strategy of the original picture size and the spliced picture strategy of different degrees. After splicing of different degrees, the target scale in the training picture is reduced;

[0016] S2.3) Count the gradient information of the current iteration, and calculate r1 and r2 respectively;

[0017] S2.4) If r1≤r2 and r1≤t, it means that the gradient information of small objects with scale s2<s<s1 is insufficient, so the next iteration selects a stitching strategy corresponding to small objects to generate training pictures;

[0018] S2.5) If r2<r1 and r2≤t, it means that the gradient information of small objects with scale s<s2 is insufficient, so the next iteration selects a stitching strategy corresponding to small objects to generate training pictures;

[0019] S2.6) If neither of the above two conditions is met, i.e., r1>t and r2>t, it means that the current iteration does not lack gradient information of small objects, so the next iteration selects a regular training picture strategy;

[0020] S2.7) Based on the gradient statistical information of the current iteration, the training image stitching strategy of the next iteration is determined, and the training set of the next iteration is adjusted.

[0021] The method of the present application oversamples the few-sample class in training, increases the supervision information in the loss function of the few-sample class, makes the model pay more attention to the few-sample class, and at the same time increases the diversity of training data through a random data augmentation strategy, avoids overfitting caused by simple repeated samples, improves the generalization of the model, and improves the accuracy of the whole detection network. In addition, the target size of the training set is dynamically and automatically adjusted according to the network training requirements, further balancing the gradient proportion of objects of each scale in the training set. Compared with the method of using only fixed training image iterative training, the accuracy of small object detection can be improved while ensuring the accuracy of large objects, and only a small amount of training calculation is increased, i.e. statistics of the gradient proportion of each scale does not affect the inference speed.

[0022] As shown in Table 1, the performance of the class and scale balanced target detection training method based on loss feedback of the present application. The data set is a tunnel scene vehicle detection data set labeled by the laboratory itself, and the target classes include cars, buses, minibuses, trucks and vans. 2000 frames of pictures were randomly selected from multiple cameras and multiple time periods as the data set, the training set included 1500 pictures, and the test set included 500 pictures. The target detection algorithm uses RetinaNet, Baseline represents the RetinaNet trained by the original training method, and the hyperparameters k and t of the class and scale balanced training method based on loss feedback are 0.2 and 0.1 respectively. As can be seen from Table 1, the class and scale balanced training method based on loss feedback can improve AP 1.8% and AP50 5.5%, greatly improving the accuracy of the detector, and the training time of each epoch is only increased by 9s.

[0023] Table 1

[0024] BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a loss feedback-based scale and category balancing vehicle detection training method schematic diagram of the present application.

[0026] Figure 2 is a loss feedback-based category balancing training method pseudocode of the present application.

[0027] Figure 3 is a loss feedback-based scale balancing training method pseudocode of the present application.

[0028] Figure 4 is an image splicing strategy schematic diagram of the present application. DETAILED DESCRIPTION

[0029] The present application proposes a loss feedback-based category and scale balancing target detection training method, which can improve the detection accuracy of small objects and few sample categories without increasing additional computational consumption in the inference stage, thereby improving the overall accuracy of the detector. The present application statistically analyzes gradient information based on loss function feedback, dynamically selects training image scales and adjusts the oversampling degree of different category samples, improves the detection accuracy of small objects and few sample categories, improves the overall accuracy of the detector, and does not introduce additional calculations in the inference stage.

[0030] As shown in Figure 1 , when iteratively training the target detection network, the present application respectively performs category balancing and scale balancing on the training samples based on the loss feedback of the loss function. The category balancing dynamically adjusts the sampling probability of each category and each sample in the iterative training according to the gradient statistical information of each category sample in the current iteration, so that the training categories and sample data are balanced, and the training set I' after category balancing is obtained. The scale balancing dynamically adjusts the scale of the training samples of the training set I' based on the loss feedback of the loss function according to the gradient statistical information of the training samples of the training set I' in the current iteration, and makes the target scale of the training set I' meet the training requirements through image splicing. The training set I'' after category balancing and scale balancing is input into the target detection network as a new iteration training sample for training. The category balancing and scale balancing of the present application are further described below in combination with the drawings and examples.

[0031] In target detection, the number of samples of each class is usually unbalanced. If the number of samples of some classes is extremely large and the number of samples of some classes is extremely small, i.e. long-tail data distribution, it will greatly damage the performance of the detector. In the embodiment of vehicle detection, the number of samples of the car class is relatively large in the vehicle detection in the urban scene, and the number of samples of the truck, van and other classes is relatively small, but there is an equal detection demand for the small sample classes. Because the sample size is small, the supervision information provided during the training process is less, and the gradient information is missing, which leads to that the small sample classes cannot be learned well. The commonly used Re-Sample and Re-Weight strategies are easy to cause the model to overfit to the small sample class data. The present application proposes a class balancing training method based on loss feedback to improve the detection accuracy of the small sample class, and the pseudo code is as shown in Figure 2 The specific process is as follows:

[0032] S1.1) define g(c) as the gradient proportion of the class c training sample in the current iteration, r(c) as the oversampling coefficient of the class c, r(i) as the oversampling coefficient of the picture i, D as the data augmentation operation set, and k as the oversampling threshold;

[0033] S1.2) For each class, the gradient proportion g(c) of the training sample in the current iteration is counted.

[0034] S1.3) The oversampling coefficient of each class and sample in the next iteration is updated according to the gradient proportion g(c) of each class. The oversampling coefficient of the class c in the next iteration is That is, when g(c)≥k, the class c does not perform oversampling, and when g(c)<k, the class c performs oversampling, and the oversampling coefficient is Where k is the oversampling threshold, which is a hyperparameter;

[0035] S1.4) The oversampling coefficient of the sample i is That is, among the classes appearing in the sample i, the maximum oversampling coefficient in all classes is taken as the oversampling coefficient of the sample i;

[0036] S1.5) Traverse the training sample set, and take the integer part of the oversampling coefficient of each sample as the repetition number of the sample in the new training sample set. In order to reduce the risk of overfitting, each sample is first transformed by a random data augmentation strategy, and then added to the new training sample set. The sample is transformed so that the repeated samples in the oversampling are different. The operation, degree and probability of data augmentation are pre-set, and are randomly selected during execution;

[0037] S1.6) Return the new training sample set I' balanced in terms of sample class based on loss feedback, which is used for the next iteration training.

[0038] The advantage of the class balancing training method based on loss feedback is that: the few-sample class is usually not well learned in the training due to insufficient gradient information, less than the gradient of the multi-sample class, so balancing the number of samples of each class in the training set according to the class loss, oversampling the few-sample class, increasing the supervision information of the few-sample class in the loss function, making the model pay more attention to the few-sample class, and increasing the diversity of training data through a random data augmentation strategy to avoid overfitting caused by simple repeated samples, improve the generalization of the model, and improve the accuracy of the detector as a whole.

[0039] The number of training samples of different scales is also imbalanced in the training, and due to the maximum IoU sample matching mechanism, the training samples of large scales are generally more, and the training samples of small scales are less. Through the statistics of the loss of different scale samples, it is found that the loss of small objects is lower than that of large objects, but the final accuracy is lower than that of large objects. Through the statistics of the gradient of different scale samples, it can be found that the gradient of different scale samples in the training process is imbalanced, and the gradient of small objects accounts for a lower proportion, so although the loss of small objects is lower, it does not mean that the model has been well learned, but the lack of supervision information causes small objects to be unable to be fully trained and learned, which is one of the important reasons why the accuracy of small objects is much lower than that of large objects. Therefore, more small-scale training samples can be provided to supplement the missing supervision and gradient, and the specific operation can be to generate small-scale training samples by splicing images, but if only the spliced images are used for training, it will cause insufficient supervision and gradient of large objects, which will damage the detection performance of large objects, so a balanced sample generation strategy needs to be sought. According to the above analysis, the scale balancing training method based on loss feedback is proposed to improve the detection accuracy of small objects, and the pseudo code is as shown in Figure 3 The specific operation is as follows:

[0040] S2.1) define s as the sample scale, s1 as the small object determination scale, s≤s1 as the small object, s2 as the extremely small object determination scale, s≤s2<s1 as the extremely small object, r1 as the gradient proportion of the training sample with a scale satisfying s2<s<s1 in the current iteration, r2 as the gradient proportion of the training sample with a scale satisfying s<s2 in the current iteration, I as a set of training image splicing strategies, and t as the gradient proportion threshold for image splicing operation, which is also a hyperparameter.

[0041] S2.2) According to the training requirements of the target size, a training picture splicing strategy set I is configured, and the strategy set includes a regular training picture strategy of an original picture size and a spliced picture strategy of different degrees. The training picture splicing strategy set of the embodiment of the application includes a regular training picture, a two-split spliced training picture, and a three-split spliced training picture. The regular training picture refers to using the original picture size, the width and the height are w and h, the two-split splicing refers to that after the width and the height of the image are changed to 1 / 2, four pictures are spliced into one training picture, and the width and the height of the spliced image are w and h, the three-split splicing refers to that after the width and the height of the image are changed to 1 / 3, nine pictures are spliced into one training picture, and the width and the height of the spliced image are w and h, as shown in FIG. 1, after splicing of different degrees, the target size is reduced. The splicing picture strategy determines the degree of splicing according to the target size required by the actual application, but the value of N in N-split splicing cannot be too large, which will lead to too small target size, and the network calculation will swallow the too small target. Generally, after three-split splicing, that is, nine-picture splicing, the target is already very small, which is sufficient to meet the demand for very small targets. Figure 4

[0042] S2.3) Statistics of the current iteration gradient information, r1 and r2 are calculated respectively, r1 is the gradient proportion of the training sample of the size s2 < s < s1 in the current iteration, and r2 is the gradient proportion of the training sample of the size s < s2 in the current iteration.

[0043] S2.4) If r1 ≤ r2 and r1 ≤ t, it is indicated that the gradient information of the small object of the size s2 < s < s1 is insufficient at this time, and therefore the two-split splicing strategy is selected to generate the training picture in the next iteration.

[0044] S2.5) If r2 < r1 and r2 ≤ t, it is indicated that the gradient information of the very small object of the size s < s2 is insufficient at this time, and therefore the three-split splicing strategy is selected to generate the training picture in the next iteration.

[0045] S2.6) If the above two conditions are not met, that is, r1 > t and r2 > t, it is indicated that the gradient information of the small object in the current iteration is not insufficient, and therefore the regular training picture is selected in the next iteration.

[0046] S2.7) According to the gradient statistical information of the current iteration, the training picture splicing strategy of the next iteration is determined, and the training set of the next iteration is adjusted, that is, the training set after the S1 sample balancing is combined with the splicing strategy, so that the training set of the next iteration is balanced in class and size.​

Claims

1. A loss feedback based class and scale balanced object detection training method, characterized in that When iteratively training the target detection network, the training samples are respectively subjected to class balancing and scale balancing based on loss feedback of the loss function. The class balancing dynamically adjusts the sampling probability of each class and each sample in the iterative training according to the gradient statistical information of each class sample in the current iteration, so as to balance the training classes and sample data, and obtain a class-balanced training set I'. The scale balancing dynamically adjusts the scale of the training samples of the training set I' according to the gradient statistical information of the training samples of the training set I' in the current iteration based on the loss feedback of the loss function, so as to make the target scale of the training set I' meet the training requirement through image splicing. The training set I'' subjected to the class balancing and the scale balancing is input into the target detection network as the training sample of a new iteration. The class balancing specifically comprises: S1.1) defining g(c) as the gradient proportion of the training sample of class c in the current iteration, r(c) as the oversampling coefficient of class c, r(i) as the oversampling coefficient of sample i, D as the data augmentation operation set, k as the oversampling threshold, and as a hyperparameter; S1.2) for each class, the gradient proportion g(c) of the training sample in the current iteration is counted; S1.3) updating the oversampling coefficients of each class and sample for the next iteration according to the gradient proportion of each class g(c), the oversampling coefficient of class c for the next iteration is That is, when g(c)≥k, the class c does not oversample, when g(c)<k, the class c oversamples, and the oversampling coefficient is S1.4) The oversampling coefficient of sample i is That is, in the categories in which sample i appears, the maximum category oversampling coefficient is taken as the oversampling coefficient of sample i; S1.5) steps S1.2)-S1.4) are performed on the training sample set of the current iteration, and the oversampling coefficient of each sample is rounded up as the repetition number of the sample in the training sample set of the next iteration; S1.6) the training set I' subjected to the sample class balancing based on the loss feedback is returned, and is used for the next iteration training.

2. The loss feedback-based class and scale balanced object detection training method according to claim 1, characterized in that In S1.5), each sample is transformed through a random data augmentation strategy and then added to the training set I'. The operation, degree and probability of the data augmentation are pre-set, and are randomly selected when executed.

3. The loss feedback based class and scale balanced object detection training method according to claim 1, characterized in that The scale balancing specifically comprises: S2.1) defining s as the sample scale, s1 as the small object determination scale, s≤s1 as a small object, s2 as the extremely small object determination scale, s≤s2<s1 as an extremely small object, r1 as the gradient proportion of the training sample with the scale satisfying s2<s<s1 in the current iteration, r2 as the gradient proportion of the training sample with the scale satisfying s<s2 in the current iteration, I as the training picture splicing strategy set, and t as the gradient proportion threshold for performing the image splicing operation; S2.2) the training picture splicing strategy set I is configured according to the training requirement of the target size, and the strategy set includes a regular training picture strategy of an original picture size and a splicing picture strategy of different degrees. After being spliced to different degrees, the target scale in the training picture is reduced; S2.3) the gradient information of the current iteration is counted, and r1 and r2 are calculated respectively; S2.4) if r1≤r2 and r1≤t, it is indicated that the gradient information of the small object with the scale satisfying s2<s<s1 is insufficient at this time, and therefore the splicing strategy corresponding to the small object is selected to generate the training picture in the next iteration; S2.5) if r2<r1 and r2≤r, it is indicated that the gradient information of the extremely small object with the scale satisfying s<s2 is insufficient at this time, and therefore the splicing strategy corresponding to the extremely small object is selected to generate the training picture in the next iteration. S2.6) If neither of the above two conditions is met, i.e. r1 > t and r2 > t, it means that the current iteration is not lacking gradient information of small objects, then the next iteration selects the regular training picture strategy; S2.7) Based on the gradient statistical information of the current iteration, determine the training image stitching strategy for the next iteration, and adjust the training set for the next iteration.

Images