Target detection method for non-motor vehicle scene

Abstract

The invention relates to the technical field of image target detection, and discloses a target detection method for a non-motor vehicle scene, which specifically comprises the following steps of: based on sampling of repetition factors, calculating the repetition factors according to the occurrence frequency of a training image containing a specific category of objects; adjusting the sampling rate of sampling the training image containing the specific category of objects from the training data set; based on mosaic balance enhancement, according to the repetition factor of each category, repeating the object of the non-motor vehicle category at the random position of the training image; and sequentially performing feature extraction on the training images by using the target detection model, generating candidate frames of possible objects, classifying the objects in the candidate frames, and finally obtaining a detection result. The non-motor vehicle instance proportion in the original data set is increased by using the sampling method based on the repetition factor and the balanced sampling method based on mosaic, and the non-motor vehicle lane scene can be well adapted under the condition that the non-motor vehicle lane data set does not need to be additionally collected.

Description

Technical Field

[0001] The present invention relates to the technical field of image target detection, and in particular to a target detection method for non-motor vehicle scenes. Background Art

[0002] At present, target detection technology is relatively mature, especially target detection on motor vehicle lanes in traffic systems. For example, the patent with application publication number CN118552929A discloses a traffic target detection method and system based on improved YOLOv8n, and improves the recognition accuracy by improving the downsampling component of YOLOv8. The patent with application publication number CN118247766A discloses a traffic target detection algorithm based on YOLOV8, which is adapted to traffic scenes by constructing a small target detection layer, introducing the TripletAttention mechanism and deformable convolution and WIoU_v3 loss function. The patent with application publication number CN117315614A discloses a traffic target detection method based on improved YOLOv7, which uses the ReduceBlock module to replace the original convolution for deep feature extraction, and embeds the SimCAM module in the head prediction module of the YOLOv7 network structure to deepen the extraction of key features, and replaces the bounding box regression loss function with the FocalDIoU function to improve the recognition accuracy in traffic scenes.

[0003] However, there is a lack of datasets and models for non-motorized vehicle lanes, and it is very expensive to re-label non-motorized vehicle lanes. For this reason, people naturally wonder whether they can use existing models and datasets on motor vehicle lanes to train a network that can perform well on non-motorized vehicle lanes. However, using this method directly cannot adapt well to non-motorized vehicle lane scenarios, because the number of motor vehicle annotations in the usual traffic target detection dataset is much larger than that of non-motorized vehicles, resulting in low non-motorized vehicle detection accuracy. In addition, due to the scenario, the computing power that can be used for non-motorized vehicle lane scenarios is lower. Summary of the invention

[0004] In response to the long-tail problem and computing power problem of non-motorized vehicle lane scenes, the present invention provides a target detection method for non-motorized vehicle lane scenes, so that the model can better perform detection on non-motorized vehicle lanes. First, the present invention uses repetition factor-based sampling and mosaic-based balanced enhancement to improve the accuracy of the model for minority classes in the motor vehicle lane data set. Secondly, the present invention uses long-tail uniform focusing loss to weight the losses of different categories to further improve the detection capability of non-motorized vehicles. Finally, the present invention uses the trained model and uses a distillation method based on random mask generation (MGD) to distill features of different scales of the model, thereby further improving the accuracy of the model without increasing the computing power of the model.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A target detection method for non-motor vehicle scenes, based on the existing target detection model, realizes target detection in non-motor vehicle scenes through sampling based on repetition factor and balanced sampling based on mosaic; specifically, it includes:

[0007] Sampling based on repetition factor: The repetition factor is calculated based on the frequency of occurrence of training images containing objects of a specific category to adjust the sampling rate of training images containing objects of a specific category from the training dataset; if the frequency of occurrence of training images containing objects of the cth category f(c) is greater than or equal to the preset threshold oversample thr , then the sampling rate of the training images containing objects of the cth category remains 1; if f(c) is less than oversample thr , then the sampling rate of the training images containing objects of the cth category will be greater than 1;

[0008] Mosaic-based balanced augmented sampling: objects of the non-motor vehicle category are repeated at random positions in the training images according to the repetition factor r(c) of each category;

[0009] Use the target detection model to extract features from the training images in turn, generate candidate boxes for possible objects, classify the objects in the candidate boxes, and finally get the detection results. The loss function of the target detection model is Including CIOU losses Bounding Box Regression Loss and classification loss

[0010]

[0011] Furthermore, the calculation of the repetition factor according to the frequency of occurrence of the training images containing the specific category of objects to adjust the sampling rate of the training images containing the specific category of objects from the training data set specifically includes:

[0012] For each category c, calculate the repetition factor r(c):

[0013]

[0014] For the training image I of the cth category, calculate the sampling rate r(I):

[0015]

[0016] Furthermore, the method of repeating the objects of the non-motor vehicle category at random positions of the training image according to the repetition factor r(c) of each category specifically includes:

[0017]

[0018] Δx, Δy = random(0, wx 1 ), random(0,hy 1 );

[0019]

[0020] Among them, p∈[0,1], p represents the probability of uniform sampling between 0 and 1, p t is the probability threshold calculated according to r(c), Δx and Δy are the random offsets of the horizontal and vertical coordinates, w and h represent the length and width of the image respectively, and IOU max Indicates that the maximum IOU value is calculated with the existing detection box, and the detection object is randomly copied with the probability of pt, and the maximum IOU value of the newly copied detection object and the existing detection object does not exceed the threshold; x′ 1 , y′ 1 , x′ 2 , y′ 2 Respectively represent the position of the new detection box after copying, x 1 ,y 1 , x 2 ,y 2 They respectively represent the positions of the original instance detection boxes.

[0021] Furthermore, the classification loss Replaced with long-tail uniform focusing loss

[0022]

[0023] Among them, γ b and α t are two hyperparameters, parameters is a variable parameter related to the degree of imbalance of the jth category; parameter g j represents the ratio of the cumulative gradient of the j-th category positive sample to the j-th category negative sample; γ represents the category coefficient calculated based on the gradient; s represents a hyperparameter used to adjust the impact of the gradient on the category coefficient; C represents the detected category;

[0024] The loss function of the target detection model Replace with:

[0025]

[0026] Furthermore, it also includes a distillation method based on random mask generation to adapt to the computing power limitations of non-motor vehicle scenarios;

[0027] Use T l ∈R C×H×W and S l ∈R C×H×W Represents the teacher feature map of the lth layer of the teacher detection network and the student feature map of the lth layer of the student detection network; set the lth random mask to cover the student feature map of the lth layer of the student detection network

[0028]

[0029] in, is a random number in (0, 1), i and j are the horizontal and vertical coordinates of the student feature map respectively; λ is a hyperparameter representing the mask ratio; then the corresponding mask is used to cover the student feature map of the student detection network, and an attempt is made to generate the teacher feature map of the teacher detection network;

[0030] Use masks based on spatial dimensions and masks based on channel dimensions to reconstruct features; based on the complementary relationship between multi-level feature pyramids of target detection, introduce global teacher feature information to assist distillation:

[0031]

[0032] Among them, G represents the use of convolutional layers, Indicates the masking of the spatial dimension of the feature, T FPN represents the features of multiple scales calculated by the teacher; f space represents a convolution module based on a 3×3 convolution kernel, f space Align the teacher feature map dimensions by masking the features in spatial dimensions; f channel represents the MLP-based linear layer, f channel The features are masked in the channel dimension and aligned with the teacher feature map; f global represents the module that aligns the global features, f global The global features of the teacher detection network are aligned to the dimension through average pooling and downsampling;

[0033] Distillation loss L dist for:

[0034]

[0035] L represents the total number of layers of the teacher detection network or the student detection network, C represents the number of feature map channels, H represents the feature map height, and W represents the length of the feature map. The feature value of the k-th channel dimension of the feature map of the l-th layer of the teacher detection network with the long coordinate i and the wide coordinate j;

[0036] The loss function of the target detection model Replace with:

[0037]

[0038] Compared with the prior art, the beneficial technical effects of the present invention are:

[0039] The present invention can also adapt well to non-motorized vehicle lane scenarios without the need to collect additional non-motorized vehicle lane data sets. First, the present invention uses sampling based on repetition factors and balanced sampling methods based on mosaics to increase the proportion of non-motorized vehicle instances in the original data set; secondly, the present invention uses a long-tail uniform focusing loss method to promote loss re-weighting, thereby increasing the contribution of non-motorized vehicle instances to the loss, and further improving the accuracy in non-motorized vehicle scenarios; then, the present invention uses a distillation method based on random masking to mask part of the student features to restore the teacher features, thereby improving the detection accuracy without increasing computing power; finally, the present invention has good performance in target detection tasks in non-motorized vehicle scenarios, and has strong generalization ability and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a target detection method for a non-motor vehicle scene in an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of a distillation generation module in an embodiment of the present invention;

[0042] Figure 3 Schematic diagram of output results of the target detection method for non-motor vehicle scene in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0044] The present invention discloses a target detection method for non-motor vehicle scenes. On the basis of using the previous YOLO target detection network, a sampling method combining repetition factor and a balanced sampling method based on mosaic is proposed. The method specifically comprises a target detection model, sampling based on repetition factor, a balanced enhancement method based on mosaic, a long-tail uniform focus loss and a distillation method based on random mask generation.

[0045] 1. Object Detection Model:

[0046] The present invention uses a basic single-stage target detection network to extract features from the input image, generate candidate boxes of possible objects, classify objects in the candidate boxes, and finally obtain the detection results. The present invention uses gold-yolo as the basic target detection network to guide the network to extract features that are conducive to target detection. The target detection loss function includes the classification loss, CIOU loss and bounding box regression loss of the network:

[0047]

[0048] 2. Sampling method based on repetition factor:

[0049] The sampling method based on the repetition factor is an image-level resampling method, which works by adjusting the sampling rate based on the frequency of the category. Specifically, for a category c, if its frequency f(c) is greater than a preset threshold oversample thr , the sampling rate remains at 1, that is, no oversampling is performed; if its frequency f(c) is less than the threshold, the sampling rate of images containing this category will be greater than 1, that is, these images will be repeatedly sampled.

[0050] Compute the replication factor for each category c:

[0051]

[0052] For each image I, calculate the sampling rate:

[0053] r(I)=max cinI r(c);

[0054] Where r(c) is the repetition factor for each category and r(I) is the sampling rate.

[0055] 3. Mosaic-based balance enhancement method:

[0056] Since the sampling method based on the repetition factor is an image-level sampling method, mosaic data enhancement will mix in a large number of head classes (motor vehicles). At the same time, the rare categories appear in limited locations on the image. During the reasoning process, when the rare categories in the training set appear in places that did not appear during training, they are easily identified as other categories. Therefore, according to the repetition factor r(c) of each class sampled by the repetition factor, the minority classes (such as non-motor vehicles) are randomly repeated at random locations on the image.

[0057]

[0058] Δx, Δy = random(0, wx 1 ), random(0,hy 1);

[0059]

[0060] Among them, p∈[0,1], p t is the probability threshold calculated based on r(c), Δx, Δy are random offsets, and IOU max It means to calculate the maximum IOU value with the existing detection box, randomly copy the detection object with the probability of pt, and the maximum IOU value between the newly copied detection object and the existing detection object does not exceed the threshold.

[0061] 4. Long-tail uniform focusing loss:

[0062] When training the motor vehicle lane dataset, there are too few non-motor vehicle instances, resulting in a small contribution of minority class samples to the loss, resulting in poor results. Therefore, the classification loss is evenly weighted. Replaced with long-tail uniform focusing loss

[0063]

[0064] where γ b and α t are two hyperparameters. is the variable parameter related to the degree of imbalance of the jth category, parameter g j It represents the ratio of the cumulative gradient of the j-th category positive sample to the j-th category negative sample. It determines how much the learning concentration is for the positive-negative imbalance problem.

[0065] After improvement, the overall loss function of the target detection model is becomes:

[0066]

[0067] 5. Distillation method based on random mask generation:

[0068] Due to the computing power limitation of non-motor vehicle scenarios, the accuracy can be further improved by knowledge distillation without increasing any computing power. l ∈R C×H×W and S l ∈R C×H×W Denote the feature maps of the lth layer of the teacher detection network and the student detection network respectively. First, set the lth random mask to cover the feature map of the lth layer of the student detection network, which can be expressed as:

[0069]

[0070] in is a random number in (0, 1), i and j are the horizontal and vertical coordinates of the feature map respectively. λ is a hyperparameter representing the mask ratio. The corresponding mask is then used to cover the feature map of the student detection network and try to generate the feature map of the teacher detection network.

[0071] The present invention uses two different masks, namely, masks based on spatial dimensions and masks based on channel dimensions, to better reconstruct features. At the same time, due to the complementary relationship between the multi-level feature pyramids of target detection, the present invention introduces global teacher feature information to assist distillation.

[0072]

[0073] where f space The feature map dimensions of the teacher detection network are aligned using a 3×3 convolutional kernel based on convolution, f channel Align the feature maps of the teacher detection network using an MLP-based linear layer, f global The distillation loss L is obtained by aligning the global features of the teacher detection network through average pooling and downsampling. dist It can be expressed as:

[0074]

[0075] In summary, the overall loss function of the target detection model is for:

[0076]

[0077] Example

[0078] The target detection model of the present invention is implemented through a single-stage detection network. The method proposed by the present invention can improve the detection capability of the target detection model on non-motorized vehicle lanes without collecting additional non-motorized vehicle lane data sets. First, the motor vehicle lane data set images and annotations are input to train the detection network so that the network has good detection capabilities; then the images to be detected of the non-motorized vehicle lanes are sent to the network to directly obtain the position and category of the objects in each image.

[0079] Step 1: Prepare the training data set for motor vehicle lanes and the test data set for non-motor vehicle lanes. Then scale the image to keep the image size at 600*600, and then use some conventional image enhancement methods for preprocessing. The final training data includes images and annotations: The data for the non-motorized lanes tested only contains images and does not require annotation:

[0080] Step 2, use the deep learning framework to establish a target detection model. The target detection model consists of a data sampling enhancement module, a student detection network with a small number of parameters, a teacher detection network with a large number of parameters, and a distillation module. The basic detection network can be a mainstream single-stage target detection network, such as the backbone can select yolov8, gold-yolo, etc., and this embodiment uses gold-yolo as the backbone of the detection network. At the same time, a teacher detection network with a large number of parameters such as gold-yolom and a lightweight student detection network such as gold-yolon are used, and then the domain discriminators are set at the middle and high-level feature outputs of the backbone to classify the features, and the domain discriminator is connected to the backbone using a gradient back propagation network. Finally, the central feature alignment module is introduced at the region proposal network to align the region proposal features of the source domain and the target domain.

[0081] Step 3, the sampling factor is calculated for a batch of input images, and the images are sampled based on the repetition factor and enhanced based on the mosaic balanced sampling method.

[0082] In step 4, the model obtains the teacher features through the teacher detection network with a large number of parameters, and obtains the student features through the target student detection network with a small number of parameters. The generation module is used to generate the teacher feature map with the student features, and then the distillation module is used to calculate the distillation loss with the teacher feature map.

[0083] Step 5: After the model calculates the classification loss, record the gradient value g of the back propagation j And mapped to the value range of 0-1, through 1-g j The category weights corresponding to different categories are calculated. The long-tail uniform focus loss mentioned above is used to further improve the model's ability to recognize rare categories of motor vehicles and pedestrians.

[0084] Step 6: After the number of iterations reaches the maximum, stop the iteration and get the final detection network. Input the test data set and calculate the effect of the model on target detection in the target domain. Figure 3 .

[0085] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0086] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for detecting a target in a non-motor vehicle scene, characterized in that: Based on the existing target detection model, target detection in non-motor vehicle scenes is achieved through sampling based on repetition factors and balanced sampling based on mosaics. Specifically, it includes: Sampling based on repetition factor: The repetition factor is calculated based on the frequency of occurrence of training images containing objects of a specific category to adjust the sampling rate of training images containing objects of a specific category from the training dataset; if the frequency of occurrence of training images containing objects of the cth category f(c) is greater than or equal to the preset threshold oversample thr , then the sampling rate of the training images containing objects of the cth category remains 1; if f(c) is less than oversample thr , then the sampling rate of the training images containing objects of the cth category will be greater than 1; Mosaic-based balanced augmented sampling: objects of the non-motor vehicle category are repeated at random positions in the training images according to the repetition factor r(c) of each category; Use the target detection model to extract features from the training images in turn, generate candidate boxes for possible objects, classify the objects in the candidate boxes, and finally get the detection results. The loss function of the target detection model is Including CIOU losses Bounding Box Regression Loss and classification loss 2. The target detection method for non-motor vehicle scenes according to claim 1, characterized in that: The method of calculating the repetition factor according to the frequency of occurrence of the training images containing the specific category of objects to adjust the sampling rate of the training images containing the specific category of objects from the training data set specifically includes: For each category c, calculate the repetition factor r(c): For the training image I of the cth category, calculate the sampling rate r(I):

3. The target detection method for non-motor vehicle scenes according to claim 1, characterized in that: The method of repeating the objects of the non-motor vehicle category at random positions of the training image according to the repetition factor r(c) of each category specifically includes: Δx,Δy=random(0,w-x1),random(0,h-y1); Among them, p∈[0,1], p represents the probability of uniform sampling between 0 and 1, p t is the probability threshold calculated according to r(c), Δx, Δy are the random offsets of the horizontal and vertical coordinates, w, h represent the length and width of the image respectively, and IOU max Indicates the maximum value of IOU calculated with the existing detection box, expressed as p t The detection object is randomly copied with a probability of , and the maximum IOU between the newly copied detection object and the existing detection object does not exceed the threshold; x′1, y′1, x′2, y′2 respectively represent the positions of the new detection frame after copying, and x1, y1, x2, y2 respectively represent the positions of the original instance detection frame.

4. The target detection method for non-motor vehicle scenes according to claim 3 is characterized in that: The classification loss Replaced with long-tail uniform focusing loss Among them, γ b and α t are two hyperparameters, parameters is a variable parameter related to the degree of imbalance of the jth category; parameter g j represents the ratio of the cumulative gradient of the j-th category positive sample to the j-th category negative sample; γ represents the category coefficient calculated based on the gradient; s represents a hyperparameter used to adjust the impact of the gradient on the category coefficient; C represents the detected category; The loss function of the target detection model Replace with:

5. The target detection method for non-motor vehicle scenes according to claim 4, characterized in that: It also includes a distillation method based on random mask generation to adapt to the computing power limitations of non-motor vehicle scenarios; Use T l ∈R C×H×W and S l ∈R C×H×W denote the teacher feature map of the lth layer of the teacher detection network and the student feature map of the lth layer of the student detection network respectively; Set the lth random mask to cover the student feature map of the lth layer of the student detection network in, is a random number in (0,1), i and j are the horizontal and vertical coordinates of the student feature map respectively; λ is a hyperparameter representing the mask ratio; then the corresponding mask is used to cover the student feature map of the student detection network, and an attempt is made to generate the teacher feature map of the teacher detection network; Use masks based on spatial dimensions and masks based on channel dimensions to reconstruct features; based on the complementary relationship between multi-level feature pyramids of target detection, introduce global teacher feature information to assist distillation: Among them, G represents the use of convolutional layers, Indicates the masking of the spatial dimension of the feature, T FPN represents the features of multiple scales calculated by the teacher; f space represents a convolution module based on a 3×3 convolution kernel, f space Align the teacher feature map dimensions by masking the features in spatial dimensions; f channel represents the MLP-based linear layer, f channel The features are masked in the channel dimension and aligned with the teacher feature map; f global represents the module that aligns the global features, f global The global features of the teacher detection network are aligned to the dimension through average pooling and downsampling; Distillation loss L dist for: L represents the total number of layers of the teacher detection network or the student detection network, C represents the number of feature map channels, H represents the feature map height, and W represents the length of the feature map. The feature value of the k-th channel dimension of the feature map of the l-th layer of the teacher detection network with the long coordinate i and the wide coordinate j; The loss function of the target detection model Replace with:

Images