A target detection method for non-motorized vehicle scenarios

Abstract

This invention relates to the field of image object detection technology and discloses a method for object detection in non-motorized vehicle scenes. Specifically, it includes: sampling based on repetition factors, calculating a repetition factor based on the frequency of occurrence of training images containing specific object categories to adjust the sampling rate of training images containing specific object categories from the training dataset; balanced enhancement based on mosaic, repeating objects of the non-motorized vehicle category at random positions in the training images according to the repetition factor of each category; using an object detection model to sequentially extract features from the training images, generate candidate boxes for possible objects, classify the objects in the candidate boxes, and finally obtain the detection result. This invention uses sampling based on repetition factors and balanced sampling based on mosaic to increase the proportion of non-motorized vehicle instances in the original dataset, and can adapt well to non-motorized vehicle lane scenes without the need for additional non-motorized vehicle lane dataset collection.

Description

Technical Field

[0001] This invention relates to the field of image target detection technology, and specifically to a target detection method for non-motorized vehicle scenes. Background Technology

[0002] Currently, object detection technology is relatively mature, especially object detection in traffic systems' motor vehicle lanes. For example, patent application CN118552929A discloses a traffic object detection method and system based on improved YOLOv8n, which improves recognition accuracy by modifying the downsampling component of YOLOv8. Patent application CN118247766A discloses a traffic object detection algorithm based on YOLOv8, which adapts to traffic scenarios by constructing a small object detection layer, introducing a TripletAttention mechanism, and using deformable convolution with the WIoU_v3 loss function. Patent application CN117315614A discloses a traffic object detection method based on improved YOLOv7, which uses a ReduceBlock module to replace the original convolution for deep feature extraction, embeds a SimCAM module in the head prediction module of the YOLOv7 network structure to deepen the extraction of key features, and changes the bounding box regression loss function to the FocalDIoU function to improve recognition accuracy in traffic scenarios.

[0003] However, datasets and models specifically designed for non-motorized vehicle lanes are scarce, and re-annotating them on non-motorized vehicle lanes is extremely costly. Therefore, it's natural to consider using existing models and datasets for motorized vehicle lanes to train a network with excellent detection performance on non-motorized vehicle lanes. However, directly using this method is not well-suited for non-motorized vehicle lane scenarios because the number of labeled motorized vehicles in typical traffic object detection datasets is far greater than that for non-motorized vehicles, resulting in lower detection accuracy for non-motorized vehicles. Furthermore, due to the limitations of the scenario, the available computing power is significantly lower for non-motorized vehicle lanes. Summary of the Invention

[0004] To address the long-tail and computational challenges of non-motorized vehicle lane scenarios, this invention provides a target detection method for these scenarios, enabling the model to perform better detection on non-motorized vehicle lanes. First, this invention uses sampling based on repetition factors and balanced augmentation based on mosaicism to improve the model's accuracy for the minority class in motorized vehicle lane datasets. Second, this invention uses long-tailed uniform focusing loss to weight the losses for different classes, further enhancing the detection capability for non-motorized vehicles. Finally, this invention utilizes the trained model and employs a random mask generation-based distillation method (MGD) to distill features at different scales, thereby further improving the model's accuracy without increasing computational cost.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A target detection method for non-motorized vehicle scenes, based on existing target detection models, achieves target detection in non-motorized vehicle scenes through sampling based on repetition factors and balanced sampling based on mosaic; specifically including:

[0007] Sampling based on repetition factor: A repetition factor is calculated based on the frequency of training images containing objects of a specific class to adjust the sampling rate of training images containing objects of a specific class from the training dataset; if the frequency f(c) of training images containing objects of the c-th class is greater than or equal to a preset threshold oversample... thr If f(c) is less than oversample, then the sampling rate of the training images containing objects of the c-th class remains at 1; thr If the sampling rate of the training images containing objects of the c-th category is greater than 1, then the sampling rate of the training images will be greater than 1.

[0008] Mosaic-based balanced augmentation sampling: Based on the repetition factor r(c) of each category, the objects of the non-motorized vehicle category are repeated at random locations in the training image.

[0009] The object detection model is used to sequentially extract features from training images, generate candidate boxes for possible objects, classify the objects within the candidate boxes, and finally obtain the detection results. The loss function of the object detection model is... Including CIOU losses Bounding box regression loss and classification loss

[0010]

[0011] Furthermore, the step of calculating a repetition factor based on the frequency of occurrence of training images containing specific categories of objects, and adjusting the sampling rate of training images containing specific categories of objects from the training dataset, specifically includes:

[0012] For each category c, calculate the repetition factor r(c):

[0013]

[0014] For the training image I of the c-th category, calculate the sampling rate r(I):

[0015]

[0016] Furthermore, the step of repeating objects of the non-motorized vehicle category at random locations in the training image according to the repetition factor r(c) for each category specifically includes:

[0017]

[0018] Δx, Δy=random(0,w-x1),random(0,h-y1);

[0019]

[0020] Where p∈[0,1], p represents the probability of uniform sampling between 0 and 1. t Let r(c) be the probability threshold calculated based on r(c), Δx and Δy be the random offsets of the x and y coordinates, w and h represent the length and width of the image, respectively, and IOU be the probability threshold. max This represents the maximum IOU calculated with the existing detection boxes. The detected object is randomly copied with a probability of pt, and the maximum IOU between the newly copied object and the existing detected object does not exceed the threshold. x′1, y′1, x′2, and y′2 represent the positions of the new detection boxes after copying, and x1, y1, x2, and y2 represent the positions of the original instance detection boxes.

[0021] Furthermore, classify losses Replace with long-tail uniform focusing loss

[0022]

[0023] Where, γ b and α t There are two hyperparameters, parameter It is a variable parameter related to the degree of imbalance in the j-th category; parameter g j γ represents the ratio of the cumulative gradients of positive samples of the j-th class to negative samples of the j-th class; γ represents the class coefficient calculated based on the gradient; s represents a hyperparameter used to adjust the influence of the gradient on the class coefficient; C represents the detected class.

[0024] The loss function of the object detection model Replace with:

[0025]

[0026] Furthermore, it also includes a distillation method based on random mask generation to adapt to the computing power limitations present in non-motorized vehicle scenarios;

[0027] Use T l ∈R C×H×W and S l ∈R C×H×WLet L represent the teacher feature map at layer l of the teacher detection network and the student feature map at layer l of the student detection network, respectively; set the l-th random mask to cover the student feature map at layer l of the student detection network.

[0028]

[0029] in, Let i be a random number in (0, 1), and j be the x and y coordinates of the student feature map, respectively; λ is a hyperparameter representing the mask ratio; then the corresponding mask is used to cover the student feature map of the student detection network, and an attempt is made to generate the teacher feature map of the teacher detection network.

[0030] Features are reconstructed using spatial-dimensional and channel-dimensional masking; global teacher feature information is introduced to assist distillation based on the complementary relationship between multi-level feature pyramids for object detection.

[0031]

[0032] Where G represents the use of convolutional layers, T represents a spatial dimension masking of the features. FPN This represents the features calculated by the teacher across multiple scales; f space This represents a convolutional module based on a 3×3 convolutional kernel, f space Align the teacher feature map dimensions by spatial dimension masking; f channel f represents a linear layer based on an MLP. channel After masking the features along the channel dimension, align them with the teacher's feature map; f global f represents a module that aligns global features. global By aligning the dimensions of the global features of the teacher detection network through average pooling and downsampling;

[0033] Distillation loss L dist for:

[0034]

[0035] L represents the total number of layers in the teacher detection network or student detection network, C represents the number of feature map channels, H represents the feature map height, and W represents the feature map length. Let i represent the feature value of the k-th channel dimension of the feature map of the l-th layer of the teacher detection network, with length coordinate i and width coordinate j.

[0036] The loss function of the object detection model Replace with:

[0037]

[0038] Compared with the prior art, the beneficial technical effects of the present invention are:

[0039] This invention adapts well to non-motorized vehicle lane scenarios without requiring additional non-motorized vehicle lane datasets. First, it employs sampling based on repetition factors and balanced sampling based on mosaicism to increase the proportion of non-motorized vehicle instances in the original dataset. Second, it uses a long-tailed uniform focusing loss method to reweight the loss, thereby increasing the contribution of non-motorized vehicle instances to the loss and further improving accuracy in non-motorized vehicle scenarios. Third, it uses a distillation method based on random masking to mask some student features to recover teacher features, improving detection accuracy without increasing computational power. Finally, this invention performs well in target detection tasks in non-motorized vehicle scenarios, demonstrating strong generalization ability and practical value. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the target detection method in a non-motorized vehicle scene according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the distillation generation module in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the output results of the target detection method in a non-motorized vehicle scene in an embodiment of the present invention. Detailed Implementation

[0043] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0044] This invention discloses a target detection method for non-motorized vehicle scenes. Based on the existing YOLO target detection network, it proposes a sampling method that combines repetition factor sampling and mosaic-based balanced sampling. Specifically, it includes a target detection model, repetition factor sampling, mosaic-based balanced enhancement method, long-tailed uniform focusing loss, and a distillation method based on random mask generation.

[0045] 1. Object detection model:

[0046] This invention uses a basic single-stage object detection network to sequentially extract features from the input image, generate candidate bounding boxes for possible objects, classify objects within the candidate boxes, and so on, ultimately obtaining the detection result. This invention uses Gold-YOLO as the basic object detection network to guide the network in extracting features beneficial for object detection. The object detection loss function includes the network's classification loss, CIOU loss, and bounding box regression loss.

[0047]

[0048] 2. Sampling methods based on repetition factors:

[0049] The repetition factor-based sampling method is an image-level resampling method that adjusts the sampling rate based on the frequency of each category. Specifically, for a given category c, if its frequency f(c) exceeds a preset threshold for oversampling... thr If the frequency f(c) is less than the threshold, the sampling rate will remain at 1, meaning no oversampling will occur; if the frequency f(c) is less than the threshold, the sampling rate of images containing that category will be greater than 1, meaning these images will be sampled repeatedly.

[0050] Calculate the repetition factor for each category c:

[0051]

[0052] For each image I, calculate the sampling rate:

[0053] r(I) = max cinI r(c);

[0054] Where r(c) is the repetition factor for each category, and r(I) is the sampling rate.

[0055] 3. Mosaic-based equilibrium enhancement method:

[0056] Since the repetition factor-based sampling method is an image-level sampling method, using mosaic data augmentation will introduce a large number of head classes (motor vehicles). Furthermore, rare classes have limited locations on the image. During inference, when a rare class from the training set appears in a location not present during training, it is easily misidentified as another class. Therefore, based on the repetition factor r(c) of each class sampled using the repetition factor, the minority classes (such as non-motor vehicles) are randomly repeated at random locations on the image.

[0057]

[0058] Δx, Δy=random(0,w-x1),random(0,h-y1);

[0059]

[0060] Where, p∈[0,1], p t Let r(c) be the probability threshold calculated from r(c), where Δx and Δy are random offsets, and IOU is... max This indicates the maximum value of the Intersection over Union (IOU) calculated with existing detection boxes. The detected object is randomly copied with a probability of pt, and the maximum IOU between the newly copied object and the existing detected object does not exceed the threshold.

[0061] 4. Long-tail uniform focusing loss:

[0062] When training on the dataset for motor vehicle lanes, the limited number of non-motorized vehicle instances results in a small contribution of minority class samples to the loss, leading to poor performance. Therefore, a uniform weighting is applied to the classification loss. Replace with long-tail uniform focusing loss

[0063]

[0064] Where γ b and α t There are two hyperparameters. Parameters The parameter g is a variable related to the imbalance of the j-th category. j This represents the ratio of the cumulative gradients of positive samples from the j-th class to those of negative samples from the j-th class. It determines the magnitude of the learning concentration in the problem of positive-negative imbalance.

[0065] The improved loss function of the overall object detection model It becomes:

[0066]

[0067] 5. Distillation method based on random masking:

[0068] Due to the computational limitations in non-motorized vehicle scenarios, accuracy can be further improved through knowledge distillation without increasing computational power. This invention uses T... l ∈R C×H×W and S l ∈R C×H×W Let represent the feature maps of the l-th layer of the teacher detection network and the student detection network, respectively. First, a random mask is set to cover the feature map of the l-th layer of the student detection network, which can be represented as:

[0069]

[0070] in Let be a random number in (0, 1), and let i and j be the x and y coordinates of the feature map, respectively. λ is a hyperparameter representing the mask ratio. Then, the feature map of the student detection network is overlaid with the corresponding mask, and an attempt is made to generate the feature map of the teacher detection network.

[0071] This invention uses two different masks, namely spatial dimension-based mask and channel dimension-based mask, to better reconstruct features. At the same time, due to the complementary relationship between the multi-level feature pyramids for object detection, this invention introduces global teacher feature information to assist distillation.

[0072]

[0073] Where f space The feature map dimensions of the teacher detection network are aligned using a 3×3 convolutional kernel based on convolution, f. channel Align the feature maps of the teacher detection network with those of the MLP-based linear layer, f global By aligning the dimensions of the global features of the teacher detection network through average pooling and downsampling, the distillation loss L dist It can be represented as:

[0074]

[0075] In summary, the overall loss function of the object detection model for:

[0076]

[0077] Example

[0078] The object detection model of this invention is implemented through a single-stage detection network. The method proposed in this invention can improve the detection capability of the object detection model on non-motorized vehicle lanes without the need to collect additional non-motorized vehicle lane datasets. First, input images from the motorized vehicle lane dataset and annotate them to train the detection network, enabling the network to have good detection capabilities. Then, input the images to be detected from the non-motorized vehicle lanes into the network to directly obtain the location and category of objects in each image.

[0079] Step 1: Prepare the training dataset for motor vehicle lanes and the test dataset for non-motor vehicle lanes. Then, scale the images to maintain a size of 600*600 pixels, and perform preprocessing using standard image enhancement techniques. The final training data includes images and annotations: The data for the non-motorized vehicle lanes tested only includes images and does not require annotation:

[0080] Step 2: Build an object detection model using a deep learning framework. The object detection model consists of a data sampling enhancement module, a student detection network with a small number of parameters, a teacher detection network with a large number of parameters, and a distillation module. The base detection network can be a mainstream single-stage object detection network, such as the backbone, which can be YOLOv8 or Gold-YOLO. In this embodiment, Gold-YOLO is used as the backbone. Simultaneously, a teacher detection network with a large number of parameters, such as Gold-YOLOm, and a lightweight student detection network, such as Gold-YOLOm, are used. Then, a neighborhood discriminator is set at the mid-level and high-level feature outputs of the backbone to classify the features. The neighborhood discriminator is connected to the backbone using a gradient backpropagation network. Finally, a center feature alignment module is introduced in the region proposal network to align the region proposal features of the source and target domains.

[0081] Step 3: Calculate the sampling factor for the input batch of images, and enhance the images by sampling based on the repetition factor and balanced sampling based on the mosaic.

[0082] Step 4: The model obtains teacher features through a teacher detection network with a large number of parameters and student features through a target student detection network with a small number of parameters. The generation module is used to generate teacher feature maps from student features. Then, the distillation module is used to calculate the distillation loss with the teacher feature maps.

[0083] Step 5: After the model calculates the classification loss, record the gradient value g of the backpropagation. j And mapped to the value range of 0-1, through 1-g j The category weights corresponding to different categories are calculated. The long-tailed uniform focusing loss described above is used to further improve the model's ability to identify rare categories such as motor vehicles and pedestrians.

[0084] Step 6: Stop iterating after reaching the maximum number of iterations to obtain the final detection network. Input the test dataset and calculate the model's performance in object detection within the target domain. See the detection results below. Figure 3 .

[0085] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0086] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A target detection method for non-motorized vehicle scenarios, characterized in that, Based on existing target detection models, target detection in non-motorized vehicle scenarios is achieved through sampling based on repetition factors and balanced sampling based on mosaicism; specifically including: Sampling based on repetition factor: A repetition factor is calculated based on the frequency of occurrence of training images containing specific object categories to adjust the sampling rate of training images containing specific object categories from the training dataset. Specifically, this includes: Calculate the repetition factor for each category c. : Training images for the c-th category Calculate the sampling rate : ; If the training images containing objects of the c-th category appear frequently Greater than or equal to the preset threshold If the sampling rate of the training images containing objects of the c-th category remains 1, then... Less than If the sampling rate of the training images containing objects of the c-th category is greater than 1, then the sampling rate of the training images will be greater than 1. Mosaic-based balanced augmentation sampling: based on the repetition factor of each class The training image repeats the process for non-motorized vehicle objects at random locations, specifically including: ; ； ； in, , This represents the probability of uniform sampling between 0 and 1. According to The calculated probability threshold, For random offsets of the x and y coordinates, These represent the length and width of the image, respectively. This indicates the maximum value of the Intersection over Union (IOU) calculated with existing bounding boxes. The probability is to randomly copy the detected object, and the maximum IOU between the newly copied detected object and the existing detected object does not exceed the threshold. These represent the positions of the new detection boxes after copying. These represent the positions of the original instance detection boxes; The object detection model is used to sequentially extract features from training images, generate candidate boxes for possible objects, classify the objects within the candidate boxes, and finally obtain the detection results. The loss function of the object detection model is... Including CIOU losses Bounding box regression loss and long-tail uniform focusing loss : ; ; ;in, and There are two hyperparameters, parameter These are variable parameters related to the degree of imbalance in the j-th category; parameters This represents the ratio of the cumulative gradient of the j-th class positive sample to the j-th class negative sample; This represents the category coefficient calculated based on the gradient; This represents a hyperparameter used to adjust the effect of the gradient on the class coefficient; Indicates the category of detection.

2. The target detection method for non-motorized vehicle scenarios according to claim 1, characterized in that, It also includes a distillation method based on random mask generation to adapt to the computing power limitations present in non-motorized vehicle scenarios; use and Representing the teacher detection network, respectively. The teacher feature map and the first layer of the student detection network Student feature map of layer; set the first layer A random mask is used to cover the first student detection network. Student Feature Map of Layers : ； in, A random number in (0,1) These are the x and y coordinates of the student feature map, respectively. It is a hyperparameter representing the mask ratio; then the corresponding mask is used to cover the student feature map of the student detection network, and an attempt is made to generate the teacher feature map of the teacher detection network; Features are reconstructed using spatial-dimensional and channel-dimensional masking; global teacher feature information is introduced to assist distillation based on the complementary relationship between multi-level feature pyramids for object detection. ； in, This indicates the use of convolutional layers. This indicates that the features are masked in a spatial dimension. This represents the features calculated by the teacher across multiple scales; Indicates based on The convolution module of the convolution kernel, Align the teacher feature map dimensions by spatial dimension masking of the features; This represents a linear layer based on an MLP. The features are aligned with the teacher's feature map after channel dimension masking. This indicates a module that aligns global features. By aligning the dimensions of the global features of the teacher detection network through average pooling and downsampling; Distillation loss for: ； This represents the total number of layers in the teacher detection network or the student detection network. Indicates the number of channels in the feature map. Indicates the feature map height. Indicates the length of the feature map. Let i represent the feature value of the k-th channel dimension of the feature map of the l-th layer of the teacher detection network, with length coordinate i and width coordinate j. The loss function of the object detection model Replace with: 。

Images