Traffic scene image small target detection method, device and equipment and storage medium

Abstract

The application relates to the technical field of deep learning, in particular to a traffic scene image small target detection method and device, equipment and a storage medium. The method comprises the following steps: acquiring a traffic scene image and a detection result of a small target in the traffic scene image, generating a training data set according to the traffic scene image and the detection result of the small target in the traffic scene image; constructing a convolutional neural network based on a feature extraction module, a feature fusion module, a sample distribution module and a positioning classification module, and optimizing the sample distribution module and the positioning classification module of the convolutional neural network to obtain an optimized convolutional neural network; training the optimized convolutional neural network by using the training data set, updating network parameters of the optimized convolutional neural network based on gradient back propagation, and stopping the training until a preset training stop condition is met; and detecting small targets in the traffic scene image by using the trained convolutional neural network. Thus, the problems of small target sample imbalance and poor positioning capability in the related art are solved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method, apparatus, device and storage medium for detecting small targets in traffic scene images. Background Technology

[0002] Small object detection is a challenging problem in the object detection branch. Its purpose is to accurately identify and locate small foreground objects with limited information in a scene. Detecting small objects in traffic scenarios has broad application prospects and important practical significance in fields such as autonomous driving technology and road traffic management.

[0003] Current methods for identifying small targets in traffic scenarios mostly employ deep neural networks, optimizing for limited feature information, multi-scale feature representation, and single issues in sample scheme design, without considering the fundamental problems such as the imbalance of small target samples and the difficulty in localization.

[0004] Specifically, sample imbalance reduces the contribution of small targets to the training process of neural network models, while inaccurate localization is caused by the higher sensitivity of small targets to disturbances compared to targets of other scales, manifested in the higher uncertainty of the coordinate localization of small target bounding boxes. Related technologies for small target detection in traffic scenarios generally enrich the information representation of small targets by using multi-scale representation and contextual information, but they do not fundamentally solve the aforementioned problems. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for detecting small targets in traffic scene images, in order to solve the problems of unbalanced small target samples and poor positioning ability in related technologies.

[0006] The first aspect of this application provides a method for detecting small targets in traffic scene images, comprising the following steps: acquiring traffic scene images and detection results of small targets in the traffic scene images; generating a training dataset based on the traffic scene images and the detection results of small targets in the traffic scene images; constructing a convolutional neural network based on a feature extraction module, a feature fusion module, a sample allocation module, and a localization and classification module, and optimizing the sample allocation module and the localization and classification module of the convolutional neural network to obtain an optimized convolutional neural network; training the optimized convolutional neural network using the training dataset, updating the network parameters of the optimized convolutional neural network based on gradient backpropagation until a preset training stopping condition is met, and detecting small targets in the traffic scene images using the trained convolutional neural network.

[0007] Optionally, the sample allocation module and localization / classification module of the convolutional neural network are optimized to obtain an optimized convolutional neural network, including: optimizing the sample allocation output of the sample allocation module using a scale adaptive method; and optimizing the localization / classification module using a quality-driven localization loss.

[0008] Optionally, the sample allocation output by the sample allocation module is optimized using a scale-adaptive method, including: obtaining the ground truth bounding boxes of small targets; calculating the Euclidean distance between preset uniform anchor points and the geometric centers of the ground truth bounding boxes; selecting the Euclidean distance as the anchor points within the target range, generating multiple pseudo boxes based on the anchor points within the target range, using the pseudo boxes as candidate samples, and generating a candidate sample set; calculating the intersection-exclusion ratio (OI) between each pseudo box and the ground truth bounding box in the candidate sample set, and allocating training samples based on the OI.

[0009] Optionally, training samples are allocated based on the intersection-over-union ratio (IoU), including: obtaining scale information of small targets; determining an IoU threshold for small targets based on the scale information of small targets; and if the IoU of the pseudo-boundary and the ground truth bounding box is greater than the IoU threshold, then the pseudo-boundary is used as a training sample.

[0010] Optionally, the detection results include the coordinate information of the predicted bounding box and the type of small target within the bounding box. The localization and classification module includes a regression branch, an uncertainty branch, and a classification branch. The regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the localization inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of small target within the bounding box.

[0011] Optionally, the localization classification module is optimized using quality-driven localization loss, including: constructing a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the localization inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution and the localization inaccuracy is used as the variance of the Gaussian distribution; calculating the KL (Kullback-Leibler Divergence) divergence between the predicted Gaussian distribution and the Dirac distribution of the ground truth bounding boxes using an objective function, and using the KL divergence as the quality-driven localization loss; and minimizing the quality-driven loss to optimize the localization classification module.

[0012] Optionally, before training the optimized convolutional neural network using the training dataset, the method further includes: segmenting the traffic scene image to obtain segmented images; scaling the segmented images to a preset size and randomly flipping the scaled images to obtain the final image; generating a training dataset based on the final image and the detection results of small targets in the traffic scene image; and training the optimized convolutional neural network using the training dataset.

[0013] A second aspect of this application provides a device for detecting small targets in traffic scene images, comprising: an acquisition module for acquiring traffic scene images and detection results of small targets in the traffic scene images, and generating a training dataset based on the traffic scene images and the detection results of small targets in the traffic scene images; a construction module for constructing a convolutional neural network based on a feature extraction module, a feature fusion module, a sample allocation module, and a localization and classification module, and optimizing the sample allocation module and the localization and classification module of the convolutional neural network to obtain an optimized convolutional neural network; and a detection module for training the optimized convolutional neural network using the training dataset, updating the network parameters of the optimized convolutional neural network based on gradient backpropagation until a preset training stopping condition is met, and detecting small targets in the traffic scene images using the trained convolutional neural network.

[0014] Optionally, the building module is further used to: optimize the sample allocation output by the sample allocation module using a scale-adaptive method; and optimize the localization and classification module using a quality-driven localization loss.

[0015] Optionally, the construction module is further used to: obtain the true bounding boxes of small targets; calculate the Euclidean distance between preset uniform anchor points and the geometric center of the true bounding boxes; select the Euclidean distance as the anchor points within the target range, generate multiple pseudo boxes based on the anchor points within the target range, use the pseudo boxes as candidate samples, and generate a candidate sample set; calculate the intersection-exclusion ratio (IoU) between each pseudo box and the true bounding box in the candidate sample set, and allocate training samples based on the IoU.

[0016] Optionally, the construction module is further used to: obtain the scale information of the small target; determine the intersection ratio threshold of the small target based on the scale information of the small target; if the intersection ratio of the pseudo box and the real labeled box is greater than the intersection ratio threshold, then the pseudo box is used as a training sample.

[0017] Optionally, the detection results include the coordinate information of the predicted bounding box and the type of small target within the bounding box. The localization and classification module includes a regression branch, an uncertainty branch, and a classification branch. The regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the localization inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of small target within the bounding box.

[0018] Optionally, the construction module is further used to: construct a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the localization inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution and the localization inaccuracy is used as the variance of the Gaussian distribution; calculate the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true bounding box using an objective function, and use the KL divergence as the quality-driven localization loss; minimize the quality-driven loss to optimize the localization and classification module.

[0019] In this embodiment of the application, it further includes: a segmentation module, used to segment the traffic scene image to obtain a segmented image before training the optimized convolutional neural network using the training dataset; to scale the segmented image to a preset size and to randomly flip the scaled image to obtain the final image; to generate a training dataset based on the final image and the detection results of small targets in the traffic scene image; and to train the optimized convolutional neural network using the training dataset.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to perform the method for detecting small targets in traffic scene images as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program or instructions stored thereon, which are executed by a processor to perform the method for detecting small targets in traffic scene images as described above.

[0022] Therefore, this application has at least the following beneficial effects:

[0023] This application embodiment can acquire traffic scene images and detection results to generate a training dataset, construct and optimize a convolutional neural network (CNN), and use the training dataset to train the optimized CNN to update its parameters. The trained CNN is then used to detect small targets in the traffic scene images. By optimizing the sample allocation module of the CNN, the quantity and quality of small target samples during the CNN training process are increased, enhancing the CNN's recognition ability. Furthermore, by optimizing the localization and classification module of the CNN, more accurate localization of small targets is achieved, further improving the detection effect. Thus, it solves the technical problems of small target sample imbalance and poor localization ability in related technologies.

[0024] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0025] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0026] Figure 1 This is a flowchart of a method for detecting small targets in traffic scene images according to an embodiment of this application;

[0027] Figure 2 This is a structural diagram of the optimized convolutional neural network provided according to an embodiment of this application;

[0028] Figure 3 This is a schematic diagram illustrating the improvement effect on the SODA-D test set of the traffic scene dataset according to the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of a device for detecting small targets in traffic scene images according to an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0032] The following description, with reference to the accompanying drawings, outlines a method, apparatus, device, and storage medium for detecting small targets in traffic scene images according to embodiments of this application. Addressing the aforementioned background art, current methods for identifying small targets in traffic scenes often employ deep neural networks, focusing on optimizing limited feature information, multi-scale feature representation, and single-factor sample design issues, without considering the fundamental problems of imbalanced small target samples and difficulty in localization. This application provides a method for detecting small targets in traffic scene images. In this method, traffic scene images and detection results are acquired to generate a training dataset. A sample allocation module and a localization / classification module of a convolutional neural network are constructed and optimized. The optimized convolutional neural network is trained using the training dataset to update its parameters. The trained convolutional neural network is then used to detect small targets in the traffic scene images. This solves the problems of imbalanced small target samples and poor localization capabilities in related technologies.

[0033] Specifically, Figure 1 This is a flowchart illustrating a method for detecting small targets in traffic scene images provided in an embodiment of this application.

[0034] like Figure 1 As shown, the method for detecting small targets in traffic scene images includes the following steps:

[0035] In step S101, traffic scene images and the detection results of small targets in the traffic scene images are obtained, and a training dataset is generated based on the traffic scene images and the detection results of small targets in the traffic scene images.

[0036] Small targets are those that are relatively small in size, such as pedestrians and traffic lights. The detection results include the coordinates of the predicted bounding box and the type of small targets within the bounding box, including pedestrians, riders, bicycles, motorcycles, cars, traffic signs, traffic lights, traffic cameras, and warning cones.

[0037] It is understood that the embodiments of this application can acquire traffic scene images and the detection results of small targets in the traffic scene images, and generate a training dataset based on the traffic scene images and the detection results of small targets, so as to train a convolutional neural network based on the training dataset in the future.

[0038] In step S102, a convolutional neural network is constructed based on the feature extraction module, feature fusion module, sample allocation module, and localization and classification module. The sample allocation module and localization and classification module of the convolutional neural network are then optimized to obtain an optimized convolutional neural network.

[0039] The feature extraction module can be ResNet-50, and the feature fusion module can be FPN.

[0040] It is understood that the embodiments of this application can construct a convolutional neural network. A network structure consisting of a feature extraction module, a feature fusion module, a sample allocation module, a localization and classification module can be constructed based on Reppoints. The sample allocation module and the localization and classification module in the convolutional neural network are optimized to obtain an optimized convolutional neural network, so that the optimized convolutional neural network can better identify small targets. The specific optimization method is described in the following embodiments.

[0041] Convolutional neural networks are divided into a feature extraction module, a feature fusion module, a sample allocation module, and a localization and classification module. The feature extraction module is used to extract target information from the image, the feature fusion module is used to realize information interaction between multi-scale features, the sample allocation module is used to allocate training samples to the ground truth bounding boxes, and the localization and classification module is used to calculate the coordinate information and classification information of the predicted bounding boxes.

[0042] In this embodiment of the application, the sample allocation module and the localization and classification module of the convolutional neural network are optimized to obtain an optimized convolutional neural network, including: optimizing the sample allocation output by the sample allocation module using a scale adaptive method; and optimizing the localization and classification module using a quality-driven localization loss.

[0043] It is understood that the embodiments of this application may use a scale-adaptive method to optimize the sample classification output by the sample allocation module and use quality-driven localization loss to optimize the localization and classification module.

[0044] Because the current sample allocation scheme results in a low quantity and quality of samples corresponding to small targets, this application introduces an adaptive sample allocation scheme that fully considers the characteristic that small targets are more sensitive to the discrimination criteria, thereby improving the quantity and quality of samples corresponding to small targets during training. A quality-driven localization loss is introduced in the localization and classification module, that is, an extra branch is added to the relevant part of the localization module to estimate the localization uncertainty of small targets, thereby achieving more accurate localization of small targets. The specific optimization method is as follows.

[0045] In this embodiment of the application, the sample allocation output by the sample allocation module is optimized using a scale-adaptive method, including: obtaining the true bounding boxes of small targets; calculating the Euclidean distance between preset uniform anchor points and the geometric center of the true bounding boxes; selecting the Euclidean distance as the anchor points within the target range, generating multiple pseudo-boxes based on the anchor points within the target range, using the pseudo-boxes as candidate samples, and generating a candidate sample set; calculating the intersection-exclusion ratio (IoU) between each pseudo-box and the true bounding box in the candidate sample set, and allocating training samples based on the IoU.

[0046] The target range can be the first n anchor points with the smallest Euclidean distance, such as the first 4 anchor points.

[0047] It is understood that the embodiments of this application can obtain the true bounding box of a small target and calculate the Euclidean distance between the preset uniform anchor point and the geometric center of the true bounding box; select anchor points within the target range with Euclidean distance for the true bounding box, generate multiple pseudo boxes based on these anchor points, use the pseudo boxes as candidate samples, generate a candidate sample set, calculate the IoU (Intersection over Union) between each pseudo box and the true bounding box in the candidate sample set, and allocate training samples based on the IoU.

[0048] In this embodiment of the application, the allocation of training samples based on the intersection-over-union ratio (IoU) includes: obtaining the scale information of the small target; determining the IoU threshold of the small target based on the scale information of the small target; and if the IoU of the pseudo-box and the real labeled box is greater than the IoU threshold, then the pseudo-box is used as a training sample.

[0049] It is understood that the embodiments of this application can determine the IoU threshold of the small target based on the scale information of the small target. If the IoU between the pseudo box and the real labeled box is greater than the IoU threshold, the pseudo box is used as a training sample.

[0050] Specifically, this application introduces a scale-adaptive sample allocation scheme for the sample allocation module. It fully considers the greater sensitivity of small targets to the discrimination criteria, designs a serialized discrimination criterion, initially expands the samples based on the Euclidean distance discrimination criterion, generates a series of pseudo-boxes based on the expanded samples, further forms a discrimination criterion for the IoU between the pseudo-boxes and the ground truth boxes, and establishes a serialized criterion. Scale-oriented discrimination is used, and different discrimination thresholds are set for small targets and other scale targets when performing IoU discrimination after the pseudo-boxes are formed. A scale-adaptive threshold is also used; when performing IoU threshold discrimination on small targets, the corresponding IoU threshold is dynamically adjusted according to the target size to achieve a more reasonable allocation of small target samples.

[0051] In this embodiment, the localization and classification module includes a regression branch, an uncertainty branch, and a classification branch. The regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the localization inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of small targets within the bounding box.

[0052] Compared to existing technologies, the localization and classification module of this application includes three branches. In addition to the original regression branch and classification branch, an uncertainty branch is introduced to estimate the localization inaccuracy of the coordinate information of the predicted bounding box, thereby enabling more accurate localization of small targets.

[0053] In this embodiment, the optimization of the localization classification module using quality-driven localization loss includes: constructing a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the localization inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution and the localization inaccuracy is used as the variance of the Gaussian distribution; calculating the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true bounding box using an objective function, and using the KL divergence as the quality-driven localization loss; and minimizing the quality-driven loss to optimize the localization classification module.

[0054] The objective function can be an NLL (Negative Log Likelihood) function.

[0055] It is understood that the embodiments of this application can construct a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the localization inaccuracy. The coordinate information is used as the mean of the Gaussian distribution, and the localization inaccuracy is used as the variance of the Gaussian distribution. The KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true bounding box is calculated using an objective function. The KL divergence is used as the quality-driven localization loss. Minimizing the KL divergence is equivalent to minimizing the quality-driven localization loss, thereby achieving more accurate localization of small targets.

[0056] Specifically, in this embodiment, a quality-driven positioning loss is introduced into the positioning classification module. An additional branch (regression branch) is added to the relevant part of the positioning module to estimate the positioning uncertainty of small targets. The predicted bounding box is explicitly modeled as a multivariate Gaussian distribution, and the negative log-likelihood function is used to measure the KL divergence between the predicted distribution and the true value box distribution. This is used as the loss function to supervise the positioning uncertainty, thereby achieving more accurate positioning of small targets.

[0057] In step S103, the optimized convolutional neural network is trained using the training dataset, and the network parameters of the optimized convolutional neural network are updated based on gradient backpropagation until the preset training stopping condition is met. The trained convolutional neural network is then used to detect small targets in traffic scene images.

[0058] It is understood that the embodiments of this application can use the training dataset to train the optimized convolutional neural network, and update the network parameters of the optimized convolutional neural network based on gradient backpropagation until the preset training stopping condition is met, and use the trained convolutional neural network to detect small targets in traffic scene images, thereby achieving more accurate small target localization.

[0059] In this embodiment of the application, before training the optimized convolutional neural network using the training dataset, the method further includes: segmenting the traffic scene image to obtain the segmented image; scaling the segmented image to a preset size and randomly flipping the scaled image to obtain the final image; generating a training dataset based on the final image and the detection results of small targets in the traffic scene image; and training the optimized convolutional neural network using the training dataset.

[0060] The preset size can be set according to specific circumstances, such as 1200*1200.

[0061] It is understood that, before training the optimized convolutional neural network using the training dataset, the traffic images in this embodiment are processed. Preprocessing not only makes the neural network easier to process but also fully preserves the effective information of small targets. Specifically, the traffic scene images are segmented: offline slicing processing is performed, dividing the original high-resolution images in the dataset into a series of 800*800 sub-images in both horizontal and vertical directions with a stride of 650 to adapt to hardware processing capabilities; data augmentation processing includes image scaling and random flipping, scaling the segmented sub-images to 1200*1200 to retain as much of the small target information as possible, and then randomly flipping the scaled images to form a richer data representation. Finally, a training dataset is generated based on the processed images and the detection results of small targets, and the optimized convolutional neural network is trained using the training dataset.

[0062] In summary, specifically, the method for detecting small targets in traffic scene images according to embodiments of this application includes the following steps:

[0063] S1: Preprocessing of small target detection dataset in traffic scenes.

[0064] The dataset consists of multiple high-resolution images rich in small object instances, including traffic lights, pedestrians, cars, traffic signs, and other nine categories, all of which are labeled. The dataset also undergoes a series of preprocessing steps to improve processing efficiency and add more sample support.

[0065] S11: Offline slicing processing, which divides the original high-resolution images in the dataset into a series of 800*800 sub-images in both the horizontal and vertical directions with a step size of 650, to adapt to the hardware processing capabilities.

[0066] S12: Data augmentation processing, including image scaling and random flipping, scales the segmented sub-images to 1200*1200 size, retaining as much small target information as possible, and then randomly flips the scaled image to form a richer data representation, which is then input into the convolutional neural network for training.

[0067] Preprocessing the original training dataset not only makes it easier for neural networks to process, but also fully preserves the effective information of small targets.

[0068] S2: Convolutional Neural Network Construction. Based on Reppoints, a network structure is constructed consisting of a feature extraction module, a feature fusion module, a sample allocation module, and a localization and classification module. These modules are used to extract target information from the image, realize information interaction between multi-scale features, allocate training samples for ground truth bounding boxes, calculate the coordinate information of predicted bounding boxes, and generate classification results. The specific structure of the convolutional neural network is as follows: Figure 2 As shown.

[0069] Preferably, the feature extraction module is ResNet-50, the feature fusion module that performs information interaction between feature layers is FPN, a progressive semantic point regression module based on L1 Loss is introduced, a classification module based on focus loss is introduced, and a sample allocation module provides samples for the classification and regression process is introduced.

[0070] S3: Sample Allocation Optimization Scheme Design: To address the issue that the current sample allocation scheme results in low quantity and quality of samples corresponding to small targets, a scale-adaptive sample allocation scheme is introduced. This scheme fully considers the characteristic that small targets are more sensitive to the discrimination criteria, designs a sequential discrimination criterion, and then combines Euclidean distance and IoU based on pseudo-boxes for sample allocation after sample expansion, thereby improving the quantity and quality of samples corresponding to small targets during training.

[0071] S4: Introduce a quality-driven localization loss. Add an extra branch to the relevant part of the localization module to estimate the localization uncertainty of small targets. Explicitly model the predicted bounding box as a multivariate Gaussian distribution and use the negative log-likelihood function to measure the KL divergence between the predicted distribution and the ground truth box distribution. Use this as the loss function to supervise the localization uncertainty and achieve more accurate localization of small targets.

[0072] S5: Use the preprocessed data from S1 to train the convolutional neural network optimized by S2 and S3, and update the network parameters through gradient backpropagation using the SGD optimizer.

[0073] S6: Use the network model trained in S5 to perform forward inference on the data in the test set to obtain the actual detection results.

[0074] The trained neural network model is used to detect a given input traffic scene image, outputting the coordinates and category information of potential targets, and the performance of the improved method on the test set is statistically analyzed for each category.

[0075] The S3 mesoscale adaptive sample allocation scheme includes:

[0076] The design of the serialization discrimination criterion first expands the samples based on the Euclidean distance criterion, generates a series of pseudo-boundaries based on the expanded samples, and further forms a discrimination criterion for the IoU between the pseudo-boundaries and the ground truth boxes, thus forming the serialization criterion. Scale-oriented discrimination sets different discrimination thresholds for small targets and other scale targets when performing IoU discrimination after the pseudo-boundaries are formed. Scale-adaptive thresholding dynamically adjusts the corresponding IoU threshold according to the target size when performing IoU threshold discrimination for small targets, in order to achieve a more reasonable allocation of small target samples.

[0077] For a given input image I, the truth box is represented as G. i =(lt i ,rb i ), where lt i rb represents the coordinates of the top-left corner of the truth box. i This represents the coordinates of the bottom right corner. For each truth box G in the image... i This application first finds the n closest anchor points based on Euclidean distance, and obtains the preliminary allocation result A. i These serve as candidate anchor points for the final positive samples, and then pseudo-boxes P are generated based on these candidate anchor points. i By maintaining the same size and aspect ratio as the corresponding truth box, the IoU between the pseudo-box and the truth box can be calculated. In this embodiment, two hyperparameters, thr, are set for small targets and other scale targets respectively. s and thr e Thr is used as the IoU threshold for distinguishing between positive and negative samples.s The specific plan is as follows:

[0078]

[0079] Where τ is a hyperparameter with a default value of 0.1, s represents the area of ​​the truth box corresponding to the sample, and ε is related to the statistical properties of the dataset, representing the definition limit of the dataset for the smallest scale range target.

[0080] The mass-driven positioning loss in S4 includes:

[0081] The regression branch design for positioning uncertainty estimation is based on the assumption that the network output follows a certain probability distribution. The network directly predicts the relevant parameters of the probability distribution, modeling the network prediction result as a multivariate Gaussian distribution. The output value of the original regression branch is associated with the mean of the Gaussian distribution, and the corresponding variance is obtained by introducing an additional regression branch. Decoding the positioning uncertainty helps obtain a more accurate predicted bounding box. This involves decoding the mean based on linear transformation while simultaneously considering the corresponding variance – the process of decoding positioning uncertainty. Finally, the positioning loss design incorporates scale information and sample quality, optimizing the loss function in S4 using the scale information of small targets and the quality of positive samples used in training.

[0082] Let the output of the original regression branch of the network be correlated with the mean of the predicted distribution, and the corresponding variance be obtained through the introduced additional regression branch. Since the representation of the predicted bounding box is generally composed of multiple variables, and considering the characteristics of deep neural networks, it is assumed that each output variable of the regression branch is independent of each other. All variables then satisfy independent multivariate Gaussian distributions and each follows its own distribution. The diagonal covariance matrix is The prediction results corresponding to the newly added regression branch represent the relevant positioning uncertainty.

[0083] Most current object detection methods encode the predicted offsets of the localization branch using different encoding schemes. The encoding process provides a learning target for the network's regression branch output. Correspondingly, the decoding process transforms the network output values ​​into predicted bounding boxes. When modeling the predicted bounding box coordinates using a Gaussian distribution, the encoding and decoding schemes should also be considered. In the baseline method Reppoints of this application, for any position (x, y) on the feature map, the regression branch predicts a series of offsets (Δx) in the horizontal and vertical directions. i ,Δy i This forms a set containing 9 prediction points. The coordinates of the predicted point are obtained by the following formula:

[0084] x i =x + Δxi ,y i =y+Δy i ;

[0085] The final predicted bounding box is also represented by the top-left and bottom-right corner points (x1, y1, x2, y2), and is composed of the predicted point set. The predicted offset, obtained through a decoding process using a certain transformation function, is equivalent to the mean of the multivariate Gaussian distribution after modeling. Therefore, the encoding and decoding process for the offset is the same as the encoding and decoding process for the mean. Naturally, the encoding and decoding process for localization uncertainty also needs to be considered. Specifically, this application, based on the baseline method, adds a 3×3 convolutional output channel with a 4-dimensional feature map to represent the variance corresponding to each mean. Supervised training for localization uncertainty is equivalent to estimating the network parameters that minimize the log-likelihood function, as shown in the following equation:

[0086]

[0087] Where Θ represents the set of learnable network model parameters, P D (x) represents the Dirac distribution followed by the truth box, P Θ (x) represents the Gaussian distribution that the network model predicts the bounding box to follow. Furthermore, the loss related to the positioning uncertainty can be... Notation:

[0088]

[0089] Where N represents the sample size, g i b represents the coordinates of the truth box. i This represents the mean value after decoding, i.e., the coordinates of the predicted bounding box after decoding. Indicates b i The corresponding variance after decoding, i.e., the positioning uncertainty;

[0090] Furthermore, considering the use of the scale information of small targets and the quality of the positive samples participating in training to optimize this loss function, the final quality-driven localization loss is denoted as:

[0091]

[0092] In the above formula, K is the hyperparameter set by this application for the stable training process, and IoU i s represents the intersection-union ratio (IoU) of the i-th positive sample and its corresponding truth box. i Let be the area of ​​the truth box corresponding to the i-th positive sample.

[0093] The overall loss function in S5 consists of the following components:

[0094] L = Lcts +L reg1 +L reg2 +γ·L QLL

[0095] Among them, L cls L represents the classification loss of the classification branch that runs parallel to the second regression stage. reg1 and L reg2 L represents the localization loss in the first regression stage and the second regression stage, respectively. QLL This represents QLL (Quality-driven Localization Loss).

[0096] This application method includes two regression stages, in the above formula, L cls L represents the classification loss of the classification branch that runs parallel to the second regression stage. reg1 and L reg2 Let L represent the localization loss in the first regression stage and the second regression stage, respectively, both obtained through L... sl1 That is, it is calculated using Smooth L1 Loss.

[0097] For a batch participating in training, let N be the total number of positive samples in the first regression phase. pos1 The total number of positive samples in the second phase is denoted as N. pos2 Q x,y and P x,y G represents the total number of samples in the first and second regression phases, respectively. x,y This represents the set of all truth boxes in this batch; the outputs of the classification branch and the regression branch are represented as follows: as well as and These represent the category label and the coordinates of the truth box after being encoded by the network detection head, respectively.

[0098] The following specific embodiment illustrates the method for detecting small targets in traffic scene images according to this application. The above steps are executed on an NVIDIA GeForce RTX-4090 graphics card with 24GB of video memory. The system environment is Ubuntu 18.04, and the software environment is Python 3.7.13 + PyTorch 1.10.0 + CUDA 11.3. The SGD optimization method is adopted, the batch size is fixed at 16, the total number of training rounds is 12, and the initial learning rate is set to 0.01.

[0099] The dataset is the SODA-D dataset, which consists of multiple high-resolution images rich in small object instances. It includes nine categories such as traffic lights, pedestrians, cars, and traffic signs, all of which are labeled. Detailed information is shown in Table 1. At the same time, a series of preprocessing steps were performed on the dataset to improve processing efficiency and add more sample support. Table 1 is a statistical information table of the number of labels in the traffic scene dataset.

[0100] Table 1

[0101] Target Category Number of instances pedestrians 35928 rider 4636 bicycle 2560 Motorcycle 3896 Car 69197 Traffic sign 85905 Traffic lights 62729 Traffic camera 7636 Warning cone 5946

[0102] This application qualitatively compares the target detection method with Reppoints from the perspectives of accuracy and efficiency. It uses two objective evaluation metrics, AP and FPS, to compare and analyze the detection performance of this application and the existing Reppoints method on the test set. Table 2 shows the overall detection accuracy (AP) and the detection accuracy (AP) of the two methods on targets at various scales. eS AP rS AP gS Compared to existing methods, the method in this application shows significant improvements in all accuracy metrics without a noticeable decrease in processing speed. The processing speed of this application is 41.4 FPS, with an average processing time of 24.2 ms per image. It can be seen that this application can effectively improve the detection accuracy of small targets without sacrificing detection efficiency. Table 2 compares the small target detection performance of this application with that of Reppoints.

[0103] Table 2

[0104] Detection methods AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AP eS ]]> <![CDATA[AP rS ]]> <![CDATA[AP gS ]]> <![CDATA[AP N ]]> FPS Reppoints 28.0 55.6 24.7 10.1 23.8 35.1 45.3 43.1 This application 29.2 57.5 25.2 10.7 25.0 36.3 46.2 41.4

[0105] To more intuitively demonstrate the improvement effect of this application on small object detection, a visual analysis of the improvement effects of the benchmark method and the method of this application in various categories was conducted, such as... Figure 3 As shown, the proposed method offers limited improvement for extremely small targets such as traffic lights and traffic cameras. Significant improvements are observed for relatively larger targets such as pedestrians, cars, and traffic signs, with the most significant improvements seen for larger target categories such as bicycles, riders, warning cones, and motorcycles. Overall, the proposed method effectively improves the detection accuracy for all categories in the SODA-D dataset.

[0106] In summary, this application addresses the imbalance of small target samples in current traffic scene image target detection datasets. From the perspective of sample allocation, it designs a scale-adaptive sample allocation scheme, fully considering the greater sensitivity of small targets to evaluation criteria during sample allocation. By combining serialization criteria, scale-oriented screening, and dynamic thresholds, it comprehensively improves the quantity and quality of small target samples, thereby increasing the proportion and quality of small targets in the training process of convolutional neural networks. To address the difficulty of small target localization, it incorporates localization uncertainty information through a multivariate Gaussian distribution, designs a quality-driven localization loss, and jointly optimizes this loss function using sample quality and target scale information. This achieves more accurate localization of small targets, effectively improving the accuracy of small target detection without significantly reducing detection efficiency, thereby enhancing road traffic safety.

[0107] The method for detecting small targets in traffic scene images proposed in this application can acquire traffic scene images and detection results to generate a training dataset, construct and optimize a convolutional neural network, and use the training dataset to train the optimized convolutional neural network to update the parameters of the convolutional neural network. The trained convolutional neural network is then used to detect small targets in traffic scene images. By optimizing the sample allocation module of the convolutional neural network, the number and quality of small targets in the training process of the convolutional neural network are improved, thereby enhancing the recognition ability of the convolutional neural network. By optimizing the localization and classification module of the convolutional neural network, more accurate localization of small targets is achieved, further improving the detection effect of small targets.

[0108] Next, referring to the accompanying drawings, a device for detecting small targets in traffic scene images according to an embodiment of this application is described.

[0109] Figure 4 This is a block diagram of a device for detecting small targets in traffic scene images according to an embodiment of this application.

[0110] like Figure 4 As shown, the detection device 10 for small targets in traffic scene images includes: an acquisition module 100, a construction module 200, and a detection module 300.

[0111] The acquisition module 100 is used to acquire traffic scene images and the detection results of small targets in the traffic scene images, and generate a training dataset based on the traffic scene images and the detection results of small targets in the traffic scene images; the construction module 200 is used to construct a convolutional neural network based on the feature extraction module, feature fusion module, sample allocation module and localization and classification module, and optimize the sample allocation module and localization and classification module of the convolutional neural network to obtain an optimized convolutional neural network; the detection module 300 is used to train the optimized convolutional neural network using the training dataset, update the network parameters of the optimized convolutional neural network based on gradient backpropagation until the preset training stopping condition is met, and use the trained convolutional neural network to detect small targets in the traffic scene images.

[0112] In this embodiment, the construction module 200 is further configured to: optimize the sample allocation output by the sample allocation module using a scale-adaptive method; and optimize the localization and classification module using a quality-driven localization loss.

[0113] In this embodiment of the application, the construction module 200 is further configured to: obtain the true bounding box of the small target; calculate the Euclidean distance between the preset uniform anchor point and the geometric center of the true bounding box; select the Euclidean distance as the anchor point within the target range, generate multiple pseudo boxes based on the anchor points within the target range, use the pseudo boxes as candidate samples, and generate a candidate sample set; calculate the intersection ratio between each pseudo box and the true bounding box in the candidate sample set, and allocate training samples based on the intersection ratio.

[0114] In this embodiment of the application, the construction module 200 is further used to: obtain the scale information of the small target; determine the intersection ratio threshold of the small target based on the scale information of the small target; if the intersection ratio of the pseudo box and the real labeled box is greater than the intersection ratio threshold, then the pseudo box is used as a training sample.

[0115] In this embodiment, the detection result includes the coordinate information of the predicted bounding box and the type of small target within the bounding box. The localization and classification module includes a regression branch, an uncertainty branch, and a classification branch. The regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the localization inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of small target within the bounding box.

[0116] In this embodiment, the construction module 300 is further configured to: construct a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the localization inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution and the localization inaccuracy is used as the variance of the Gaussian distribution; calculate the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true bounding box using an objective function, and use the KL divergence as the quality-driven localization loss; and minimize the quality-driven loss to optimize the localization classification module.

[0117] In this embodiment of the application, the apparatus 10 further includes a segmentation module.

[0118] The segmentation module is used to segment traffic scene images to obtain segmented images before training the optimized convolutional neural network using the training dataset; scale the segmented images to a preset size and randomly flip the scaled images to obtain the final images; generate a training dataset based on the final images and the detection results of small targets in the traffic scene images; and train the optimized convolutional neural network using the training dataset.

[0119] It should be noted that the explanation of the above-described method for detecting small targets in traffic scene images also applies to the device for detecting small targets in traffic scene images in this embodiment, and will not be repeated here.

[0120] The device for detecting small targets in traffic scene images according to the embodiments of this application can acquire traffic scene images and detection results to generate a training dataset, construct and optimize a convolutional neural network, and use the training dataset to train the optimized convolutional neural network to update the parameters of the convolutional neural network. The trained convolutional neural network is then used to detect small targets in traffic scene images. By optimizing the sample allocation module of the convolutional neural network, the number and quality of small targets in the training process of the convolutional neural network are improved, thereby enhancing the recognition ability of the convolutional neural network. By optimizing the localization and classification module of the convolutional neural network, more accurate localization of small targets is achieved, further improving the detection effect of small targets.

[0121] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0122] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0123] When the processor 502 executes the program, it implements the method for detecting small targets in traffic scene images provided in the above embodiments.

[0124] Furthermore, electronic devices also include:

[0125] Communication interface 503 is used for communication between memory 501 and processor 502.

[0126] The memory 501 is used to store computer programs that can run on the processor 502.

[0127] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0128] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0129] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0130] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0131] This application also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the above-described method for detecting small targets in traffic scene images.

[0132] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0134] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0135] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0136] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

Claims

1. A method for detecting small targets in traffic scene images, characterized in that, Includes the following steps: Acquire traffic scene images and the detection results of small targets in the traffic scene images, and generate a training dataset based on the traffic scene images and the detection results of small targets in the traffic scene images, wherein the detection results include the coordinate information of the predicted bounding boxes and the type of small targets within the bounding boxes; A convolutional neural network (CNN) is constructed based on a feature extraction module, a feature fusion module, a sample allocation module, and a localization and classification module. The sample allocation and localization and classification modules of the CNN are then optimized to obtain an optimized CNN. The localization and classification module includes a regression branch, an uncertainty branch, and a classification branch. The regression branch outputs the coordinate information of the predicted bounding box; the uncertainty branch estimates the localization inaccuracy of the predicted bounding box coordinate information; and the classification branch predicts the type of small targets within the bounding box. Optimizing the sample allocation and localization and classification modules of the CNN to obtain the optimized CNN includes: optimizing the sample allocation output by the sample allocation module using a scale-adaptive method; and optimizing the localization and classification module using a quality-driven localization loss. The optimization of the sample allocation output by the sample allocation module using a scale-adaptive method includes: The process involves: obtaining the ground truth bounding box of the small target; calculating the Euclidean distance between preset uniform anchor points and the geometric center of the ground truth bounding box; selecting the Euclidean distance as anchor points within the target range; generating multiple pseudo-bounding boxes based on the anchor points within the target range; using the pseudo-bounding boxes as candidate samples and generating a candidate sample set; calculating the intersection-union ratio (IUU) between each pseudo-bounding box and the ground truth bounding box in the candidate sample set; and allocating training samples based on the IUUU. The optimization of the localization classification module using quality-driven localization loss includes: constructing a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the localization inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution and the localization inaccuracy is used as the variance of the Gaussian distribution; calculating the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the ground truth bounding box using an objective function; using the KL divergence as the quality-driven localization loss; and minimizing the quality-driven loss to optimize the localization classification module. The optimized convolutional neural network is trained using the training dataset, and the network parameters of the optimized convolutional neural network are updated based on gradient backpropagation until a preset training stopping condition is met. The trained convolutional neural network is then used to detect small targets in traffic scene images.

2. The method for detecting small targets in traffic scene images according to claim 1, characterized in that, The allocation of training samples based on the intersection ratio includes: Obtain the scale information of the small target; The intersection ratio threshold of the small target is determined based on the scale information of the small target; If the intersection ratio of the pseudo-boundary and the real labeled bounding box is greater than the intersection ratio threshold, then the pseudo-boundary is used as the training sample.

3. The method for detecting small targets in traffic scene images according to claim 1, characterized in that, Before training the optimized convolutional neural network using the training dataset, the method further includes: The traffic scene image is segmented to obtain the segmented image; The segmented image is scaled to a preset size, and the scaled image is randomly flipped to obtain the final image; A training dataset is generated based on the detection results of small targets in the final image and the traffic scene image. The optimized convolutional neural network is trained using the training dataset.

4. A device for detecting small targets in traffic scene images, characterized in that, include: The acquisition module is used to acquire traffic scene images and the detection results of small targets in the traffic scene images, and generate a training dataset based on the traffic scene images and the detection results of small targets in the traffic scene images. The detection results include the coordinate information of the predicted bounding boxes and the type of small targets within the bounding boxes. A construction module is used to build a convolutional neural network based on a feature extraction module, a feature fusion module, a sample allocation module, and a localization and classification module. The sample allocation and localization and classification modules of the convolutional neural network are then optimized to obtain an optimized convolutional neural network. The localization and classification module includes a regression branch, an uncertainty branch, and a classification branch. The regression branch outputs the coordinate information of the predicted bounding box; the uncertainty branch estimates the localization inaccuracy of the predicted bounding box coordinate information; and the classification branch predicts the type of small targets within the bounding box. The optimization of the sample allocation and localization and classification modules of the convolutional neural network to obtain the optimized convolutional neural network includes: optimizing the sample allocation output by the sample allocation module using a scale-adaptive method; optimizing the localization and classification module using a quality-driven localization loss; and optimizing the sample allocation output by the sample allocation module using a scale-adaptive method. The process includes: acquiring the ground truth bounding boxes of the small target; calculating the Euclidean distance between preset uniform anchor points and the geometric center of the ground truth bounding boxes; selecting the Euclidean distance as anchor points within the target range, generating multiple pseudo-bounding boxes based on the anchor points within the target range, using the pseudo-bounding boxes as candidate samples, and generating a candidate sample set; calculating the intersection-union ratio (IUU) between each pseudo-bounding box and the ground truth bounding box in the candidate sample set, and allocating training samples based on the IUUU; the optimization of the localization classification module using quality-driven localization loss includes: constructing a predicted Gaussian distribution based on the coordinate information of the predicted bounding boxes and the localization inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution, and the localization inaccuracy is used as the variance of the Gaussian distribution; calculating the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the ground truth bounding boxes using an objective function, and using the KL divergence as the quality-driven localization loss; minimizing the quality-driven loss to optimize the localization classification module. The detection module is used to train the optimized convolutional neural network using the training dataset, update the network parameters of the optimized convolutional neural network based on gradient backpropagation until a preset training stopping condition is met, and use the trained convolutional neural network to detect small targets in traffic scene images.

5. An electronic device, characterized in that, include: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for detecting small targets in traffic scene images as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by a processor to implement the method for detecting small targets in traffic scene images as described in any one of claims 1-3.

Images