Traffic scene image small target detection method and device, equipment and storage medium

Abstract

The invention relates to the technical field of deep learning, in particular to a traffic scene image small target detection method and device, equipment and a storage medium, and the method comprises the steps: obtaining a traffic scene image and a detection result of a small target in the traffic scene image, generating a training data set according to the traffic scene image and the detection result of the small target in the traffic scene image; constructing a convolutional neural network based on the feature extraction module, the feature fusion module, the sample distribution module and the positioning classification module, and optimizing the sample distribution module and the positioning classification module of the convolutional neural network to obtain an optimized convolutional neural network; and training the optimized convolutional neural network by using the training data set, updating network parameters of the optimized convolutional neural network based on gradient back propagation until a preset training stop condition is met, and detecting a small target of the traffic scene image by using the trained convolutional neural network. Therefore, the problems of small target sample imbalance, poor positioning capability and the like in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method, device, equipment and storage medium for detecting small targets in traffic scene images. Background Art

[0002] Small target detection is a challenging problem in the target detection branch task. Its purpose is to accurately identify and locate small-sized foreground targets with less information in the scene, and detect small targets in traffic scenes. It has broad application prospects and important practical significance in the fields of autonomous driving technology and road traffic management.

[0003] At present, most methods for identifying small targets in traffic scenes use deep neural networks, which optimize single problems such as limited feature information, multi-scale feature expression, and sample scheme design, but do not consider essential problems such as the imbalance of small target samples and the difficulty of positioning.

[0004] Specifically, sample imbalance will reduce the contribution of small targets to the training process of the neural network model, and inaccurate positioning is caused by the fact that small targets are more sensitive to disturbances than targets of other scales, which is reflected in the high uncertainty of the coordinate positioning of the small target bounding box. In related technologies, the detection of small targets in traffic scenes generally enriches the information representation of small targets from the aspects of multi-scale expression and context information utilization, but does not fundamentally solve the above problems. Summary of the invention

[0005] The present application provides a method, device, equipment and storage medium for detecting small targets in traffic scene images to solve the problems of imbalance of small target samples and poor positioning capability in related technologies.

[0006] The first aspect of the present application provides a method for detecting small targets in traffic scene images, including the following steps: obtaining detection results of traffic scene images and small targets in the traffic scene images, and generating a training data set based on the detection results of traffic scene images and small targets in the traffic scene images; constructing a convolutional neural network based on a feature extraction module, a feature fusion module, a sample allocation module and a positioning classification module, and optimizing the sample allocation module and the positioning classification module of the convolutional neural network to obtain an optimized convolutional neural network; training the optimized convolutional neural network using the training data set, updating the network parameters of the optimized convolutional neural network based on gradient back propagation until a preset training stop condition is met, and using the trained convolutional neural network to detect small targets in traffic scene images.

[0007] Optionally, the sample allocation module and the positioning classification module of the convolutional neural network are optimized to obtain an optimized convolutional neural network, including: optimizing the sample allocation output by the sample allocation module using a scale adaptive method; optimizing the positioning classification module using a quality-driven positioning loss.

[0008] Optionally, a scale-adaptive method is used to optimize the sample allocation output by the sample allocation module, including: obtaining a true annotation box of a small target; calculating the Euclidean distance between a preset uniform anchor point and the geometric center of the true annotation box; selecting the Euclidean distance as an anchor point within the target range, generating multiple pseudo boxes based on the anchor points within the target range, using the pseudo boxes as candidate samples, and generating a candidate sample set; calculating the union-intersection ratio of each pseudo box in the candidate sample set to the true annotation box, and allocating training samples based on the union-intersection ratio.

[0009] Optionally, allocating training samples based on the intersection-and-parallel ratio includes: obtaining scale information of the small target; determining an intersection-and-parallel ratio threshold of the small target based on the scale information of the small target; if the intersection-and-parallel ratio of the pseudo box to the real labeled box is greater than the intersection-and-parallel ratio threshold, using the pseudo box as a training sample.

[0010] Optionally, the detection result includes the coordinate information of the predicted bounding box and the type of the small target in the bounding box, and the positioning classification module includes: a regression branch, an uncertainty branch and a classification branch, wherein the regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the positioning inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of the small target in the bounding box.

[0011] Optionally, the quality-driven positioning loss is used to optimize the positioning classification module, including: constructing a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the positioning inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution and the positioning inaccuracy is used as the variance of the Gaussian distribution; using the objective function to calculate the KL (Kullback-Leibler Divergence) divergence between the predicted Gaussian distribution and the Dirac distribution of the true annotation box, and using the KL divergence as the quality-driven positioning loss; minimizing the quality-driven loss to optimize the positioning classification module.

[0012] Optionally, before using the training data set to train the optimized convolutional neural network, it also includes: segmenting the traffic scene image to obtain segmented images; scaling the segmented image to a preset size, and randomly flipping the scaled image to obtain the final image; generating a training data set based on the final image and the detection results of small targets in the traffic scene image; and using the training data set to train the optimized convolutional neural network.

[0013] The second aspect of the present application provides a detection device for small targets in traffic scene images, including: an acquisition module, used to obtain the detection results of traffic scene images and small targets in the traffic scene images, and generate a training data set according to the detection results of traffic scene images and small targets in the traffic scene images; a construction module, used to construct a convolutional neural network based on a feature extraction module, a feature fusion module, a sample allocation module and a positioning classification module, and optimize the sample allocation module and the positioning classification module of the convolutional neural network to obtain an optimized convolutional neural network; a detection module, used to train the optimized convolutional neural network with the training data set, update the network parameters of the optimized convolutional neural network based on gradient back propagation until a preset training stop condition is met, and use the trained convolutional neural network to detect small targets in traffic scene images.

[0014] Optionally, the construction module is further used to: optimize the sample allocation output by the sample allocation module using a scale adaptive method; and optimize the positioning classification module using a quality driven positioning loss.

[0015] Optionally, the construction module is further used to: obtain the true annotation box of the small target; calculate the Euclidean distance between the preset uniform anchor point and the geometric center of the true annotation box; select the Euclidean distance as the anchor point within the target range, generate multiple pseudo boxes based on the anchor points within the target range, use the pseudo boxes as candidate samples, and generate a candidate sample set; calculate the union-intersection ratio of each pseudo box in the candidate sample set to the true annotation box, and allocate training samples based on the union-intersection ratio.

[0016] Optionally, the construction module is further used to: obtain scale information of the small target; determine the intersection-and-union ratio threshold of the small target based on the scale information of the small target; if the intersection-and-union ratio of the pseudo box and the real labeled box is greater than the intersection-and-union ratio threshold, use the pseudo box as a training sample.

[0017] Optionally, the detection result includes the coordinate information of the predicted bounding box and the type of the small target in the bounding box, and the positioning classification module includes: a regression branch, an uncertainty branch and a classification branch, wherein the regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the positioning inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of the small target in the bounding box.

[0018] Optionally, the construction module is further used to: construct a predicted Gaussian distribution based on the coordinate information and positioning accuracy of the predicted bounding box, wherein the coordinate information is used as the mean of the Gaussian distribution and the positioning accuracy is used as the variance of the Gaussian distribution; use the objective function to calculate the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true annotation box, and use the KL divergence as the quality-driven positioning loss; minimize the quality-driven loss to optimize the positioning classification module.

[0019] In an embodiment of the present application, it also includes: a segmentation module, which is used to segment the traffic scene image to obtain a segmented image before using the training data set to train the optimized convolutional neural network; scaling the segmented image to a preset size, and randomly flipping the scaled image to obtain a final image; generating a training data set based on the final image and the detection results of small targets in the traffic scene image; and using the training data set to train the optimized convolutional neural network.

[0020] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform a method for detecting small targets in traffic scene images as described in the above embodiment.

[0021] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program or instruction is stored. The computer program or instruction is executed by a processor to perform a method for detecting small targets in traffic scene images as described in the above embodiment.

[0022] Therefore, this application has at least the following beneficial effects:

[0023] The embodiment of the present application can obtain traffic scene images and detection results to generate training data sets, build and optimize convolutional neural networks, and use the training data sets to train the optimized convolutional neural networks to update the parameters of the convolutional neural networks. The trained convolutional neural networks are used to detect small targets in traffic scene images. By optimizing the sample allocation module of the convolutional neural network, the sample quantity and quality of small targets in the convolutional neural network training process are improved, and the recognition ability of the convolutional neural network is enhanced. By optimizing the positioning and classification module of the convolutional neural network, more accurate positioning of small targets is achieved, and the detection effect of small targets is further improved. Therefore, technical problems such as imbalance of small target samples and poor positioning ability in related technologies are solved.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flowchart of a method for detecting small targets in traffic scene images provided according to an embodiment of the present application;

[0027] Figure 2 A structural diagram of an optimized convolutional neural network provided according to an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the improvement effect on a category-by-category basis on a traffic scene dataset SODA-D test set provided according to an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a small target detection device for a traffic scene image provided according to an embodiment of the present application;

[0030] Figure 5 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0032] The following describes the detection method, device, equipment and storage medium of small targets in traffic scene images of the embodiments of the present application with reference to the accompanying drawings. In view of the fact that the current methods for identifying small targets in traffic scenes mentioned in the background technology center mostly use deep neural networks, and optimize single problems in limited feature information, multi-scale feature expression and sample scheme design, without considering the essential problems such as the imbalance of small target samples and positioning difficulties, the present application provides a method for detecting small targets in traffic scene images, in which a training data set can be obtained to generate a training data set for traffic scene images and detection results, and a sample allocation module and a positioning classification module of a convolutional neural network are constructed and optimized, and the optimized convolutional neural network is trained using the training data set to update the parameters of the convolutional neural network, and the trained convolutional neural network is used to detect small targets in traffic scene images. In this way, the problems of imbalance of small target samples and poor positioning ability in the related technology are solved.

[0033] Specifically, Figure 1 A schematic flow chart of a method for detecting small targets in traffic scene images provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the method for detecting small targets in traffic scene images includes the following steps:

[0035] In step S101, traffic scene images and detection results of small targets in the traffic scene images are obtained, and a training data set is generated according to the traffic scene images and the detection results of small targets in the traffic scene images.

[0036] Among them, small targets are targets of smaller size, such as pedestrians, traffic lights, etc.; the detection results include the coordinate information of the predicted bounding box and the type of small targets in the bounding box, including pedestrians, riders, bicycles, motorcycles, cars, traffic signs, traffic lights, traffic cameras, warning cones, etc.

[0037] It can be understood that the embodiments of the present application can obtain the detection results of traffic scene images and small targets in the traffic scene images, and generate a training data set based on the detection results of the traffic scene images and small targets, so as to subsequently train the convolutional neural network based on the training data set.

[0038] In step S102, a convolutional neural network is constructed based on the feature extraction module, the feature fusion module, the sample allocation module and the positioning classification module, and the sample allocation module and the positioning classification module of the convolutional neural network are optimized to obtain an optimized convolutional neural network.

[0039] Among them, the feature extraction module can be ResNet-50, and the feature fusion module can be FPN.

[0040] It can be understood that the embodiments of the present application can construct a convolutional neural network, and can construct a network structure consisting of a feature extraction module, a feature fusion module, a sample allocation module, a positioning and classification module based on Reppoints, and optimize the sample allocation module and the positioning and classification module in the convolutional neural network to obtain an optimized convolutional neural network, so that the optimized convolutional neural network can better identify small targets. The specific optimization method is described in the following embodiments.

[0041] The convolutional neural network is divided into feature extraction module, feature fusion module, sample allocation module, positioning and classification module. The feature extraction module is used to extract target information in the image, the feature fusion module is used to realize information interaction between multi-scale features, the sample allocation module is used to allocate training samples to the real annotation box, and the positioning and classification module is used to calculate the coordinate information and classification information of the predicted bounding box.

[0042] In an embodiment of the present application, the sample allocation module and the positioning classification module of the convolutional neural network are optimized to obtain an optimized convolutional neural network, including: optimizing the sample allocation output by the sample allocation module using a scale adaptive method; optimizing the positioning classification module using quality-driven positioning loss.

[0043] It is understandable that the embodiments of the present application can use a scale adaptive method to optimize the sample classification output by the sample allocation module, and use a quality-driven positioning loss to optimize the positioning classification module.

[0044] Since the current sample allocation scheme causes the problem of low quantity and quality of samples corresponding to small targets, the embodiment of the present application introduces an adaptive sample allocation scheme, which fully considers the characteristics of small targets that are more sensitive to the discrimination criteria, and improves the quantity and quality of samples corresponding to small targets during training; introduces a quality-driven positioning loss in the positioning classification module, that is, adds an additional branch to the relevant part of the positioning module to estimate the positioning uncertainty of the small target, so as to achieve more accurate small target positioning. The specific optimization method is as follows.

[0045] In an embodiment of the present application, a scale-adaptive method is used to optimize the sample allocation output by a sample allocation module, including: obtaining a true annotation box of a small target; calculating the Euclidean distance between a preset uniform anchor point and the geometric center of the true annotation box; selecting an anchor point within the target range as the Euclidean distance, generating multiple pseudo boxes based on the anchor points within the target range, using the pseudo boxes as candidate samples, and generating a candidate sample set; calculating the union-intersection ratio of each pseudo box in the candidate sample set to the true annotation box, and allocating training samples based on the union-intersection ratio.

[0046] The target range may be the first n anchor points with the smallest Euclidean distance, such as the first 4 anchor points.

[0047] It can be understood that the embodiments of the present application can obtain the true annotation box of the small target, and calculate the Euclidean distance between the preset uniform anchor point and the geometric center of the true annotation box; select anchor points with Euclidean distance within the target range for the true annotation box, generate multiple pseudo boxes based on these anchor points, use the pseudo boxes as candidate samples, and generate a candidate sample set, calculate the IoU (Intersection over Union) of each pseudo box in the candidate sample set and the true annotation box, and allocate training samples based on the IoU ratio.

[0048] In an embodiment of the present application, training samples are allocated based on the intersection-and-parallel ratio, including: obtaining scale information of the small target; determining the intersection-and-parallel ratio threshold of the small target based on the scale information of the small target; if the intersection-and-parallel ratio of the pseudo box to the real labeled box is greater than the intersection-and-parallel ratio threshold, then using the pseudo box as a training sample.

[0049] It can be understood that the embodiment of the present application can determine the IoU threshold of the small target based on the scale information of the small target. If the IoU between the pseudo box and the real labeled box is greater than the IoU threshold, the pseudo box is used as a training sample.

[0050] Specifically, the embodiment of the present application introduces a scale-adaptive sample allocation scheme for the sample allocation module, fully considers the characteristics of small targets that are more sensitive to the discrimination criteria, designs a serialization discrimination criterion, preliminarily expands the samples based on the Euclidean distance discrimination criterion, generates a series of pseudo-frames according to the expanded samples, and further forms the IoU discrimination criterion between the pseudo-frame and the true value frame to form a serialization criterion, scale-oriented discrimination, and sets different discrimination thresholds for small targets and other scale targets when performing IoU discrimination after the pseudo-frame is formed; scale-adaptive threshold, when performing IoU threshold discrimination on small targets, dynamically adjusts the corresponding IoU threshold according to the target size to achieve more reasonable small target sample allocation.

[0051] In an embodiment of the present application, the positioning classification module includes: a regression branch, an uncertainty branch and a classification branch, wherein the regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the positioning inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of small targets within the bounding box.

[0052] Compared with the prior art, the positioning and classification module of the embodiment of the present application includes three branches. In addition to the original regression branch and classification branch, an uncertainty branch is introduced to estimate the positioning inaccuracy of the coordinate information of the predicted bounding box, so as to more accurately locate small targets.

[0053] In an embodiment of the present application, a positioning classification module is optimized using quality-driven positioning loss, including: constructing a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the positioning inaccuracy, wherein the coordinate information is used as the mean of the Gaussian distribution and the positioning inaccuracy is used as the variance of the Gaussian distribution; using an objective function to calculate the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true annotation box, and using the KL divergence as the quality-driven positioning loss; and minimizing the quality-driven loss to optimize the positioning classification module.

[0054] The objective function may be a NLL (Negative Log Likelihood) function.

[0055] It can be understood that the embodiments of the present application can construct a predicted Gaussian distribution based on the coordinate information and positioning accuracy of the predicted bounding box, with the coordinate information serving as the mean of the Gaussian distribution and the positioning accuracy serving as the variance of the Gaussian distribution. The objective function is used to calculate the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true annotation box, and the KL divergence is used as the quality-driven positioning loss to minimize the KL divergence, that is, to minimize the quality-driven positioning loss, thereby achieving more accurate small target positioning.

[0056] Specifically, the embodiment of the present application introduces quality-driven positioning loss in the positioning classification module, adds an additional branch (regression branch) to the relevant part of the positioning module to estimate the positioning uncertainty of small targets, explicitly models the predicted bounding box as a multivariate Gaussian distribution, and uses a negative log-likelihood function to measure the KL divergence between the predicted distribution and the true value box distribution, which is used as a loss function to supervise the positioning uncertainty, thereby achieving more accurate small target positioning.

[0057] In step S103, the optimized convolutional neural network is trained using the training data set, and the network parameters of the optimized convolutional neural network are updated based on gradient back propagation until the preset training stop condition is met, and the trained convolutional neural network is used to detect small targets in traffic scene images.

[0058] It can be understood that the embodiments of the present application can use the training data set to train the optimized convolutional neural network, and update the network parameters of the optimized convolutional neural network based on gradient back propagation until the preset training stop condition is met, and use the trained convolutional neural network to detect small targets in traffic scene images, thereby achieving more accurate small target positioning.

[0059] In an embodiment of the present application, before using the training data set to train the optimized convolutional neural network, it also includes: segmenting the traffic scene image to obtain segmented images; scaling the segmented image to a preset size, and randomly flipping the scaled image to obtain the final image; generating a training data set based on the final image and the detection results of small targets in the traffic scene image; and using the training data set to train the optimized convolutional neural network.

[0060] The preset size may be set according to specific circumstances, such as 1200*1200.

[0061] It can be understood that the embodiment of the present application processes the traffic image before using the training data set to train the optimized convolutional neural network. After preprocessing, it is not only easier for the neural network to process, but also can fully retain the effective information of the small target. Specifically, the traffic scene image is segmented: offline slicing processing, the original high-resolution image in the data set is sliced ​​into a series of 800*800 sub-images in horizontal and vertical directions with a step size of 650 to adapt to the hardware processing capabilities; data enhancement processing: including image scaling and random flipping, scaling the segmented sub-image to 1200*1200, retaining as complete small target information as possible, and then randomly flipping the scaled image to form a richer data representation, and finally generating a training data set based on the processed image and the detection results of the small target, and using the training data set to train the optimized convolutional neural network.

[0062] In summary, specifically, the method for detecting small targets in traffic scene images of the embodiment of the present application includes the following steps:

[0063] S1: Preprocessing of small target detection dataset in traffic scenes.

[0064] The dataset consists of multiple high-resolution images rich in small target instances, including 9 categories such as traffic lights, pedestrians, cars, traffic signs, etc., all of which are labeled. At the same time, the dataset is preprocessed in a series to improve processing efficiency and add more sample support.

[0065] S11: Offline slicing processing, the original high-resolution image in the dataset is sliced ​​into a series of 800*800 sub-images in horizontal and vertical directions with a step size of 650 to adapt to the hardware processing capability.

[0066] S12: Data enhancement processing, including image scaling and random flipping. The segmented sub-image is scaled to a size of 1200*1200, retaining as complete small target information as possible. The scaled image is then randomly flipped to form a richer data representation, which is then input into the convolutional neural network for training.

[0067] After preprocessing, the original training data set is not only easier to process by the neural network, but also can fully retain the effective information of small targets.

[0068] S2: Convolutional neural network construction. Based on Reppoints, a network structure consisting of feature extraction module, feature fusion module, sample allocation module, positioning and classification module is constructed. These modules are used to extract target information from images, realize information interaction between multi-scale features, allocate training samples to real annotation boxes, and calculate the coordinate information and classification results of predicted bounding boxes. The specific structure of the convolutional neural network is as follows: Figure 2 shown.

[0069] Preferably, the feature extraction module is ResNet-50, the feature fusion module that performs information interaction between feature layers is FPN, a progressive semantic point regression module based on L1 Loss, a classification module based on focal loss, and a sample allocation module that provides samples for classification and regression processes are introduced.

[0070] S3: Design of sample allocation optimization scheme: To address the problem that the number and quality of samples corresponding to small targets are low due to the current sample allocation scheme, a scale-adaptive sample allocation scheme is introduced. This scheme fully considers the fact that small targets are more sensitive to the discrimination criteria, designs a serialized discrimination criterion, and allocates samples based on Euclidean distance and pseudo-frame-based IoU after sample expansion, thereby improving the number and quality of samples corresponding to small targets during training.

[0071] S4: A quality-driven localization loss is introduced. An additional branch is added to the relevant part of the localization module to estimate the localization uncertainty of small targets. The predicted bounding box is explicitly modeled as a multivariate Gaussian distribution, and the negative log-likelihood function is used to measure the KL divergence between the predicted distribution and the true value box distribution. This is used as the loss function to supervise the localization uncertainty and achieve more accurate small target localization.

[0072] S5: Use the preprocessed data in S1 to train the convolutional neural network optimized by S2 and S3, and use the SGD optimizer to perform gradient back propagation to update the network parameters.

[0073] S6: Use the network model trained in S5 to perform forward reasoning on the data in the test set to obtain the actual detection results.

[0074] The trained neural network model is used to detect the given input traffic scene image, output the coordinates and category information of potential targets, and statistically analyze the effect of the improved method on the test set category by category.

[0075] Among them, the S3 mesoscale adaptive sample allocation scheme includes:

[0076] The serialization discrimination criterion design first performs a preliminary expansion of the samples based on the Euclidean distance discrimination criterion, generates a series of pseudo-frames according to the expanded samples, and further forms the discrimination criterion of IoU between the pseudo-frame and the true value frame to form a serialization criterion; scale orientation discrimination, when performing IoU discrimination after the pseudo-frame is formed, different discrimination thresholds are set for small targets and other scale targets; scale adaptive threshold, when performing IoU threshold discrimination on small targets, the corresponding IoU threshold is dynamically adjusted according to the target size to achieve a more reasonable small target sample allocation.

[0077] For a given input image I, the ground-truth box is represented as G i =(lt i ,rb i ), where lt i Indicates the coordinates of the upper left corner of the truth box, rb i Represents the coordinates of the lower right corner. For each true value box G in the image i , this application first finds the n closest anchor points according to the Euclidean distance and obtains the preliminary allocation result A i , as the candidate anchor points of the final positive sample, and then generate pseudo boxes P based on these candidate anchor points i , while keeping the same size and aspect ratio as the corresponding true value box, the IoU between the pseudo box and the true value box can be calculated. This embodiment of the application sets two hyperparameters thr for small targets and other scale targets respectively. s and thr e , as the IoU threshold for positive and negative sample discrimination, thrs The specific plan is as follows:

[0078]

[0079] Among them, τ is a hyperparameter with a default value of 0.1, s represents the area of ​​the true value box corresponding to the sample, and ε is related to the statistical characteristics of the dataset, indicating the definition limit of the dataset for the minimum scale range target.

[0080] The quality-driven positioning losses in S4 include:

[0081] The regression branch design for positioning uncertainty estimation is based on the assumption that the network output obeys a certain probability distribution. The network is used to directly predict the relevant parameters of the probability distribution, and the network prediction result is modeled as a multivariate Gaussian distribution. The output value of the original regression branch of the network is associated with the mean of the Gaussian distribution, and the corresponding variance is obtained by introducing an additional regression branch; the decoding processing of positioning uncertainty helps to obtain a more accurate prediction bounding box, and is aimed at mean decoding based on linear transformation, and the corresponding variance is simultaneously considered, that is, the decoding process of positioning uncertainty; the positioning loss design combines scale information and sample quality, and uses the scale information of small targets and the quality of positive samples participating in training to optimize the loss function in S4.

[0082] The output of the original regression branch of the network is associated with the mean of the predicted distribution, and the corresponding variance is obtained by introducing an additional regression branch. Since the representation of the predicted bounding box is generally composed of multiple variables, combined with the characteristics of deep neural networks, it is assumed that each output variable of the regression branch is independent of each other, and all variables satisfy independent multivariate Gaussian distributions and obey The diagonal covariance matrix is Corresponding to the prediction result of the newly added regression branch, it represents the relevant positioning uncertainty.

[0083] Most current target detection methods encode the predicted offset of the localization branch through different encoding schemes. The encoding process provides a learning target for the output of the network regression branch. Correspondingly, the decoding process realizes the conversion of the network output value to the predicted bounding box. When Gaussian distribution modeling is performed on the predicted bounding box coordinates, the encoding and decoding scheme should also be taken into consideration. In the benchmark method Reppoints of this application, for any position (x, y) on the feature map, the regression branch predicts a series of offsets (Δx i ,Δy i ), forming a set of 9 prediction points The coordinates of the predicted point are obtained by the following formula:

[0084] x i =x+Δxi ,y i =y+Δy i ;

[0085] The final predicted bounding box is also represented by the upper left corner point and the lower right corner point (x1, y1, x2, y2), and is represented by the predicted point set It is obtained through a certain conversion function through the decoding process. After the multivariate Gaussian distribution modeling is performed, the predicted offset generated by the above encoding scheme is equivalent to the mean of the multivariate Gaussian distribution. The encoding and decoding process of the offset is the encoding and decoding process of the mean. Naturally, the encoding and decoding process of the positioning uncertainty also needs to be considered. Specifically, on the basis of the benchmark method, this application adds a 3×3 convolution output channel dimension of 4 feature map to represent the variance corresponding to each mean. The supervised training of the positioning uncertainty is equivalent to estimating the network parameters that minimize the log-likelihood function, as shown in the following formula:

[0086]

[0087] Among them, Θ represents the set of learnable network model parameters, P D (x) represents the Dirac distribution obeyed by the truth box, P Θ (x) indicates that the network model predicts that the bounding box obeys the Gaussian distribution. Furthermore, the positioning uncertainty-related loss can be Denoted as:

[0088]

[0089] Where N represents the number of samples, g i represents the coordinates of the true value box, b i represents the decoded mean, i.e. the decoded predicted bounding box coordinates, Indicates and b i The corresponding decoded variance is the positioning uncertainty;

[0090] Furthermore, we consider optimizing the loss function by using the scale information of small targets and the quality of positive samples involved in training. The final quality-driven positioning loss is recorded as:

[0091]

[0092] In the above formula, K is the hyperparameter set by this application to stabilize the training process, and IoU i represents the intersection-over-union ratio of the i-th positive sample and its corresponding true value box, s i is the area of ​​the true value box corresponding to the i-th positive sample.

[0093] The overall loss function in S5 is composed as follows:

[0094] L=Lcts +L reg1 +L reg2 +γ·L QLL

[0095] Among them, L cls represents the classification loss of the classification branch in parallel with the second regression stage, L reg1 and L reg2 They represent the positioning loss in the first regression stage and the second regression stage respectively, and L QLL Represents QLL (Quality-driven Localization Loss).

[0096] The present application method includes two regression stages. In the above formula, L cls represents the classification loss of the classification branch in parallel with the second regression stage, L reg1 and L reg2 They represent the positioning losses of the first regression stage and the second regression stage, and both are calculated by L sl1 , that is, Smooth L1 Loss is calculated.

[0097] For a batch participating in training, all positive samples in the first regression stage are recorded as N pos1 , all positive samples in the second stage are recorded as N pos2 ;Q x,y and P x,y Represent all samples in the first and second regression stages, G x,y It represents the set of all true value frames in the batch; the output results of the classification branch and the output results of the regression branch are respectively expressed as as well as and They represent the category label and the true value box coordinates after being encoded by the network detection head.

[0098] The following is an example of a specific embodiment to illustrate the method for detecting small targets in traffic scene images of the present application. The above steps are executed on an NVIDIA Geforce RTX-4090 graphics card with 24GB of video memory, the system environment is Ubuntu 18.04, the software environment is Python 3.7.13 + PyTorch 1.10.0 + CUDA 11.3, and the SGD optimization method is used. The batch_size is fixed to 16, the total number of training rounds is 12, and the initial learning rate is set to 0.01.

[0099] The dataset is the SODA-D dataset, which consists of multiple high-resolution images rich in small target instances, including traffic lights, pedestrians, cars, traffic signs, etc., a total of 9 categories, all of which are labeled. For detailed information, see Table 1. At the same time, a series of preprocessing is performed on the dataset to improve processing efficiency and add more sample support. Among them, Table 1 is a statistical information table of the number of labels in the traffic scene dataset.

[0100] Table 1

[0101] Target Category Number of instances People 35928 Rider 4636 Bicycle 2560 Motorcycle 3896 Vehicle 69197 Traffic sign 85905 Traffic light 62729 Traffic camera 7636 Warning cone 5946

[0102] This application qualitatively compares the target detection representative method Reppoints from the perspectives of accuracy and efficiency, and uses two objective evaluation indicators, AP and FPS, to compare and analyze the detection effects of this application and the existing Reppoints method on the test set. Table 2 shows the overall detection accuracy AP of the two methods and the detection accuracy AP of targets in various scale ranges. eS , A.P. rS , A.P. gS Compared with the existing methods, the method of this application has greatly improved in various accuracy indicators, and there is no obvious decrease in processing speed. The processing speed of the method of this application is 41.4FPS, and the average processing time for each image is 24.2ms. It can be seen that this application can effectively improve the detection accuracy of small targets without sacrificing detection efficiency. Among them, Table 2 is a comparison of the small target detection effect of this application and Reppoints.

[0103] Table 2

[0104] Detection Methods AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AP eS ]]> <![CDATA[AP rS ]]> <![CDATA[AP gS ]]> <![CDATA[AP N ]]> FPS Reppoints 28.0 55.6 24.7 10.1 23.8 35.1 45.3 43.1 This application 29.2 57.5 25.2 10.7 25.0 36.3 46.2 41.4

[0105] In order to more intuitively present the improvement effect of this application on small target detection, a visual analysis of the improvement effect of the baseline method and the method of this application is performed in terms of each category. Figure 3 As shown in the figure, the proposed method has limited improvement on two extremely small targets, traffic lights and traffic cameras, but has obvious improvement on relatively large small targets such as pedestrians, cars, and traffic signs. The improvement is most obvious on relatively large target categories such as bicycles, riders, warning cones, and motorcycles. Overall, the proposed method effectively improves the detection accuracy of each category in the SODA-D dataset.

[0106] In summary, this application aims to solve the problem of small target sample imbalance in the current traffic scene image target detection dataset. From the perspective of sample allocation, this application designs a scale-adaptive sample allocation scheme, fully considering the characteristics of small targets that are more sensitive to the evaluation criteria during the sample allocation process, and combines serialization criteria, scale-oriented screening, and dynamic thresholds to comprehensively improve the number and quality of small target samples, and improve the sample proportion and sample quality of small targets in the convolutional neural network training process; in order to solve the problem of difficulty in positioning small targets, the positioning uncertainty information is taken into consideration through multivariate Gaussian distribution, a quality-driven positioning loss is designed, and the loss function is jointly optimized using sample quality and target scale information to achieve more accurate positioning of small targets, and effectively improve the accuracy of small target detection without significantly reducing detection efficiency, thereby improving road traffic safety.

[0107] According to the method for detecting small targets in traffic scene images proposed in the embodiment of the present application, traffic scene images and detection results can be obtained to generate a training data set, a convolutional neural network can be constructed and optimized, and the optimized convolutional neural network can be trained using the training data set to update the parameters of the convolutional neural network. The trained convolutional neural network can be used to detect small targets in traffic scene images. By optimizing the sample allocation module of the convolutional neural network, the number and quality of samples of small targets in the training process of the convolutional neural network are improved, and the recognition ability of the convolutional neural network is enhanced. By optimizing the positioning and classification module of the convolutional neural network, more accurate positioning of small targets is achieved, and the detection effect of small targets is further improved.

[0108] Next, a detection device for small targets in traffic scene images proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0109] Figure 4 It is a block diagram of a detection device for small targets in traffic scene images according to an embodiment of the present application.

[0110] like Figure 4 As shown, the detection device 10 for small targets in traffic scene images includes: an acquisition module 100 , a construction module 200 and a detection module 300 .

[0111] Among them, the acquisition module 100 is used to obtain the detection results of traffic scene images and small targets in traffic scene images, and generate a training data set based on the detection results of traffic scene images and small targets in traffic scene images; the construction module 200 is used to construct a convolutional neural network based on the feature extraction module, the feature fusion module, the sample allocation module and the positioning classification module, and optimize the sample allocation module and the positioning classification module of the convolutional neural network to obtain an optimized convolutional neural network; the detection module 300 is used to train the optimized convolutional neural network using the training data set, update the network parameters of the optimized convolutional neural network based on gradient back propagation until the preset training stop condition is met, and use the trained convolutional neural network to detect small targets in traffic scene images.

[0112] In the embodiment of the present application, the construction module 200 is further used to: optimize the sample allocation output by the sample allocation module using a scale adaptive method; and optimize the positioning classification module using a quality driven positioning loss.

[0113] In an embodiment of the present application, the construction module 200 is further used to: obtain a true annotation box of a small target; calculate the Euclidean distance between a preset uniform anchor point and the geometric center of the true annotation box; select an anchor point within the target range as the Euclidean distance, generate multiple pseudo boxes based on the anchor points within the target range, use the pseudo boxes as candidate samples, and generate a candidate sample set; calculate the intersection / union ratio of each pseudo box in the candidate sample set to the true annotation box, and allocate training samples based on the intersection / union ratio.

[0114] In an embodiment of the present application, the construction module 200 is further used to: obtain the scale information of the small target; determine the intersection-and-union ratio threshold of the small target based on the scale information of the small target; if the intersection-and-union ratio of the pseudo frame to the real labeled frame is greater than the intersection-and-union ratio threshold, then use the pseudo frame as a training sample.

[0115] In an embodiment of the present application, the detection result includes the coordinate information of the predicted bounding box and the type of the small target in the bounding box, and the positioning classification module includes: a regression branch, an uncertainty branch and a classification branch, wherein the regression branch is used to output the coordinate information of the predicted bounding box; the uncertainty branch is used to estimate the positioning inaccuracy of the coordinate information of the predicted bounding box; and the classification branch is used to predict the type of the small target in the bounding box.

[0116] In an embodiment of the present application, the construction module 300 is further used to: construct a predicted Gaussian distribution based on the coordinate information and positioning accuracy of the predicted bounding box, wherein the coordinate information is used as the mean of the Gaussian distribution and the positioning accuracy is used as the variance of the Gaussian distribution; use the objective function to calculate the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true annotation box, and use the KL divergence as the quality-driven positioning loss; minimize the quality-driven loss to optimize the positioning classification module.

[0117] In the embodiment of the present application, the device 10 of the embodiment of the present application further includes: a segmentation module.

[0118] Among them, the segmentation module is used to segment the traffic scene image to obtain segmented images before using the training data set to train the optimized convolutional neural network; scale the segmented image to a preset size, and randomly flip the scaled image to obtain the final image; generate a training data set based on the final image and the detection results of small targets in the traffic scene image; and use the training data set to train the optimized convolutional neural network.

[0119] It should be noted that the aforementioned explanation of the embodiment of the method for detecting small targets in traffic scene images is also applicable to the device for detecting small targets in traffic scene images of this embodiment, and will not be repeated here.

[0120] According to the detection device for small targets in traffic scene images proposed in the embodiment of the present application, it is possible to obtain traffic scene images and detection results to generate a training data set, build and optimize a convolutional neural network, and use the training data set to train the optimized convolutional neural network to update the parameters of the convolutional neural network. The trained convolutional neural network is used to detect small targets in traffic scene images. By optimizing the sample allocation module of the convolutional neural network, the number and quality of samples of small targets in the training process of the convolutional neural network are improved, and the recognition ability of the convolutional neural network is enhanced. By optimizing the positioning and classification module of the convolutional neural network, more accurate positioning of small targets is achieved, and the detection effect of small targets is further improved.

[0121] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0122] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0123] When the processor 502 executes the program, the method for detecting small targets in traffic scene images provided in the above embodiment is implemented.

[0124] Furthermore, the electronic device further comprises:

[0125] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0126] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0127] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0128] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0129] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0130] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0131] An embodiment of the present application also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the above-mentioned method for detecting small targets in traffic scene images is implemented.

[0132] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0133] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0134] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0135] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

Claims

1. A method for detecting small targets in traffic scene images, characterized in that: The following steps are involved: Acquire a traffic scene image and a detection result of a small target in the traffic scene image, and generate a training data set according to the traffic scene image and the detection result of the small target in the traffic scene image; Constructing a convolutional neural network based on a feature extraction module, a feature fusion module, a sample allocation module and a positioning classification module, and optimizing the sample allocation module and the positioning classification module of the convolutional neural network to obtain an optimized convolutional neural network; The optimized convolutional neural network is trained using the training data set, and the network parameters of the optimized convolutional neural network are updated based on gradient back propagation until a preset training stop condition is met, and the trained convolutional neural network is used to detect small targets in traffic scene images.

2. The method for detecting small targets in traffic scene images according to claim 1, characterized in that: The optimized convolutional neural network obtained by optimizing the sample allocation module and the positioning classification module of the convolutional neural network includes: Optimizing the sample allocation output by the sample allocation module using a scale adaptive method; The localization classification module is optimized using a quality-driven localization loss.

3. The method for detecting small targets in traffic scene images according to claim 2, characterized in that: The method of optimizing the sample allocation output by the sample allocation module by using a scale adaptive method includes: Obtaining a real annotation frame of the small target; Calculating the Euclidean distance between a preset uniform anchor point and the geometric center of the real annotation box; Selecting the Euclidean distance as an anchor point within the target range, generating multiple pseudo boxes based on the anchor points within the target range, using the pseudo boxes as candidate samples, and generating a candidate sample set; The intersection-and-union ratio of each pseudo-frame in the candidate sample set to the real labeled frame is calculated, and training samples are allocated based on the intersection-and-union ratio.

4. The method for detecting small targets in traffic scene images according to claim 3, characterized in that: The allocating the training samples based on the parallel-crossover ratio includes: Obtaining scale information of the small target; Determining a parallel-intersection ratio threshold of the small target based on scale information of the small target; If the union-intersection ratio of the pseudo box and the real marked box is greater than the union-intersection ratio threshold, the pseudo box is used as the training sample.

5. The method for detecting small targets in traffic scene images according to claim 2, characterized in that: The detection result includes the coordinate information of the predicted bounding box and the type of the small target in the bounding box. The positioning classification module includes: a regression branch, an uncertainty branch and a classification branch, wherein: The regression branch is used to output coordinate information of the predicted bounding box; The uncertainty branch is used to estimate the positioning inaccuracy of the coordinate information of the predicted bounding box; The classification branch is used to predict the type of the small object in the bounding box.

6. The method for detecting small targets in traffic scene images according to claim 5, characterized in that: The method of optimizing the positioning classification module by using quality-driven positioning loss includes: constructing a predicted Gaussian distribution based on the coordinate information of the predicted bounding box and the positioning inaccuracy, wherein the coordinate information serves as a mean of the Gaussian distribution and the positioning inaccuracy serves as a variance of the Gaussian distribution; Calculating the KL divergence between the predicted Gaussian distribution and the Dirac distribution of the true annotation box using the objective function, and using the KL divergence as the quality-driven positioning loss; The quality-driven loss is minimized to optimize the positioning classification module.

7. The method for detecting small targets in traffic scene images according to claim 1, characterized in that: Before using the training data set to train the optimized convolutional neural network, the method further includes: Segmenting the traffic scene image to obtain segmented images; The segmented image is scaled to a preset size, and the scaled image is randomly flipped to obtain the final image; Generating a training data set according to the detection results of small objects in the final image and the traffic scene image; The optimized convolutional neural network is trained using the training data set.

8. A detection device for small targets in traffic scene images, characterized in that: include: An acquisition module, used to acquire a traffic scene image and a detection result of a small target in the traffic scene image, and generate a training data set according to the traffic scene image and the detection result of the small target in the traffic scene image; A construction module, used to construct a convolutional neural network based on a feature extraction module, a feature fusion module, a sample allocation module and a positioning classification module, and optimize the sample allocation module and the positioning classification module of the convolutional neural network to obtain an optimized convolutional neural network; The detection module is used to train the optimized convolutional neural network using the training data set, update the network parameters of the optimized convolutional neural network based on gradient back propagation until the preset training stop condition is met, and use the trained convolutional neural network to detect small targets in traffic scene images.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for detecting small targets in traffic scene images as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: The computer program or instruction is executed by a processor to implement a method for detecting small targets in traffic scene images as claimed in any one of claims 1 to 7.

Images