An end-to-end target detection method based on target probability density map

Abstract

The application discloses an end-to-end target detection method based on a target probability density map, and comprises the following steps: generating a target probability density map of a to-be-detected image; based on the target probability density map, obtaining a center point and a width and height of a small target region in the to-be-detected image; based on the center point and the width and height of the small target region, the small target region is intercepted through an affine transformation; target detection is performed on the small target region, and a target type and a position are output. The application utilizes the affine transformation to intercept the target region, can realize end-to-end training of a neural network, greatly simplifies a network training process and a network structure, and enables the end-to-end network to make the model from original input to final output, gives the model more space which can be automatically adjusted according to data, and increases overall fitting degree of the model; the application can reduce a missed detection rate and a false detection rate of small targets without reducing a large target detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to an end-to-end target detection method based on target probability density maps. Background Technology

[0002] In recent years, object detection in computer vision has been used in an increasing number of engineering and academic fields, leading to its rapid development. Typically, the targets to be detected are large objects that are clearly visible to the naked eye. However, with the continuous enrichment and diversification of application scenarios, the detection of small objects is becoming increasingly important. There are two main ways to define small objects in the industry: one is relative size definition, for example, an object whose length and width are 0.1 times the length and width of the image can be considered a small object; the other is absolute size definition, which defines objects with a size smaller than 32×32 pixels as small objects.

[0003] Currently, most object detection methods are based on data-driven neural network approaches. These methods first require creating a large number of object detection labels, then using these labels as ground truth inputs to train the neural network, and finally using the trained network to detect objects in specific scenes. While many neural network-based object detection methods can quickly and effectively detect large objects, small objects often suffer from missed detections and false positives. Therefore, developing detection methods specifically for small objects is crucial.

[0004] The main idea behind existing neural network-based small object detection methods is to transform small objects into larger ones. There are two main approaches: one is relative enlargement, often achieved by segmenting the image into multiple smaller images, where the smaller images become larger, and a trained neural network is used to detect the small object in each smaller image; the other approach is feature enhancement of the small object to give it clearer features similar to those of a larger object. Since small objects, as input to the neural network, undergo scaling and pooling operations, their features are gradually "diluted." Therefore, many researchers use methods such as pyramids to obtain features of small objects from various dimensions for feature fusion and enhancement.

[0005] The former method is further divided into two approaches. The first uses uniform image segmentation, but its main drawback is that it consumes a large amount of computational resources, making it a crude full-image search mode. The second approach first performs initial localization of the target region, then crops that region and feeds it into the target detector for target detection. This method fragments the training of the neural network, making end-to-end training impossible. In addition, the method of feature enhancement for small targets has very limited detection effectiveness. Summary of the Invention

[0006] In view of this, the present invention provides an end-to-end target detection method based on target probability density maps, so as to reduce the false detection rate and false detection rate of small targets without reducing the detection accuracy of large targets.

[0007] This invention discloses an end-to-end target detection method based on target probability density maps, comprising the following steps:

[0008] Step 1: Generate the target probability density map of the image to be detected;

[0009] Step 2: Based on the target probability density map, obtain the center point and width and height of the small target region in the image to be detected;

[0010] Step 3: Based on the center point and width and height of the small target region, extract the small target region through affine transformation;

[0011] Step 4: Perform target detection on the small target area and output the target type and location.

[0012] Furthermore, if there is no small target in the image to be detected in step 2, a new image is input as the image to be detected, and steps 1 to 2 are repeated until there is a small target in the image to be detected;

[0013] Step 4 also includes:

[0014] A single-stage detection network is used to perform target detection on the image to be detected, and the type and location of the detected target are output.

[0015] Furthermore, the target probability density map is as follows:

[0016]

[0017]

[0018] Where, σ i Let x be the standard deviation of the Gaussian function. k It is the center of the target, δ(xx) k ) represents the density point. Denotes the Gaussian kernel, δ(xx) k )and Convolution yields the target probability density map D(x), where k = 1, 2, ..., N represents the target index; i and j represent the i-th category and the j-th target within the i-th category, respectively; H i and W i Let h represent the average height and width of the i-th type of target, respectively. ij and w ij Let represent the average height and width of the j-th target of the i-th species, respectively. η is used to balance the contribution of the overall size and individual size of the species to the filtering parameter σ.

[0019] Furthermore, step 1 penalizes background regions in the image to be detected using a loss function:

[0020]

[0021] and

[0022] Among them, Loss density Let i = 1, 2, ..., N represent the indices of the input images, and j = 0, 1, ..., M be the loss function of the neural network. i D(X) represents the index of the background pixel of the i-th input image. i ;Θ) represents the input X i The target probability density map, Θ represents the parameters of the target probability density map generation model, D i (j) = 0 indicates that the j-th pixel in the i-th target probability density map is a background pixel. β is the penalty coefficient. W and H represent the width and height of the target probability density map. Therefore, the more background pixels there are, the larger the penalty coefficient β will be.

[0023] Further, step 2 includes:

[0024] Step 21: Use a sliding window to perform a non-overlapping traversal of the target probability density map, compare the pixel values ​​in the sliding window with a preset threshold, and assign values ​​to the pixel values ​​in the sliding window based on the comparison results;

[0025] Step 22: Cluster the target probability density map after assignment;

[0026] Step 23: Use the centroid method to calculate the centroid of each target region after clustering.

[0027] Further, step 21 includes:

[0028] During training, the size of the sliding window is the average of all target sizes. The average pixel value m in each sliding window is compared with a pre-set threshold. If m is less than the threshold, all pixel values ​​in that sliding window are set to 0; otherwise, they are set to m.

[0029] Further, step 22 includes:

[0030] The DBSCAN algorithm is used to cluster non-zero regions in the target probability density map after thresholding. If the sliding window size is w... s ×h s Then the radius of the cluster scan is 2×max(w) s ,hs After clustering, regions that are grouped into one category represent a target region to be segmented;

[0031] Step 23 includes:

[0032] For each clustered target region R, the centroid (x) is calculated using the centroid method. c ,y c ).

[0033] Furthermore, the center of gravity (x) c ,y c The formula for calculating ) is:

[0034]

[0035]

[0036] Where, d ij This represents the probability density value in R, i.e., the pixel value of the target probability density map; if the size of the target region R is w×h, then the calculation formula is:

[0037]

[0038]

[0039] Center of gravity (x) c ,y c The x becomes the center of the new region, and finally the region to be clipped, R = (x) c ,y c ,w,h), parameters in R for d ij Differentiable.

[0040] Further, step 3 includes:

[0041] Affine transformation is used to extract the target region. The coordinates of the region R to be extracted are represented as (x...). o ,y o The coordinates of the extracted target area are represented as (x... t ,y t The affine transformation is:

[0042]

[0043] In affine transformations, only (x o ,y o The target region is unknown, while the coordinates and transformation matrix of the target region are known. An analytical representation of the target probability density map to be clipped and the target region after clipping can be established through affine transformation, thus making the differentiability of the training process continuous.

[0044] Furthermore, (xt ,y t (x) are all integer coordinates, obtained by inverse transformation. o ,y o The coordinates of the image to be detected are not all integers, and bilinear interpolation is required to obtain the pixel values ​​corresponding to the coordinates of the image after inverse transformation.

[0045] Due to the adoption of the above technical solutions, this invention has the following advantages: This invention proposes a method for small target detection based on target probability density maps for target region localization. It can locate the area where the target exists, reducing the false negative and false positive rates of small targets without reducing the accuracy of large target detection. This invention utilizes affine transformation for target region extraction, enabling end-to-end training of the neural network. This greatly simplifies the network training process and network structure. The end-to-end network allows the model to proceed from the original input to the final output, giving the model more room to automatically adjust based on data, increasing the overall fit of the model. This invention can be seamlessly connected with other neural networks, including different detection networks and neural networks for other tasks such as target recognition and classification. This invention can find application scenarios in multiple fields, including military and civilian applications. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0047] Figure 1 This is a flowchart illustrating a small target detection method according to an embodiment of the present invention;

[0048] Figure 2(a) is a schematic diagram of the original input image in an embodiment of the present invention;

[0049] Figure 2(b) is a schematic diagram of the target probability density map corresponding to Figure 2(a);

[0050] Figure 3 This is a schematic diagram of a target area positioning process according to an embodiment of the present invention;

[0051] Figure 4(a) is a schematic diagram of the effect after threshold processing in Figure 2(b);

[0052] Figure 4(b) is a schematic diagram of the effect of clustering Figure 4(a);

[0053] Figure 5 This is a schematic flowchart of an end-to-end target detection method based on a target probability density map according to an embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further described in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0055] This invention provides an embodiment of an end-to-end target detection method based on a target probability density map, which includes target probability density map generation, target region localization, target region extraction, and target detection. Its flowchart is shown in Figure 1.

[0056] Target probability density map generation: This is used to generate a target probability density map. The brighter the area in the map, the higher the probability of the existence of a small target. The left image is the original input image, as shown in Figure 2(a), and the right image is the generated target probability density map, as shown in Figure 2(b). The brighter the area in the right image, the higher the probability of the existence of a small target (person).

[0057] Target region localization: Based on the generated target probability density map, small target regions are located through threshold filtering and clustering, and the center point and width and height of the small target regions are obtained.

[0058] Target region extraction: Based on the localized small target region, the small target region is extracted through affine transformation and sent to the target detector.

[0059] Target detection: Performs target detection on the input small target region and outputs the type and location of the detected target.

[0060] S1. Target probability density map generation

[0061] The generation of the target probability density map first requires training the network, drawing inspiration from the CSRNet method. CSRNet is used to generate crowd distribution density maps. The CSRNet network mainly consists of a front-end network and a back-end network. The front-end network uses VGG-16 with fully connected layers removed. Increasing the number of convolutional layers leads to a smaller output image size, making it more difficult to generate the density map. Therefore, CSRNet uses a dilated convolutional neural network as the back-end network to expand the receptive field while maintaining resolution, generating high-quality crowd distribution density maps.

[0062] During training, the target probability density map is Gaussian blurred at each density point on the original density map to improve the robustness of target prediction. The formula is as follows:

[0063]

[0064] Where, x k It is the center of the target, δ(xx) k ) represents the density point. Denotes the Gaussian kernel, δ(xx) k )and Convolution yields the probability density map D(x). k = 1, 2, ..., N represents the target index.

[0065] The value of σ changes dynamically according to the size of the target, and its formula is as follows:

[0066]

[0067] Where i and j represent the i-th category and the j-th target within the i-th category, respectively, H i and W i Let h represent the average height and width of the i-th type of target, respectively. ij and w ij Let represent the average height and width of the j-th target of the i-th type, respectively. η is used to balance the contribution of the size of the type's median and the individual size to the filtering parameter σ; experiments show that 0.7 is suitable. In this way, when the target size is small, the area of ​​small targets after Gaussian filtering can be increased by the overall size of the type; when the target size is large, the area of ​​small targets after Gaussian filtering can be relatively reduced, achieving a balancing effect. By dynamically changing the filter kernel size, the probability density map obtained after filtering shows that areas with larger values ​​have a higher probability of containing small targets.

[0068] The CSRNet method addresses the problem of dense target counting, while this method addresses the problem of small target detection. The challenge is that the background region, devoid of targets, constitutes a large proportion of the target area. Therefore, the loss function penalizes the background region, as shown below:

[0069]

[0070] and

[0071] Where i = 1, 2, ..., N represents the index of the input image, and j = 0, 1, ..., M i This represents the index of the background pixel in the i-th input image. D(X) i ;Θ) represents the input X i The probability density plot, Θ represents the parameters of the probability density plot generation model, D i (j) = 0 indicates that the j-th pixel in the i-th probability density map is a background pixel. β is the penalty coefficient, and W and H represent the width and height of the probability density map, respectively. Therefore, the more background pixels there are, the larger the penalty coefficient β becomes.

[0072] S2, Target Area Location

[0073] This step will locate the target region based on the target probability density map generated in S1. As shown in Figure 2(b), the brighter the area in the target probability density map, the higher the probability of the presence of the target. To further reduce the impact of background and noise on target detection, a series of measures will be taken to process the target probability density map to obtain the target region. First, a sliding window is used to perform a non-overlapping traversal of the density map. During training, the size of the sliding window is the average of all target sizes. The average pixel value m in each sliding window is compared with a pre-set threshold. If m is less than the threshold, all pixel values ​​in that sliding window are set to 0; otherwise, they are set to m. Then, the DBSCAN algorithm is used to cluster the non-zero regions in the thresholded density map. If the size of the sliding window is w... s ×h s Then the radius of the cluster scan is 2×max(w) s ,h s After clustering, regions grouped into a single cluster represent a target region to be segmented. Then, for each clustered target region R, the centroid (x0) is calculated using the centroid method. c ,y c The formula is as follows:

[0074]

[0075]

[0076] Where, d ij This represents the probability density value in R, i.e., the pixel value of the target probability density map. If the size of the target region R is w×h, then the calculation formula is:

[0077]

[0078]

[0079] Thus the center of gravity (x) c ,y c The x-axis becomes the center of the new region, and finally the region to be clipped is R = (x-axis). c ,y c ,w,h), parameters in R for d ij Differentiable. Target area localization, such as... Figure 3 As shown in the diagram, the target area positioning effect is as follows: Figure 4(a) and 4(b) As shown.

[0080] S3, Target Area Capture

[0081] The target region has been obtained in S2, and it needs to be cropped and fed into the target detector for target detection. Traditional region cropping uses templates, which leads to non-differentiable training of the entire network. Therefore, this method uses affine transformation for target region cropping. The coordinates of the region R to be cropped are represented as (x... o ,y o The coordinates of the extracted target area are represented as (x... t ,y t The affine transformation is:

[0082]

[0083] Affine transformations only have (x o ,y o The target region is unknown, while the coordinates and transformation matrix of the target region are known. Therefore, an analytical representation of the target probability density map to be clipped and the target region after clipping can be established through affine transformation, thus making the differentiability of the training process continuous.

[0084] Given (x) t ,y t (x) are all integer coordinates, obtained by inverse transformation. o ,y o Since the coordinates are not all integers, bilinear interpolation is needed to obtain the pixel values ​​corresponding to the coordinates of the original image after inverse transformation.

[0085] S4, Target Detection

[0086] Object detection is divided into two branches, such as Figure 5 As shown, the top branch uses a single-stage detection network to detect targets in the original input image. This detection method often misses or falsely detects some small targets, but it can achieve good detection results for non-small targets. The bottom branch is specifically for detecting small targets, and the same single-stage detection network can be used. Finally, the detection results from the two branches are combined to obtain the final detection result.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An end-to-end object detection method based on target probability density map, characterized in that, The method comprises the following steps: Step 1: generating a target probability density map of an image to be detected; Step 2: obtaining a center point and a width and height of a small target region in the image to be detected based on the target probability density map; Step 3: intercepting the small target region through affine transformation based on the center point and the width and height of the small target region; Step 4: performing target detection on the small target region to output a target type and a position; The target probability density map is: where σ i is the standard deviation of the Gaussian function, x k is the center of the object, δ(x-x k ) represents the density point, represents the Gaussian kernel, δ(x-x k ) is convolved with to obtain the object probability density map D(x), k = 1, 2, …, N represents the serial number of the object; i, j respectively represent the i-th category and the j-th object in the i-th category, H i and W i respectively represent the average height and width of the i-th category object, h ij and w ij respectively represent the average height and width of the j-th object in the i-th category, and η is used to balance the contribution of the overall size and individual size of the category to the filtering parameter σ.

2. The method of claim 1, wherein, If there is no small target in the image to be detected in the step 2, a new image is input as the image to be detected, and the steps 1 and 2 are executed again until there is a small target in the image to be detected; The step 4 further comprises: performing target detection on the image to be detected by using a single-stage detection network to output a detected target type and position.

3. The method of claim 1, wherein, The step 1 punishes a background region in the image to be detected by using a loss function: where Loss density is the loss function of the neural network, i = 1, 2, …, N represents the serial number of the input image, j = 0, 1, …, M i represents the serial number of the background pixel point of the i-th input image, D(X i ; Θ) represents the target probability density map of the input X i , Θ represents the parameters of the target probability density map generation model, D i (j) = 0 represents that the j-th pixel point of the i-th generated target probability density map is a background point, β is a penalty term coefficient, and W and H represent the width and height of the target probability density map, so the more the background points, the greater the penalty coefficient β.

4. The method of claim 1, wherein, The step 2 comprises: Step 21: using a sliding window to perform non-overlapping traversal on the target probability density map, comparing pixel values in the sliding window with a preset threshold, and assigning values to the pixel values in the sliding window based on a result of the comparison; Step 22: clustering the target probability density map after the assignment; Step 23: calculating a barycenter of each target region after the clustering by using a barycenter method.

5. The method of claim 4, wherein, The step 21 comprises: In a training process, a size of the sliding window is an average value of all target sizes, a pixel average value m in each sliding window is compared with a preset threshold threshold, if the pixel average value m is less than the threshold, all pixel values in the sliding window are set to 0, otherwise, the pixel values are set to m.

6. The method of claim 5, wherein, The step 22 comprises: The DBSCAN algorithm is used to cluster the regions with non-zero target probability density in the threshold-processed target probability density map. If the size of the sliding window is w s ×h s , the radius of the clustering scan is 2×max(w s ,h s ), and the clustered regions that belong to the same class represent a target region to be segmented. The step 23 comprises: The center of gravity (x c ,y c ) is calculated for each target region R using the center of gravity method. c c 7. The method of claim 6, wherein, The formula for calculating the center of gravity (x c ,y c ) is: where d ij represents the probability density value in R, i.e. the pixel value of the target probability density map; if the size of the target region R is w x h, the calculation formula is: Center of gravity (x) c ,y c The x becomes the center of the new region, and finally the region to be clipped, R = (x c ,y c ,w,h), parameters in R for d ij Differentiable.

8. The method of claim 1, wherein, The step 3 comprises: The target region is intercepted by using affine transformation. The coordinates of the region R to be intercepted are represented as (x o ,y o ), the coordinates of the target region after interception are represented as (x t ,y t ), and the affine transformation is as follows: x t = 0,1,..., w - 1 y t = 0,1,..., h - 1 In affine transformation, only (x o ,y o ) is unknown, the coordinates of the target region and the transformation matrix are known, and the analytical expression of the target probability density map to be cropped region and the target region after cropping can be established by affine transformation, so that the continuity of the derivative of the training process is achieved.

9. The method of claim 8, wherein, (x t ,y t ) are integer coordinates, (x o ,y o ) obtained by inverse transformation are not all integers, and the pixel values corresponding to the coordinates of the image to be detected after inverse transformation need to be obtained by bilinear interpolation.

Images