A method and system for accelerating target detection inference based on slice screening
By performing slice screening and detection of bird's eye view, using convolutional neural network to predict the target probability and combining the original image detection, the problem of small object detection accuracy and low efficiency in the bird's eye view scene is solved, and efficient object detection is achieved.
Patent Information
- Application Number
- CN202211004041.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-19
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2042-08-19
AI Technical Summary
The prior art small object detection in bird's-eye view scenarios has the problems of low detection accuracy and low inference efficiency, especially due to the small and sparse target scales and excessive detector burden.
By performing slice screening on the bird's-eye view, using a convolutional neural network to predict the probability of targets in the slice, setting a double threshold to filter out the slices with obvious and potentially containing targets, and testing them in combination with the original image, and finally the fusion detection results are suppressed by non-maximum value.
While maintaining high accuracy, it significantly improves detection speed, saves time cost of training models, and is versatile, suitable for different detectors.
Smart Images
Figure CN115375969B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a method for accelerating target detection inference based on slice screening. Background Art
[0002] Object detection is one of the classic tasks in the field of computer vision, aiming to locate the position of the target object of interest in the image and identify the specific category of the object. In the field of computer vision, the data samples of object detection can include any object in the real world, such as people, animals, cars, buildings, etc. In the training stage of the whole task, a large number of data samples are input, and a convolutional neural network is trained to extract high-level semantic features related to the target category and output the position and category results of the object, that is, to search for and solve the optimal parameters of the model. When the model parameters have been solved, using and deploying the model is called the inference stage.
[0003] With the development of artificial intelligence technology and the continuous expansion of data samples, researchers have begun to focus on using deep learning technology to solve the object detection problem of high-altitude bird's-eye view images such as remote sensing, drones, and satellites. In scenes where the objects in bird's-eye view images (remote sensing, drone images) are usually small and sparse, the object detection targets are mostly ground objects with rich natural and social attributes, such as airplanes, vehicles, ships, buildings, etc., which have clear boundaries and can be clearly distinguished from the background environment. Compared with general scenes, the bird's-eye view scene is quite different in at least two aspects: (1) The scale of the target is much smaller than that of the targets in general data sets (such as MS COCO). Most of the information of these small targets may be lost or disappear during the downsampling process of general detectors, resulting in a low accuracy of general detectors in these scenes and making it difficult to achieve high-precision full-image detection. (2) The objects are sparsely distributed in the image and usually only gather in local areas. For example, vehicles are usually concentrated on roads, and oil tanks are often concentrated in factories. There are large areas of background in other areas of the image, such as forests, farmlands, and oceans. Due to these characteristics, object detection in bird's-eye view scenes is very challenging. Existing general detectors cannot achieve satisfactory detection accuracy in the bird's-eye view scene of small targets, and training a detector specifically designed for small targets from scratch requires time and technical costs.
[0004] The existing block detection process is a target detection method based on block parallelism disclosed in CN113421253A. Based on an image analysis model, an image with a large scale or an extreme ratio can be divided into multiple sub-blocks for detection, which can prevent the scaling operation from affecting the detection result and achieve the purpose of effectively detecting targets in an image with a large scale or an extreme ratio. Although the existing method can effectively improve the detection accuracy in this scenario, it does not consider the small scale and sparse distribution of object targets in the bird's-eye view scenario, and sends a large number of slices that do not contain target objects to the detector, resulting in slice redundancy and low inference efficiency. Summary of the Invention
[0005] Aiming at the problem of low inference efficiency of detecting and reasoning all slices in the existing slice division method, the present invention proposes a target detection inference acceleration method and system based on slice screening, adding a slice screening mechanism on the basis of the existing slice division detection method to filter out slices that do not contain targets, enabling the data in the bird's-eye view scenario to achieve high accuracy on a general detector while accelerating the inference process.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] On the one hand, the present invention provides a target detection inference acceleration method based on slice screening, including the following steps:
[0008] S1. Divide the original bird's-eye view image into slices of the same size in a way that adjacent slices have a certain overlap according to a set size;
[0009] S2. For the divided slices, use a convolutional neural network to predict the probability that each region in the slice contains a target, output a probability matrix about the slices, and take the maximum value among all probabilities as the probability that each slice contains a target;
[0010] S3. Use a double-threshold screening method to screen the probability that each slice contains a target to obtain slices that obviously and potentially contain targets;
[0011] S4. Perform target detection on the screened slices that obviously and potentially contain targets, and at the same time use the original image as an auxiliary prediction to output the detection results relative to the slices and the original image;
[0012] S5. Map the detection results relative to the slices onto the original image, and then post-process all the detection results to remove redundant items to obtain the final retrieved result after fusion.
[0013] Further, in step S1, the method for setting the slice size is as follows: Based on the ratio of the target area to the original image area in the pre-trained model dataset and the statistical results of the target sizes in the dataset, keep the ratio of the target sizes of the dataset and the pre-trained model dataset to the size of the image fed into the detector consistent.
[0014] Further, the overlap ratio of adjacent slices in step S1 is 0.25.
[0015] Further, in step S1, the slices cut out are recorded in row and column order according to the division order as indicating that the slice comes from the i-th row and j-th column. The upper-left, upper-right, lower-left, and lower-right coordinates of each block are (X min , Y min ), (X max , Y min ), (X min , Y max ), (X max , Y max ), where:
[0016] X min = i * S width - i * Overlap ratio
[0017] Y min = j * S height - j * Overlap ratio
[0018] X max = X min + S width
[0019] Y max = Y min + S height
[0020] where Overlap ratio is the overlap ratio, taking 0.25; S width is the slice width, S height is the slice height, I width is the width of the original image, I height is the height of the original image.
[0021] Further, in step S2, the slice is the smallest square covering more than 85% of the target size in the dataset.
[0022] Further, in step S3, the threshold is set to 0.5, and the slices with a probability greater than or equal to 0.5 are initially screened out as the slices S obvious that obviously contain the target. Then, a probability threshold of 0.3 is set around the slices that obviously contain the target, and the slices S that potentially contain the target are further screened outpotential .
[0023] Further, in step S5, for each detection result O ij for each slice S i (X o , Y o , H, W) is mapped to the original image through the following formula to obtain O i ′(X o ′, Y o ′, H, W);
[0024] X′ o = X o + X min
[0025] Y′ o = Y o + Y min
[0026] wherein, X o , Y o , H, W are respectively the upper left coordinates and the height and width of the i-th detection result box O i , and X o ′, Y o ′, H, W are respectively the upper left coordinates and the height and width of the i-th detection result box O i after being mapped to the original image, and X i ′, Y min ′ are respectively the upper left coordinates of the slice where the i-th detection result box O min is located. i
[0027] Further, in step S5, the post-processing method uses non-maximum suppression or soft non-maximum suppression method.
[0028] On the other hand, the present invention also provides a target detection inference acceleration system based on slice screening, including the following modules to implement the method of any one of the above:
[0029] An image segmentation module, configured to divide the original image into slices in a manner that adjacent slices have a certain overlap according to a set size;
[0030] A probability prediction module, configured to use a region-based probability prediction model composed of a convolutional neural network to predict the probability that each region in the divided slice contains a target, and output a probability matrix of the slice by the model, and take the maximum value of all probabilities as the probability that each slice contains a target;
[0031] A slice screening module, configured to screen the probability that each slice contains a target obtained based on the probability prediction module to obtain slices that obviously and potentially contain targets;
[0032] A target detection module, which is used to detect the slices containing the target screened by the slice screening module, and at the same time use the original image as an auxiliary prediction to output the detection results relative to the slices and the original image;
[0033] A result fusion module, which is used to map the detection results relative to the slices onto the original image, and then use non-maximum suppression to post-process all the detection results to obtain the final retrieved result after fusion.
[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0035] The method for accelerating target detection inference based on slice screening proposed by the present invention additionally designs a probability prediction module and a slice screening module for screening slices. First, the original image is divided into slices, and the probability of whether the slice contains the target is obtained through a classification network for the slices, and then by setting double thresholds, the slices containing the target and the original image are selected and sent to the detector for target detection. In the scenario where the targets are usually small and sparse in bird's-eye view images (such as remote sensing and drone images), the inference process is accelerated while maintaining high accuracy. Compared with the method of directly detecting the original image, the accuracy of the method of the present invention has been greatly improved; compared with the method of detecting all slices after dividing the slices, the accuracy of the method of the present invention has been slightly improved and the inference speed has been significantly improved. The target detection inference acceleration system based on slice screening proposed by the present invention can be fine-tuned to a bird's-eye view dataset on the basis of a pre-trained model obtained on a general target detection dataset, without training the model from scratch, saving time costs. In addition, the system proposed by the present invention can replace different detectors and has universality in the scenario where the targets are small and sparse. Description of the Drawings
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings.
[0037] Figure 1 It is a flowchart of the method for accelerating target detection inference based on slice screening provided by an embodiment of the present invention.
[0038] Figure 2 It is a functional module diagram of the target detection inference acceleration system provided by an embodiment of the present invention.
[0039] Figure 3 It is a flowchart of the result fusion module provided by an embodiment of the present invention. Detailed Embodiments
[0040] For small and sparse detection targets, the present invention designs an object detection inference acceleration method based on slice screening. The main steps are as follows: given an aerial view image and splitting it, using a convolutional neural network to predict the regional probabilities of these slices to obtain the probability matrix corresponding to the slices, using a dual-threshold screening method to screen the slices, and feeding the screened slices containing targets into the object detection model in parallel. At the same time, using the original image as an auxiliary prediction, and finally obtaining the fused final retrieval result through post-processing methods such as non-maximum suppression.
[0041] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.
[0042] The object detection inference acceleration method based on slice screening of the present invention has a process as Figure 1 shown, including the following steps:
[0043] S1. Divide the original aerial view image into slices of the same size in a way that adjacent slices have a certain overlap according to a set size.
[0044] S2. For the divided slices, use a convolutional neural network to predict the probability that each region in the slice contains a target, output the probability matrix of the slice, and take the maximum value among all probabilities as the probability that each slice contains a target.
[0045] S3. Use the dual-threshold screening method to screen the probability that each slice contains a target to obtain slices that obviously and potentially contain targets.
[0046] S4. Perform object detection on the screened slices that obviously and potentially contain targets, and at the same time use the original image as an auxiliary prediction to output the detection results relative to the slices and the original image.
[0047] S5. Map the detection results relative to the slices onto the original image, and then use non-maximum suppression to post-process all detection results to obtain the fused final retrieval result.
[0048] The above method is implemented by the object detection inference acceleration system based on slice screening provided by the present invention, as Figure 2 shown, the system includes the following modules:
[0049] (1) Image splitting module, used to divide the original image into slices in a way that adjacent slices have a certain overlap according to a set size.
[0050] Specifically, first, the image splitting module calculates the appropriate slice size (S based on the ratio of the target area in the pre-trained model dataset to the original image area and the statistical results of the target sizes in the dataset, while keeping the ratio of the target sizes in the dataset and the pre-trained model dataset to the image size fed into the detector consistent. height,S width )。
[0051] Then, the image segmentation module divides the original image with a height and width of (I height ,I width ) into slices in the order from left to right and from top to bottom, with an overlap ratio of Overlap ratio. The sliced slices are recorded in row and column order according to the division order as indicating that the slice comes from the i-th row and j-th column. The upper-left, upper-right, lower-left, and lower-right coordinates of each block are (X min ,Y min ), (X max ,Y min ), (X min ,Y max ), (X max ,Y max ), expressed as:
[0052] X min = i * S width - i * Overlap ratio
[0053] Y min = j * S height - j * Overlap ratio
[0054] X max = X min + S width
[0055] Y max = Y min + S height
[0056] where, Overlap ratio is the overlap ratio, and in this embodiment, 0.25 is adopted; S width is the slice width, S height is the slice height, I width is the original image width, and I height is the original image height.
[0057] (2) The probability prediction module is used to predict the probability that each region in the slice contains the target using a region-based probability prediction model composed of a convolutional neural network for the divided slices, and the model outputs a probability matrix for the slice, and takes the maximum value among all probabilities as the probability that each slice contains the target.
[0058] Specifically, first, calculate the minimum grid size that can cover more than 85% of the target size in the dataset. Then, send the slices into the region-based probability prediction model composed of a convolutional neural network in parallel. The module outputs a probability matrix for the slices. Each probability value in the matrix corresponds to the probability of whether each grid region of the slice contains the target. Then, take the maximum value among all the probabilities as the probability P that each slice contains the target. ij 。
[0059] In addition to obtaining probabilities for each slice using the region-based probability prediction method and screening, methods such as clustering can also be used to pre-classify the regions with targets in the figure. The slices formed by clustering have no restrictions on size and position, and have a larger learnable space compared to the slices formed by artificially dividing the grid.
[0060] (3) Slice screening module, used to screen the probability that each slice contains the target obtained from the probability prediction module to obtain slices that obviously and potentially contain the target.
[0061] Specifically, according to the probability P that each slice contains the target obtained from the probability prediction module ij , in this embodiment, the threshold is set to 0.5, and the slices with probabilities greater than or equal to 0.5 are initially screened out as slices S that obviously contain the target obvious , then a probability threshold of 0.3 is set around the slices that obviously contain the target, and the slices S that potentially contain the target are further screened out potential . The formula is as follows:
[0062] S obvious ={S ij |i=m,j=n}
[0063] S potential ={S ij |P ij ≥0.3,i=m - 1 or m + 1,j=n - 1 or n + 1}.
[0064] (4) Target detection module, used to detect the slices containing the target screened out by the slice screening module, and at the same time use the original image as an auxiliary prediction to output the detection results relative to the slices and the original image.
[0065] Specifically, send the slices S that obviously contain the target screened out by the slice screening module obvious and the slices S that potentially contain the target potential into the target detection model in parallel, and also send the original image into the target detection model. The output of the model is the detection results relative to these slices and the original image.
[0066] (5) The result fusion module is used to map the detection results relative to the slices onto the original image, and then perform post-processing on all the detection results using non-maximum suppression to obtain the final retrieved result after fusion.
[0067] Specifically, as Figure 3 shown, the result fusion module maps each detection result O ij for each slice S i (X o , Y o , H, W) onto the original image through the following formula to obtain O i ′(X o ′, Y o ′, H, W), and then perform post-processing on all the detection results using non-maximum suppression.
[0068] X′ o = X o + X min
[0069] Y′ o = Y o + Y min
[0070] where X o , Y o , H, and W are respectively the upper-left coordinates and the height and width of the bounding box of the i-th detection result O i , and X o ′, Y o ′, H, and W are respectively the upper-left coordinates and the height and width of the bounding box of O i ′ after mapping the i-th detection result O i onto the original image, and X min , Y min are respectively the upper-left coordinates of the slice where the i-th detection result O i is located. In the result fusion module, in addition to using non-maximum suppression to remove redundant items of the detection results, a softened non-maximum suppression method can also be used for removal to obtain the final detection result.
[0071] To verify the object detection inference acceleration method based on slice screening proposed by the present invention, a task-aligned single-stage object detection head is used on the VisDrone-DET2021 and Oiltank datasets. Compared with the method of directly detecting the original image, the accuracy of the method of the present invention can be significantly improved by up to 28%; compared with the method of dividing slices and detecting all slices, the accuracy of the method of the present invention can be improved by 2.9% and the inference speed can be increased by up to 5 times.
[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features, but these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for accelerating object detection inference based on slice screening, characterized in that, It includes the following steps: S1. Divide the original bird's-eye view image into slices of the same size in a way that adjacent slices have a certain overlap according to the set size; S2. For the divided slices, use a convolutional neural network to predict the probability that each region in the slice contains the target, output a probability matrix for the slice, and take the maximum value among all probabilities as the probability that each slice contains the target; S3. Use a double-threshold screening method to screen the probability that each slice contains the target to obtain slices that obviously and potentially contain the target; S4. Perform target detection on the slices that are screened out and obviously and potentially contain the target, and use the original image as an auxiliary prediction at the same time, and output the detection results relative to the slices and the original image; S5. Map the detection results relative to the slices onto the original image, and then post-process all the detection results to remove redundant items to obtain the final fused retrieval result.
2. The method for accelerating target detection inference based on slice screening according to claim 1, wherein In step S1, the method for setting the slice size is as follows: Based on the ratio of the target area to the original image area in the pre-trained model dataset and the statistical results of the target sizes in the dataset, keep the ratio of the target sizes in the dataset and the pre-trained model dataset to the image size fed into the detector consistent.
3. The method for accelerating target detection inference based on slice screening according to claim 1, wherein In step S1, the overlap ratio of adjacent slices is 0.
25.
4. The method for accelerating target detection inference based on slice screening according to claim 1, wherein In step S1, the cut slices are recorded in row-column order according to the divided order as indicating that the slice comes from the i-th row and j-th column. The upper-left, upper-right, lower-left, and lower-right coordinates of each block are (X min , Y min ), (X max , Y min ), (X min , Y max ), (X max , Y max ), where: X min = i * S width - i * Overlap ratio Y min = j * S height - j * Overlap ratio X max = X min + S width Y max = Y min + S height Among them, the Overlap ratio is the overlap proportion, and 0.25 is adopted; S width is the slice width, S height is the slice height, I width is the original image width, I height is the original image height.
5. The method for accelerating target detection inference based on slice screening according to claim 1, wherein In step S2, the slice is the smallest square that covers more than 85% of the target size in the dataset.
6. The method for accelerating target detection inference based on slice screening according to claim 1, wherein In step S3, set the threshold to 0.5, and initially screen out the slices with a probability greater than or equal to 0.5 as the slices S that obviously contain the target obvious , then set the probability threshold to 0.3 around the slices that obviously contain the target, and further screen out the slices S that potentially contain the target potential .
7. The method for accelerating target detection inference based on slice screening according to claim 1, wherein In step S5, for each detection result O ij with respect to each slice S i (X o , Y o , H, W) is mapped onto the original image by the following formula to obtain O i ′(X o ′, Y o ′, H, W); X′ o = X o + X min Y′ o = Y o + Y min Among them, X o , Y o , H, and W are respectively the upper left coordinates and the height and width of the i-th detection result box O i . X o ′, Y o ′, H, and W are respectively the upper left coordinates and the height and width of the i-th detection result box O i after being mapped to the original image O i ′. X min and Y min are respectively the upper left coordinates of the slice where the i-th detection result box O i is located.
8. The method for accelerating target detection inference based on slice screening according to claim 1, wherein In step S5, the post-processing method uses non-maximum suppression or softened non-maximum suppression method.
9. A target detection inference acceleration system based on slice screening, characterized in that, It includes the following modules to implement the method described in any one of claims 1-8: An image segmentation module for dividing the original image into slices in a way that adjacent slices have a certain overlap according to the set size; A probability prediction module for using a region-based probability prediction model composed of a convolutional neural network to predict the probability that each region in the slice contains the target for the divided slices, and the model outputs a probability matrix for the slice, and takes the maximum value among all probabilities as the probability that each slice contains the target; A slice screening module for screening the probability that each slice contains the target obtained from the probability prediction module to obtain slices that obviously and potentially contain the target; A target detection module for detecting the slices containing the target screened out by the slice screening module, and using the original image as an auxiliary prediction at the same time, and outputting the detection results relative to the slices and the original image; A result fusion module for mapping the detection results relative to the slices onto the original image, and then using non-maximum suppression to post-process all the detection results to obtain the final fused retrieval result.