Detection method for small target image
Through the improved YOLOv8 neural network structure, the sparse matrix module and dynamic weighted fusion module are used to detect small target images, which solves the problems of difficulty in extracting small target features and low detection efficiency in the prior art, and achieves efficient small target detection.
Patent Information
- Application Number
- CN202411826971.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art is difficult to effectively capture and represent the characteristics of small targets, resulting in poor small target detection performance, and the evaluation mechanism based on cross-border ratio is not suitable for small target detection, resulting in the potential small target being misjudged or ignored.
Using the improved YOLOv8 neural network structure, the neck network includes a sparse matrix module and the original neck network. The shallowest and deepest feature maps output by the backbone network are fused through the sparse matrix module, and the feature fusion is fused through the dynamic weighted fusion module, and finally output to the detection head network for prediction.
It improves the efficiency and sensitivity of small object detection, ensures the model prediction effect, and is also suitable for the detection of large object images.
Smart Images
Figure CN119649147A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and recognition, and in particular to a detection method for small target images. Background Art
[0002] To date, object detection and recognition remains a highly-regarded research topic in the field of computer vision, and its broad application prospects have promoted the rapid development and technological innovation in this field.
[0003] At present, general neural networks have strong detection and recognition capabilities for common target recognition tasks. However, existing neural networks often face challenges in detecting small targets with a pixel size of less than 32*32, because they cannot effectively capture and represent the features of small targets, resulting in poor detection performance.
[0004] At the same time, the evaluation mechanism based on the intersection over union (IOU) is not suitable for small target detection, which will cause a large number of potential small targets to be misjudged or ignored. In order to improve the accuracy and precision of small target detection, high-resolution images are used. This method will increase the amount of neural network calculations and reduce the efficiency of target detection.
[0005] Therefore, there is an urgent need for a technical solution for detecting and identifying small targets. Summary of the invention
[0006] In view of the above analysis, an embodiment of the present invention aims to provide a detection method for small target images, so as to solve the problems of difficulty in extracting small target features and low detection efficiency in the prior art.
[0007] An embodiment of the present invention provides a detection method for a small object image, the detection method comprising:
[0008] Preprocessing the image to be detected to obtain the image to be detected in a set format;
[0009] Input the image to be detected into the pre-trained image detection model to obtain the predicted target result of the image to be detected; wherein the image detection model is an improved YOLOv8 neural network structure; the neck network in the improved YOLOv8 neural network structure includes a sparse matrix module and an original neck network; the original neck network is used to fuse the feature maps output by the backbone network except the shallowest feature map; the sparse matrix module is used to fuse the shallowest feature map and the deepest feature map output by the backbone network, and the fused feature map and the fused feature map obtained by the original neck network are further fused in the neck network and then output;
[0010] The predicted target results are parsed to determine the number of targets, target categories, and target bounding boxes in the image to be detected, and displayed on a visualization screen.
[0011] Based on the further improvement of the above detection method, the image detection model includes:
[0012] The backbone network is used to extract features from the input image to be detected, and input the obtained second feature map, third feature map, fourth feature map and fifth feature map from shallow to deep into the neck network;
[0013] The neck network also includes a dynamic weighted fusion module; the sparse matrix module fuses the second feature map and the fifth feature map to output a tenth feature map, and the original neck network fuses the third to fifth feature maps to obtain an eleventh feature map and a twelfth feature map;
[0014] The dynamic weighted fusion module is used to fuse the tenth feature map and the twelfth feature map to output a thirteenth feature map;
[0015] The original neck network is also used to further fuse the thirteenth feature map, the eleventh feature map and the fifth feature map to obtain the sixth feature map, the seventh feature map and the eighth feature map, and input the sixth feature map, the seventh feature map and the eighth feature map into the detection head network;
[0016] The detection head network is used to obtain the predicted target result of the image to be detected according to the sixth feature map, the seventh feature map and the eighth feature map; wherein the predicted target result includes the predicted target category and the predicted target bounding box.
[0017] Based on the further improvement of the above detection method, the sparse matrix module includes a double upsampling layer, a nonlinear activation layer, a first upsampling layer, a multiplication module and a first convolution module;
[0018] A double upsampling layer, used to upsample the fifth feature map of the input twice, and input the twice upsampled feature map into the nonlinear activation layer;
[0019] A non-linear activation layer is used to perform non-linear activation on the input feature map to obtain a sparse matrix and input it into the first upsampling layer;
[0020] A first upsampling layer, used for upsampling the sparse matrix to obtain a ninth feature map and inputting it into a multiplication module;
[0021] A multiplication module, used for multiplying the input second feature map and the ninth feature map, and inputting the multiplied feature map into the first convolution module;
[0022] The first convolution module is used to downsample and nonlinearly represent the input feature map to obtain a tenth feature map.
[0023] Based on the further improvement of the above detection method, the original neck network is used to upsample the fifth feature map, the upsampled fifth feature map is spliced and fused with the fourth feature map, and the spliced and fused feature map is subjected to a residual convolution operation to obtain an eleventh feature map;
[0024] The eleventh feature map is upsampled, and the obtained feature map is concatenated and fused with the third feature map to obtain the twelfth feature map.
[0025] Based on the further improvement of the above detection method, the dynamic weighted fusion module is used to fuse the tenth feature map and the twelfth feature map to output a thirteenth feature map, including:
[0026] Obtaining a first learnable parameter corresponding to the tenth feature map and a second learnable parameter corresponding to the twelfth feature map; wherein the sum of the first learnable parameter and the second learnable parameter is 1;
[0027] A thirteenth feature map is obtained by summing a product of the tenth feature map and the first learnable parameter and a product of the twelfth feature map and the second learnable parameter, wherein the first learnable parameter and the second learnable parameter are obtained through network training.
[0028] Based on the further improvement of the above detection method, in the original neck network,
[0029] The thirteenth feature map is subjected to a residual convolution operation to obtain the sixth feature map.
[0030] Based on the further improvement of the above detection method, in the original neck network,
[0031] The sixth feature map is downsampled and nonlinearly represented by the second convolution module, and the obtained feature map is spliced with the eleventh feature map. The spliced feature map is subjected to a residual convolution operation to obtain the seventh feature map.
[0032] Based on the further improvement of the above detection method, in the original neck network,
[0033] The seventh feature map is downsampled and nonlinearly represented by the second convolution module, and the obtained feature map is spliced with the fifth feature map. The spliced feature map is subjected to a residual convolution operation to obtain the eighth feature map.
[0034] Based on the further improvement of the above detection method, the detection head network includes a first detection head, a second detection head, a third detection head and a post-processing module;
[0035] The first detection head, the second detection head, and the third detection head obtain corresponding prediction box results according to the sixth feature map, the seventh feature map, and the eighth feature map respectively;
[0036] Among them, each prediction box result includes the center point coordinates, width, height, confidence of each category and the probability of containing the target of the prediction box;
[0037] The post-processing module is used to filter the prediction box results to obtain the prediction target results.
[0038] Based on the further improvement of the above detection method, the prediction box result is screened to obtain the predicted target result, including:
[0039] Delete the prediction boxes whose probability of containing the target in the prediction box results does not meet the target threshold;
[0040] For the remaining prediction boxes, traverse all combinations of two prediction boxes and calculate the intersection-over-union ratio of the two prediction boxes in each combination; when the intersection-over-union ratio is greater than the same target threshold, compare the highest category confidences of the two prediction boxes and delete the prediction box corresponding to the smaller highest category confidence; when the intersection-over-union ratio is less than or equal to the same target threshold, both prediction boxes are retained;
[0041] The category corresponding to the highest category confidence in the remaining prediction box is used as the predicted target category of the predicted target result, and the center point coordinates, width and height of the prediction box are used as the predicted target bounding box of the predicted target result.
[0042] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0043] 1. The shallowest feature map and the deepest feature map output by the backbone network are fused through the sparse matrix module of the improved YOLOv8 neural network structure. During the fusion, only the feature map area where the target may exist is calculated. The efficiency of the model prediction is improved as much as possible while ensuring the model prediction effect, so that the model can quickly detect and identify the image;
[0044] 2. The learnable parameters corresponding to the tenth feature map and the twelfth feature map are obtained through network training. The learnable parameters can be adaptively adjusted by fully considering the border sizes of different images. It is not only suitable for the detection of large target images, but also more conducive to improving the sensitivity of small target image detection.
[0045] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0047] Figure 1 A schematic diagram of a flow chart of a method for detecting small target images provided by an embodiment of the present invention;
[0048] Figure 2 A schematic diagram of the structure of an image detection model provided by an embodiment of the present invention;
[0049] Figure 3 A schematic diagram of the structure of a sparse matrix module provided in an embodiment of the present invention;
[0050] Figure 4 A schematic diagram of the structure of a detection head and a post-processing module provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0052] A specific embodiment of the present invention discloses a detection method for small target images, such as Figure 1 As shown, the detection method comprises:
[0053] Step S1: preprocessing the image to be detected to obtain the image to be detected in a set format;
[0054] Step S2: input the image to be detected into a pre-trained image detection model to obtain a predicted target result of the image to be detected; wherein the image detection model is an improved YOLOv8 neural network structure; the neck network in the improved YOLOv8 neural network structure includes a sparse matrix module and an original neck network; the original neck network is used to fuse other feature maps output by the backbone network except the shallowest feature map; the sparse matrix module is used to fuse the shallowest feature map and the deepest feature map output by the backbone network, and the fused feature map and the fused feature map obtained by the original neck network are further fused in the neck network and then output;
[0055] Step S3: Analyze the predicted target results, determine the number of targets, target categories and target bounding boxes in the image to be detected, and display them on the visualization screen.
[0056] Specifically, Figure 1As shown, an image to be detected is obtained and preprocessed. The preprocessing method may be graying, geometric transformation, image enhancement and filtering, etc., which helps to eliminate irrelevant information, restore real information, enhance the detectability of relevant information, and simplify the data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition.
[0057] Specifically, after preprocessing the image to be detected in step S1, an image to be detected in a set format is obtained.
[0058] It is worth noting that, in the present invention, the image to be detected needs to be identified to determine the category of each target and the bounding box of each target included in the image to be detected.
[0059] Specifically, Figure 1 As shown, in step S2, the image to be detected that has been preprocessed in step S1 is input into a pre-trained image detection model to obtain a predicted target result of the image to be detected. The predicted target result includes all targets included in the image to be detected, as well as bounding box information and category information of each target.
[0060] Specifically, the image detection model is an improved YOLOv8 neural network structure, that is, a neural network structure improved on the basis of the YOLOv8 neural network structure. In the improved YOLOv8 neural network structure, the neck network includes a sparse matrix module in addition to the original neck network.
[0061] Specifically, in the image detection model, the backbone network is used to extract multi-scale features of the image to be detected, and then output to the neck network. In the neck network, the original neck network fuses the feature maps output by the backbone network except the shallowest feature map; the sparse matrix module is used to fuse the shallowest feature map and the deepest feature map output by the backbone network, and the fused feature map is further fused with the fused feature map generated by the original neck network and output to the detection head network. The detection head network predicts based on the multiple fused feature maps output by the neck network to obtain the predicted target result of the image to be detected.
[0062] Specifically, the image detection model outputs the predicted target result of the image to be detected obtained in the detection head.
[0063] Preferably, if Figure 2 As shown, the image detection model includes:
[0064] The backbone network is used to extract features from the input image to be detected, and input the obtained second feature map, third feature map, fourth feature map and fifth feature map from shallow to deep into the neck network;
[0065] The neck network also includes a dynamic weighted fusion module; the sparse matrix module fuses the second feature map and the fifth feature map to output a tenth feature map, and the original neck network fuses the third to fifth feature maps to obtain an eleventh feature map and a twelfth feature map;
[0066] The dynamic weighted fusion module is used to fuse the tenth feature map and the twelfth feature map to output a thirteenth feature map;
[0067] The original neck network is also used to further fuse the thirteenth feature map, the eleventh feature map and the fifth feature map to obtain the sixth feature map, the seventh feature map and the eighth feature map, and input the sixth feature map, the seventh feature map and the eighth feature map into the detection head network;
[0068] The detection head network is used to obtain the predicted target result of the image to be detected according to the sixth feature map, the seventh feature map and the eighth feature map; wherein the predicted target result includes the predicted target category and the predicted target bounding box.
[0069] Specifically, Figure 2 As shown, the preprocessed image to be detected is input into the backbone network of the image detection model, and features of the image to be detected are extracted in the backbone network to obtain a first feature map P1, a second feature map P2, a third feature map P3, a fourth feature map P4 and a fifth feature map P5. For example, Figure 2 As shown, a 640*640*3 image to be detected is input into the backbone network, and a 320*320*64 feature map is generated through the convolution module ConvModule as the first feature map P1; the first feature map P1 is passed through the convolution module ConvModule to generate a 160*160*128 feature map, and then passed through the residual convolution module CSPLayer to generate a 160*160*128 feature map as the second feature map P2; the second feature map P2 is passed through the convolution module ConvModule and the residual convolution module CSPLayer to generate a 80*80*256 feature map as the third feature map P3; the third feature map is passed through the convolution module ConvModule and the residual convolution module CSPLayer to generate a 40*40*512 feature map as the fourth feature map P4; the fourth feature map P4 is passed through the convolution module ConvModule, the residual convolution module CSPLayer and the spatial pyramid pooling module SPPF to generate a 20*20*512 feature map as the fifth feature map.
[0070] It can be understood that SPPF is fixed-space pyramid pooling, which further processes the deep feature map through multi-layer pooling and splicing operations without changing the shape of the input feature map.
[0071] Specifically, Figure 2As shown, the backbone network inputs the obtained second feature map P2, third feature map P3, fourth feature map P4 and fifth feature map P5 into the neck network.
[0072] Specifically, the neck network includes an original neck network, a sparse matrix module and a dynamic weighted fusion module; the original neck network receives the third feature map P3, the fourth feature map P4 and the fifth feature map P5, and fuses them to obtain the eleventh feature map P11 and the twelfth feature map P12; the sparse matrix module receives the second feature map P2 and the fifth feature map P5 output by the backbone network and fuses them to obtain the tenth feature map P10; the dynamic weighted fusion module receives the tenth feature map P10 output by the sparse matrix module and the twelfth feature map P12 output by the original neck network, and fuses them to obtain the thirteenth feature map P13.
[0073] It is worth mentioning that the shallow high-resolution feature map P2 can effectively extract the features of small targets, and at the same time, the sparse matrix can be used to reduce the amount of calculation and achieve faster acquisition of the features of small targets.
[0074] Specifically, in the original neck network, the thirteenth feature map P13, the eleventh feature map P11 and the fifth feature map P5 are fused to obtain the sixth feature map P6, the seventh feature map P7 and the eighth feature map P8, and the sixth feature map P6, the seventh feature map P7 and the eighth feature map P8 are input into the detection head network.
[0075] Specifically, in the detection head network, the predicted target results corresponding to the image to be detected are obtained according to the sixth feature map P6, the seventh feature map P7 and the eighth feature map P8; the predicted target results include the predicted target category and the predicted target bounding box, that is, the category and bounding box of each target in the image to be detected.
[0076] Preferably, if Figure 3 As shown, the sparse matrix module includes a double upsampling layer, a nonlinear activation layer, a first upsampling layer, a multiplication module and a first convolution module;
[0077] A double upsampling layer, used to upsample the fifth feature map of the input twice, and input the twice upsampled feature map into the nonlinear activation layer;
[0078] A non-linear activation layer is used to perform non-linear activation on the input feature map to obtain a sparse matrix and input it into the first upsampling layer;
[0079] The first upsampling layer is used to upsample the sparse matrix to obtain the ninth feature map and input it into the multiplication module;
[0080] A multiplication module, used for multiplying the input second feature map and the ninth feature map, and inputting the multiplied feature map into the first convolution module;
[0081] The first convolution module is used to downsample and nonlinearly represent the input feature map to obtain a tenth feature map.
[0082] Specifically, Figure 3 As shown, the fifth feature map P5 is input into the sparse matrix module, and passes through the double upsampling layer, the nonlinear activation layer, and the first upsampling layer in sequence to obtain the ninth feature map P9; the second feature map P2 and the ninth feature map P9 input into the sparse matrix module are multiplied point by point in the multiplication module, and the multiplied feature map is output to the first convolution module ConvModule1 to obtain the tenth feature map P10.
[0083] For example, Figure 3 As shown, the fifth feature map P5 is 20*20*512, and is upsampled twice in the double upsampling layer to obtain a feature map 80*80*512, which only changes the size of the feature map, and does not change the number of channels of the feature map; the feature map 80*80*512 is nonlinearly activated in the nonlinear activation layer to obtain a feature map 80*80*512 as a sparse matrix; the sparse matrix 80*80*512 is input to the first upsampling layer for upsampling to obtain 160*160*128, as the ninth feature map P9; the second feature map P2 is 160*160*128, and the ninth feature map P9 and the second feature map P2 are multiplied point by point in the multiplication module to obtain a feature map 160*160*128; the feature map 160*160*128 after point-by-point multiplication is downsampled and nonlinearly represented in the first convolution module ConvModule1 to obtain 80*80*768, as the tenth feature map P10.
[0084] Preferably, if Figure 2 As shown, the original neck network is used to upsample the fifth feature map, and the upsampled fifth feature map is concatenated and fused with the fourth feature map. The concatenated and fused feature map is subjected to a residual convolution operation to obtain an eleventh feature map;
[0085] The eleventh feature map is upsampled, and the obtained feature map is concatenated and fused with the third feature map to obtain the twelfth feature map.
[0086] For example, Figure 2As shown, the fifth feature map P5 is input to the original neck network, and is upsampled by the second upsampling layer in the original neck network to obtain a feature map of 40*40*512, and then passes through splicing 2, splicing 2 receives the fourth feature map P4 input by the backbone network, and in splicing 2, the fourth feature map P4 is spliced with the feature map 40*40*512 after the upsampling layer to obtain a feature map 40*40*1024, that is, the size remains unchanged in the splicing layer, and the channels are spliced. The feature map 40*40*1024 obtained by splicing 2 is input to the residual convolution layer CSPLayer for residual convolution operation to obtain a feature map 40*40*512 as the eleventh feature map P11. The eleventh feature map is upsampled by the second upsampling layer to obtain a feature map of 80*80*512, and then passes through splicing 1. Splicing 1 also receives the third feature map P3 output by the backbone network. In splicing 3, the feature map 80*80*512 and the third feature map P3 are spliced and fused to obtain a feature map of 80*80*768 as the twelfth feature map.
[0087] Preferably, if Figure 2 As shown, the dynamic weighted fusion module is used to fuse the tenth feature map and the twelfth feature map to output a thirteenth feature map, including:
[0088] Obtaining a first learnable parameter corresponding to the tenth feature map and a second learnable parameter corresponding to the twelfth feature map; wherein the sum of the first learnable parameter and the second learnable parameter is 1;
[0089] A thirteenth feature map is obtained by summing a product of the tenth feature map and the first learnable parameter and a product of the twelfth feature map and the second learnable parameter, wherein the first learnable parameter and the second learnable parameter are obtained through network training.
[0090] Specifically, Figure 2 As shown, the tenth feature map P10 and the twelfth feature map P12 are fused in the dynamic weighted fusion module to obtain the thirteenth feature map P13.
[0091] For example, Figure 2 As shown, the pre-trained first learnable parameter α and the second learnable parameter (1-α) are determined, the tenth feature map P10 is point-by-point multiplied with the first learnable parameter α, the twelfth feature map P12 is point-by-point multiplied with the second learnable parameter (1-α), and the tenth feature map P10 and the twelfth feature map P12 after the point-by-point multiplication are added and summed to obtain a feature map 80*80*768, which is used as the thirteenth feature map P13.
[0092] Preferably, if Figure 2 As shown, in the original neck network,
[0093] The thirteenth feature map is subjected to a residual convolution operation to obtain the sixth feature map.
[0094] Preferably, if Figure 2 As shown, in the original neck network,
[0095] The sixth feature map is downsampled and nonlinearly represented by the second convolution module, and the obtained feature map is spliced with the eleventh feature map. The spliced feature map is subjected to a residual convolution operation to obtain the seventh feature map.
[0096] Preferably, if Figure 2 As shown, in the original neck network,
[0097] The seventh feature map is downsampled and nonlinearly represented by the second convolution module, and the obtained feature map is spliced with the fifth feature map. The spliced feature map is subjected to a residual convolution operation to obtain the eighth feature map.
[0098] Specifically, Figure 2 As shown, in the original neck network, the thirteenth feature map P13 is received from the dynamic weighted fusion module, and the thirteenth feature map P13 is subjected to residual convolution operation by the residual convolution module CSPLayer to obtain a feature map 80*80*256 as the sixth feature map P6; the sixth feature map P6 is subjected to the second convolution module ConvModule2 to obtain a feature map 40*40*256, which is output to splicing 3; the eleventh feature map P11 is received in splicing 3, and is spliced and fused with the feature map 40*40*256 to obtain a feature map 40*40*768, which is then subjected to the residual convolution layer CS PLayer obtains a feature map of 40*40*512 as the seventh feature map P7; the seventh feature map P7 passes through the second convolution module ConvModule2 to obtain a feature map of 20*20*512, which is input to splicing 4. Splicing 4 receives the fourth feature map P4 output by the backbone network. In splicing 4, the fifth feature map P5 and the feature map 20*20*512 are spliced and fused to obtain a feature map of 20*20*1024. The feature map 20*20*1024 passes through the residual convolution module CSPLayer to obtain a feature map of 20*20*512 as the eighth feature map P8. The sixth feature map P6, the seventh feature map and the eighth feature map P8 are output to the detection head network and predicted in the detection head network.
[0099] It is worth noting that in the neck network, the second convolution module ConvModule2 downsamples and nonlinearly represents the input feature map, the residual convolution module performs residual convolution operation on the input feature map, the second upsampling layer upsamples the input feature map, splicing 1, splicing 2 and splicing 3 respectively splice and fuse the two input feature maps, and finally obtain the sixth feature map P6, the seventh feature map P7 and the eighth feature map P8.
[0100] Preferably, if Figure 2 As shown, the detection head network includes a first detection head, a second detection head, a third detection head and a post-processing module;
[0101] The first detection head, the second detection head, and the third detection head obtain corresponding prediction box results according to the sixth feature map, the seventh feature map, and the eighth feature map respectively;
[0102] Among them, each prediction box result includes the center point coordinates, width, height, confidence of each category and the probability of containing the target of the prediction box;
[0103] The post-processing module is used to filter the prediction box results to obtain the prediction target results.
[0104] Specifically, Figure 2 As shown, the first detection head Head1 makes predictions based on the input sixth feature map P6, the second detection head Head2 makes predictions based on the input seventh feature map P7, and the third detection head Head3 makes predictions based on the input eighth feature map P8.
[0105] Specifically, Figure 4 As shown in the figure, the input feature map passes through the convolution module CBS and the convolution layer Conv2D in sequence. The convolution module CBS downsamples and nonlinearly represents the input feature map, and the convolution layer Conv2D performs a convolution operation on the input feature map to change the number of channels of the feature map.
[0106] Specifically, Head represents any one of the first detection head Head1, the second detection head Head2 or the third detection head Head3. Exemplarily, when the first detection head Head1 is represented, the frame prediction result 80*80*4 is predicted according to the sixth feature map P6, where 80*80 represents the predicted 6400 prediction boxes, 4 represents the center point coordinates, width and height of each prediction box, and the category prediction result 80*80*(cls_num+1) is predicted according to the sixth feature map P6, where 80*80 represents the 6400 prediction boxes, cls_num represents the confidence of each category, and +1 represents the probability of containing the target in each prediction box.
[0107] Specifically, the three detection heads, the first detection head, the second detection head and the third detection head, obtain a total of 6400+1600+400 prediction box results, each of which includes the center point coordinates, width, height, confidence of each category and the probability of containing the target. Figure 4 As shown, the prediction box result of 8400*(cls_num+5) is input into the post-processing module, and in the post-processing module, the prediction box result is screened to obtain the final prediction target result.
[0108] Preferably, the prediction box results are screened to obtain the predicted target results, including:
[0109] Delete the prediction boxes whose probability of containing the target in the prediction box results does not meet the target threshold;
[0110] For the remaining prediction boxes, traverse all combinations of two prediction boxes and calculate the intersection-over-union ratio of the two prediction boxes in each combination; when the intersection-over-union ratio is greater than the same target threshold, compare the highest category confidences of the two prediction boxes and delete the prediction box corresponding to the smaller highest category confidence; when the intersection-over-union ratio is less than or equal to the same target threshold, both prediction boxes are retained;
[0111] The category corresponding to the highest category confidence in the remaining prediction box is used as the predicted target category of the predicted target result, and the center point coordinates, width and height of the prediction box are used as the predicted target bounding box of the predicted target result.
[0112] Specifically, in the post-processing module, it is first determined whether the probability of each prediction box containing a target is greater than a preset target threshold. If it is greater than the target threshold, the prediction box is considered to contain a target. Otherwise, the prediction box is considered to not contain a target and the prediction box is deleted.
[0113] For the remaining prediction frames, the combination of any two prediction frames is judged to see whether the two prediction frames correspond to the same target. Specifically, the intersection-over-union ratio of the two prediction frames is calculated. If the intersection-over-union ratio is greater than the preset threshold for the same target, the two prediction frames are considered to correspond to the same target, otherwise they correspond to different targets. When the two prediction frames correspond to the same target, the category confidences corresponding to the two prediction frames are checked, and the prediction frame with the smaller highest category confidence is deleted, and the prediction frame with the larger highest category confidence is retained. That is, when the two prediction frames correspond to the same target, the prediction frame with the higher category confidence is retained.
[0114] Specifically, among the remaining prediction boxes after screening, each prediction box corresponds to the center point coordinates, width and height, which serves as the predicted target bounding box of the target; the category corresponding to the highest category confidence corresponding to each prediction box is used as the predicted target category of the target.
[0115] Specifically, the image detection model is trained through the following steps:
[0116] ① Obtain the AI-TOD2 dataset for screening so that each sample includes small targets. It is worth noting that a small target is a target whose width is less than 1 / 20 of the image width, or whose height is less than 1 / 20 of the image height, or whose area is less than 1 / 400 of the image area in an image. Divide the screened dataset into training set, test set, and validation set.
[0117] ② Randomly select multiple samples from the training set as training samples for each iteration round, input them into the improved YOLOv8 neural network structure with initialized parameters, and use the loss back propagation algorithm to update the parameters of the improved YOLOv8 neural network structure. At the end of each iteration round, the improved YOLOv8 neural network structure is evaluated through the validation set.
[0118] ③ After 100 rounds of training, based on the evaluation of each round of training, the improved YOLOv8 neural network structure with the highest evaluation is selected as the trained image detection model.
[0119] Specifically, Figure 1 As shown, in step S3, the predicted target results of the image to be detected are analyzed to determine the number of targets included in the image to be detected, as well as the category of each target and the bounding box of each target, and the information of the predicted target results is displayed on a visualization screen for user viewing.
[0120] Specifically, the predicted target result includes multiple targets, each of which corresponds to a predicted target category, center point coordinates, width and height. After parsing, the number of targets in the image to be detected, the bounding box of each target and the category of each target can be obtained.
[0121] Compared with the prior art, a detection method for small target images provided by an embodiment of the present invention fuses the shallowest feature map and the deepest feature map output by a backbone network through a sparse matrix module of an improved YOLOv8 neural network structure. During the fusion, only the feature map area where the target may exist is calculated, so as to improve the efficiency of model prediction while ensuring the model prediction effect as much as possible, so that the model can quickly detect and identify the image; at the same time, the learnable parameters corresponding to the tenth feature map and the twelfth feature map are obtained through network training, and the learnable parameters can be adaptively adjusted by fully considering the border size of different images. It is not only suitable for the detection of large target images, but also more conducive to improving the sensitivity of small target image detection.
[0122] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0123] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A detection method for small target images, characterized in that: The detection method comprises: Preprocessing the image to be detected to obtain the image to be detected in a set format; Input the image to be detected into the pre-trained image detection model to obtain the predicted target result of the image to be detected; wherein the image detection model is an improved YOLOv8 neural network structure; the neck network in the improved YOLOv8 neural network structure includes a sparse matrix module and an original neck network; the original neck network is used to fuse the feature maps output by the backbone network except the shallowest feature map; the sparse matrix module is used to fuse the shallowest feature map and the deepest feature map output by the backbone network, and the fused feature map and the fused feature map obtained by the original neck network are further fused in the neck network and then output; The predicted target results are parsed to determine the number of targets, target categories, and target bounding boxes in the image to be detected, and displayed on a visualization screen.
2. The detection method according to claim 1, characterized in that: The image detection model includes: The backbone network is used to extract features from the input image to be detected, and input the obtained second feature map, third feature map, fourth feature map and fifth feature map from shallow to deep into the neck network; The neck network also includes a dynamic weighted fusion module; the sparse matrix module fuses the second feature map and the fifth feature map to output a tenth feature map, and the original neck network fuses the third to fifth feature maps to obtain an eleventh feature map and a twelfth feature map; The dynamic weighted fusion module is used to fuse the tenth feature map and the twelfth feature map to output a thirteenth feature map; The original neck network is also used to further fuse the thirteenth feature map, the eleventh feature map and the fifth feature map to obtain the sixth feature map, the seventh feature map and the eighth feature map, and input the sixth feature map, the seventh feature map and the eighth feature map into the detection head network; The detection head network is used to obtain the predicted target result of the image to be detected according to the sixth feature map, the seventh feature map and the eighth feature map; wherein the predicted target result includes the predicted target category and the predicted target bounding box.
3. The detection method according to claim 2, characterized in that: The sparse matrix module includes a double upsampling layer, a nonlinear activation layer, a first upsampling layer, a multiplication module and a first convolution module; A double upsampling layer, used to upsample the fifth feature map of the input twice, and input the twice upsampled feature map into the nonlinear activation layer; A non-linear activation layer is used to perform non-linear activation on the input feature map to obtain a sparse matrix and input it into the first upsampling layer; A first upsampling layer, used for upsampling the sparse matrix to obtain a ninth feature map and inputting it into a multiplication module; A multiplication module, used for multiplying the input second feature map and the ninth feature map, and inputting the multiplied feature map into the first convolution module; The first convolution module is used to downsample and nonlinearly represent the input feature map to obtain a tenth feature map.
4. The detection method according to claim 3, characterized in that: The original neck network is used to upsample the fifth feature map, and the upsampled fifth feature map is spliced and fused with the fourth feature map. The spliced and fused feature map is subjected to a residual convolution operation to obtain an eleventh feature map; The eleventh feature map is upsampled, and the obtained feature map is concatenated and fused with the third feature map to obtain the twelfth feature map.
5. The detection method according to any one of claims 2 to 4, characterized in that: The dynamic weighted fusion module is used to fuse the tenth feature map and the twelfth feature map to output a thirteenth feature map, including: Obtaining a first learnable parameter corresponding to the tenth feature map and a second learnable parameter corresponding to the twelfth feature map; wherein the sum of the first learnable parameter and the second learnable parameter is 1; A thirteenth feature map is obtained by summing a product of the tenth feature map and the first learnable parameter and a product of the twelfth feature map and the second learnable parameter, wherein the first learnable parameter and the second learnable parameter are obtained through network training.
6. The detection method according to claim 5, characterized in that: In the original neck network, The thirteenth feature map is subjected to a residual convolution operation to obtain the sixth feature map.
7. The detection method according to claim 6, characterized in that: In the original neck network, The sixth feature map is downsampled and nonlinearly represented by the second convolution module, and the obtained feature map is spliced with the eleventh feature map. The spliced feature map is subjected to a residual convolution operation to obtain the seventh feature map.
8. The detection method according to claim 7, characterized in that: In the original neck network, The seventh feature map is downsampled and nonlinearly represented by the second convolution module, and the obtained feature map is spliced with the fifth feature map. The spliced feature map is subjected to a residual convolution operation to obtain the eighth feature map.
9. The detection method according to claim 2, characterized in that: The detection head network includes a first detection head, a second detection head, a third detection head and a post-processing module; The first detection head, the second detection head, and the third detection head obtain corresponding prediction box results according to the sixth feature map, the seventh feature map, and the eighth feature map respectively; Among them, each prediction box result includes the center point coordinates, width, height, confidence of each category and the probability of containing the target of the prediction box; The post-processing module is used to filter the prediction box results to obtain the prediction target results.
10. The detection method according to claim 9, characterized in that: The prediction frame results are screened to obtain the prediction target results, including: Delete the prediction boxes whose probability of containing the target in the prediction box results does not meet the target threshold; For the remaining prediction boxes, traverse all combinations of two prediction boxes and calculate the intersection-over-union ratio of the two prediction boxes in each combination; when the intersection-over-union ratio is greater than the same target threshold, compare the highest category confidences of the two prediction boxes and delete the prediction box corresponding to the smaller highest category confidence; when the intersection-over-union ratio is less than or equal to the same target threshold, both prediction boxes are retained; The category corresponding to the highest category confidence in the remaining prediction box is used as the predicted target category of the predicted target result, and the center point coordinates, width and height of the prediction box are used as the predicted target bounding box of the predicted target result.