Small target detection method based on slice-assisted reasoning and adaptive spatial feature fusion

Abstract

This invention relates to a small target detection method based on slice-assisted inference and adaptive spatial feature fusion, belonging to the field of computer vision. The method first processes the original image data through a slice-assisted inference module to slice the original image. Then, the sliced ​​image blocks are fed into the input of an improved YOLOv5 network model. After passing through Conv and C3 in the backbone, and with CCSA modules added to the backbone and neck, respectively, the neck and head are subjected to adaptive spatial feature fusion using the FPN+ASFF module. The fused features are then fed into the output, and an improved Alpha-ICIoU loss function is used to evaluate the model and provide feedback for parameter tuning. This invention improves the accuracy of small target detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically the field of object detection, and specifically relates to a small object detection method based on slice-assisted reasoning and adaptive spatial feature fusion. Background Technology

[0002] In recent years, object detection has become a key area of ​​computer vision and digital image processing, widely applied in various fields such as face detection, industrial software testing, video surveillance, autonomous vehicles, and aerospace. In this field, many methods employing deep learning architectures have achieved high accuracy, including two-stage region proposal methods such as Fast R-CNN, Faster R-CNN, and Cascade R-CNN; and single-stage detectors such as RetinaNet, SSD, and YOLO. Generally, single-stage methods are faster than two-stage methods, while the latter offer higher accuracy. These recent detectors are trained and evaluated on well-known datasets such as ImageNet, Pascal VOC12, and MS COCO. These datasets mostly involve low-resolution images (640×480) containing fairly large objects and large pixel coverage (averaging 60% of the image height). While trained models have successful detection performance on these types of input data, in high-resolution images generated by high-end drones and surveillance cameras, very small objects may only contain a few pixels within their bounding boxes, lacking sufficient detail, making them difficult for traditional detectors to detect. Therefore, the accuracy of the aforementioned detectors for small object detection tasks is significantly lower.

[0003] Recent advances in drone, 4K camera, and deep learning research have enabled long-range target detection that meets the Detect, Observe, and Recognize (DORI) standard. The DORI standard defines minimum pixel heights for objects for different tasks: 10% of the image height is required for object detection, and 20% (108 pixels in full HD video) is required for object recognition. This relatively small pixel coverage limits the capabilities of CNN-based object detection methods; furthermore, high-resolution images place higher demands on memory.

[0004] Therefore, there is an urgent need for a method that can improve the detection accuracy of small targets. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a small target detection method based on slice-assisted reasoning and adaptive spatial feature fusion, so as to solve the problem of low accuracy in small target detection in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A small target detection method based on slice-assisted reasoning and adaptive spatial feature fusion, the method includes the following steps:

[0008] S1: The original image data is processed by the slice-assisted reasoning module to perform slice operations on the original image;

[0009] S2: The sliced ​​image patches are fed into the input of the improved YOLOv5 network model. After passing through Conv and C3 in the backbone, and adding CCSA modules in the backbone and neck, the features of the neck and head are adaptively fused using the FPN+ASFF module. The fused features are then fed into the output, and the improved Alpha-ICIoU loss function is used to evaluate the model and provide feedback for parameter tuning. Here, CCSA represents coordinated channel and spatial attention, and ASFF represents adaptive spatial feature fusion.

[0010] Furthermore, the slice-assisted inference module, the improved YOLOv5 network model backbone, the improved YOLOv5 network model neck, and the improved YOLOv5 network model head are connected sequentially. The output of the slice-assisted inference module is connected to the input of the improved YOLOv5 network model backbone, the output of the improved YOLOv5 network model backbone is connected to the input of the improved YOLOv5 network model neck, and the output of the improved YOLOv5 network model neck is connected to the input of the improved YOLOv5 network model head.

[0011] Furthermore, in step S1, the slice-assisted reasoning module specifically includes: cutting the original image into M×N image blocks, adjusting the size of each image block while maintaining the aspect ratio; feeding the sliced ​​image blocks and the original image into the input of the neural network for independent prediction, and merging overlapping detection boxes through non-maximum suppression.

[0012] Furthermore, in step S2, CCSA is added to the improved YOLOv5 network model in the 9th layer of the backbone and the 19th, 23rd and 27th layers of the neck.

[0013] Furthermore, in step S2, the calculation formula for the channel attention module in the CCSA module is as follows:

[0014]

[0015] Among them, M C (·) represents the channel attention feature of the output, x c (i,j) represents the coordinate (i,j) in the input feature map C×W×H, where C is the number of channels, W is the image width, and H is the image height; and The attention weight is calculated using the following formula:

[0016] g h =σ(F h (f h ))

[0017] g w =σ(F w (f w ))

[0018] Where σ is the sigmoid activation function, f h with f w This indicates that f is divided into two independent tensors along the spatial dimension; F h With F w This is a 1×1 convolution transform function used to transform f h with f w Convert them into tensors with the same number of channels respectively;

[0019]

[0020] Where, x c The original features are the input. This indicates that the input features are averaged in the W direction, and it is the output of the c-th channel with height h. The input features are averaged in the H direction, and the output of the c-th channel with width w is given. F1 represents the shared 1×1 convolution transformation function, and f represents the z-axis. h With z w The output result after nonlinear transformation by the convolution kernel;

[0021] The calculation equation for the spatial attention module is as follows:

[0022] M S (F)=σ(F7([AvgPool(x s MaxPool(x) s )]))

[0023] Among them, M S (F) represents the spatial attention feature of the output, σ is the sigmoid activation function, F7 is the 7×7 convolution transformation function, and x s The features output by the channel attention module are represented by AvgPool and MaxPool, which represent average pooling and max pooling, respectively.

[0024] Furthermore, in step S2, in the improved YOLOv5 network model, the ASFF module replaces the element-wise add operation of the original PANet in YOLOv5. That is, the features from layers 19, 23 and 27 are convolved with 1×1 to adjust the number of channels to be consistent, and then upsampled or upsampled to adjust the feature size from the three layers to be consistent; then multiplied by the weight parameters respectively and the three are connected to obtain the fused features.

[0025] Furthermore, in step S2, the formula for calculating the improved Alpha-ICIoU loss function is as follows:

[0026]

[0027] Among them, L α-ICIOU Indicates Alpha-ICIoU loss;

[0028]

[0029]

[0030] Where IoU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box, w gt h gt and w p h p ρ represents the width and height of the ground truth box and the prediction box, respectively. 2 (b p ,b gt ) represents the Euclidean distance between the centers of the predicted bounding box and the ground truth bounding box; c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes; β is a weighting function; and α is a positive trade-off parameter; v v It is an intermediate value used to measure the difference in aspect ratio.

[0031] The beneficial effects of this invention are as follows: By introducing a slice-assisted inference module, the input image is divided into overlapping small blocks, resulting in a relatively large pixel region for small targets relative to the input network image, while maintaining high memory utilization; by adding a Coordinating Channel and Spatial Attention (CCSA) mechanism, the sensitivity of the model to information channels and space is increased so that the model can better capture global information; by adding an Adaptive Spatial Feature Fusion (ASFF) module, the problem of inconsistent gradient backpropagation when filtering conflicting information in space is solved, the feature scaling invariance is improved, the target detection effect is improved, and the inference cost is reduced; by improving the loss function to the Alpha-CIoU loss function, the accuracy of BBox regression is adaptively improved, and better robustness is provided for small datasets and noisy BBoxes.

[0032] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0034] Figure 1 This is a network structure diagram of the small target detection method based on slice-assisted reasoning and adaptive spatial feature fusion of the present invention;

[0035] Figure 2 This is a flowchart of the small target detection method based on slice-assisted reasoning and adaptive spatial feature fusion of the present invention;

[0036] Figure 3 Here is a block diagram of the principle of the slice-assisted reasoning module;

[0037] Figure 4 A flowchart to aid reasoning by slicing;

[0038] Figure 5 Diagram of the coordination mechanism between channels and spatial attention;

[0039] Figure 6 The structure diagram is as follows: FPN+ASFF;

[0040] Figure 7 A schematic diagram illustrating the calculation of parameters for the Alpha-ICIoU loss function in an example. Detailed Implementation

[0041] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0042] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0043] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0044] Please see Figures 1 to 7 This invention first presents a small target detection method based on slice-assisted inference and adaptive spatial feature fusion. The original image data is first processed by a slice-assisted inference module to slice the original image. The sliced ​​images are then fed into the input of an improved YOLOv5 network model. After passing through the backbone's Conv and C3 modules, as well as the added Coordination Channel and Spatial Attention Mechanism (CCSA) module, the neck and head features are adaptively fused using FPN+ASFF. The fused features are then fed into the output terminal, where the Alpha-ICIoU loss function is used to evaluate the model and provide feedback for parameter tuning.

[0045] The basic implementation process of one embodiment of the present invention is as follows: Based on the Visdrone2019 dataset containing small target datasets with preset labels respectively, which is captured by various drone cameras, the training set and the test set are divided in a 4:1 ratio.

[0046] The original images in the dataset are first processed through a slice-assisted inference module. The sliced ​​images are then fed into the neural network for independent prediction, while the original images undergo full inference. Coordination Channel and Spatial Attention (CCSA) are added to the backbone and neck region, and the Adaptive Spatial Feature Fusion (ASFF) module replaces the PANet in the original YOLOv5. Finally, the loss function CIoU at the output of the YOLOv5 neural network is improved to the Alpha-ICIoU loss function.

[0047] The improved YOLOv5 network structure is as follows: Figure 1 As shown, improvements have been made to the original YOLOv5 version 6.0, with modifications to the input, backbone, neck, and head of the YOLOv5 network structure. For example... Figure 2 As shown, the specific steps for constructing the improved YOLOv5 network structure are as follows:

[0048] S1: Based on the YOLOv5 neural network, a slice-assisted inference (SAHI) module is added to allow the sliced ​​image data to be input into the YOLOv5 neural network.

[0049] S2: Add CCSA modules to the backbone and neck of the YOLOv5 neural network;

[0050] S3: Add an adaptive spatial feature fusion (ASFF) module to the head part of the YOLOv5 neural network to construct an improved YOLOv5 neural network head.

[0051] S4: Change the loss function of the YOLOv5 output from CIoU to Alpha-ICIoU loss function to construct an improved YOLOv5 neural network output;

[0052] S5: Sequentially connect the slice-assisted inference module, the improved YOLOv5 neural network backbone, the improved YOLOv5 neural network neck, and the improved YOLOv5 neural network head. The output of the slice-assisted inference module is connected to the input of the improved YOLOv5 neural network backbone, the output of the improved YOLOv5 neural network backbone is connected to the input of the improved YOLOv5 neural network neck, and the output of the improved YOLOv5 neural network neck is connected to the input of the improved YOLOv5 neural network head.

[0053] S6: Train the improved YOLOv5 network model by feeding the preset dataset into it to obtain the optimal model. Use the optimal model to detect on the test set and obtain the results.

[0054] Slice-assisted reasoning module, such as Figure 3 As shown, the original image I is cut into M×N small image blocks. While maintaining the aspect ratio, the size of each image patch is adjusted so that the image width and height are between 800 and 1333 pixels. Prediction is performed independently for each overlapping image patch. The workflow is as follows: Figure 4 As shown.

[0055] The sliced ​​image blocks pass sequentially through the trunk and neck of YOLOv5.

[0056] In step S2, the Coordination Channel and Spatial Attention (CCSA) mechanism was added to the 9th layer of the skeletal structure and the 19th, 23rd, and 27th layers of the cervical spine, such as... Figure 1 As shown.

[0057] Coordinating channel and spatial attention mechanisms, such as Figure 5 As shown, Figure 5 In this context, C represents the number of channels, W represents the image width, H represents the image height, and r is a constant. The calculation equation for the channel attention module in CCSA is as follows:

[0058]

[0059] Among them, M C (·) represents the channel attention feature of the output, and the input x c (i,j) represents the value of the coordinate (i,j) in the feature map C×W×H. and This represents the attention weight. Its formula is as follows:

[0060] g h =σ(F h (f h ))

[0061] g w =σ(F w (f w ))

[0062] Where σ is the sigmoid activation function, f h with f w This indicates that f is divided into two independent tensors along the spatial dimension. h With F w This is a 1×1 convolution transform function used to transform f h with f w Convert each into a tensor with the same number of channels.

[0063]

[0064] Where, x c The original features are the input. This indicates that the input features are average pooled in the W direction, and the output is the c-th channel of height h. The input features are average pooled in the H direction, and the output is the c-th channel with width w. F1 represents the shared 1×1 convolution transformation function, and f represents z. h With z w The output result after nonlinear transformation by convolution kernel.

[0065] The spatial attention module in CCSA is calculated using the following equation:

[0066] M S (F)=σ(F7([AvgPool(x s MaxPool(x) s )]))

[0067] Among them, M s (F) Spatial attention features of the output, σ is the sigmoid activation function, F7 is the 7×7 convolution transformation function, x s The features output by the channel attention module are represented by AvgPool and MaxPool, which represent average pooling and max pooling, respectively.

[0068] The Adaptive Spatial Feature Fusion (ASFF) module replaces the element-wise add operation in the original PANet of YOLOv5. For features from (… Figure 1 In the middle layer, the features from Layer 19, Layer 23, and Layer 27 are subjected to 1×1 convolution to ensure the number of channels is consistent. Then, upsampling or downsampling is performed to adjust the feature sizes from the three layers to be consistent.

[0069] The adaptive spatial feature fusion process described above involves the following steps:

[0070] S31: As Figure 6 As shown, the features of layers C3, C4, and C5 are processed by the FPN (YOLOv5 backbone) to output features C3′, C4′, and C5′. C5′ is then subjected to a 1×1 convolution to adjust the number of channels to be the same as C3′, followed by a 4x upsampling (4×UpSample), and finally multiplied by the weight parameter α1. C4′ is then subjected to a 1×1 convolution to adjust the number of channels to be the same as C3′, followed by a 2x upsampling (2×UpSample), and finally multiplied by the weight parameter β1. C3′ is multiplied by the weight parameter γ1 and then concatenated with the two previously processed terms to obtain feature P3.

[0071] S32: Perform a 1×1 convolution on C5′ to adjust the number of channels to be the same as C4′, then perform a 2x upsampling (2×UpSample), and finally multiply by the weight parameter α2; multiply C4′ by the weight parameter β2; perform a 1×1 convolution on C3′ to adjust the number of channels to be the same as C4′, then perform downsampling (DownSample) to change the size back to the original. Finally, multiply by the weight parameter γ2; concatenate the three to obtain feature P4.

[0072] S33: Multiply C5′ by the weight parameter α3; perform a 1×1 convolution on C4′ to adjust the number of channels to be the same as C5′, then perform downsampling to change the size back to the original. Finally, multiply by the weight parameter β3; then perform a 1×1 convolution on C3′ to adjust the number of channels to be the same as C5′, and then perform downsampling to change the size back to the original. Finally, multiply by the weight parameter γ3; concatenate the three to obtain feature P5.

[0073] The calculation formulas for steps S31 to S33 above are as follows:

[0074]

[0075] in:

[0076]

[0077]

[0078] by The softmax function is used to define the control parameters. Use a 1×1 convolutional layer from x 1→l x 2→l x 3→l Calculate the weight scalar mapping Therefore, it can be learned through standard backpropagation. This method adaptively aggregates features from all levels at each scale. The output {y1, y2, y3} is used for object detection.

[0079] The improved Alpha-ICIoU standardizes the form of IoU-based methods by adding an α power and renaming CIoU to ICIoU. The Alpha-ICIoU loss function is calculated as follows:

[0080]

[0081] in,

[0082]

[0083]

[0084] Where IoU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box, w gt h gt and w p h p ρ represents the width and height of the ground truth box and the prediction box, respectively. 2 (b p ,b gt ) represents the Euclidean distance between the centers of the predicted bounding box and the ground truth bounding box. c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes, such as... Figure 7As shown. β is a weighting function, α is a positive trade-off parameter, and v v It is an intermediate value used to measure the difference in aspect ratio.

[0085] The partitioned dataset is fed into the improved model (e.g.) Figure 1 The model is trained (as shown) and the optimal model is obtained, which can be used to detect small targets.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A small target detection method based on slice-assisted reasoning and adaptive spatial feature fusion, characterized in that, The method comprises the following steps: S1: the original image data is subjected to a slice auxiliary inference module to perform a slicing operation on the original picture; The slice auxiliary inference module specifically comprises: cutting the original image into MxN image blocks, adjusting the size of each image block while maintaining the aspect ratio; the sliced image blocks and the original image are sent to the input end of the neural network for independent prediction, and the overlapping detection boxes are merged through non-maximum suppression; S2: the sliced image blocks are sent to the input end of the improved YOLOv5 network model, after the Conv, C3 of the backbone part, and the addition of the CCSA module at the 9th layer of the backbone part, and the 19th layer, the 23rd layer and the 27th layer of the neck part, the neck part and the head part adopt the FPN+ASFF module to perform adaptive spatial feature fusion on the features, the fused features are sent to the output end, and the improved Alpha-ICIoU loss function is used to evaluate and feedback the parameters of the model; wherein, CCSA represents coordinated channel and spatial attention, and ASFF represents adaptive spatial feature fusion; The ASFF module adjusts the channel number to be consistent by 1x1 convolution from the features of the 19th layer, the 23rd layer and the 27th layer, and then performs upsampling or upsampling to adjust the feature sizes from the three layers to be consistent; then, the three are multiplied by weight parameters and connected to obtain the fused features; The adaptive spatial feature fusion process comprises the following steps: S31: The features from layers C3, C4, and C5 are output after passing through the FPN. , , ;Will After 1×1 convolution, the number of channels is adjusted and The same result is then upsampled by a factor of 4, and finally multiplied by the weight parameter. ;Will After 1×1 convolution, the number of channels is adjusted and The same result is then upsampled by a factor of 2, and finally multiplied by the weight parameter. ;Will Multiply by weight parameters The feature P3 is obtained by concatenating the two previously processed items. S32: multiply by weight parameter After 1x1 convolution, adjust the number of channels and same, then perform 2 times upsampling, and finally multiply by weight parameter ; multiply by weight parameter ; multiply by weight parameter ; multiply by weight parameter After 1x1 convolution, adjust the number of channels and same, then perform downsampling to change the size to , and finally multiply by weight parameter ; splice the three to obtain feature P4; S33: multiply by a weight parameter ; adjust the number of channels by 1x1 convolution ; and down-sample to change the size to , and finally multiply by a weight parameter ; adjust the number of channels by 1x1 convolution ; and down-sample to change the size to , and finally multiply by a weight parameter ; adjust the number of channels by 1x1 convolution ; and down-sample to change the size to , and finally multiply by a weight parameter ; and splice the three to obtain the feature P5.

2. The small object detection method of claim 1, wherein In step S2, in the CCSA module, the calculation formula of the channel attention module is: wherein, is the output channel attention feature, is the value of the coordinate in the input feature map CxWxH, C is the number of channels, W is the image width, and H is the image height; and is the attention weight, and the calculation formula is as follows: wherein, sigmoid activation function, with denotes splitting f into two independent tensors along the spatial dimension; with is a 1 x 1 convolutional transformation function for transforming with into tensors of the same number of channels, respectively; wherein, is the input raw feature; represents the input feature in the W direction average pooling, which is the output of the cth channel with the height of h ; c represents the input feature in the H direction average pooling, which is the output of the cth channel with the width of w, represents a shared 1x1 convolution transformation function, f represents and the output result after the nonlinear transformation of the convolution kernel;​ The calculation equation of the spatial attention module is as follows: wherein, is the spatial attention feature output, is a sigmoid activation function, is a 7x7 convolutional transformation function, is the feature output by the channel attention module, and denote average pooling and max pooling, respectively.

3. The small object detection method of claim 1, wherein In step S2, the calculation formula of the improved Alpha-ICIoU loss function is: wherein, denotes the Alpha-ICIoU loss; where, IoU denotes the intersection over union of the predicted and ground truth boxes, , and , denotes the width and height of the ground truth and predicted boxes, respectively, denotes the Euclidean distance between the centers of the predicted and ground truth boxes; c denotes the diagonal distance of the smallest closed region that can contain both the predicted and ground truth boxes, is a weight function, is a positive trade-off parameter; is an intermediate value that measures the difference in aspect ratio.

4. The small object detection method of claim 1, wherein The slice auxiliary inference module, the improved YOLOv5 network model backbone, the improved YOLOv5 network model neck, and the improved YOLOv5 network model head are sequentially connected, the output end of the slice auxiliary inference module is connected to the input end of the improved YOLOv5 network model backbone, the output end of the improved YOLOv5 network model backbone is connected to the input end of the improved YOLOv5 network model neck, and the output end of the improved YOLOv5 network model neck is connected to the input end of the improved YOLOv5 network model head.

Images