A small target detection method and system based on feature fusion

Abstract

The application belongs to the field of computer vision, and particularly relates to a small target detection method and system based on feature fusion, which comprises the following steps: inputting an image into a backbone network for feature extraction to obtain a multi-scale feature map; inputting the multi-scale feature map into CIAM for feature information integration to obtain feature information containing target context; cross-layer connecting the output features of the 37th and 50th layers in the backbone network to a fusion network to obtain a fusion feature map combining the original feature information of the target; and inputting the fusion feature map into a detection network to obtain a detection result. The application integrates CIAM into the feature extraction network of yolov7 to integrate the target features and their related background information in the multi-scale feature map, and simultaneously uses a cross-layer feature fusion network to retain the original features of the target, thereby solving the problem of low recognition accuracy of small targets caused by insufficient feature information and positioning error sensitivity.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a small target detection method and system based on feature fusion. Background Technology

[0002] Object detection is a popular research area in computer vision, primarily tasked with detecting and locating target objects from massive amounts of images and videos. As a fundamental task in computer vision fields such as object tracking and behavior recognition, it involves multiple techniques including image preprocessing, feature extraction, and feature fusion. In recent years, with the increasing demand for intelligence across various fields, object detection technology has developed rapidly and is widely used in many areas such as autonomous driving, drone navigation, and medical image analysis.

[0003] Traditional object detection methods identify targets of interest by extracting candidate regions, manually constructing features, and classifying candidate regions. Representative algorithms include Histogram of Oriented Gradients (HOG) and Deformable Component Model (DPM). However, these methods perform poorly in complex scenes and lack robustness when dealing with large changes in target scale, failing to meet the diverse needs of real-world applications. Existing deep learning-based methods divide object detection into three main steps: image feature extraction, multi-scale feature fusion, and target localization and recognition. Algorithms such as R-CNN and the YOLO series automatically perform feature extraction and classification using convolutional neural networks. This end-to-end approach allows for full utilization of large-scale datasets for training, thereby learning more generalizable feature representations.

[0004] In recent years, mainstream object detection models have made significant progress in handling detection tasks in complex environments; however, they often perform poorly in identifying small objects. Specifically, these models suffer from the following problems in small object detection: 1) Existing models employ feature fusion strategies, extracting features from different levels of the original image and fusing them. However, small-scale objects, due to their small area and lack of feature information, are prone to feature loss during the fusion process. 2) The location loss function of existing models does not consider the sensitivity of different scale targets to localization, causing the models to often overlook smaller targets. Therefore, improving the performance of models in small object detection has become an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a small target detection method based on feature fusion, which can effectively enhance the feature representation of small targets and improve the accuracy and recall of small target detection. The method includes: acquiring image data to be detected; inputting the image data into a trained small target detection network; and having the detection network perform classification and regression to output a recognition result. The specific process is as follows:

[0006] The original image data with RGB three channels is input into the backbone network for feature extraction to obtain multi-scale feature maps;

[0007] The multi-scale feature map is input into the Context Information Aggregation Module (CIAM) to integrate the target features with relevant background information, thereby obtaining feature information containing the target context.

[0008] The enhanced multi-scale feature map is input into the Cross Layer Featurefusion Network (CLFNet) for two feature fusions to obtain the fused feature map.

[0009] The fused feature map is input into the corresponding detection head to obtain the target recognition result.

[0010] Furthermore, the process of obtaining target contextual feature information includes:

[0011] The feature maps X extracted from the backbone network at different scales are processed by three 3×3 dilated convolutions with different receptive fields, where Conv1 has an expansion rate of 1 and a receptive field size of 3, Conv2 has an expansion rate of 3 and a receptive field size of 7, and Conv3 has an expansion rate of 5 and a receptive field size of 11; then the Concat operation is used to obtain the fused feature map X′.

[0012] The fused feature map X′ is input into the path aggregation network to integrate feature information, resulting in an enhanced feature map X containing the target context. * .

[0013] Furthermore, the process of obtaining multi-scale fused feature maps includes:

[0014] The original feature maps of layers 37 and 50 in the backbone network that have not undergone context information integration are channel-adjusted using a 1×1 Conv and then fused across layers into the MP-2 module. The fused features are then processed by ELAN to obtain the fused feature map N = {N1, N2, N3}.

[0015] Furthermore, the process of inputting the target recognition result into the detection head includes:

[0016] The feature maps fused at each scale are reparameterized by the RepConv module and input into the detection head to calculate the target's position loss, confidence loss, and classification loss. The position loss is calculated using point distance (Shpae-DotD), and finally, the recognition result is output after non-maximum suppression (NMS).

[0017]

[0018] Shpae-DotD is defined as the normalized Euclidean distance between the center points of the detection box and the ground truth box, used to calculate the target's position loss. This method can effectively reduce the impact of localization errors on small targets, thereby improving the accuracy of the recognition results.

[0019] Another aspect of the present invention provides a small target detection system based on feature fusion, the system comprising: an input module, a feature extraction module, a context information aggregation module, a feature fusion module, a detection module, and an output module;

[0020] The input module is used to acquire the image data to be detected;

[0021] The feature extraction module is used to extract features from image data and generate multi-scale feature maps with different receptive field sizes.

[0022] The context information aggregation module is used to integrate target feature information and relevant background information in the multi-scale feature map to obtain an enhanced feature map;

[0023] The feature fusion module is used to fuse the enhanced feature maps of different scales to obtain a fused feature map;

[0024] The detection module is used to perform target detection on the fused feature map to obtain the detection result;

[0025] The output module is used to output the detection results.

[0026] The beneficial effects of this invention are:

[0027] 1. The Context Information Aggregation (CIAM) module is integrated into the feature extraction network of YOLOv7 to integrate target features and related background information in multi-scale feature maps. By extending convolution and path aggregation, the context features of the target can be effectively extracted, which solves the problem of low recognition accuracy of small targets due to lack of feature information.

[0028] 2. The original YOLOv7 feature fusion network is replaced by a cross-layer feature fusion network (CLFNet) structure. This structure effectively preserves the original features of small-scale targets by connecting unoptimized feature maps in the backbone network across layers to the fusion network, thereby solving the problem of information loss of small targets during the fusion process.

[0029] 3. Replacing the original CIoU position loss function with Shpae-DotD point distance helps address the sensitivity of small targets to localization errors, guiding the model to focus more on the detection of small-scale targets. This significantly improves the accuracy of small target detection. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the present invention;

[0031] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0032] Figure 3 This is a schematic diagram of the Context Information Aggregation Module (CIAM) structure of the present invention;

[0033] Figure 4 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] A small target detection method based on feature fusion, such as Figure 1 As shown, the method includes: acquiring image data to be detected, inputting the image data into a trained small target detection network to obtain recognition results; the small target detection network includes a backbone network, a context information aggregation module (CIAM), a cross-layer feature fusion network (CLFNet), and a detection network; training the small target detection network includes: constructing an original image dataset; inputting the images in the dataset into the backbone network for feature extraction to obtain multi-scale feature maps; inputting the multi-scale feature maps into CIAM for feature information integration to obtain feature information containing target context; inputting the feature information containing target context into CLFNet for multi-scale feature fusion to obtain a fused feature map; inputting the fused feature map into the detection network, using Shpae-DotD to calculate the target's position loss, completing the model training, and obtaining the detection results.

[0036] A specific implementation of a small target detection method based on feature fusion, the method comprising:

[0037] S1: Construct a small target detection network that includes a backbone network, a neck network, and a detection head. This invention uses YOLOv7 as the base network for illustration.

[0038] S2: Input the original image data with RGB three channels into the backbone network of the detection model, and after a series of convolution and pooling operations, obtain multi-scale feature maps with different receptive field sizes;

[0039] S3: For the feature maps of different scales generated by the backbone network in process S2, the context information aggregation module (CIAM) is used to integrate the feature information of the target with the relevant background information to obtain feature information containing the target context;

[0040] S4: Use a cross-layer feature fusion network (CLFNet) to fuse feature maps of different scales. In the first fusion process, the deep feature map with context information enhancement is upsampled to the same size as the shallow feature map and fused. In the second fusion process, the original feature map of the backbone network is connected across layers to the fusion module to obtain fused feature maps for detecting large, medium and small scale targets respectively.

[0041] S5: Input the fused feature map generated in S4 into the detection network. First, it is reparameterized by the RepConv module. Then, the location loss, confidence loss, and classification loss are calculated by the detection head. The location loss is calculated by point distance (Shpae-DotD) and the recognition result is obtained.

[0042] Figure 2 This is a schematic diagram of the network structure of the small target detection method based on feature fusion in this invention. The method uses YOLOv7 as the basic network architecture and includes a feature extraction module, a context information aggregation module, a feature fusion module, and a detection module. Figure 2 In this process, the image data to be detected is processed by the YOLOv7 backbone network to extract features and obtain multi-scale feature maps. The multi-scale feature maps are input into CIAM to obtain context-enhanced target feature information. The enhanced multi-scale features are input into CLFNet for feature information fusion to obtain multi-scale fused features. The fused features are input into a detection head optimized for small targets to obtain target recognition results.

[0043] In this embodiment, the multi-scale feature map extraction of the image using the backbone network includes: using two different CBS modules to alternately perform feature extraction and downsampling processing on the input image to obtain an initial feature map; the CBS module consists of a convolutional layer, a batch normalization layer and a Silu activation layer, wherein the 3×3 convolutional stride CBS module with a stride of 1 is used for feature extraction, and the 3×3 convolutional stride CBS module with a stride of 2 is used for downsampling.

[0044] The initial features are passed sequentially through the ELAN module, MP-1 module, and SPPCSPC module to obtain feature maps B = {B1, B2, B3} at different scales.

[0045] In this embodiment, the ELAN module is used to balance the relationship between the network gradient path length and gradient propagation efficiency to avoid gradient loss caused by channel changes, thereby giving the network stronger learning ability and robustness. This structure consists of four parallel branches. The first and second branches each undergo a 1×1 convolution to adjust the number of channels. The third branch receives the output of the second branch and extracts features through two 3×3 convolutions. The fourth branch receives the output of the third branch and extracts features using two 3×3 convolutions. Finally, the outputs of the four branches are merged through a concat operation and then passed through a 1×1 convolution to obtain the final output features.

[0046] The MP-1 module combines two downsampling methods to accelerate computation and improve feature representation. The first branch of this structure first undergoes a 2×2 max pooling downsampling, followed by a 1×1 convolutional channel transformation. The second branch undergoes a 1×1 convolutional channel transformation, followed by a 3×3 convolutional downsampling with a stride of 2. Finally, the results are concatenated to obtain the downsampling result.

[0047] The SPPCSPC module uses max pooling to expand the receptive field, improving the model's adaptability to images of different resolutions. The first branch halves the channels using a 1×1 convolution. The second branch first optimizes the features using three convolutions, then uses max pooling of sizes 5, 9, 13, and 1 to obtain different receptive fields, and uses two convolutions to adjust the channels. Finally, the two branches are merged to obtain the output.

[0048] In this embodiment, the process of obtaining target contextual feature information includes: for the large, medium, and small-scale feature maps extracted by the backbone network, CIAM is used to integrate the target features and their related background information respectively; the input feature map X is first processed by three 3×3 dilated convolutions with different receptive field sizes, where Conv1 has an expansion rate of 1 and a receptive field size of 3, Conv2 has an expansion rate of 3 and a receptive field size of 7, and Conv2 has an expansion rate of 5 and a receptive field size of 11; then Concat is executed to fuse the three output feature maps to obtain X′; the receptive field size is calculated as follows:

[0049]

[0050] RF i+1 RF is the receptive field size of the current layer. i K is the size of the receptive field of the previous layer, and K is the size of the convolution kernel. It is the product of the first i step sizes.

[0051] The fused feature map X′ is then processed through two 1×1 Conv branches to halve the channels. The third branch receives the output of the second branch and integrates the feature information through a 3×3 convolution. The fourth branch receives the output of the third branch and uses a 3×3 convolution to optimize the features. Finally, the output feature maps from the four branches are concatenated through a Concat operation and then processed through a 1×1 convolution to adjust the channels, resulting in the final output feature map X. * .

[0052] CIAM module structure reference Figure 3 As shown, this module extracts the contextual features of the target by extending convolution, and integrates the extracted target features with relevant background information using a path aggregation network, thereby obtaining feature information containing the target context.

[0053] In this embodiment, the process of CLFNet fusing multi-scale feature maps across layers includes: merging the enhanced feature maps after the context feature aggregation module... Perform the first top-down feature fusion; where the feature maps Upsampling to feature map After merging channels to the same size, the channels are concatenated using the Concat operation. The specific fusion process is as follows:

[0054]

[0055] Where output is and The fused output feature map uses Upsample (nearest neighbor interpolation upsampling) and Concat (channel concatenation).

[0056] After the first fusion, the output feature map P = {P1, P2, P3} is obtained. Considering that the highly integrated features in the deep network reduce the detail features such as small object edges and textures, the original features from the backbone network are introduced across layers in the second feature fusion process. The specific fusion process is as follows:

[0057] output=Concat(P2,MP2(P3),B2)

[0058] Where output is the output feature after cross-layer fusion, P2 and P3 are the features after the first fusion, B2 is the original feature in the backbone network, MP2 is used for downsampling and channel adjustment, and Concat is for channel splicing.

[0059] The improved CLFNet structure can retain the original feature information of small targets in low-to-medium resolution feature maps, which helps to enhance the network's ability to detect small targets. At the same time, increasing the fused feature dimension enables the network to have a stronger feature learning ability.

[0060] In this embodiment, the process of the detection network processing the fused feature map includes: passing the output feature map N = {N1, N2, N3} through the RepConv module, where the RepConv module consists of three branches. During training, the input features are passed through a 3×3 convolution and batch normalization (BN) layer, a 1×1 convolution and BN layer, and a single BN layer, and the output result is obtained by addition. During inference, reparameterization is performed, and the 1×1 convolution and identity mapping branch is optimized into a 3×3 convolution layer, thereby significantly improving the inference speed of the model.

[0061] The detection head is used to calculate the target's location loss, classification loss, and confidence loss, and uses non-maximum suppression (NMS) to filter out redundant detection boxes for target classification and localization. This embodiment uses the Shape-DotD loss function to calculate the target's location loss, and its formula is:

[0062]

[0063] Where Shape-DotD is the point distance loss, D represents the Euclidean distance between the center points of two bounding boxes, and S represents the average size of all objects in the dataset, which is normalized using exponential form.

[0064] The calculation method for the center point distance and the average size of the target is as follows:

[0065]

[0066] Where (x) c ,y c ), , where w and hh are the center coordinates of the detection and ground truth bounding boxes, respectively; ww and hh are the weight coefficients in the horizontal and vertical directions, respectively; M represents the number of images in the dataset; N i w represents the number of marked targets in the i-th image. ij h ij This indicates the width and height of the corresponding detection box.

[0067] The calculation method for the horizontal and vertical weighting coefficients is as follows:

[0068]

[0069] Where scale is the scaling factor, w gt h gt The width and height of the actual bounding box.

[0070] Shape-DotD calculates the normalized Euclidean distance between the center points of the bounding box as the target's position loss, effectively reducing the negative samples generated by the traditional IoU method due to its sensitivity to small target localization errors, and providing the detector with high-quality positive samples, thereby improving the model's recall rate for small targets.

[0071] A small target detection system based on feature fusion, such as Figure 4 As shown, the system includes: an input module, a feature extraction module, a context information aggregation module, a feature fusion module, a detection module, and an output module;

[0072] Input module: Acquires the image data to be detected;

[0073] Feature extraction module: includes the YOLOv7 backbone network, used to extract features from the original image data and generate multi-scale feature maps with different receptive field sizes;

[0074] Context information aggregation module: including CIAM, which integrates target feature information with relevant background information in multi-scale feature maps to obtain enhanced feature maps;

[0075] Feature fusion module: including CLFNet, which preserves the detailed information of the target by connecting the original feature maps of the backbone network across layers to the fusion network, thus obtaining the fused feature maps;

[0076] Detection module: includes detection layers at different scales, used to classify and locate targets in the fused feature map to obtain target detection results;

[0077] Output module: Outputs the target detection results.

[0078] The specific implementation methods of the system and method of the present invention are the same.

[0079] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0080] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A small target detection method based on feature fusion, characterized in that, include: The image data to be detected is acquired and input into the trained small object detection network to obtain the recognition result. The small object detection network includes a backbone network, a context information aggregation module CIAM, a cross-layer feature fusion network CLFNet, and a detection network. Training the small object detection network includes: constructing the original image dataset; inputting the images from the dataset into the backbone network for feature extraction to obtain multi-scale feature maps; inputting the multi-scale feature maps into CIAM for feature information integration to obtain feature information containing the target context; inputting the feature information containing the target context into the CLFNet network for multi-scale feature fusion to obtain a fused feature map; inputting the fused feature map into the detection network to obtain the detection results; calculating the model's loss function based on the detection results and updating the model parameters through backpropagation; when the model converges, the training of the small object detection network is complete. The multi-scale feature maps are input into the CIAM module for feature information integration, including: extracting background feature information of the target from the feature maps at different scales using dilated convolutions with different dilation rates, to obtain the fused feature map X. ′ ; Fusion feature map X ′ After two 1×1 Conv branches to halve the channels, the third branch receives the output of the second branch and integrates the feature information through a 3×3 convolution. The fourth branch receives the output of the third branch and uses a 3×3 convolution to optimize the features. Finally, the output feature maps of the four branches are concatenated through the Concat operation and then adjusted through a 1×1 convolution to obtain feature information containing the target context. Using CLFNet to perform multi-scale feature fusion of feature information containing target context includes: fusing the fused feature map B containing target context feature information. ′ After being input into the feature enhancement module, the enhanced feature map is obtained. For the enhanced feature map Perform upsampling and compare with Channels are concatenated using the Concat operation. The specific fusion process is as follows: Where output is and The fused output feature map uses Upsample (nearest neighbor interpolation upsampling) and Concat (channel concatenation). The concatenated feature map is then passed through an ELAN-W and Conv module for the next fusion operation. After the first top-down feature fusion, the output feature map P = {P1, P2, P3} is obtained. In the second feature fusion process, original features from the backbone network are introduced across layers. The specific fusion process is: output = Concat(P2, MP2(P3), B2), where output is the output feature after cross-layer fusion, P2 and P3 are the features after the first fusion, B2 is the original feature from the backbone network, MP2 is used for downsampling and channel adjustment, and Concat is for channel concatenation. The large-scale feature map P3 is directly used to detect small-scale targets. Other feature maps are used to detect medium- and large-scale targets after cross-layer feature fusion through the MP-2 module.

2. The small target detection method based on feature fusion according to claim 1, characterized in that, The backbone network is the backbone network of the YOLOv7 network.

3. The small target detection method based on feature fusion according to claim 1, characterized in that, The model uses point distance loss as its loss function.

4. The small target detection method based on feature fusion according to claim 3, characterized in that, The point distance loss expression is: Among them, DotD shape For point distance loss, D represents the Euclidean distance between the center points of two bounding boxes, and S represents the average size of all objects in the dataset, which is normalized using exponential form.

5. The small target detection method based on feature fusion according to claim 4, characterized in that, The calculation method for the center point distance and the average size of the target is as follows: Among them, (x c ,y c ), , where w and hh are the center coordinates of the detection and ground truth bounding boxes, respectively; ww and hh are the weight coefficients in the horizontal and vertical directions, respectively; M represents the number of images in the dataset; N i w represents the number of marked targets in the i-th image. ij h ij This indicates the width and height of the corresponding detection box.

6. The small target detection method based on feature fusion according to claim 5, characterized in that, The calculation method for the horizontal and vertical weighting coefficients is as follows: Where scale is the scaling factor, w gt h gt The width and height of the actual bounding box.

7. A small target detection system based on feature fusion, the system being used to execute the small target detection method based on feature fusion according to any one of claims 1 to 6, characterized in that, The system includes: an input module, a feature extraction module, a context information aggregation module, a feature fusion module, a detection module, and an output module; The input module is used to acquire the image data to be detected; The feature extraction module is used to extract features from image data and generate multi-scale feature maps with different receptive field sizes. The context information aggregation module is used to integrate target feature information and relevant background information in the multi-scale feature map to obtain an enhanced feature map; The feature fusion module is used to fuse the enhanced feature maps of different scales to obtain a fused feature map; The detection module is used to perform target detection on the fused feature map to obtain the detection result; The output module is used to output the detection results.

Images