Small target detection method and system based on feature fusion

Abstract

The invention belongs to the field of computer vision, and particularly relates to a small target detection method and system based on feature fusion, and the method comprises the steps: inputting an image into a backbone network for feature extraction, and obtaining a multi-scale feature map; inputting the multi-scale feature map into CIAM for feature information integration to obtain feature information containing a target context; connecting output features of the 37th layer and the 50th layer in the backbone network to a fusion network in a cross-layer manner to obtain a fusion feature map combined with the original feature information of the target; and inputting the fused feature map into a detection network to obtain a detection result. According to the method, the CIAM is integrated into the yolov7 feature extraction network to integrate the target features and related background information in the multi-scale feature map, and meanwhile, the cross-layer feature fusion network is used to reserve the original features of the target, so that the problem of low recognition precision of the small target caused by lack of feature information and sensitive positioning error is solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a small target detection method and system based on feature fusion. Background Art

[0002] Object detection is a popular research direction in the field of computer vision. Its main task is to detect and locate target objects from massive images and videos. As a basic task in the field of computer vision such as target tracking and behavior recognition, it involves multiple technologies such as image preprocessing, feature extraction, and feature fusion. In recent years, with the growing demand for intelligence in various fields, object detection technology has developed rapidly and has been widely used in many fields such as intelligent driving, drone navigation, and medical image analysis.

[0003] Traditional target detection methods identify targets of interest by extracting candidate regions, manually constructing features, and classifying candidate regions. Representative algorithms include the Histogram of Oriented Gradients (HOG) and the Deformable Component Model (DPM). This method performs poorly when dealing with detection tasks in complex scenarios, and has poor robustness when the target scale changes too much, and cannot meet the diverse needs in actual application scenarios. Existing deep learning-based methods divide the target detection task into three main steps: image feature information extraction, multi-scale feature information fusion, and target positioning and recognition. Algorithms represented by the R-CNN and YOLO series automatically extract and classify features through convolutional neural networks. This end-to-end approach can make full use of large-scale data sets for training, thereby learning feature representations with more generalization capabilities.

[0004] In recent years, mainstream target detection models have made significant progress in handling detection tasks in complex environments. However, they are often less effective in identifying small targets. Specifically, the following problems exist in the model for small target detection: 1) Existing models use feature fusion strategies to extract features of different levels from the original image for fusion. However, small-scale targets are prone to feature information loss during the fusion process due to their small area and lack of feature information. 2) The position loss function of existing models does not take into account the sensitivity of targets of different scales to positioning, which makes the model often ignore targets of smaller scales. Therefore, how to improve the performance of the model in small target detection has become an urgent problem to be solved. Summary of the invention

[0005] To solve the above problems, the present invention provides a small target detection method based on feature fusion, which can effectively enhance the feature representation of small targets and improve the accuracy and recall rate of small target detection. The method comprises: obtaining image data to be detected, inputting the image data into a trained small target detection network, and the detection network outputs a recognition result through classification regression. The specific process is as follows:

[0006] The original image data with RGB three channels is input into the backbone network for feature extraction to obtain a multi-scale feature map;

[0007] The multi-scale feature map is input into the Context Information Aggregation Module (CIAM) to integrate the target features and related background information to obtain feature information containing the target context;

[0008] The enhanced multi-scale feature map is input into the Cross Layer Feature Fusion Network (CLFNet) for two feature fusions to obtain a fused feature map.

[0009] The fused feature map is input into the corresponding detection head to obtain the target recognition result.

[0010] Furthermore, the process of obtaining target context feature information includes:

[0011] The feature maps X of different scales extracted by the backbone network are processed by three 3×3 dilated convolutions with different receptive fields, where Conv1 has a dilation rate of 1 and a receptive field size of 3, Conv2 has a dilation rate of 3 and a receptive field size of 7, and Conv3 has a dilation rate of 5 and a receptive field size of 11. Then the Concat operation is used to obtain the fused feature map X′.

[0012] The fused feature map X′ is input into the path aggregation network for feature information integration to obtain the enhanced feature map X containing the target context * .

[0013] Furthermore, the process of obtaining a multi-scale fusion feature map includes:

[0014] The original feature maps of the 37th and 50th layers in the backbone network without context information integration are channel-adjusted through 1×1 Conv, and then fused across layers to the MP-2 module. The fused features are processed by ELAN to obtain the fused feature map N={N1,N2,N3}.

[0015] Furthermore, the process of inputting the target recognition result into the detection head includes:

[0016] The fused feature maps of each scale are re-parameterized by the RepConv module and input into the detection head to calculate the position loss, confidence loss, and classification loss of the target. The position loss is calculated by the dot distance (Shpae-DotD), and finally the recognition result is output after non-maximum suppression (NMS).

[0017]

[0018] Shpae-DotD is defined as the normalized Euclidean distance between the center point of the detection box and the true box, which is used to calculate the position loss of the target. This method can effectively reduce the impact of positioning errors on small targets, thereby improving the accuracy of recognition results.

[0019] On the other hand, the present invention provides a small target detection system based on feature fusion, the system comprising: an input module, a feature extraction module, a context information aggregation module, a feature fusion module, a detection module and an output module;

[0020] The input module is used to obtain image data to be detected;

[0021] The feature extraction module is used to extract features of image data and generate multi-scale feature maps with different receptive field sizes;

[0022] The context information aggregation module is used to integrate the target feature information and the related background information in the multi-scale feature map to obtain an enhanced feature map;

[0023] The feature fusion module is used to fuse the enhanced feature maps of different scales to obtain a fused feature map;

[0024] The detection module is used to perform target detection on the fused feature map to obtain a detection result;

[0025] The output module is used to output the detection result.

[0026] Beneficial effects of the present invention:

[0027] 1. The context information aggregation module (CIAM) is integrated into the feature extraction network of yolov7 to integrate the target features and related background information in the multi-scale feature map. Through extended convolution and path aggregation, the context features of the target can be effectively extracted, solving the problem of low recognition accuracy of small targets due to lack of feature information.

[0028] 2. The cross-layer feature fusion network (CLFNet) structure is used to replace the feature fusion network of the original yolov7. It effectively retains the original features of small-scale targets by connecting the unoptimized feature maps in the backbone network to the fusion network across layers, thereby solving the problem of small target information loss in the fusion process.

[0029] 3. Use point distance Shpae-DotD to replace the original CIoU position loss function. This improvement helps solve the problem that small targets are sensitive to positioning errors and guides the model to pay more attention to the detection of small-scale targets. In this way, the accuracy of small target detection is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall flow chart of the present invention;

[0031] Figure 2 It is a schematic diagram of the network structure of the present invention;

[0032] Figure 3 Schematic diagram of the context information aggregation module (CIAM) structure of the present invention;

[0033] Figure 4 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] A small target detection method based on feature fusion, such as Figure 1 As shown, the method includes: obtaining image data to be detected, inputting the image data into a trained small target detection network to obtain a recognition result; the small target detection network includes a backbone network, a context information aggregation module (CIAM), a cross-layer feature fusion network (CLFNet) and a detection network; training the small target detection network includes: constructing an original image data set; inputting the image in the data set into the backbone network for feature extraction to obtain a multi-scale feature map; inputting the multi-scale feature map into CIAM for feature information integration to obtain feature information containing target context; inputting the feature information containing target context into CLFNet for multi-scale feature fusion to obtain a fused feature map; inputting the fused feature map into the detection network, using Shpae-DotD to calculate the target position loss, completing the model training to obtain the detection result.

[0036] A specific implementation of a small target detection method based on feature fusion, the method comprising:

[0037] S1: Construct a small target detection network including a backbone network, a neck network and a detection head. The present invention uses yolov7 as the basic network for illustration;

[0038] S2: The original image data with RGB three channels is input into the backbone network of the detection model. After a series of convolution and pooling operations, a multi-scale feature map with different receptive field sizes is obtained.

[0039] S3: For the feature maps of different scales generated by the backbone network of the S2 process, the context information aggregation module (CIAM) is used to integrate the feature information of the target with the relevant background information to obtain the feature information containing the target context;

[0040] S4: Use a cross-layer feature fusion network (CLFNet) to fuse feature maps of different scales. In the first fusion process, the deep feature map enhanced by context information is upsampled to the same size as the shallow feature map and fused. In the second fusion process, the original feature map of the backbone network is connected to the fusion module across layers to obtain fused feature maps for detecting large, medium and small scale targets respectively.

[0041] S5: The fused feature map generated by the S4 process is input into the detection network, first re-parameterized by the RepConv module, and then the position loss, confidence loss, and classification loss are calculated by the detection head. The position loss is calculated by the point distance (Shpae-DotD) and the recognition result is obtained.

[0042] Figure 2 This is a schematic diagram of the network structure of the small target detection method based on feature fusion in the present invention. The method uses yolov7 as the basic network architecture and includes a feature extraction module, a context information aggregation module, a feature fusion module and a detection module. Figure 2 In the method, the image data to be detected is subjected to the yolov7 backbone network to extract features and obtain a multi-scale feature map; the multi-scale feature map is input into CIAM to obtain context-enhanced target feature information; the enhanced multi-scale features are input into CLFNet for feature information fusion to obtain multi-scale fusion features; the fusion features are input into the detection head optimized for small targets to obtain the target recognition result.

[0043] In this embodiment, extracting a multi-scale feature map of an image using a backbone network includes: using two different CBS modules to alternately perform feature extraction and downsampling processing on an input image to obtain an initial feature map; the CBS module consists of a convolution layer, a Batch normalization layer, and a Silu activation layer, wherein a CBS module with a 3×3 convolution step size of 1 is used to extract features, and a CBS module with a 3×3 convolution step size of 2 is used for downsampling.

[0044] The initial features are passed through the ELAN module, MP-1 module and SPPCSPC module in turn to obtain feature maps B = {B1, B2, B3} of different scales.

[0045] In this embodiment, the ELAN module is used to balance the relationship between the network gradient path length and the gradient propagation efficiency to avoid gradient loss caused by channel changes, so that the network has stronger learning ability and robustness. The structure consists of four parallel branches. The first and second branches are both adjusted by a 1×1 convolution for the number of channels. The third branch accepts the output of the second branch and extracts features through two 3×3 convolutions. The fourth branch accepts the output of the third branch and uses two 3×3 convolutions to extract features. Finally, the outputs of the four branches are merged through a concat operation and then a 1×1 convolution is performed to obtain the final output features.

[0046] The MP-1 module combines two downsampling methods to accelerate calculation and improve feature expression capabilities. The first branch of the structure first undergoes a 2×2 maximum pooling downsampling, then a 1×1 convolution channel change, and the second branch undergoes a 1×1 convolution channel change, a 3×3 convolution downsampling with a step size of 2, and finally concatenates to obtain the downsampling result.

[0047] The SPPC module uses the maximum pooling operation to expand the receptive field to improve the model's adaptability to images of different resolutions. The first branch uses 1×1 convolution to halve the channel, and the second branch first optimizes the features through 3 convolutions, then obtains different receptive fields through maximum pooling of sizes 5, 9, 13, and 1, and uses 2 convolutions to adjust the channel. Finally, the two branches are merged to obtain the output result.

[0048] In this embodiment, the process of obtaining the target context feature information includes: for the large, medium and small scale feature maps extracted by the backbone network, CIAM is used to integrate the target features and their related background information respectively; the input feature map X is first processed by three 3×3 dilated convolutions with different receptive field sizes, where Conv1 has an expansion rate of 1 and a receptive field size of 3, Conv2 has an expansion rate of 3 and a receptive field size of 7, and Conv2 has an expansion rate of 5 and a receptive field size of 11; then Concat is executed to fuse the three output feature maps to obtain X′; the calculation method of the receptive field size is:

[0049]

[0050] Among them, RF i+1 is the receptive field size of the current layer, RF i is the receptive field size of the previous layer, K is the convolution kernel size, is the product of the previous i stride sizes.

[0051] The fused feature map X′ then passes through two 1×1 Conv branches to halve the channels. The third branch accepts the output of the second branch and integrates the feature information through a 3×3 convolution. The fourth branch accepts the output of the third branch and uses a 3×3 convolution to optimize the features. Finally, the output feature maps of the last four branches are spliced ​​through the Concat operation and then adjusted through a 1×1 convolution to obtain the final output feature map X * .

[0052] CIAM module structure reference Figure 3 As shown in the figure, the module extracts the contextual features of the target through dilated convolution, and uses a path aggregation network to integrate the extracted target features with relevant background information to obtain feature information containing the target context.

[0053] In this embodiment, the process of CLFNet cross-layer fusion of multi-scale feature maps includes: Perform the first top-down feature fusion; the feature map Upsampling to feature map After the same size, the channels are concatenated through the Concat operation. The specific fusion process is as follows:

[0054]

[0055] The output is and The fused output feature map, Upsample is the nearest neighbor interpolation upsampling operation, and Concat is the channel concatenation operation.

[0056] After the first fusion, the output feature map P = {P1, P2, P3} is obtained. Considering that the highly integrated features in the deep network reduce the detail features such as the edge and texture of small targets, the original features from the backbone network are introduced across layers in the second feature fusion process. The specific fusion process is:

[0057] output=Concat(P2,MP2(P3),B2)

[0058] Among them, output is the output feature after cross-layer fusion, P2 and P3 are the features after the first fusion, B2 is the original feature in the backbone network, MP2 is used for downsampling and channel adjustment, and Concat is channel splicing.

[0059] The improved CLFNet structure can retain the original feature information of small targets in medium and low resolution feature maps, which helps to enhance the network's detection ability for small targets. At the same time, increasing the fused feature dimension can enable the network to have stronger feature learning ability.

[0060] In this embodiment, the process of the detection network processing the fused feature map includes: passing the output feature map N = {N1, N2, N3} through the RepConv module, where the RepConv module consists of three branches. During the training process, the input features are respectively passed through a 3×3 convolution and batch normalization (BN) layer, a 1×1 convolution and BN layer, and a single BN layer, and the output result is obtained by adding; during the reasoning process, re-parameterization is performed, and the 1×1 convolution and identity mapping branches are optimized to a 3×3 convolution layer, thereby significantly improving the reasoning speed of the model.

[0061] The detection head is used to calculate the position loss, classification loss, and confidence loss of the target, and to filter out redundant detection frames through non-maximum suppression (NMS) to classify and locate the target. This embodiment uses the Shape-DotD loss function to calculate the position loss of the target, and its formula is:

[0062]

[0063] Among them, Shape-DotD is the point distance loss, D represents the Euclidean distance between the center points of the two bounding boxes, S represents the average size of all objects in the dataset, and it is normalized using an exponential form.

[0064] The calculation method of the center point distance and the average target size is:

[0065]

[0066] Where (x c ,y c ), are the center coordinates of the detection box and the real box respectively, ww and hh are the weight coefficients in the horizontal and vertical directions respectively, M represents the number of images in the dataset, and N i represents the number of targets marked in the i-th image, w ij 、h ij Indicates the width and height of the corresponding detection box.

[0067] The calculation method of the horizontal and vertical weight coefficients is:

[0068]

[0069] Where scale is the scale factor, w gt 、h gt is the width and height of the real box.

[0070] Shape-DotD calculates the normalized Euclidean distance between the center points of the bounding box as the position loss of the target, which effectively reduces the negative samples generated by the traditional IoU method due to its sensitivity to small target positioning errors, and provides high-quality positive samples for the detector, thereby improving the model's recall rate for small targets.

[0071] A small target detection system based on feature fusion, such as Figure 4 As shown, the system includes: an input module, a feature extraction module, a context information aggregation module, a feature fusion module, a detection module and an output module;

[0072] Input module: obtain the image data to be detected;

[0073] Feature extraction module: including the yolov7 backbone network, which is used to extract the features of the original image data and generate multi-scale feature maps with different receptive field sizes;

[0074] Contextual Information Aggregation Module: including CIAM, which is used to integrate the target feature information in the multi-scale feature map with the relevant background information to obtain an enhanced feature map;

[0075] Feature fusion module: including CLFNet, which preserves the detailed information of the target by connecting the original feature map of the backbone network to the fusion network across layers to obtain the fused feature map;

[0076] Detection module: includes detection layers of different scales, which are used to classify and locate the targets in the fused feature map to obtain target detection results;

[0077] Output module: outputs target detection results.

[0078] The system implementation of the present invention is the same as the specific implementation of the method.

[0079] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0080] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A small target detection method based on feature fusion, characterized in that: include: Obtain the image data to be detected, input the image data into the trained small target detection network, and obtain the recognition result; The small target detection network includes the backbone network, CIAM, CLFNet and the detection network; Training the small target detection network includes: constructing an original image dataset; inputting the images in the dataset into the backbone network for feature extraction to obtain a multi-scale feature map; The multi-scale feature map is input into CIAM for feature information integration to obtain feature information containing the target context; the feature information containing the target context is input into the CLFNet network for multi-scale feature fusion to obtain a fused feature map; the fused feature map is input into the detection network to obtain the detection result; the loss function of the model is calculated according to the detection result, and the model parameters are updated through back propagation. When the model reaches convergence, the training of the small target detection network is completed.

2. The small target detection method based on feature fusion according to claim 1, characterized in that: The backbone network is the backbone network of the yolov7 network.

3. The small target detection method based on feature fusion according to claim 1, characterized in that: Inputting the multi-scale feature map into the CIAM module for feature information integration includes: using dilated convolution with different expansion rates to extract the background feature information of the target in the feature map of different scales, and obtaining the fused feature map X ′ ; Fusion feature map X ′ After two 1×1 Conv branches, the channel is halved. The third branch receives the output of the second branch and integrates the feature information through a 3×3 convolution. The fourth branch receives the output of the third branch and uses a 3×3 convolution to optimize the features. Finally, the output feature maps of the last four branches are concatenated through the Concat operation, and then the channels are adjusted through a 1×1 convolution to obtain feature information containing the target context.

4. The small target detection method based on feature fusion according to claim 1, characterized in that: Using CLFNet to perform multi-scale feature fusion on feature information containing target context includes: fusion feature map B containing target context feature information ′ After input into the feature enhancement module, the enhanced feature map is obtained For the enhanced feature map Upsample and compare The channels are spliced ​​through the Concat operation; the spliced ​​feature map passes through an ELAN-W and Conv module for the next fusion operation; after the first top-down feature fusion, the output feature map P = {P1, P2, P3} is obtained, among which the large-scale feature map P3 is directly used to detect small-scale targets; the other feature maps are used to detect medium and large-scale targets respectively after cross-layer feature fusion by the MP-2 module.

5. The small target detection method based on feature fusion according to claim 1, characterized in that: The loss function of the model adopts point distance loss.

6. The small target detection method based on feature fusion according to claim 5, characterized in that: The point distance loss expression is: Among them, Shape-DotD is the point distance loss, D represents the Euclidean distance between the center points of the two bounding boxes, S represents the average size of all objects in the dataset, and is normalized using an exponential form.

7. The small target detection method based on feature fusion according to claim 6, characterized in that: The distance between the center point and the average size of the target is calculated as: Among them, (x c ,y c ), are the center coordinates of the detection box and the real box respectively, ww and hh are the weight coefficients in the horizontal and vertical directions respectively, M represents the number of images in the dataset, and N i represents the number of targets marked in the i-th image, w ij 、h ij Indicates the width and height of the corresponding detection box.

8. The small target detection method based on feature fusion according to claim 7, characterized in that: The calculation method of the horizontal and vertical weight coefficients is: Where scale is the scale factor, w gt 、h gt is the width and height of the real box.

9. A small target detection system based on feature fusion, the system is used to execute the small target detection method based on feature fusion according to any one of claims 1 to 8, characterized in that: The system includes: an input module, a feature extraction module, a context information aggregation module, a feature fusion module, a detection module and an output module; The input module is used to obtain image data to be detected; The feature extraction module is used to extract features of image data and generate multi-scale feature maps with different receptive field sizes; The context information aggregation module is used to integrate the target feature information and the related background information in the multi-scale feature map to obtain an enhanced feature map; The feature fusion module is used to fuse the enhanced feature maps of different scales to obtain a fused feature map; The detection module is used to perform target detection on the fused feature map to obtain a detection result; The output module is used to output the detection result.

Images