Small Target Detection Method Based on Full-Link Multi-Scale Fusion Network

Abstract

The present invention proposes a small target detection method based on a full-link multi-scale fusion network, and the implementation steps are as follows: obtaining a training sample set and a test sample set; constructing a small target detection model including a full link with an attention-guided feature cross-aggregation encoder, a decoder, and a Head block cascaded in sequence, and a branch network with a cooperative super-resolution block connected between the input end of the decoder and the input end of the Head block; initializing parameters; training the small target detection model; updating the parameters of the small target detection model; and obtaining small target detection results. In the multi-scale fusion network of the present invention, the feature map with multi-scale feature information generated by the depth-weighted multi-scale attention block is fused with the feature maps of different levels generated by the cross-layer feature fusion block, and a high-resolution multi-scale fusion feature map is obtained through the Head block, which can fully exploit the full-scale information of the small target image and effectively improve the accuracy of small target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a small target detection method, specifically to a small target detection method based on a full-link multi-scale fusion network, which can be used in systems such as forest fire detection and maritime rescue. Background Art

[0002] Small target detection is a technology that uses computer technology to identify targets from images and belongs to the field of image recognition in computer vision. Usually, targets with an occupancy ratio of 0.08% to 0.58% of the entire image are called small targets. With the continuous development of deep learning technology, existing algorithms have shown good detection effects on large and medium targets. However, due to the small pixels of small targets in images, and their low resolution, contrast, and signal-to-noise ratio, it is difficult to fully extract the feature information of small targets during the detection process, which poses a challenge to small target detection. To improve the accuracy of small target detection, more and more innovative methods have been proposed.

[0003] For example, in the patent document "A Small Target Detection Method Based on Multi-Scale Feature Compensation and Gating Enhancement" (patent application number: CN202310283681.6, publication number: CN116342868A) applied by Xidian University, a small target detection method based on multi-scale feature compensation and gating enhancement is proposed. The small target detection model based on multi-scale feature compensation and gating enhancement includes a convolutional module, a transposed module, a gating enhancement module, and a prediction module connected in sequence, and a multi-scale feature compensation module loaded between the convolutional module and the transposed module. The multi-scale feature compensation module is used to extract shallow and deep features before and after the downsampling operation, and use the shallow features to compensate the detailed features of the deep features, so that the detailed features are propagated to the deepest part of the network to compensate for the target features lost in the downsampling operation and improve the representation ability of multi-scale context. The gating enhancement module is used to enhance the target features and suppress background noise. Due to the easy loss of small target feature information during the sampling operation, and the small pixels of small target features, and their low resolution, contrast, and signal-to-noise ratio, the detection accuracy is still low. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned deficiencies of the prior art and provide a small target detection method based on a full-link multi-scale fusion network to solve the technical problem of low detection accuracy existing in the prior art.

[0005] To achieve the above purpose, the technical solutions adopted by the present invention include the following steps:

[0006] (1) Obtain a training sample set and a test sample set:

[0007] Obtain K small target images, label the small target objects in each small target image, then form a training sample set R1 with M small target images and their corresponding labels, and form a test sample set E1 with the remaining K - M small target images and their corresponding labels, where K ≥ 500.

[0008] (2) Construct a small target detection model O based on a full-link multi-scale fusion network:

[0009] Construct a small target detection model O that includes a full-link with an attention-guided feature cross-aggregation encoder, a decoder, and a Head block in sequential cascade, and a branch network with a cooperative super-resolution block connected between the input end of the decoder and the input end of the Head block; the attention-guided feature cross-aggregation encoder includes N cascaded residual blocks and a multi-scale fusion network, where the multi-scale fusion network consists of a total of N - 1 depth-weighted multi-scale attention blocks loaded between every two adjacent residual blocks and a cross-layer feature fusion block connected to its output end; the depth-weighted multi-scale attention block is used to generate fusion feature maps with different scale information; the cross-layer feature fusion block is used to fuse the feature information of different depth-weighted multi-scale attention blocks; the cooperative super-resolution block is used to reconstruct the low-resolution image output by the encoder, where N ≥ 2.

[0010] (3) Initialize the parameters:

[0011] Initialize the number of iterations as t, the maximum number of iterations as T, T ≥ 1000, and the weights and bias parameters of the small target detection model O in the t-th iteration are w t 、b t respectively, and let t = 0, O t = O; t =

[0012] (4) Train the small target detection model O:

[0013] Randomly and with replacement select L training samples from the training sample set R1 as the input of the small target detection model O for forward propagation to obtain L small target detection results, where 1 ≤ L ≤ M.

[0014] (5) Update the parameters of the small target detection model:

[0015] Update the weights and bias parameters w t 、b t 、 t of the small target detection model O with the L small target detection results obtained in step (4) to obtain the network model O of this iteration t; and determine whether t≥T holds. If so, obtain the trained small target detection model O*, otherwise, set t = t + 1, O t = O, and execute step (4);

[0016] (6) Obtain the small target detection result:

[0017] Use the test sample set E1 as the input of the trained small target detection model O* for forward propagation to obtain the small target detection results corresponding to K - M test samples.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] During the process of training the small target detection model and obtaining the small target detection results in the present invention, the multi - scale fusion network fuses the feature maps with multi - scale feature information generated by the depth - weighted multi - scale attention block and the feature maps of different levels generated by the cross - layer feature fusion block, and the Head block fuses each reconstructed image with its corresponding high - resolution multi - scale fusion feature map, which can fully exploit the full - scale information of small target images. Experimental results show that the present invention can effectively improve the accuracy of small target detection. Description of the Drawings

[0020] Figure 1 is the implementation flowchart of the present invention;

[0021] Figure 2 is the structural schematic diagram of the small target detection model adopted in the embodiment of the present invention;

[0022] Figure 3 is the structural schematic diagram of the depth - weighted multi - scale attention module adopted in the embodiment of the present invention;

[0023] Figure 4 is the structural schematic diagram of the cross - layer feature fusion module adopted in the embodiment of the present invention.

[0024] Figure 5 is the structural schematic diagram of the residual block adopted in the embodiment of the present invention. Detailed Embodiments

[0025] The following further describes the present invention in detail with reference to the drawings and specific embodiments.

[0026] Refer to Figure 1 , the present invention includes the following steps:

[0027] Step 1) Obtain the training sample set and the test sample set:

[0028] Obtain 1000 small-target images included in the IRSTD-1k dataset, annotate the small targets in each small-target image, and then form a training sample set R1 with 600 small-target images and their corresponding labels, and form a test sample set E1 with the remaining 400 small-target images and their corresponding labels;

[0029] Step 2) Construct a small-target detection model O based on the full-link multi-scale fusion network:

[0030] Refer to Figure 2 to further describe the small-target detection model based on the full-link multi-scale fusion network adopted in the embodiments of the present invention.

[0031] Construct a small-target detection model O including the full link with an attention-guided feature cross-aggregation encoder, a decoder, and a Head block connected in series in sequence, and a cooperative super-resolution block connected between the input end of the decoder and the input end of the Head block as a branch; the attention-guided feature cross-aggregation encoder and the cooperative super-resolution branch are loaded between the feature extraction block and the Head block; the attention-guided feature cross-aggregation encoder includes three residual blocks, two depth-weighted multi-scale attention modules, and a cross-layer feature fusion module; the output end of the first residual block is connected to the input end of the first depth-weighted multi-scale attention module, the output end of the first depth-weighted multi-scale attention module is connected to the input end of the second residual block, the output end of the second residual block is connected to the input end of the second depth-weighted multi-scale attention module, the output end of the second depth-weighted multi-scale attention module is connected to the input end of the third residual block, the output ends of the first depth-weighted multi-scale attention module and the second depth-weighted multi-scale attention module are both connected to the input end of the cross-layer feature fusion block, the input ends of the decoder and the cooperative super-resolution block are both connected to the third residual block, and the output ends are both connected to the Head block; the residual block includes an eighth convolutional layer, a first normalization layer, a first non-linear activation layer, a ninth convolutional layer, a second normalization layer, and a second non-linear activation layer connected in sequence; the depth-weighted multi-scale attention block includes a stacked first pooling layer, a first convolutional layer, and second, third, fourth, fifth, and sixth convolutional layers and a second pooling layer arranged in parallel; the branch of the cross-layer feature fusion block connected to the output end of the first depth-weighted multi-scale attention module includes a stacked third pooling layer, a seventh convolutional layer, ReLU, a second deconvolutional layer, and ReLU, and the branch connected to the output end of the second depth-weighted multi-scale attention module includes a stacked third deconvolutional layer, ReLU, a first deconvolutional layer, and ReLU; the Head module includes a tenth convolutional layer, a third normalization layer, a third non-linear activation layer, a random dropout layer, and an eleventh convolutional layer connected in sequence;

[0032] The specific parameters are set as follows: the convolution kernel size of the first convolutional layer is 3*3, the stride is 1, and the padding is 1; the convolution kernel size of the second convolutional layer is 1*1, and the stride is 1; the convolution kernel size of the third convolutional layer is 3*3, the stride is 1, and the padding is 1; the fourth, fifth, and sixth convolutional layers all use atrous convolution; the convolution kernel size of the fourth convolutional layer is 7*7, the stride is 2, and the padding is 1; the convolution kernel size of the fifth convolutional layer is 5*5, the stride is 2, and the padding is 1; the convolution kernel size of the sixth convolutional layer is 3*3, the stride is 2, and the padding is 1; the convolution kernel size of the seventh convolutional layer is 3*3, the stride is 1, and the padding is 1; the convolution kernel sizes of the eighth and ninth convolutional layers are both 3*3, the stride is 1, and the padding is 1; the first pooling layer uses global average pooling; the second pooling layer uses Atrous Spatial Pyramid Pooling; the third pooling layer uses max pooling; the normalization layers all use layer normalization; the non-linear activation layer uses the ReLU function;

[0033] Referring to Figure 3 , a further description is made of the depth-weighted multi-scale attention module adopted in the embodiments of the present invention.

[0034] The feature map C1 is processed by the softmax function after being generated by the first pooling layer and the first convolutional layer to generate a depth vector C depth , and the expression is as follows:

[0035] C depth = S(Conv(AvgPooling(C1))),

[0036] where C depth represents the depth vector obtained after processing, Conv(·) is the convolution function, AvgPooling(·) is the global pooling function, S(·) is the softmax function, and C1 represents the feature map output by the first residual block to the first depth-weighted multi-scale attention module;

[0037] In order to explore the context features of the image at multiple scales, the feature map C1 is processed by the second convolutional layer, the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer, and the second pooling layer respectively to generate 6 multi-scale features f with the same resolution but different feature information m (m = 1, 2, 3, 4, 5, 6), and this process can be expressed as:

[0038] f1 = δ(Β(Conv 1×1 (C1))),

[0039] f2 = δ(Β(Conv 3×3 (C1))),

[0040] f3 = δ(Β(AConv d＝7(C1))),

[0041] f4 = δ(Β(AConv d＝5 (C1))),

[0042] f5 = δ(Β(AConv d＝3 (C1))),

[0043] f6 = Upsample(δ(Β(AConv 1×1 (APooling(APooling(C1)))))),

[0044] where Conv(·), AConv(·) and APooling(·) are the convolution function, Atreus convolution function and Atreus spatial pyramid pooling function respectively, δ(·) is the rectified linear function, Β(·) is the batch normalization function, and f1, f2, f3, f4, f5 and f6 represent the feature maps output by the second convolutional layer, third convolutional layer, fourth convolutional layer, fifth convolutional layer, sixth convolutional layer and second pooling layer respectively;

[0045] Next, the depth vector is used as a weight to guide the multi-scale feature fusion of f1, f2, f3, f4, f5 and f6 to generate a new feature image, and this operation can be expressed as:

[0046]

[0047] where F1 is the image output by the first depth-weighted multi-scale attention module through multi-scale feature fusion, is the weight of the m-th feature map during the fusion process;

[0048] Refer to Figure 4 for a further description of the cross-layer feature fusion module adopted in the embodiments of the present invention.

[0049] First, the feature map from the first depth-weighted multi-scale attention block is applied to the third pooling layer and the seventh convolutional layer, and then passes through the ReLU layer to form a feature map with the same resolution as the feature map of the second depth-weighted multi-scale attention block; subsequently, this feature map is fused with and superimposed on the feature map from the second depth-weighted multi-scale attention block, and after passing through the second deconvolutional layer and the ReLU layer, it is linked to the decoder.

[0050] Similarly, the feature map from the second depth-weighted multi-scale attention block passes through the third deconvolutional layer and the ReLU layer to form a feature map with the same resolution as the feature map of the first depth-weighted multi-scale attention block; subsequently, this feature map is fused with and superimposed on the feature map from the first depth-weighted multi-scale attention block, and after passing through the first deconvolutional layer and the ReLU layer, it is linked to the decoder. This process can be expressed as follows:

[0051] N2 = δ(Deconv(δ(Conv(MaxPooling(F1))) + F2)),

[0052] N1 = δ(Deconv(δ(Deconv(F2)) + F1)),

[0053] Wherein, F1 and F2 respectively represent the feature maps output by the first depth-weighted multi-scale attention module and the second depth-weighted multi-scale attention module to the cross-layer feature fusion block, N1 and N2 respectively represent the feature maps output by the cross-layer feature fusion block to the first-level decoder and the second-level decoder, Deconv(·) and MaxPooling(·) respectively represent the deconvolution function and the max pooling function, δ(·) is the rectified linear function, and Β(·) is the batch normalization function;

[0054] Step 3) Initialize parameters:

[0055] Initialize the number of iterations as t, the maximum number of iterations as T = 1000, and the weights and bias parameters of the small target detection model O in the t-th iteration are w t 、b t 、respectively, and let t = 0, O t = O; t

[0056] Step 4) Train the small target detection model O:

[0057] Randomly select 32 training samples from the training sample set R1 with replacement as the input of the small target detection model O for forward propagation:

[0058] (4a) The first residual block in the attention-guided feature cross-aggregation encoder downsamples each input image to obtain the feature map C1, the second residual block downsamples the feature map from the first depth-weighted multi-scale attention block to obtain the feature map C2, the third residual block downsamples the feature map from the second depth-weighted multi-scale attention block to obtain the feature map C3, the first depth-weighted multi-scale attention block performs multi-scale feature fusion on each feature map from the first residual block to generate the fused feature map F1, the second depth-weighted multi-scale attention block performs multi-scale feature fusion on the feature map from the second residual block to generate the fused feature map F2, and the cross-layer feature fusion block fuses the fused feature map F1 from the first depth-weighted multi-scale attention block and the fused feature map F2 from the second depth-weighted multi-scale attention block to respectively generate the feature maps N1 and N2 containing multi-scale semantic information;

[0059] ​(4b) The cooperative super-resolution block reconstructs each input low-resolution feature map, provides high-resolution features for the decoder, and cooperates with the decoder to upsample each image from the attention-guided feature cross-aggregation encoder to obtain a high-resolution multi-scale fusion feature map;

[0060] (4c) The Head block performs object detection on each high-resolution multi-scale fusion feature map to obtain 32 small object detection results.

[0061] Step 5) Update the parameters of the small object detection model:

[0062] Based on the 32 small object detection results obtained in step (4), update the weights and bias parameters w t and b t of the small object detection model O t to obtain the network model O t of this iteration:

[0063] (5a) Using the Dice loss function, calculate the loss value L of the small object detection model through the 32 small object detection results generated and the labels of the corresponding 32 images Dice :

[0064]

[0065] where p l represents the probability that each pixel of the l-th image in the small object detection result belongs to the label category, r l represents the category of each pixel in the l-th input image, and γ is a correction factor, taking any real number within the range of (0, 0.1), and its function is to prevent the denominator of the fraction from being zero;

[0066] (5b) Calculate the partial derivatives of L Dice with respect to the weight parameter ω t and the bias parameter b t through the chain rule and Finally, according to and update ω t and b t :

[0067]

[0068]

[0069] where ω t and b t represent the weights and bias parameters of all learnable parameters of O t , ω t ' and bt ' represents ω t , b t 's update result, α represents the learning rate; determine whether t≥T holds, if so, obtain the trained small target detection model O * , otherwise, let t=t + 1, O t =O, and execute step (4).

[0070] Step 6) Obtain the small target detection result:

[0071] Use the test sample set E1 as the input of the trained small target detection model O* for forward propagation to obtain the small target detection results corresponding to 400 test samples.

[0072] The technical effects of the present invention are described below in combination with simulation experiments

[0073] Simulation conditions, content and result analysis:

[0074] The hardware platform for the simulation experiment is: the processor is an Intel(R) Core i9-9900K CPU with a main frequency of 3.5GHz, the memory is 32GB, and the graphics card is an NVIDIA GeForce RTX 2080Ti. The software platform for the simulation experiment is: the Ubuntu 16.04 operating system, the python version is 3.7, and the Pytorch version is 1.7.1.

[0075] Use the intersection over union (IoU) evaluation metric to compare the detection accuracy of the patent document "A Small Target Detection Method Based on Multi-Scale Feature Compensation and Gated Enhancement" (patent application number: CN202310283681.6, publication number: CN116342868A) and the present invention on test samples. The intersection over union (IoU) of the small target detection results of the existing method is 64.87%, and the intersection over union (IoU) of the small target detection results of the present invention is 77.1%. Compared with the prior art, the detection accuracy of the present invention has been significantly improved.

Claims

1. A small target detection method based on a full-link multi-scale fusion network, characterized in that It includes the following steps: (1) Obtain a training sample set and a test sample set: Obtain K small target images, annotate the small target objects in each small target image, and then form a training sample set R1 with M small target images and their corresponding labels, and form a test sample set E1 with the remaining K - M small target images and their corresponding labels, where K ≥ 500. (2) Construct a small target detection model O based on a full-link multi-scale fusion network: Construct a small target detection model O that includes an attention-guided feature cross-aggregation encoder, a decoder, and a Head block in series as the full link, and a cooperative super-resolution block connected between the input end of the decoder and the input end of the Head block as the branch network; the attention-guided feature cross-aggregation encoder includes N cascaded residual blocks and a multi-scale fusion network, where the multi-scale fusion network consists of a total of N - 1 depth-weighted multi-scale attention blocks loaded between every two adjacent residual blocks and a cross-layer feature fusion block connected to its output end; the depth-weighted multi-scale attention block is used to generate fused feature maps with different scale information; The cross-layer feature fusion block is used to fuse the feature information of different depth-weighted multi-scale attention blocks; the cooperative super-resolution block is used to reconstruct the low-resolution image output by the encoder, where N ≥ 2; (3) Initialize the parameters: Initialize the number of iterations as t, the maximum number of iterations as T, where T ≥ 1000, and the small target detection model O in the t-th iteration t The weight and bias parameters in t are w t , b t respectively, and let t = 0, O t = O; (4) Train the small target detection model O: Randomly and with replacement, select L training samples from the training sample set R1 as the input of the small target detection model O for forward propagation to obtain L small target detection results, where 1 ≤ L ≤ M; (5) Update the parameters of the small target detection model: For the L small target detection results obtained through step (4), for the small target detection model O t update the weights and bias parameters w t , b t to obtain the network model O of this iteration t ; and determine whether t≥T holds. If so, obtain the trained small target detection model O*, otherwise, set t=t + 1, O t =O, and execute step (4); (6) Obtain the small target detection results: Use the test sample set E1 as the input of the trained small target detection model O* for forward propagation to obtain the small target detection results corresponding to K - M test samples.

2. The method according to claim 1, wherein Regarding the small target detection model O described in step (2), where: The residual block includes multiple stacked composite layers composed of a convolutional layer, a normalization layer, and a non-linear activation layer stacked in sequence; The Head block includes a convolutional layer, a normalization layer, a non-linear activation layer, a random dropout layer, and a convolutional layer stacked in sequence; The depth-weighted multi-scale attention block includes a stacked pooling layer and a convolutional layer, and J parallelly arranged pooling layers and convolutional layers with different expansion rates and different kernel sizes loaded between the input end of the pooling layer and the output end of the convolutional layer, where J ≥ 2.

3. The method according to claim 1, characterized in that, Regarding the training of the small target detection model O described in step (4), the implementation steps are: (4a) The N residual blocks in the attention-guided feature cross-aggregation encoder perform downsampling on each image to obtain L feature maps. The N - 1 depth-weighted multi-scale attention blocks perform feature fusion on each feature map at J scales to obtain L fused feature maps. The cross-layer feature fusion block fuses each fused feature map generated by different depth-weighted multi-scale attention blocks to generate L low-resolution feature maps containing multi-scale semantic information; (4b) The cooperative super-resolution block reconstructs each input low-resolution feature map, provides high-resolution features for the decoder, and cooperates with the decoder to upsample each image from the attention-guided feature cross-aggregation encoder to obtain L high-resolution multi-scale fusion feature maps; (4b) The cooperative super-resolution block reconstructs each low-resolution feature map into a high-resolution feature map, and at the same time, the decoder upsamples each low-resolution feature map to obtain L high-resolution multi-scale fusion feature maps; (4c) The Head block fuses each reconstructed image with its corresponding high-resolution multi-scale fusion feature map, and performs object detection on the fusion result to obtain L small object detection results.

4. The method according to claim 3, wherein The L fused feature maps described in step (4a), where the l-th fused feature map F l is expressed as: Among them, f j is the single-scale feature image output by the j-th convolutional layer or pooling layer, is the weight of f j .

5. The method according to claim 1, characterized in that The weight and bias parameters w t and b t of the small target detection model O t in step (5) are updated, and the implementation steps are as follows: (5a) Using the Dice loss function, calculate the loss value L of the small object detection model based on the L generated small object detection results and the labels of the corresponding L images Dice : Among them, p l represents the probability that each pixel in the l-th image of the small target detection result belongs to the label category, and r l represents the category of each pixel in the l-th input image, and γ is the correction factor; (5b) Calculate L through the chain rule Dice For the weight parameter ω t and the bias parameter b t partial derivative and and according to and for ω t 、b t perform the update: where ω′ t , b′ t represent the updated results of ω t , b t , and α represents the learning rate.

Images