A small target detection method and model based on multi-scale feature fusion

Abstract

The present application belongs to the field of image processing technology, and discloses a small target detection method and model based on multi-scale feature fusion, which introduces a shallow fusion module, a middle fusion module and a deep fusion module to perform feature enhancement and fusion processing on preliminary feature images of different scales extracted from an input image, which is beneficial to maximize the extraction of features at different levels for subsequent target detection, thereby improving the accuracy and robustness of small target detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a small target detection method and model based on multi-scale feature fusion. Background Art

[0002] Object detection is an important task in the field of computer vision, which is mainly divided into two tasks: positioning and classification. Small object detection belongs to the subfield of object detection, which focuses on the detection of small objects in images. Small objects generally refer to target objects that occupy relatively few pixels in an image, and the pixels they occupy are generally less than 32*32. Small object detection is used in many places. In the field of autonomous driving, small object detection is used to identify and track small traffic signs, pedestrians, bicycles and motorcycles on the road to improve the perception and decision-making capabilities of autonomous driving vehicles and enhance driving safety; in pedestrian recognition and monitoring systems, small object detection can be used for crowd management, video surveillance, behavior analysis and security alarms; in addition, small object detection also has important applications in drone applications, remote sensing target detection, robot vision, medical image analysis and video content analysis.

[0003] The existing feature pyramid network (FPN) has good results in multi-scale object detection, but it still has shortcomings in small object detection. These shortcomings include:

[0004] Traditional FPN lacks effective information enhancement methods in the feature fusion process, resulting in poor feature extraction results for small targets;

[0005] Traditional FPN does not fully utilize the attention mechanism to enhance the expressiveness of feature maps, resulting in insufficient accuracy in small target detection;

[0006] The fusion strategy between feature maps of traditional FPN is relatively simple, and it fails to make full use of multi-scale feature information for effective fusion, resulting in small targets being easily ignored or misdetected in complex backgrounds;

[0007] The above-mentioned deficiencies lead to obvious limitations of the existing feature pyramid network in dealing with the complexity and diversity of small target detection, and the accuracy and robustness of small target detection are poor. Summary of the invention

[0008] The purpose of this application is to provide a small target detection method and model based on multi-scale feature fusion, which can improve the accuracy and robustness of small target detection.

[0009] In a first aspect, the present application provides a small target detection model based on multi-scale feature fusion, including a backbone network and an improved feature pyramid network; the improved feature pyramid network includes a scale adjustment unit, a feature enhancement and fusion unit, and a feature image output unit;

[0010] The backbone network has a plurality of feature extraction layers, and the backbone network gradually extracts features of the input image through the plurality of feature extraction layers to obtain a plurality of preliminary feature images of different scales, and outputs the plurality of preliminary feature images to the scale adjustment unit;

[0011] The scale adjustment unit is used to adjust the scale of the input preliminary feature images to obtain pre-fusion feature images corresponding to each preliminary feature image, and the number of channels of all the pre-fusion feature images is the same;

[0012] The feature enhancement and fusion unit includes a shallow fusion module, a middle fusion module and a deep fusion module. The shallow fusion module, the middle fusion module and the deep fusion module each perform feature enhancement and fusion processing on a plurality of feature images before fusion to obtain a fused feature image; the feature images before fusion fused by the shallow fusion module, the middle fusion module and the deep fusion module are not completely the same;

[0013] The feature image output unit is used to generate and output a plurality of output feature images of different scales using all the fused feature images; the number of the output feature images is not less than the number of the fused feature images.

[0014] The model introduces shallow fusion module, middle fusion module and deep fusion module to perform feature enhancement and fusion processing on preliminary feature images of different scales extracted from the input image, which is conducive to maximizing the extraction of features at different levels for subsequent target detection, thereby improving the accuracy and robustness of small target detection.

[0015] Preferably, the backbone network is a ResNet50 network, and the ResNet50 network includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer and a fifth feature extraction layer connected in sequence from front to back; the output ends of the third feature extraction layer, the fourth feature extraction layer and the fifth feature extraction layer are all connected to the scale adjustment unit to output the corresponding preliminary feature image to the scale adjustment unit.

[0016] Preferably, the scale adjustment unit includes a first convolution block, a second convolution block and a third convolution block;

[0017] The input end of the first convolution block is connected to the output end of the third feature extraction layer, and the output end of the first convolution block is connected to the input ends of the middle layer fusion module and the deep layer fusion module;

[0018] The input end of the second convolution block is connected to the output end of the fourth feature extraction layer, and the output end of the second convolution block is connected to the input ends of the shallow fusion module, the middle fusion module and the deep fusion module;

[0019] The input end of the third convolution block is connected to the output end of the fifth feature extraction layer, and the output end of the third convolution block is connected to the input ends of the shallow fusion module and the middle layer fusion module.

[0020] Through the above connection structure, the shallow fusion module is conducive to extracting more shallow features for fusion, the middle-level fusion module is conducive to extracting more middle-level features for fusion, and the deep fusion module is conducive to extracting more deep features for fusion, thereby improving the model's ability to express shallow features, middle-level features and deep features of the input image.

[0021] Preferably, the feature image output unit includes a fourth convolution block, a fifth convolution block, a sixth convolution block, a seventh convolution block and an eighth convolution block;

[0022] The input end of the fourth convolution block is connected to the output end of the deep fusion module, the input end of the fifth convolution block is connected to the output end of the middle fusion module, the input end of the sixth convolution block is connected to the output end of the shallow fusion module, the input end of the seventh convolution block is connected to the output end of the sixth convolution block, and the input end of the eighth convolution block is connected to the output end of the seventh convolution block; the fourth convolution block, the fifth convolution block, the sixth convolution block, the seventh convolution block and the eighth convolution block are each used to output one of the output feature images.

[0023] The feature image output unit can output feature images of more scales for subsequent target detection, which is beneficial to further improve the accuracy and robustness of small target detection.

[0024] Preferably, the shallow fusion module includes a first SE attention module, a first convolutional attention module and a first maximum pooling layer;

[0025] The input end of the first SE attention module is connected to the output end of the third convolution block, the first convolution attention module is connected to the first maximum pooling layer front and back, the input end of the first convolution attention module is connected to the output end of the second convolution block, the output of the first SE attention module is multiplied element by element with the output of the first maximum pooling layer and then added element by element with the output of the third convolution block, and the addition result is used as the output of the shallow fusion module.

[0026] Preferably, the middle-layer fusion module includes a second SE attention module, a first sub-pixel convolutional layer, a first convolutional layer, a second convolutional attention module, and a second maximum pooling layer;

[0027] The second SE attention module and the first sub-pixel convolution layer are connected front and back, the input end of the second SE attention module is connected to the output end of the third convolution block, the input end of the first convolution layer is connected to the output end of the second convolution block, the second convolution attention module and the second maximum pooling layer are connected front and back, the input end of the second convolution attention module is connected to the output end of the first convolution block, the output of the first sub-pixel convolution layer, the output of the first convolution layer, the output of the second convolution block and the output of the second maximum pooling layer are added element by element, and the addition result is used as the output of the middle-layer fusion module.

[0028] Preferably, the first convolutional attention module and the second convolutional attention module each include a first input layer, a channel attention module, a spatial attention module and a first output layer;

[0029] The first input layer and the channel attention module are connected front and back, the output of the channel attention module and the output of the first input layer are multiplied element by element to obtain a first feature map, the first feature map is input into the spatial attention module, and the output of the spatial attention module is multiplied element by element with the first feature map and then input into the first output layer.

[0030] Preferably, the deep fusion module includes a second convolution layer, a second sub-pixel convolution layer and a CE feature enhancement module;

[0031] The second convolution layer and the second sub-pixel convolution layer are connected front and back, the input end of the second convolution layer is connected to the output end of the second convolution block, the output of the first convolution block and the output of the second sub-pixel convolution layer are added element by element and then input into the CE feature enhancement module, the output of the CE feature enhancement module and the output of the second sub-pixel convolution layer are added element by element, and the addition result is used as the output of the deep fusion module.

[0032] Preferably, the CE feature enhancement module includes a first two-dimensional convolutional layer, a first BN layer, a ReLU activation function layer, a second two-dimensional convolutional layer and a second BN layer which are sequentially connected from front to back.

[0033] In a second aspect, the present application provides a small target detection method based on multi-scale feature fusion, comprising the steps of:

[0034] A1. Constructing an initial small target detection model based on multi-scale feature fusion; the initial small target detection model based on multi-scale feature fusion is the small target detection model based on multi-scale feature fusion described above;

[0035] A2. training the initial small target detection model based on multi-scale feature fusion to obtain a trained small target detection model based on multi-scale feature fusion;

[0036] A3. Input the image to be identified into the trained small target detection model based on multi-scale feature fusion to obtain a plurality of output feature images output by the trained small target detection model based on multi-scale feature fusion;

[0037] A4. Utilize the plurality of output feature images to perform target detection.

[0038] Beneficial effects: The small target detection method and model based on multi-scale feature fusion provided in the present application introduces a shallow fusion module, a middle fusion module and a deep fusion module to perform feature enhancement and fusion processing on preliminary feature images of different scales extracted from the input image, which is conducive to maximizing the extraction of features at different levels for subsequent target detection, thereby improving the accuracy and robustness of small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the structure of a small target detection model based on multi-scale feature fusion provided in an embodiment of the present application.

[0040] Figure 2 Schematic diagram of the structure of the shallow fusion module.

[0041] Figure 3 It is a structural diagram of the middle-level fusion module.

[0042] Figure 4 Schematic diagram of the structure of the deep fusion module.

[0043] Figure 5 Schematic diagram of the structure of the first convolutional attention module and the second convolutional attention module.

[0044] Figure 6 Schematic diagram of the structure of the channel attention module.

[0045] Figure 7 Schematic diagram of the structure of the spatial attention module.

[0046] Figure 8 It is a structural diagram of the CE feature enhancement module.

[0047] Fig. 9 A flowchart of a small target detection method based on multi-scale feature fusion provided in an embodiment of the present application.

[0048] Fig.10 This is a graph of experimental results of the small target detection method based on multi-scale feature fusion provided in an embodiment of the present application.

[0049] Fig.11 This is the experimental result diagram of the existing target detection method based on the FPN algorithm.

[0050] Explanation of reference numerals: 1. Backbone network; 101. First feature extraction layer; 102. Second feature extraction layer; 103. Third feature extraction layer; 104. Fourth feature extraction layer; 105. Fifth feature extraction layer; 2. Improved feature pyramid network; 201. Scale adjustment unit; 202. Feature enhancement and fusion unit; 203. Feature image output unit; 3. Shallow fusion module; 301. First SE attention module; 302. First convolutional attention module; 303. First maximum pooling layer; 4. Middle fusion module; 401. Second SE attention module; 402. First sub-pixel convolutional layer; 403. First convolutional layer; 404. Second convolutional attention module; 405. Second maximum pooling layer; 5. Deep fusion module; 501. Second convolutional layer; 502. Second sub-pixel convolutional layer; 503. CE feature enhancement module; 6. First convolutional block; 7. Second convolution block; 8. Third convolution block; 9. Fourth convolution block; 10. Fifth convolution block; 11. Sixth convolution block; 12. Seventh convolution block; 13. Eighth convolution block; 14. First input layer; 15. Channel attention module; 1501. Second input layer; 1502. First average pooling layer; 1503. Third maximum pooling layer; 1504. Multilayer perceptron; 1505. First activation function layer; 1506. Second output layer; 16. Spatial attention module; 1601. Third input layer; 1602. Fourth maximum pooling layer; 1603. Second average pooling layer; 1604. Third convolution layer; 1605. Second activation function layer; 1606. Third output layer; 17. First output layer; 18. First two-dimensional convolution layer; 19. First BN layer; 20. ReLU activation function layer; 21. Second two-dimensional convolution layer; 22. Second BN layer. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0052] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0053] Please refer to Figure 1 , Figure 1 It is a small target detection model based on multi-scale feature fusion in some embodiments of the present application, including a backbone network 1 and an improved feature pyramid network 2; the improved feature pyramid network 2 includes a scale adjustment unit 201, a feature enhancement and fusion unit 202 and a feature image output unit 203;

[0054] The backbone network 1 has multiple feature extraction layers. The backbone network 1 gradually extracts the features of the input image through the multiple feature extraction layers to obtain multiple preliminary feature images of different scales, and converts the multiple preliminary feature images (which can be partial preliminary feature images or all preliminary feature images, for example) into Figure 1 C3, C4 and C5 in the figure are partial preliminary feature images) are output to the scale adjustment unit 201;

[0055] The scale adjustment unit 201 is used to adjust the scale of the input preliminary feature images to obtain the pre-fusion feature images corresponding to each preliminary feature image (for example Figure 1 P3', P4' and P5' in the figure are the feature images before fusion corresponding to C3, C4 and C5 respectively), and the number of channels of all feature images before fusion is the same;

[0056] The feature enhancement and fusion unit 202 includes a shallow fusion module 3, a middle fusion module 4 and a deep fusion module 5. The shallow fusion module 3, the middle fusion module 4 and the deep fusion module 5 each perform feature enhancement and fusion processing on a plurality of pre-fusion feature images to obtain a fused feature image (e.g. Figure 1 , P3, P4 and P5 are fused feature images corresponding to P3', P4' and P5' respectively); the feature images before fusion fused by the shallow fusion module 3, the middle fusion module 4 and the deep fusion module 5 are not completely the same;

[0057] The feature image output unit 203 is used to generate and output multiple output feature images of different scales using all fused feature images (for example Figure 1 O3, O4, O5, O6, and O7 in are all output feature images); the number of output feature images is not less than the number of fused feature images.

[0058] The model introduces a shallow fusion module 3, a middle fusion module 4 and a deep fusion module 5 to perform feature enhancement and fusion processing on preliminary feature images of different scales extracted from the input image, which is beneficial to maximize the extraction of features at different levels for subsequent target detection, thereby improving the accuracy and robustness of small target detection (that is, the small target detection model based on multi-scale feature fusion is used to extract output feature images of multiple scales from a single input image, which are used as input data for other target detection algorithms, thereby improving the accuracy and robustness of target detection).

[0059] The scale of an image in the present application includes width, height and number of channels, and the different scales of two images refer to at least one of the width, height and number of channels of the two images being different.

[0060] Among them, in backbone network 1, from front to back (based on the direction of information transmission), the width and height of the preliminary feature images extracted by each feature extraction layer gradually decrease, and the number of channels gradually increases.

[0061] The specific number of feature extraction layers of the backbone network 1 and the number of preliminary feature images output to the scale adjustment unit 201 can be set according to actual needs and are not limited here.

[0062] For example, in some possible implementations, see Figure 1 , the backbone network 1 is a ResNet50 network, which includes a first feature extraction layer 101 (i.e., the stage0 layer of the ResNet50 network), a second feature extraction layer 102 (i.e., the stage1 layer of the ResNet50 network), a third feature extraction layer 103 (i.e., the stage2 layer of the ResNet50 network), a fourth feature extraction layer 104 (i.e., the stage3 layer of the ResNet50 network), and a fifth feature extraction layer 105 (i.e., the stage4 layer of the ResNet50 network) connected in sequence from front to back; the output ends of the third feature extraction layer 103, the fourth feature extraction layer 104, and the fifth feature extraction layer 105 are all connected to the scale adjustment unit 201 to output the corresponding preliminary feature image (i.e., Figure 1 C3, C4 and C5 in ). Among them, the structure of each feature extraction layer (each stage layer) of the ResNet50 network is the existing technology and will not be described in detail here.

[0063] The scale of the preliminary feature image extracted by each feature extraction layer can be set according to actual needs. For example, the scale of the preliminary feature image extracted by the first feature extraction layer 101 is 320*320*64 (where the first number is the width, the second number is the height, and the third number is the number of channels), the scale of the preliminary feature image extracted by the second feature extraction layer 102 is 160*160*256, the scale of the preliminary feature image extracted by the third feature extraction layer 103 is 80*80*512, the scale of the preliminary feature image extracted by the fourth feature extraction layer 104 is 40*40*1025, and the scale of the preliminary feature image extracted by the fifth feature extraction layer 105 is 20*20*2048; but it is not limited thereto.

[0064] Further, see Figure 1 , the scale adjustment unit 201 includes a first convolution block 6, a second convolution block 7 and a third convolution block 8;

[0065] The input end of the first convolution block 6 is connected to the output end of the third feature extraction layer 103, and the output end of the first convolution block 6 is connected to the input ends of the middle layer fusion module 4 and the deep layer fusion module 5;

[0066] The input end of the second convolution block 7 is connected to the output end of the fourth feature extraction layer 104, and the output end of the second convolution block 7 is connected to the input ends of the shallow fusion module 3, the middle fusion module 4 and the deep fusion module 5;

[0067] The input end of the third convolution block 8 is connected to the output end of the fifth feature extraction layer 105 , and the output end of the third convolution block 8 is connected to the input ends of the shallow fusion module 3 and the middle fusion module 4 .

[0068] Through the above connection structure, the shallow fusion module 3 is conducive to extracting more shallow features for fusion, the middle-level fusion module 4 is conducive to extracting more middle-level features for fusion, and the deep fusion module 5 is conducive to extracting more deep features for fusion, thereby improving the model's ability to express shallow features, middle-level features and deep features of the input image.

[0069] Among them, the first convolution block 6, the second convolution block 7 and the third convolution block 8 are all two-dimensional convolution blocks with a convolution kernel size of 1*1 and a convolution step size of 1.

[0070] The scales of the feature images before fusion output by the first convolution block 6, the second convolution block 7, and the third convolution block 8 can be set according to actual needs. For example, the scale of the feature image before fusion output by the first convolution block 6 is 80*80*256, the scale of the feature image before fusion output by the second convolution block 7 is 40*40*256, and the scale of the feature image before fusion output by the third convolution block 8 is 20*20*256. At this time, the number of channels of all feature images before fusion is 256; but it is not limited to this.

[0071] The number of output feature images output by the feature image output unit 203 can be set according to actual needs.

[0072] In some possible implementations, see Figure 1 , the feature image output unit 203 includes a fourth convolution block 9, a fifth convolution block 10, a sixth convolution block 11, a seventh convolution block 12 and an eighth convolution block 13;

[0073] The input end of the fourth convolution block 9 is connected to the output end of the deep fusion module 5, the input end of the fifth convolution block 10 is connected to the output end of the middle fusion module 4, the input end of the sixth convolution block 11 is connected to the output end of the shallow fusion module 3, the input end of the seventh convolution block 12 is connected to the output end of the sixth convolution block 11, and the input end of the eighth convolution block 13 is connected to the output end of the seventh convolution block 12; the fourth convolution block 9, the fifth convolution block 10, the sixth convolution block 11, the seventh convolution block 12 and the eighth convolution block 13 are each used to output an output feature image.

[0074] The feature image output unit 203 can output feature images of more scales for subsequent target detection, which is beneficial to further improve the accuracy and robustness of small target detection.

[0075] Among them, the fourth convolution block 9, the fifth convolution block 10 and the sixth convolution block 11 are all two-dimensional convolution blocks with a convolution kernel size of 3*3 and a convolution step of 1, and the seventh convolution block 12 and the eighth convolution block 13 are both two-dimensional convolution blocks with a convolution kernel size of 3*3, a convolution step of 2 and a padding number of 1 (i.e., padding is 1).

[0076] The scales of the output feature images output by the fourth convolution block 9, the fifth convolution block 10, the sixth convolution block 11, the seventh convolution block 12, and the eighth convolution block 13 can be set according to actual needs. For example, the scale of the output feature image output by the fourth convolution block 9 is 80*80*256, the scale of the output feature image output by the fifth convolution block 10 is 40*40*256, the scale of the output feature image output by the sixth convolution block 11 is 20*20*256, the scale of the output feature image output by the seventh convolution block 12 is 10*10*256, and the scale of the output feature image output by the eighth convolution block 13 is 5*5*256; but it is not limited thereto.

[0077] Specifically, see Figure 2, the shallow fusion module 3 includes a first SE attention module 301 (SE attention module is Squeeze-and-Excitation module), a first convolutional attention module 302 (convolutional attention module is CBAM module, Convolutional Block Attention Module) and a first maximum pooling layer 303;

[0078] The input end of the first SE attention module 301 is connected to the output end of the third convolution block 8, the first convolution attention module 302 is connected to the first maximum pooling layer 303 front and back, the input end of the first convolution attention module 302 is connected to the output end of the second convolution block 7, the output of the first SE attention module 301 is multiplied element by element with the output of the first maximum pooling layer 303, and then added element by element with the output of the third convolution block 8, and the addition result is used as the output of the shallow fusion module 3.

[0079] When working, the feature image P5' before fusion is recalibrated by the first SE attention module 301; the feature image P4' before fusion is pooled by the first maximum pooling layer 303 after the attention information is extracted by the first convolutional attention module 302; the output of the first SE attention module 301 is multiplied element by element with the output of the first maximum pooling layer 303 to obtain the fusion feature, and finally the fusion feature is added element by element to the original feature image P5' before fusion, and the fusion feature image P5 is output. The shallow fusion module 3 enhances the expression ability of shallow features and improves the effect of feature fusion by combining the attention mechanism and feature recalibration.

[0080] Among them, the structure of the first SE attention module 301 is the existing technology and will not be described in detail here.

[0081] Specifically, see Figure 3 , the middle layer fusion module 4 includes a second SE attention module 401, a first sub-pixel convolution layer 402 (sub-pixel convolution layer, i.e., Subconv layer), a first convolution layer 403, a second convolution attention module 404, and a second maximum pooling layer 405;

[0082] The second SE attention module 401 and the first sub-pixel convolution layer 402 are connected front to back, the input end of the second SE attention module 401 is connected to the output end of the third convolution block 8, the input end of the first convolution layer 403 is connected to the output end of the second convolution block 7, the second convolution attention module 404 and the second maximum pooling layer 405 are connected front to back, the input end of the second convolution attention module 404 is connected to the output end of the first convolution block 6, the output of the first sub-pixel convolution layer 402, the output of the first convolution layer 403, the output of the second convolution block 7 and the output of the second maximum pooling layer 405 are added element by element, and the addition result is used as the output of the middle-layer fusion module 4.

[0083] When working, the feature image P5' before fusion is recalibrated by the second SE attention module 401, and then the sub-pixel convolution upsampling operation is performed; the feature image P4' before fusion is processed by the first convolution layer 403 to enhance the feature expression; the feature image P3' before fusion is extracted by the second convolution attention module 404, and then the second maximum pooling layer 405 performs a pooling operation; the output of the first sub-pixel convolution layer 402, the output of the first convolution layer 403, the original feature image P4' before fusion and the output of the second maximum pooling layer 405 are added element by element to obtain the fused feature image P4. The middle-layer fusion module 4 enhances the expressive power of the middle-layer features and improves the effect of feature fusion by fusing multi-level features.

[0084] Among them, the structures of the second SE attention module 401 and the first sub-pixel convolution layer 402 are both existing technologies and will not be described in detail here.

[0085] Specifically, see Figure 5 , the first convolutional attention module 302 and the second convolutional attention module 404 both include a first input layer 14, a channel attention module 15, a spatial attention module 16 and a first output layer 17;

[0086] The first input layer 14 and the channel attention module 15 are connected front and back, and the output of the channel attention module 15 and the output of the first input layer 14 are multiplied element by element to obtain a first feature map. The first feature map is input into the spatial attention module 16, and the output of the spatial attention module 16 is multiplied element by element with the first feature map and then input into the first output layer 17.

[0087] The first convolutional attention module 302 and the second convolutional attention module 404 can effectively filter background information and highlight the features of small targets by combining the spatial attention mechanism and the channel attention mechanism, so that small targets can be better detected in complex backgrounds, which significantly improves the robustness of the detection model in complex scenes.

[0088] Further, see Figure 6 , the channel attention module 15 includes a second input layer 1501, a first average pooling layer 1502, a third maximum pooling layer 1503, a multilayer perceptron 1504 (i.e., MLP, Multilayer Perceptron), a first activation function layer 1505, and a second output layer 1506;

[0089] The input ends of the first average pooling layer 1502 and the third maximum pooling layer 1503 are both connected to the output end of the second input layer 1501, and the output ends of the first average pooling layer 1502 and the third maximum pooling layer 1503 are both connected to the input end of the multilayer perceptron 1504. The two outputs of the multilayer perceptron 1504 (the two outputs of the multilayer perceptron 1504 correspond to the two inputs of the multilayer perceptron 1504, that is, the multilayer perceptron 1504 processes the output features of the first average pooling layer 1502 and the third maximum pooling layer 1503, respectively, to obtain two processed output features) are added element by element and then input into the first activation function layer 1505. The output end of the first activation function layer 1505 is connected to the input end of the second output layer 1506.

[0090] The multilayer perceptron 1504 is a prior art and will not be described in detail herein. The activation function of the first activation function layer 1505 can be set according to actual needs, such as a Sigmoid activation function, but is not limited thereto.

[0091] Further, see Figure 7 The spatial attention module 16 includes a third input layer 1601, a fourth maximum pooling layer 1602, a second average pooling layer 1603, a third convolutional layer 1604, a second activation function layer 1605 and a third output layer 1606 which are connected sequentially from front to back.

[0092] The activation function of the second activation function layer 1605 can be set according to actual needs, for example, a Sigmoid activation function, but not limited thereto.

[0093] Specifically, see Figure 4 , the deep fusion module 5 includes a second convolution layer 501, a second sub-pixel convolution layer 502 and a CE feature enhancement module 503 (i.e., Context Encoding Module);

[0094] The second convolution layer 501 and the second sub-pixel convolution layer 502 are connected front to back, the input end of the second convolution layer 501 is connected to the output end of the second convolution block 7, the output of the first convolution block 6 and the output of the second sub-pixel convolution layer 502 are added element by element and then input into the CE feature enhancement module 503, the output of the CE feature enhancement module 503 and the output of the second sub-pixel convolution layer 502 are added element by element, and the addition result is used as the output of the deep fusion module 5.

[0095] When working, the feature image P4' before fusion is subjected to feature extraction by the second convolution layer 501, and then subjected to downsampling operation by the second sub-pixel convolution layer 502 to reduce the spatial size of the feature map; the feature image P3' before fusion is added element by element with the output of the second sub-pixel convolution layer 502 to obtain the second feature map; the second feature map is input into the CE feature enhancement module 503 for feature enhancement; the output of the CE feature enhancement module 503 and the output of the second sub-pixel convolution layer 502 are added element by element to obtain the fused feature image P3. The deep fusion module 5 improves the expression ability of deep features and the effect of feature fusion through feature enhancement and multi-level fusion.

[0096] Among them, the second sub-pixel convolution layer 502 is a prior art and will not be described in detail here.

[0097] Further, see Figure 8 The CE feature enhancement module 503 includes a first two-dimensional convolution layer 18, a first BN layer 19 (BN layer is Batch Norm layer, also known as batch normalization layer), a ReLU activation function layer 20, a second two-dimensional convolution layer 21 and a second BN layer 22, which are connected in sequence from front to back. The CE feature enhancement module 503 can effectively enhance the features of the input feature map.

[0098] Please refer to Fig. 9 , the present application provides a small target detection method based on multi-scale feature fusion, comprising the steps of:

[0099] A1. Construct an initial small target detection model based on multi-scale feature fusion; the initial small target detection model based on multi-scale feature fusion is the small target detection model based on multi-scale feature fusion mentioned above;

[0100] A2. Train the initial small target detection model based on multi-scale feature fusion to obtain a trained small target detection model based on multi-scale feature fusion;

[0101] A3. Input the image to be identified into the trained small target detection model based on multi-scale feature fusion to obtain multiple output feature images output by the trained small target detection model based on multi-scale feature fusion;

[0102] A4. Utilize multiple output feature images for target detection.

[0103] Among them, step A1 includes: constructing a model with the same structure as the small target detection model based on multi-scale feature fusion in the previous text, and initializing the model parameters of the model to obtain an initial small target detection model based on multi-scale feature fusion.

[0104] Among them, the existing technology can be used to train the initial small target detection model based on multi-scale feature fusion. In some possible implementations, step A2 includes:

[0105] A201. Obtain a training data set including a plurality of sample data; the sample data includes feature data (ie, sample images) and corresponding label data;

[0106] A202. Divide the training data set into a training set and a test set;

[0107] A203. Input the feature data of the sample data of the training set into the initial small target detection model based on multi-scale feature fusion to obtain the output feature image output by the initial small target detection model based on multi-scale feature fusion;

[0108] A204. Use a preset target detector (Head) to perform target detection on the output feature image output by the initial small target detection model based on multi-scale feature fusion to obtain the corresponding target detection result;

[0109] A205. Calculate the loss function according to the target detection result and the corresponding label data, and update the model parameters of the initial small target detection model based on multi-scale feature fusion according to the loss function to obtain an updated small target detection model based on multi-scale feature fusion;

[0110] A206. If the loss function converges, execute step A207, otherwise, return to step A203;

[0111] A207. Use the test set to test the updated small target detection model based on multi-scale feature fusion. If it passes the test, end the training and use the updated small target detection model based on multi-scale feature fusion as the trained small target detection model based on multi-scale feature fusion. Otherwise, return to execute step A203.

[0112] In step A201, sample data can be extracted from the open source Visdrone dataset, and data enhancement (such as horizontal flipping, vertical flipping, etc.) can be performed to obtain a training dataset.

[0113] Among them, in step A205, the Adam optimizer can be used to update the model parameters of the initial small target detection model based on multi-scale feature fusion, wherein the initial learning rate can be set according to actual needs (for example, 0.0002, but not limited to this), and the learning rate is reduced through the cosine annealing scheme.

[0114] Wherein, step A4 includes: using a preset target detector to perform target detection on multiple output feature images output by the trained small target detection model based on multi-scale feature fusion. Wherein, the preset target detector can be an existing target detector, and the recognition result can be used for target classification and / or target positioning.

[0115] The target detection experiment was conducted on sample images in the Visdrone dataset using the small target detection method based on multi-scale feature fusion of the present application and the existing target detection method based on the FPN algorithm, and the performance of the two methods was evaluated using the mean average precision (mAP) as the evaluation index. The targets in the sample images in the Visdrone dataset include cars, buses, trucks, vans, pedestrians, motorcycles, tricycles, covered tricycles, bicycles, ignored areas, and people.

[0116] The experimental results are as follows Fig.10 and Fig.11 As shown, AP is the average precision corresponding to a single target type, and mAP is the average precision of all target types; it can be seen from the figure that the evaluation index of the small target detection method based on multi-scale feature fusion in this application is significantly better than the existing target detection method based on the FPN algorithm, and the mAP is improved by 5.35%, and the accuracy of each target type is better than the existing target detection method based on the FPN algorithm. It is worth noting that the detection accuracy of vehicle targets (cars, buses, trucks, vans) has been significantly improved, especially the AP of trucks and vans has increased by 8% and 7% respectively. The detection accuracy of pedestrian targets (pedestrians, motorcycles, tricycles, covered tricycles, bicycles) has been improved to varying degrees, especially the AP of pedestrians and motorcycles has increased by 3% and 5% respectively. The detection accuracy of small targets (such as bicycles and covered tricycles) has been particularly improved, and the AP of bicycles has increased from 0.05 to 0.10, an increase of 100%. It can be seen that the present application significantly enhances the multi-scale feature fusion capability and effectively improves the detection accuracy of small targets by introducing the shallow fusion module 3, the middle fusion module 4 and the deep fusion module 5.

[0117] In summary, this application has the following advantages:

[0118] 1. Significant feature enhancement effect: The present invention introduces a CE feature enhancement module 503 in the deep fusion module 5, which significantly enhances the expression ability of the high-resolution underlying feature map, making the position of small targets more prominent, and effectively improving the detection accuracy of small targets;

[0119] 2. Multi-scale information fusion is more comprehensive; the SE spatial attention module and the convolutional attention module are introduced in the middle-level fusion module 4, which can better combine the multi-scale information of the medium-scale feature map, improve the detection ability of small targets, and make the detection model use the multi-scale information more comprehensively and fully;

[0120] 3. Background interference filtering is more effective; the convolutional attention module and SE attention module are introduced in the shallow fusion module 3. By combining the spatial attention mechanism and the channel attention mechanism, the background information is effectively filtered and the features of small targets are highlighted. In addition, the sub-pixel convolution module and CE feature enhancement module are introduced in the deep fusion module, which also have a certain background filtering effect, so that small targets can be better detected in complex backgrounds, which significantly improves the robustness of the detection model in complex scenes;

[0121] 4. The detection accuracy is significantly improved; through the introduction of the above three attention mechanisms and the optimization of the multi-level fusion module, the detection accuracy of small targets is significantly improved; the experimental results show that the shallow fusion module 3, the middle fusion module 4 and the deep fusion module 5 all have significant improvements on the small target detection accuracy, which is particularly suitable for small target detection tasks in UAV images.

[0122] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0123] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0125] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0126] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A small target detection model based on multi-scale feature fusion, characterized in that: It comprises a backbone network (1) and an improved feature pyramid network (2); the improved feature pyramid network (2) comprises a scale adjustment unit (201), a feature enhancement and fusion unit (202) and a feature image output unit (203); The backbone network (1) has a plurality of feature extraction layers, and the backbone network (1) gradually extracts features of an input image through the plurality of feature extraction layers to obtain a plurality of preliminary feature images of different scales, and outputs the plurality of preliminary feature images to the scale adjustment unit (201); The scale adjustment unit (201) is used to adjust the scale of the input preliminary feature images to obtain pre-fusion feature images corresponding to each preliminary feature image, and the number of channels of all the pre-fusion feature images is the same; The feature enhancement and fusion unit (202) comprises a shallow fusion module (3), a middle fusion module (4) and a deep fusion module (5); the shallow fusion module (3), the middle fusion module (4) and the deep fusion module (5) respectively perform feature enhancement and fusion processing on a plurality of pre-fusion feature images to obtain fused feature images; the pre-fusion feature images fused by the shallow fusion module (3), the middle fusion module (4) and the deep fusion module (5) are not completely the same; The feature image output unit (203) is used to generate and output a plurality of output feature images of different scales using all the fused feature images; the number of the output feature images is not less than the number of the fused feature images; The backbone network (1) is a ResNet50 network, and the ResNet50 network comprises a first feature extraction layer (101), a second feature extraction layer (102), a third feature extraction layer (103), a fourth feature extraction layer (104), and a fifth feature extraction layer (105) which are connected in sequence from front to back; the output ends of the third feature extraction layer (103), the fourth feature extraction layer (104), and the fifth feature extraction layer (105) are all connected to the scale adjustment unit (201) to output the corresponding preliminary feature image to the scale adjustment unit (201); The scale adjustment unit (201) comprises a first convolution block (6), a second convolution block (7) and a third convolution block (8); The input end of the first convolution block (6) is connected to the output end of the third feature extraction layer (103), and the output end of the first convolution block (6) is connected to the input ends of the middle layer fusion module (4) and the deep layer fusion module (5); The input end of the second convolution block (7) is connected to the output end of the fourth feature extraction layer (104), and the output end of the second convolution block (7) is connected to the input ends of the shallow fusion module (3), the middle fusion module (4) and the deep fusion module (5); The input end of the third convolution block (8) is connected to the output end of the fifth feature extraction layer (105), and the output end of the third convolution block (8) is connected to the input ends of the shallow layer fusion module (3) and the middle layer fusion module (4); The feature image output unit (203) comprises a fourth convolution block (9), a fifth convolution block (10), a sixth convolution block (11), a seventh convolution block (12) and an eighth convolution block (13); The input end of the fourth convolution block (9) is connected to the output end of the deep fusion module (5), the input end of the fifth convolution block (10) is connected to the output end of the middle fusion module (4), the input end of the sixth convolution block (11) is connected to the output end of the shallow fusion module (3), the input end of the seventh convolution block (12) is connected to the output end of the sixth convolution block (11), and the input end of the eighth convolution block (13) is connected to the output end of the seventh convolution block (12); the fourth convolution block (9), the fifth convolution block (10), the sixth convolution block (11), the seventh convolution block (12) and the eighth convolution block (13) are each used to output one of the output feature images; The shallow fusion module (3) includes a first SE attention module (301), a first convolutional attention module (302) and a first maximum pooling layer (303); The input end of the first SE attention module (301) is connected to the output end of the third convolution block (8), the first convolution attention module (302) is connected to the first maximum pooling layer (303) front and back, the input end of the first convolution attention module (302) is connected to the output end of the second convolution block (7), the output of the first SE attention module (301) is multiplied element by element with the output of the first maximum pooling layer (303), and then is added element by element with the output of the third convolution block (8), and the addition result is used as the output of the shallow fusion module (3); The middle layer fusion module (4) includes a second SE attention module (401), a first sub-pixel convolution layer (402), a first convolution layer (403), a second convolution attention module (404) and a second maximum pooling layer (405); The second SE attention module (401) and the first sub-pixel convolution layer (402) are connected front to back, the input end of the second SE attention module (401) is connected to the output end of the third convolution block (8), the input end of the first convolution layer (403) is connected to the output end of the second convolution block (7), the second convolution attention module (404) and the second maximum pooling layer (405) are connected front to back, the input end of the second convolution attention module (404) is connected to the output end of the first convolution block (6), the output of the first sub-pixel convolution layer (402), the output of the first convolution layer (403), the output of the second convolution block (7) and the output of the second maximum pooling layer (405) are added element by element, and the addition result is used as the output of the middle layer fusion module (4); The first convolutional attention module (302) and the second convolutional attention module (404) each include a first input layer (14), a channel attention module (15), a spatial attention module (16) and a first output layer (17); The first input layer (14) and the channel attention module (15) are connected front to back, the output of the channel attention module (15) and the output of the first input layer (14) are multiplied element by element to obtain a first feature map, the first feature map is input into the spatial attention module (16), and the output of the spatial attention module (16) is multiplied element by element with the first feature map and then input into the first output layer (17); The deep fusion module (5) comprises a second convolution layer (501), a second sub-pixel convolution layer (502) and a CE feature enhancement module (503); The second convolution layer (501) and the second sub-pixel convolution layer (502) are connected front to back, the input end of the second convolution layer (501) is connected to the output end of the second convolution block (7), the output of the first convolution block (6) and the output of the second sub-pixel convolution layer (502) are added element by element and then input into the CE feature enhancement module (503), the output of the CE feature enhancement module (503) and the output of the second sub-pixel convolution layer (502) are added element by element, and the addition result is used as the output of the deep fusion module (5); The CE feature enhancement module (503) comprises a first two-dimensional convolutional layer (18), a first BN layer (19), a ReLU activation function layer (20), a second two-dimensional convolutional layer (21) and a second BN layer (22) which are sequentially connected from front to back.

2. A small target detection method based on multi-scale feature fusion, characterized in that: Includes steps: A1. Constructing an initial small target detection model based on multi-scale feature fusion; the initial small target detection model based on multi-scale feature fusion is the small target detection model based on multi-scale feature fusion according to claim 1; A2. training the initial small target detection model based on multi-scale feature fusion to obtain a trained small target detection model based on multi-scale feature fusion; A3. Input the image to be identified into the trained small target detection model based on multi-scale feature fusion to obtain a plurality of output feature images output by the trained small target detection model based on multi-scale feature fusion; A4. Utilize the plurality of output feature images to perform target detection.

Images