A small target detection method based on intra-layer and inter-layer fusion and related products

Abstract

The present invention discloses a small target detection method based on intra-layer-inter-layer fusion and related products, belonging to the field of image processing technology. The method includes: natural images of multiple small targets, marking the categories and positions of the small targets in the natural images, and constructing a small target data set; constructing a target detection network model to extract features from the small target data, and obtaining multiple groups of multi-scale feature maps; using a feature fusion method based on intra-layer-inter-layer fusion, the extracted multi-scale feature maps are fused to obtain an intra-layer-inter-layer fusion feature map; using the enhanced intra-layer-inter-layer fusion feature map to make predictions and evaluate the pre-stored structure to obtain a trained target detection network model and perform target prediction on the image to be detected. The present invention can not only solve the problem that the traditional inter-layer fusion method cannot learn the fine local representation of small targets, but also improve the unstable position of small target detection caused by the IoU-based metric method.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to a small target detection method based on intra-layer and inter-layer fusion and related products. Background Art

[0002] Among the two popular target detection methods, most of the one-stage target detection methods and the two-stage target detection methods require the use of convolutional neural networks to extract image features. However, when actually used in small target detection, due to the fuzzy edge information and little semantic information of small target objects, it is difficult to obtain complete feature information of small targets in most cases. The feature information of small target objects is lost, resulting in unsatisfactory detection results of small targets.

[0003] The shallow network of the convolutional neural network extracts spatial information, which is beneficial for positioning; the deep network extracts semantic information, which is beneficial for classification. Multi-scale feature fusion is one of the effective methods to improve the performance of small target detection. At present, the target detection methods based on deep convolutional neural networks for feature fusion mainly focus on the fusion of inter-layer features. There is a large gap between the receptive fields of features at different levels, and the representation learned through inter-layer feature fusion is relatively rough, which hinders the precise positioning of small targets. The gap between the receptive fields of intra-layer features is relatively small. Fusion of intra-layer features can extract more refined local representations and retain more accurate spatial information, which is very important for small target positioning.

[0004] Therefore, how to improve the integrity of small target location information feature extraction is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a small target detection method based on intra-layer and inter-layer fusion and related products, which can accurately determine the position and category of small targets when facing small target object detection tasks, and give the confidence of the small target object category.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] The present invention firstly discloses a small target detection method based on intra-layer and inter-layer fusion, which mainly consists of two parts: target detection network model training and small target detection.

[0008] Specifically, the target detection network model training steps include the following steps:

[0009] Step 1: Obtain a natural image including a variety of small objects, mark the categories and positions of the small objects in the natural image, and construct a small object dataset;

[0010] Step 2: construct a target detection network model, wherein the target detection network model includes a backbone feature extraction network, an intra-layer-inter-layer fusion module, and a prediction module connected in sequence;

[0011] Step 3: Input the small target data set into the backbone feature extraction network for feature extraction to obtain multiple groups of multi-scale feature maps;

[0012] Step 4: Input each set of multi-scale feature maps obtained into the intra-layer-inter-layer fusion module to perform intra-layer feature fusion to obtain the corresponding intra-layer fusion feature map;

[0013] Step 5: The intra-layer fusion feature map is fused from top to bottom according to the level of the layer to generate an intra-layer-inter-layer fusion feature map;

[0014] Step 6: Input the generated intra-layer-inter-layer fusion feature map into the prediction module to obtain the target center point prediction heat map, target size prediction map and target center point offset prediction map corresponding to the natural image, and obtain the prediction result;

[0015] Step 7: Evaluate the obtained prediction results through the loss function, and train the backbone feature extraction network and the intra-layer-inter-layer feature fusion module to obtain a trained target detection model;

[0016] Small target detection steps include:

[0017] The size of the image to be detected is unified to the preset size, and the trained target detection model is input to obtain the location and category information of small targets in the image to be detected.

[0018] Furthermore, in step 2, the backbone feature extraction network includes a ResNet50 network; the ResNet50 network includes a convolution module and an identity mapping module, wherein the convolution module is used to change the dimension of the feature; and the identity mapping module is used to deepen the depth of the backbone feature extraction network.

[0019] Furthermore, the step 3 of obtaining multiple groups of multi-scale feature maps specifically includes:

[0020] The small target dataset is input into the ResNet50 network, and four groups of multi-scale feature maps T1, T2, T3 and T4 with different depths are extracted, with their sizes halved and their channels doubled.

[0021] Furthermore, in step 4, intra-layer feature fusion is performed, which specifically includes the following steps:

[0022] Step 41: Obtain each set of multi-scale feature maps T i , i=1,2,3,4 is divided into four branches according to the number of channels f i1 、fi2 、f i3 、f i4 ;

[0023] Step 42: First Branch f i1 Perform 1×1 convolution dimensionality reduction and 3×3 convolution to obtain the first branch f i1 Output

[0024] Step 43: Second Branch f i2 After 1×1 convolution and dimension reduction, it is combined with the first branch f i1 The output L1 is concatenated and then 3×3 convolution is performed to obtain the intermediate fusion feature Perform 3×3 convolution again to get the second branch f i2 Output

[0025] Step 44: The Third Branch f i3 After 1×1 convolution and dimension reduction, the second branch f i2 Intermediate fusion features Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the second branch f i2 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again to get the third branch f i3 Output Intermediate fusion features That is, fusion f i1 、f i2 With f i3 The characteristics of the latter;

[0026] Step 45: The fourth branch f i4 After 1×1 convolution and dimension reduction, the third branch f i3 Intermediate fusion features Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the third branch f i3 The intermediate fusion feature C3 is concatenated and then 3×3 convolution is performed to obtain the intermediate fusion feature Intermediate fusion features With the third branch f i3 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again to get the fourth branch fi4 Output Intermediate fusion features That is, fusion f i1 、f i2 、f i3 With f i4 The characteristics of the latter;

[0027] Step 46: Output features of the four branches Splicing, performing 1×1 convolution, and obtaining a multi-scale feature map T i The corresponding intra-layer fusion feature map Z i , i=1,2,3,4.

[0028] Furthermore, in step 5, the intra-layer fusion feature map is subjected to inter-layer feature fusion from top to bottom according to the level of the layer, including the following steps:

[0029] Step 51: upsample the intra-layer fusion feature map Z4 to obtain an upsampled feature map, and then concatenate it with the intra-layer fusion feature map Z3 to obtain a fusion feature map U1;

[0030] Step 52: upsample the fused feature map U1 and concatenate it with the fused feature map Z2 within the layer to obtain the fused feature map U2;

[0031] Step 53: Upsample the fused feature map U2 and concatenate it with the intra-layer fused feature map Z1 to obtain a fused feature map U3;

[0032] Step 54: The fused feature map U3 is subjected to a 3×3 convolution to obtain a completely fused intra-layer and inter-layer fused feature map.

[0033] Further, in step 6, the prediction module specifically includes three branches, namely: a target heat map prediction branch, a target width and height prediction branch, and a target center point offset prediction branch;

[0034] Among them, the output layer number of the target heat map prediction branch is set to the number of target categories; the width and height prediction branch is used to output the width and height probability distribution vector of the target, and the output layer number of the target width and height prediction branch is 2*K, K is the vector length of the target width and height probability distribution; the output layer number of the target center point offset prediction branch is 2.

[0035] Furthermore, the step 7 specifically includes:

[0036] S71: obtaining a training set according to the small object data set, annotating each object in each image of the training set with a labeling box, and obtaining the bounding box coordinates and category information c of each object in the training image;

[0037] S72: Calculate the center point coordinate p of the rectangular bounding box of each category c;

[0038]

[0039] Where x1 is the horizontal coordinate of the upper left corner of the target bounding box; y1 is the vertical coordinate of the upper left corner of the target bounding box; x2 is the horizontal coordinate of the lower right corner of the target bounding box; y2 is the vertical coordinate of the lower right corner of the target bounding box;

[0040] S73: down-sample the center point, and generate a Gaussian heat map as the GT heat map with the coordinates of the down-sampled center point as the center;

[0041]

[0042] Where R is the downsampling factor, is the center point corresponding to the low resolution, σ p is the standard deviation of the adaptive size;

[0043] S74: GT heat map Y and prediction heat map Apply the normalized Gaussian wassertein distance loss function L w ;

[0044]

[0045] in, represents the second-order wassertein distance;

[0046] S75: Applying L1 loss function to the size prediction map and the center point offset prediction map;

[0047]

[0048] Among them, L size represents the size prediction graph loss function, L off represents the center point offset prediction graph loss function, N is the number of center points, s k represents the size of the kth target, Indicates that the center point position is p k The target size, Indicates the offset value predicted by the network;

[0049] S76: Add the loss functions to obtain a total loss function, and train the model by minimizing the total loss function, wherein the total loss function includes:

[0050] L=L w +λ size L size +λ off Loff

[0051] Among them, λ size =0.1,λ off =1.

[0052] On the other hand, the present invention further discloses a detection system for a small target detection method based on intra-layer-inter-layer fusion according to any one of the present inventions, comprising an image acquisition module, an image preprocessing module, a feature extraction module, a feature fusion module, a prediction module, an evaluation training module and a detection module;

[0053] An image acquisition module, used for acquiring natural images including a variety of small targets;

[0054] The image preprocessing module is used to adjust the collected natural images to a uniform preset scale and generate a small object data set;

[0055] The feature extraction module is used to receive the small target data set, perform feature extraction, and output multi-scale features;

[0056] The intra-layer-inter-layer fusion module is used to receive multi-scale features for feature fusion and output the fused intra-layer-inter-layer fusion feature map;

[0057] A prediction module is used to receive the generated intra-layer-inter-layer fusion feature map, obtain the target center point prediction heat map, target size prediction map and target center point offset prediction map corresponding to the natural image, and obtain the prediction result;

[0058] An evaluation training module is used to evaluate the obtained prediction results through a loss function, and to train the backbone feature extraction network and the intra-layer-inter-layer feature fusion module to obtain a trained target detection model;

[0059] The detection module is used to detect the image to be detected using the trained target detection model to obtain the location and category information of small targets in the image to be detected.

[0060] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any method described in the present invention are implemented.

[0061] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a small target detection method, system, storage medium and product based on intra-layer-inter-layer fusion, including: overlapping and cutting the data set images into a uniform size to perform data preprocessing; inputting the preprocessed data set images into the feature extraction network to extract the multi-scale features of the image; performing intra-layer-inter-layer feature fusion on the extracted multi-scale features to generate a fusion feature map; inputting the fusion feature map into the Head layer in the CenterNet model to obtain the center point prediction heat map, size prediction map and center point offset prediction map corresponding to the image to be detected; evaluating the obtained prediction results through an improved loss function, and training the model. Based on the feature fusion module of intra-layer-inter-layer feature fusion, a more refined local representation can be extracted, and the spatial feature information of small targets can be extracted as accurately as possible. The present invention can accurately judge the number and category of targets in the image (especially small targets) when facing a picture that needs to detect a target, and give a prediction score for target classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0063] Figure 1 A schematic diagram of the overall process of a small target detection method based on intra-layer and inter-layer fusion provided by an embodiment of the present invention;

[0064] Figure 2 A schematic diagram of a framework of a small target detection method based on intra-layer and inter-layer fusion provided in an embodiment of the present invention;

[0065] Figure 3 A schematic diagram of the intra-layer and inter-layer feature fusion process provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] On the one hand, an embodiment of the present invention discloses a small target detection method based on intra-layer-inter-layer fusion, which mainly consists of two parts: target detection network model training and small target detection.

[0068] Specifically, Figure 1 As shown in the figure, the target detection network model training steps include the following steps:

[0069] Step 1: Obtain natural images including multiple small targets, mark the categories and positions of the small targets in the natural images, and construct a small target dataset. In this step, the natural images need to be preprocessed first, including unifying the sizes of multiple natural images, and then using json files to classify and label the natural images to obtain a small target dataset.

[0070] Step 2: Construct a target detection network model, which includes a backbone feature extraction network, an intra-layer-inter-layer fusion module, and a prediction module connected in sequence.

[0071] In the constructed target detection network model, the backbone feature extraction network includes a ResNet50 network; the ResNet50 network includes a convolution module and an identity mapping module, wherein the convolution module is used to change the dimension of the feature; and the identity mapping module is used to deepen the depth of the backbone feature extraction network.

[0072] Step 3: Input the small target dataset into the backbone feature extraction network for feature extraction to obtain multiple groups of multi-scale feature maps.

[0073] Specifically, in step 3, the small target dataset is input into the ResNet50 network, and four groups of multi-scale feature maps T1, T2, T3 and T4 of different depths are extracted, with their sizes halved and their channels doubled.

[0074] Step 4: Input each set of multi-scale feature maps obtained into the intra-layer-inter-layer fusion module to perform intra-layer feature fusion to obtain the corresponding intra-layer fusion feature map.

[0075] like Figure 3 As shown, in a specific embodiment, the intra-layer feature fusion specifically includes the following steps:

[0076] Step 41: Obtain each set of multi-scale feature maps T i , i=1,2,3,4 is divided into four branches according to the number of channels f i1 、f i2 、f i3 、f i4 ;

[0077] Step 42: First Branch f i1 Perform 1×1 convolution dimensionality reduction and 3×3 convolution to obtain the first branch f i1 Output

[0078] Step 43: Second Branch f i2 After 1×1 convolution and dimension reduction, it is combined with the first branch f i1 The output L1 is concatenated and then 3×3 convolution is performed to obtain the intermediate fusion feature Perform 3×3 convolution again to get the second branch f i2 Output

[0079] Step 44: The Third Branch f i3 After 1×1 convolution and dimension reduction, the second branch f i2 Intermediate fusion features Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the second branch f i2 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again and fuse f i1 、f i2 With f i3 The characteristics of the third branch f i3 Output

[0080] Step 45: Fourth Branch f i4 After 1×1 convolution and dimension reduction, the third branch f i3 Intermediate fusion features Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the third branch f i3 Intermediate fusion features Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the third branch f i3 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again and fuse f i1 、f i2 、f i3 With f i4 The characteristics of the fourth branch f i4 Output

[0081] Step 46: Output features of the four branches Splicing, performing 1×1 convolution, and obtaining a multi-scale feature map T iThe corresponding intra-layer fusion feature map Z i , i=1,2,3,4.

[0082] Step 5: The intra-layer fusion feature map is fused from top to bottom according to the level of the layer to generate an intra-layer-inter-layer fusion feature map;

[0083] like Figure 3 As shown, in a specific embodiment, the following steps are included:

[0084] Step 51: upsample the intra-layer fusion feature map Z4 to obtain an upsampled feature map, and then concatenate it with the intra-layer fusion feature map Z3 to obtain a fusion feature map U1;

[0085] Step 52: upsample the fused feature map U1 and concatenate it with the fused feature map Z2 within the layer to obtain the fused feature map U2;

[0086] Step 53: Upsample the fused feature map U2 and concatenate it with the intra-layer fused feature map Z1 to obtain a fused feature map U3;

[0087] Step 54: The fused feature map U3 is subjected to a 3×3 convolution to obtain a completely fused intra-layer and inter-layer fused feature map.

[0088] Step 6: Input the generated intra-layer-inter-layer fusion feature map into the prediction module to obtain the target center point prediction heat map, target size prediction map and target center point offset prediction map corresponding to the natural image, and obtain the prediction result.

[0089] In this step, the prediction module specifically includes three branches: target heat map prediction branch, target width and height prediction branch, and target center point offset prediction branch;

[0090] The number of output layers of the target heat map prediction branch is set to the number of target categories; the width and height prediction branch is used to output the width and height probability distribution vector of the target, and the number of output layers of the target width and height prediction branch is 2*K, where K is the vector length of the target width and height probability distribution; the number of output layers of the target center point offset prediction branch is 2

[0091] Step 7: Evaluate the prediction results obtained through the loss function, and train the backbone feature extraction network and the intra-layer-inter-layer feature fusion module to obtain a trained target detection model.

[0092] The specific step 7 is implemented by the following steps:

[0093] S71: obtaining a training set according to the small object data set, annotating each object in each image of the training set with a labeling box, and obtaining the bounding box coordinates and category information c of each object in the training image;

[0094] S72: Calculate the center point coordinate p of the rectangular bounding box of each category c;

[0095]

[0096] Where x1 is the horizontal coordinate of the upper left corner of the target bounding box; y1 is the vertical coordinate of the upper left corner of the target bounding box; x2 is the horizontal coordinate of the lower right corner of the target bounding box; y2 is the vertical coordinate of the lower right corner of the target bounding box;

[0097] S73: down-sample the center point, and generate a Gaussian heat map as the GT heat map with the coordinates of the down-sampled center point as the center;

[0098]

[0099] Where R is the downsampling factor, is the center point corresponding to the low resolution, σ p is the standard deviation of the adaptive size;

[0100] S74: GT heat map Y and prediction heat map Apply the normalized Gaussian wassertein distance loss function L w ;

[0101]

[0102] in, represents the second-order wassertein distance;

[0103] S75: Applying L1 loss function to the size prediction map and the center point offset prediction map;

[0104]

[0105] Among them, L size represents the size prediction graph loss function, L off represents the center point offset prediction graph loss function, N is the number of center points, s k represents the size of the kth target, Indicates that the center point position is p k The target size, Indicates the offset value predicted by the network.

[0106] S76: Add the loss functions to obtain a total loss function, and train the model by minimizing the total loss function, wherein the total loss function includes:

[0107] L=L w +λ size L size +λoff L off

[0108] Among them, λ size =0.1,λ off =1.

[0109] Use the trained target detection network model to detect small targets, including:

[0110] The size of the image to be detected is unified to the preset size, and the trained target detection network model is input to obtain the location and category information of small targets in the image to be detected.

[0111] In a specific embodiment, a small target detection method based on intra-layer and inter-layer fusion is provided. When faced with a picture that needs to detect a target, the method can accurately determine the number and classification of targets in the image (especially small targets), and give a predicted score for the target classification.

[0112] The following is a detailed description of the small target detection method:

[0113] Adjust the image size of the dataset and overlap and cut the original image into 512×512 pixels. Figure 2 As shown in the figure, feature maps of four scales are extracted through the ResNet50 feature extraction network.

[0114] Specifically, the dataset image size is adjusted to 512×512. Based on the ResNet50 feature extraction network, the dimension of feature map T1 is 256*256; the dimension of feature map T2 is 128*128; the dimension of feature map T3 is 64*64; and the dimension of feature map T4 is 32*32. Due to the complex correlation between the extracted feature maps of the four scales, in order to more effectively fuse multi-scale features and thus more effectively express the image content, a feature fusion method based on intra-layer and inter-layer feature fusion is adopted.

[0115] Reference Figure 2 and Figure 3 As shown, Figure 2 IIFF stands for intra-layer-inter-layer feature fusion; IIFF means that the feature maps of each scale are subjected to the same intra-layer fusion operation, and then inter-layer fusion is performed step by step to obtain the final intra-layer-inter-layer fusion feature map.

[0116] Reference Figure 3 As shown, the intra-layer and inter-layer feature fusion is as follows: each set of feature maps T1, T2, T3, and T4 are divided into four branches f i1 、f i2 、f i3 、f i4, i represents the group number of the feature map, i = 1, 2, 3, 4; the first branch f in each group of feature maps i1 Perform 1×1 convolution dimensionality reduction and 3×3 convolution to obtain the first branch f i1 Output The second branch f i2 After 1×1 convolution and dimension reduction, it is combined with the first branch f i2 Output Splicing, then 3×3 convolution processing, fusion f i1 With f i2 The features of the intermediate fusion feature are obtained Perform 3×3 convolution again to get the second branch f i2 Output The third branch f i3 After 1×1 convolution and dimension reduction, the second branch f i2 Intermediate fusion features in Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the second branch f i2 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again to get the third branch f i3 Output Intermediate fusion features That is, fusion f i1 、f i2 With f i3 The fourth branch f i4 After 1×1 convolution and dimension reduction, the third branch f i3 Intermediate fusion features in Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the third branch f i3 Intermediate fusion features in Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the third branch f i3 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again to get the fourth branch f i4 Output Intermediate fusion features That is, fusion f i1 、fi2 、f i3 With f i4 The output features of the four branches Splicing, performing 1×1 convolution, and obtaining a multi-scale feature map T i The corresponding intra-layer fusion feature map Z i , i=1,2,3,4; upsample the feature map Z4 to obtain the upsampled feature map, and then concatenate it with the feature map Z3 to obtain the fused feature map U1; upsample the fused feature map U1 to obtain the upsampled feature map, and then concatenate it with the feature map Z2 to obtain the fused feature map U2; upsample the fused feature map U2 to obtain the upsampled feature map, and then concatenate it with the feature map Z1 to obtain the fused feature map U3; perform 3×3 convolution on the fused feature map U3 to obtain a completely fused intra-layer and inter-layer fusion feature map.

[0117] The intra-layer and inter-layer fusion feature maps are input into the Head layer of the CenterNet model to obtain the target center prediction heat map, target size prediction map, and target center offset prediction map corresponding to the image to be detected; the prediction results are evaluated through the improved loss function, and the model is trained;

[0118] Specifically, the intra-layer and inter-layer fusion feature map is input into the Head layer of the CenterNet model, and the predicted center point prediction heat map, size prediction map and center point offset prediction map are obtained through three fully connected layers respectively; then the loss values ​​of the predicted center point prediction heat map, size prediction map and center point offset prediction map are calculated respectively to determine the position and size of the target, and the final classification and positioning heat map is generated;

[0119] Since the current measurement method based on intersection over union (IoU) is very sensitive to the position change of small objects, a small position change of a small object will cause a large change in IoU. Therefore, IoU is not a good measurement method for small objects. In order to detect the position of small objects more accurately, the wassertein distance between two Gaussian distributions is used to replace IoU.

[0120] Specifically, we use the annotation box to annotate each target in each image in the training set, obtain the bounding box (x1, y1, x2, y2) and category information c of each target in the training image; calculate the center point coordinates of the target bounding box of each category c; downsample the center point, and generate a Gaussian heat map as the GT heat map with the downsampled center point coordinates as the center; perform the GT heat map Y and the predicted heat map Apply the normalized Gaussian wassertein distance loss function L w; Apply L1 loss function to the size prediction map and the center point offset prediction map; Add the loss functions to get the total loss function, train the model by minimizing the total loss function, determine the position and size of the target, and generate the final classification and positioning heat map. In this process, the annotation box is the target box provided in the dataset, and the bounding box is the target box detected by the model. The category information here is the predicted target category, and the real category is marked in step 1. The images in the training set are preprocessed natural images, and the category and position are annotated in the json annotation file in the dataset.

[0121] The evaluation training calculation process based on the improved loss function is as follows:

[0122] The coordinates of the center point of the target bounding box of each category c are calculated by the following formula:

[0123]

[0124] The GT heat map is calculated by the following formula:

[0125]

[0126] In the above formula, R represents the downsampling factor. represents the low-resolution center point, σ p Indicates the standard deviation of the adaptive size.

[0127] The GT heat map Y and the predicted heat map are calculated by the following formula Gaussian wassertein distance loss function:

[0128]

[0129] The L1 loss function of the size prediction graph is calculated by the following formula:

[0130]

[0131] In the above formula, s k represents the size of the kth target, Represents the center point position p k The target size.

[0132] The L1 loss function of the center point offset prediction graph is calculated using the following formula:

[0133]

[0134] In the above formula, R represents the downsampling factor. Represents the low-resolution center point.

[0135] The total loss function is calculated by the following formula:

[0136] L=L w +λ size L size +λ off L off

[0137] In the above formula, λ size =0.1,λ off =1.

[0138] Multi-scale feature fusion is one of the important methods to improve the performance of small target detection. In the small target detection method based on multi-scale feature fusion, if only the inter-layer feature fusion module is used, there is a large gap between the receptive fields of features at different levels, and the learned image representation is relatively rough, which seriously affects the detection performance of small targets. This embodiment uses an intra-layer-inter-layer feature extraction network to extract a more refined local representation, which can retain more accurate spatial information.

[0139] In the face of the intra-layer and inter-layer feature fusion map obtained, if a measurement method based on the intersection over union (IoU) is adopted, it is very sensitive to changes in the positions of small targets, thereby affecting the detection performance of small targets. This embodiment adopts a Gaussian measurement method based on the wassertein distance to replace IoU, so that the position detection of small targets is more stable.

[0140] This method can not only solve the problem that the traditional inter-layer fusion method cannot learn the fine local representation of small targets, but also improve the unstable position of small targets detected by the IoU-based measurement method.

[0141] On the other hand, an embodiment of the present invention further discloses a detection system according to any one of the small target detection methods based on intra-layer-inter-layer fusion according to the present invention, comprising an image acquisition module, an image preprocessing module, a feature extraction module, a feature fusion module, a prediction module, an evaluation training module and a detection module;

[0142] An image acquisition module, used for acquiring natural images including a variety of small targets;

[0143] The image preprocessing module is used to adjust the collected natural images to a uniform preset scale and generate a small object data set;

[0144] The feature extraction module is used to receive the small target data set, perform feature extraction, and output multi-scale features;

[0145] The intra-layer-inter-layer fusion module is used to receive multi-scale features for feature fusion and output the fused intra-layer-inter-layer fusion feature map;

[0146] A prediction module is used to receive the generated intra-layer-inter-layer fusion feature map, obtain the target center point prediction heat map, target size prediction map and target center point offset prediction map corresponding to the natural image, and obtain the prediction result;

[0147] An evaluation training module is used to evaluate the obtained prediction results through a loss function, and to train the backbone feature extraction network and the intra-layer-inter-layer feature fusion module to obtain a trained target detection model;

[0148] The detection module is used to detect the image to be detected using the trained target detection model to obtain the location and category information of small targets in the image to be detected.

[0149] The embodiment of the present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods described in the present invention are implemented.

[0150] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0151] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A small target detection method based on intra-layer and inter-layer fusion, characterized in that: The steps include: The target detection network model training steps include: Step 1: Obtain a natural image including a variety of small objects, mark the categories and positions of the small objects in the natural image, and construct a small object dataset; Step 2: construct a target detection network model, wherein the target detection network model includes a backbone feature extraction network, an intra-layer-inter-layer fusion module, and a prediction module connected in sequence; Step 3: Input the small target data set into the backbone feature extraction network for feature extraction to obtain multiple groups of multi-scale feature maps; Step 4: Input each group of multi-scale feature maps obtained into the intra-layer-inter-layer fusion module to perform intra-layer feature fusion to obtain the corresponding intra-layer fusion feature map; In step 4, intra-layer feature fusion is performed, which specifically includes the following steps: Step 41: Obtain each set of multi-scale feature maps T i , i=1,2,3,4 is divided into four branches according to the number of channels f i1 、f i2 、f i3 、f i4 ; Step 42: First Branch f i1 Perform 1×1 convolution dimensionality reduction and 3×3 convolution to obtain the first branch f i1 Output Step 43: Second Branch f i2 After 1×1 convolution and dimension reduction, it is combined with the first branch f i1 Output Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Perform 3×3 convolution again to get the second branch f i2 Output Step 44: The Third Branch f i3 After 1×1 convolution and dimension reduction, the second branch f i2 Intermediate fusion features in Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the second branch f i2 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again to get the third branch f i3 Output Step 45: The fourth branch f i4 After 1×1 convolution and dimension reduction, the third branch f i3 Intermediate fusion features Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the third branch f i3 Intermediate fusion features in Splicing is performed, and then 3×3 convolution is performed to obtain the intermediate fusion features Intermediate fusion features With the third branch f i3 Output Splice and perform 3×3 convolution again to obtain the intermediate fusion features Perform 3×3 convolution again to get the fourth branch f i4 Output Step 46: Output features of the four branches Splicing, performing 1×1 convolution, and obtaining a multi-scale feature map T i The corresponding intra-layer fusion feature map Z i , i=1,2,3,4; Step 5: The intra-layer fusion feature map is fused from top to bottom according to the level of the layer to generate an intra-layer-inter-layer fusion feature map; Step 6: Input the generated intra-layer-inter-layer fusion feature map into the prediction module to obtain the target center point prediction heat map, target size prediction map and target center point offset prediction map corresponding to the natural image, and obtain the prediction result; Step 7: Evaluate the obtained prediction results through the loss function, and train the backbone feature extraction network and the intra-layer-inter-layer feature fusion module to obtain a trained target detection model; Small target detection steps include: The size of the image to be detected is unified to the preset size, and the trained target detection model is input to obtain the location and category information of small targets in the image to be detected.

2. According to claim 1, a small target detection method based on intra-layer and inter-layer fusion is characterized in that: In step 2, the backbone feature extraction network includes a ResNet50 network; the ResNet50 network includes a convolution module and an identity mapping module, wherein the convolution module is used to change the dimension of the feature; and the identity mapping module is used to deepen the depth of the backbone feature extraction network.

3. The small target detection method based on intra-layer and inter-layer fusion according to claim 2 is characterized in that: In step 3, obtaining multiple groups of multi-scale feature maps specifically includes: The small target dataset is input into the ResNet50 network, and four groups of multi-scale feature maps T1, T2, T3 and T4 with different depths are extracted, with their sizes halved and their channels doubled.

4. The small target detection method based on intra-layer and inter-layer fusion according to claim 1 is characterized in that: In step 5, the intra-layer fusion feature map is subjected to inter-layer feature fusion from top to bottom according to the level of the layer, including the following steps: Step 51: upsample the intra-layer fusion feature map Z4 to obtain an upsampled feature map, and then concatenate it with the intra-layer fusion feature map Z3 to obtain a fusion feature map U1; Step 52: upsample the fused feature map U1 and concatenate it with the fused feature map Z2 within the layer to obtain the fused feature map U2; Step 53: Upsample the fused feature map U2 and concatenate it with the intra-layer fused feature map Z1 to obtain a fused feature map U3; Step 54: The fused feature map U3 is subjected to a 3×3 convolution to obtain a completely fused intra-layer and inter-layer fused feature map.

5. The small target detection method based on intra-layer and inter-layer fusion according to claim 1 is characterized in that: In step 6, the prediction module specifically includes three branches, namely: a target heat map prediction branch, a target width and height prediction branch, and a target center point offset prediction branch; Among them, the output layer number of the target heat map prediction branch is set to the number of target categories; the target width and height prediction branch is used to output the width and height probability distribution vector of the target, and the output layer number of the target width and height prediction branch is 2*K, K is the vector length of the target width and height probability distribution; the output layer number of the target center point offset prediction branch is 2.

6. The small target detection method based on intra-layer and inter-layer fusion according to claim 1 is characterized in that: The step 7 specifically includes: S71: obtaining a training set according to the small object data set, annotating each object in each image of the training set with a labeling box, and obtaining the bounding box coordinates and category information c of each object in the training image; S72: Calculate the center point coordinate p of the rectangular bounding box of each category c; Where x1 is the horizontal coordinate of the upper left corner of the target bounding box; y1 is the vertical coordinate of the upper left corner of the target bounding box; x2 is the horizontal coordinate of the lower right corner of the target bounding box; y2 is the vertical coordinate of the lower right corner of the target bounding box; S73: Downsample the center point and generate a Gaussian heat map as the GT heat map with the coordinates of the downsampled center point as the center: Where R is the downsampling factor, is the center point corresponding to the low resolution, σ p is the standard deviation of the adaptive size; S74: GT heat map Y and prediction heat map Apply the normalized Gaussian wassertein distance loss function L w : in, represents the second-order wassertein distance; S75: Applying L1 loss function to the size prediction map and the center point offset prediction map; Among them, L size represents the size prediction graph loss function, L off represents the center point offset prediction graph loss function, N is the number of center points, s k represents the size of the kth target, Indicates that the center point position is p k The target size, Indicates the offset value predicted by the network; S76: Add the loss functions to obtain a total loss function, and train the model by minimizing the total loss function, wherein the total loss function includes: L=L w +λ size L size +λ off L off Among them, l size =0.1,λ off =1.

7. A detection system for a small target detection method based on intra-layer and inter-layer fusion according to any one of claims 1 to 6, characterized in that: It includes image acquisition module, image preprocessing module, feature extraction module, feature fusion module, prediction module, evaluation training module and detection module; An image acquisition module, used for acquiring natural images including a variety of small targets; The image preprocessing module is used to adjust the collected natural images to a uniform preset scale and generate a small object data set; The feature extraction module is used to receive the small target data set, perform feature extraction, and output multi-scale features; The intra-layer-inter-layer fusion module is used to receive multi-scale features for feature fusion and output the fused intra-layer-inter-layer fusion feature map; A prediction module is used to receive the generated intra-layer-inter-layer fusion feature map, obtain the target center point prediction heat map, target size prediction map and target center point offset prediction map corresponding to the natural image, and obtain the prediction result; An evaluation training module is used to evaluate the obtained prediction results through a loss function, and to train the backbone feature extraction network and the intra-layer-inter-layer feature fusion module to obtain a trained target detection model; The detection module is used to detect the image to be detected using the trained target detection model to obtain the location and category information of small targets in the image to be detected.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the small target detection method based on intra-layer-inter-layer fusion described in any one of claims 1-6 is implemented.

Images