A small target detection method based on image feature fusion

Abstract

The present invention provides a small target detection method based on image feature fusion. A T-shaped backbone network is constructed based on feature reuse, the lowest feature layer of the initial backbone network is upsampled, and shallow features are fused with deep features, and finally four feature layers are output, so that the problem of feature information loss occurring as the network deepens is alleviated while retaining more small target area features; the present invention constructs a four-branch enhanced feature extraction network according to four feature maps output by the T-shaped backbone network, and mines out more potential features of small targets by continuously upsampling and downsampling feature maps of different scales and fusing them; the present invention constructs a global relationship in an image through an HDA attention module, so as to enrich the global position information of target instances in the image, and finally improves the detection accuracy of small targets.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a small target detection method based on image feature fusion. Background Art

[0002] Object detection is the core task of computer vision. In the past decade, the application areas of object detection have been expanding. For example, it plays an important role in the fields of autonomous driving, face recognition, intelligent security, environmental monitoring, interactive recognition, intelligent medical care, and the defense industry. With the improvement of computing power, object detection technology has made a leap from manually designed solutions to using deep learning to solve object detection problems. Compared with traditional object detection technology, object detection methods based on deep learning technology have higher robustness, more accurate detection results, and faster detection speed.

[0003] Object detection technology based on deep learning can be divided into single-stage methods and two-stage methods. The single-stage method is represented by the YOLO (You Only Look Once) series. The single-stage method converts the object detection task into a regression problem and calculates the probability of each type of object. The two-stage method includes well-known methods such as RCNN (Region based Convolutional Neural Network) and Fast RCNN (Fast Region based Convolutional Neural Network). The two-stage method first divides the area most likely to contain the object and classifies it, and then fine-tunes the bounding box containing the object. Summary of the invention

[0004] The embodiment of the present invention provides a small target detection method based on image feature fusion, which is used to solve the problems existing in the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme.

[0006] A small target detection method based on image feature fusion, comprising:

[0007] S1 uses the dataset to train the small target detection model by setting the training method and training parameters;

[0008] S2 uses mosaic data augmentation operations on the dataset as it is traversed;

[0009] S3 initializes the trained small target detection model, determines whether to use the pre-trained model weights, and clears the optimizer gradient;

[0010] S4 obtains the resized image and the corresponding label;

[0011] S5 inputs the resized image into the small target detection model that has executed step S3, obtains image feature layers of different sizes through the T-shaped backbone network of the small target detection model, extracts features from the image feature layers of different sizes through the four-branch enhanced feature extraction network of the small target detection model, and transmits the output of the four-branch enhanced feature extraction network to the HDA decoupling detection head of the small target detection model to classify, identify and locate the small targets in the image;

[0012] S6 calculates the multi-classification loss value of the small target obtained in step S5 through the BCEWithLogitsLoss loss function, and uses the iou loss function to calculate the distance between the predicted box and the real box of the small target;

[0013] S7 inputs the multi-classification loss value obtained by executing step S6 into the small target detection model for back propagation to update the gradient features, and adaptively obtains the optimizer learning rate of the small target detection model, and performs a model parameter update operation through the gradient descent of the small target detection model;

[0014] S8 uses the validation set in the data set to obtain a multi-classification loss value of the target obtained using the validation set by executing steps S5 and S6; if the multi-classification loss value is less than a preset threshold, the small target detection model of this step is saved;

[0015] S9, repeatedly executing steps S4 to S8 until a preset number of iterations is met, and a converged small target detection model is obtained;

[0016] S10 performs small target detection on image data through a converged small target detection model.

[0017] Preferably, the T-shaped backbone network of the small target detection model includes an initial network and a fusion operation unit, wherein the fusion operation unit is used to perform a matrix addition operation on the output of the initial network to obtain four image feature layers that are downsampled by 4 times, 8 times, 16 times and 32 times respectively;

[0018] The four-branch enhanced feature extraction network of the small target detection model is specifically used for:

[0019] Extracting features, upsampling and stacking operations are performed on the image feature layers with downsampling ratios of 8 times, 16 times and 32 times output by the T-shaped backbone network;

[0020] Downsampling, feature extraction and stacking operations are performed on the feature layers with downsampling ratios of 4 times, 8 times and 16 times output by the T-shaped backbone network;

[0021] Based on the operation results of the above two processes, an enhanced feature map is obtained;

[0022] The small target detection model also has an HDA decoupling detection head for:

[0023] Based on the enhanced feature map, constructing a global relationship of features of the enhanced feature map through three set linear layers;

[0024] The K, Q and V matrices are obtained by setting the convolution kernels to 3×3 and 5×5; the K, Q and V matrices are obtained by the formula

[0025] K = Conv2D kernel＝3 (Linear(LayerNorm(X)w1)+b1) (1)

[0026] Q = Conv2D kernel＝5 (Linear(LayerNorm(X)w2)+b2) (2)

[0027] V = Conv2D kernel＝3 (Linear(LayerNorm(X)w3)+b3) (3)

[0028] express;

[0029] Perform matrix addition operation on K matrix and Q matrix, perform matrix dot multiplication operation on V matrix, and perform residual link with input features;

[0030] By setting the calculation formula of the HDA attention module

[0031]

[0032] HDA=MLP(LayerNorm(((Q+K)⊙V+X)))+((Q+K)⊙V+X) (5) Get all the context information of the enhanced feature map; where Conv2D kernel＝1 (·) indicates a 2D convolution with a convolution kernel of 1×1, Conv2D kernel＝3 (·) indicates a 2D convolution with a convolution kernel of 3×3, Conv2D kernel＝5 (·) indicates a 2D convolution with a kernel of 5×5, Linear(·) indicates a linear layer, LayerNorm(·) indicates layer normalization, X indicates input features, w1, w2, and w3 are weights of the linear layer, b1, b2, and b3 are biases of the linear layer, + and ⊙ indicate matrix addition and matrix dot multiplication, respectively, kernel indicates the convolution kernel, kernel=1 indicates a convolution kernel of 1, kernel=3 indicates a convolution kernel of 3, and kernel=5 indicates a convolution kernel of 5;

[0033] The multi-layer perceptron is set up to extract features layer by layer from all the context information of the enhanced feature map, and stack the information to obtain the key context information of the enhanced feature map.

[0034] The key context information of the enhanced feature map is used to input into the standard decoupled detection head of the small object detection model for classification and regression operations.

[0035] Preferably, the processing process of the four-branch enhanced feature extraction network of the small target detection model specifically includes:

[0036] For the four image feature layers feat1, feat2, feat3 and feat4 output by the T-shaped backbone network, the following three sub-steps are performed:

[0037] The features of the E1 image feature layer fe4 are adjusted and upsampled for the first time and then stacked with the features of the image feature layer fe3;

[0038] The features of the E2 image feature layer feat4 and the image feature layer feat3 are stacked, and the features are adjusted for the second time and stacked with the features of the image feature layer feat2 after upsampling;

[0039] E3: The features of the stacked image feature layer feat4, image feature layer feat3 and image feature layer feat2 are adjusted for the third time and stacked with the features of the image feature layer feat1 after upsampling;

[0040] Based on the features after stacking the execution results of sub-steps E1 to E4, it is first directly output as an enhanced feature map feat1', then the first downsampling and feature channel adjustment are performed and the output is an enhanced feature map feat2', then the second downsampling and feature channel adjustment are performed and the output is an enhanced feature map feat3', and finally the third downsampling and feature channel adjustment are performed and the output is an enhanced feature map feat4'.

[0041] Preferably, the dataset is a combination of the VisDrone2019 dataset and the SHWD (Safety Helmet Wearing Dataset) dataset, and has a training set, a test set, and a validation set.

[0042] Preferably, in step S1, the process of setting the training mode includes: determining whether there is an available GPU, and if so, using the GPU, otherwise, using the CPU;

[0043] The process of setting training parameters includes:

[0044] Fixed random number seed to 1;

[0045] Use mixed precision training;

[0046] The input image size is 1024×1024;

[0047] The total number of training epochs is 300; batch_size is 8;

[0048] No pre-trained weights are used;

[0049] Use adaptive learning rate with a maximum learning rate of 0.01 and a minimum learning rate of 0.0001;

[0050] The stochastic gradient descent (SGD) optimizer is used with a momentum of 0.937 and a weight decay of 0.0005.

[0051] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention provides a small target detection method based on image feature fusion, constructs a T-shaped backbone network based on feature reuse, upsamples the lowest feature layer of the initial backbone network, and fuses the shallow features with the deep features, and finally outputs four feature layers, which retains more small target area features while alleviating the problem of feature information loss that occurs as the network deepens; the present invention constructs a four-branch enhanced feature extraction network based on the four feature maps output by the T-shaped backbone network, and continuously upsamples and downsamples feature maps of different scales and fuses them to mine more potential features of small targets; the present invention constructs global relationships in the image through the HDA attention module, thereby enriching the global position information of the target instance in the image, and ultimately improves the detection accuracy of the network model and the detection device for small targets.

[0052] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0054] Figure 1 A processing flow chart of a small target detection method based on image feature fusion provided by the present invention;

[0055] Figure 2 A schematic diagram of a detection model of a small target detection method based on image feature fusion provided by the present invention;

[0056] Figure 3 A T-shaped backbone network structure diagram of a small target detection method based on image feature fusion provided by the present invention;

[0057] Figure 4 A schematic diagram of the structure of an enhanced feature extraction network for a small target detection method based on image feature fusion provided by the present invention;

[0058] Figure 5 A schematic diagram of the structure of an HDA decoupling detection head for a small target detection method based on image feature fusion provided by the present invention;

[0059] Figure 6 A flowchart of a preferred embodiment of a small target detection method based on image feature fusion provided by the present invention;

[0060] Figure 7 A distribution diagram of the number of targets in each category in the VisDrone data set of an experimental example of a small target detection method based on image feature fusion provided by the present invention;

[0061] Figure 8 The frame lengths of target instances of each category in the VisDrone dataset of an experimental example of a small target detection method based on image feature fusion provided by the present invention;

[0062] Fig. 9 The figure shows the distribution of target instance types of each category in the SHWD dataset of an experimental example of a small target detection method based on image feature fusion provided by the present invention. DETAILED DESCRIPTION

[0063] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0064] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0065] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0066] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0067] The present invention provides a small target detection method based on image feature fusion, which is used to solve the following technical problems existing in the prior art:

[0068] Small target detection has always been a difficult problem in target detection technology. Small target detection can be used to identify traffic lights and license plates in smart city transportation; it can be used to build a rapid on-site evaluation system for microcells in smart healthcare; it can be used to identify vehicles, airports, and buildings on the ground in the aviation field, and it can also be used to identify targets such as ships and islands on the sea. Small target detection in aerial images is crucial for port scheduling, natural resource exploration, and the detection of ships and aircraft; targets such as aircraft, ships, bridges, oil tanks, and ports can be detected in synthetic aperture radar images. However, although target detection methods based on deep learning can accurately locate common targets, these methods cannot accurately locate and identify small targets (small targets: small targets are defined in the Microsoft COCO[1] dataset as targets with a bounding box smaller than 32*32 pixels). There are two reasons for this situation:

[0069] First, small targets occupy fewer pixels and have limited effective information, which is not enough to support deep neural networks to make accurate decisions on the category and location of the target.

[0070] Second: Object detection technology requires the original image information to be used to extract features through a deep neural network. However, in the process of the algorithm network performing feedforward from shallow to deep, the original features are inevitably lost.

[0071] In addition, the detection methods in the prior art mainly adopt the following two methods:

[0072] Super-resolution: Low-resolution images contain insufficient feature information, and target detection methods cannot make accurate judgments based on this information. High-resolution images contain rich feature information, which helps target detection methods make final decisions. Therefore, some methods perform super-resolution operations on images and image features to restore the feature details of small target areas in the image, thereby improving the detection performance of target detection technology for small targets.

[0073] Multi-scale learning: Deep neural networks will sample the original image multiple times, and the lower network space may contain only a few small target features, or even no small target features, which is an important factor that hinders target detection technology from detecting small targets. Some methods combine shallow spatial features with deep spatial features to avoid the loss of small target features, and some methods use convolution kernels of different sizes to extract features in parallel to enrich the extracted image features through different field of view features.

[0074] However, the solutions of the prior art still have the following problems:

[0075] 1. The two main methods of super-resolution technology are image super-resolution and feature super-resolution. For image super-resolution, other negative sample areas will inevitably be magnified, which will cause the model to be more inclined to obtain the features of negative samples during training; for feature super-resolution, this will cause the feature image to lack contextual information, which will greatly limit the detection performance of small objects.

[0076] 2. Multi-scale learning extracts features from images of each scale and generates multi-scale image features. These scales greatly enrich the image features extracted in the target detection method. However, in these different scales, it is also difficult for low-level feature maps to have sufficient semantic information. Even if the target location can be roughly identified, it is still easy to misclassify it.

[0077] In response to the above-mentioned problems that hinder the detection of small targets, the present invention proposes a small target detection method based on feature fusion technology. In order to protect the effective information of the small target area, the present invention upsamples the low-resolution feature map when extracting image features to improve the resolution of the area where the small target is located, so that more small target feature information can be retained. In order to alleviate the negative impact of feature loss in deep neural networks, the present invention fuses shallow features with deep features in the network space when extracting image features (feature addition, multiplication or stacking) to obtain richer features in the original image. Both of these solutions are reused on the basis of existing feature layers, which can not only maximize the use of computing resources, but also effectively improve the accuracy of small target detection.

[0078] See also Figure 1 The present invention provides a small target detection method based on image feature fusion, comprising the following steps:

[0079] S1 uses the dataset to train the small target detection model by setting the training method and training parameters;

[0080] S2 uses mosaic data augmentation operations on the dataset as it is traversed;

[0081] S3 initializes the trained small target detection model, determines whether to use the pre-trained model weights, and clears the optimizer gradient;

[0082] S4 obtains the resized image and the corresponding label;

[0083] S5 inputs the resized image into the small target detection model that has executed step S3, obtains image feature layers of different sizes through the T-shaped backbone network of the small target detection model, extracts features from the image feature layers through the four-branch enhanced feature extraction network of the small target detection model, and transmits the output of the four-branch enhanced feature extraction network to the HDA decoupling detection head of the small target detection model to classify, identify and locate small targets in the image;

[0084] S6 calculates the multi-classification loss value of the small target obtained in step S5 through the BCEWithLogitsLoss loss function, and uses the iou loss function to calculate the distance between the predicted box and the real box of the small target;

[0085] S7 inputs the multi-classification loss value obtained by executing step S8 into the small target detection model for back propagation to update the gradient features, and adaptively obtains the optimizer learning rate of the small target detection model, and performs a model parameter update operation through the gradient descent of the small target detection model;

[0086] S8 uses the validation set in the data set to obtain a multi-classification loss value of the target obtained using the validation set by executing steps S5 and S6; if the multi-classification loss value is less than a preset threshold, the small target detection model of this step is saved;

[0087] S9, repeatedly executing steps S4 to S8 until a preset number of iterations is met, and a converged small target detection model is obtained;

[0088] S10 performs small target detection on image data through a converged small target detection model to obtain small target recognition results in the image.

[0089] The small target detection method provided by the present invention achieves the goal of retaining more small target feature information in the deep space of the network by improving the resolution of the feature map in the backbone network of target detection; by fusing the shallow features with the deep features in the network space when extracting image features, so as to alleviate the negative impact caused by the feature loss that occurs as the network deepens in the convolutional neural network; by constructing an HDA (HaDamard Attention) attention module to calculate the global relationship in the image, so as to obtain the context information of the target area; by repeatedly utilizing the multi-level features extracted by the network, so as to maximize the utilization of computing resources.

[0090] In the preferred embodiment provided by the present invention, the following improvements are proposed for the small target detection model: first, there is a T-shaped backbone network in which the image feature layer originally output by the backbone network is reused: the lowest layer features are upsampled to improve the resolution of the feature map, and the low-resolution feature map is fused with the high-resolution feature map; secondly, the four feature layers output by the T-shaped backbone network are used in the four-branch enhanced feature extraction network to mine potential small target image features; finally, four decoupled detection heads combined with the HDA attention module are used to identify the target. The architecture diagram of the detection model is shown in FIG. Figure 2 shown.

[0091] (1) T-shaped backbone network

[0092] In the initial backbone, only the image is downsampled. In the present invention, the T-shaped backbone network performs three 2-fold upsampling on the lowest feature map in the original network, and performs matrix addition operation on the output feature layer of the initial backbone network and the feature layer obtained by upsampling, and finally outputs four feature layers that are downsampled by 4, 8, 16 and 32 times, respectively. In the backbone network, more features of small target areas are obtained by upsampling operations, and shallow features are fused with deep features to alleviate the problem of original feature loss in the image caused by deepening the network.

[0093] like Figure 3As shown, the image in the initial network backbone outputs three feature layers after passing through block3, block4 and block5, and the t-shaped backbone network in the present invention performs three upsamplings after block5 to generate three high-resolution feature maps in block6, block7 and block8 to obtain more image information of small target areas. In order to alleviate the loss of image features caused by the network space being too deep, the t-shaped backbone network fuses the image features output by block2, block3 and block4 with block6, block7 and block8 respectively (matrix addition operation), and finally blocks5, block6, block7 and block8 output four image feature layers feat4, feat3, feat2 and feat1. In the t-shaped backbone network, different blocks are stacked, and each block is composed of BaseConv and CSPLayer, such as Figure 2 Displayed on the right.

[0094] (2) Four-branch enhanced feature extraction network

[0095] After the input image passes through the T-shaped backbone network, four feature layers with downsampling times of 4, 8, 16 and 32 are output. First, the feature layers with downsampling times of 8, 16 and 32 are extracted, upsampled and stacked. Then, the feature layers with downsampling times of 4, 8 and 16 are downsampled, extracted and stacked. Finally, a feature map with the same shape as the input feature map is output.

[0096] like Figure 4 As shown, the enhanced feature extraction network in the present invention receives four image feature layers feat1, feat2, feat3 and feat4 output from the backbone network. These four feature layers undergo three upsampling, three downsampling and feature stacking operations respectively to maximize the use of all information in the image features. Finally, the enhanced feature extraction network outputs four enhanced feature maps feat1, feat2, feat3 and feat4. In the embodiments provided by the present invention, these image feature layers are referred to as enhanced feature maps.

[0097] The specific processing process is as follows Figure 4 As shown, in the four-branch enhanced feature extraction network, the four image feature layers feat1, feat2, feat3 and feat4 output by the T-shaped backbone network are operated through the following processes respectively:

[0098] The features of the image feature layer feat4 are adjusted and upsampled for the first time ( Figure 4 The lower left layer group) is then stacked with the features of the image feature layer feat3;

[0099] The features of the image feature layer feat4 and the image feature layer feat3 are stacked and the second feature channel adjustment and upsampling are performed ( Figure 4 The layer group on the lower left performs the second operation) and the features of the image feature layer feat2 are stacked ( Figure 4 Layer group in the middle left);

[0100] The features of the image feature layer feat4, image feature layer feat3 and image feature layer feat2 are stacked and the third feature channel adjustment and upsampling are performed ( Figure 4 The layer group in the middle left) is then stacked with the features of the image feature layer feat1 ( Figure 4 upper left layer group);

[0101] Based on the features obtained by stacking the results of the above three operations, the enhanced feature map feat1' is directly output first, and then the first downsampling and feature channel adjustment are performed ( Figure 4 The output is the enhanced feature map feat2', and then the second downsampling and feature channel adjustment are performed ( Figure 4 The output is the enhanced feature map feat3', and finally the third downsampling and feature channel adjustment are performed ( Figure 4 The output of the layer group on the lower right is the enhanced feature map feat4'.

[0102] (3) HDA decoupling detection head

[0103] The HDA attention module is first constructed in the detection head. The purpose is to make the input features construct the global position information in the image through the HDA module. This position information can assist the decoupled detection head to classify and identify the target more accurately.

[0104] like Figure 5 As shown, the present invention constructs an HDA attention module in front of the standard decoupled detection head. The final output of the HDA attention module is a feature map with contextual information. Relative to the enhanced feature map output by the four-branch enhanced feature extraction network, the HAD attention module calculates the contextual information for each enhanced feature map output by the four-branch enhanced feature extraction network, and uses this information to assist the detection head in identifying the target in the feature map. Therefore, these feature layers contain richer information than the output of the four-branch enhanced feature extraction network.

[0105] In the HDA decoupled detection head, the features output by the four-branch enhanced feature extraction network first pass through three linear layers to build the global relationship in the image features, and then use the convolution kernels of 3×3 and 5×5 to calculate the three matrices K (key), Q (query) and V (value). K and Q are added and then dot-multiplied with V, and residually linked with the input features. K, Q and V can be expressed by formulas (1.1), (1.2) and (1.3):

[0106] K = Conv2D kernel＝3 (Linear(LayerNorm(X)w1)+b1) (1)

[0107] Q = Conv2D kernel＝5 (Linear(LayerNorm(X)w2)+b2) (2)

[0108] V = Conv2D kernel＝3 (Linear(LayerNorm(X)w3)+b3) (3)

[0109] The HDA attention module can be expressed as:

[0110] MLP(X) = Conv2D kernel＝1 (Conv2D kernel＝3 (Conv2D kernel＝1 (X))+Conv2D kernel＝1 (X))(4)

[0111] HDA = MLP (LayerNorm (((Q+K)⊙V+X))) + ((Q+K)⊙V+X) (5) where Conv2D kernel＝1 (·) indicates a 2D convolution with a convolution kernel of 1×1, Conv2D kernel＝3 (·) indicates a 2D convolution with a convolution kernel of 3×3, Conv2D kernel＝5 (·) indicates a 2D convolution with a kernel of 5×5. Linear(·) indicates a linear layer, LayerNorm(·) indicates layer normalization, and X indicates input features. w1, w2, and w3 are weights of the linear layer, and b1, b2, and b3 are biases of the linear layer. + and ⊙ indicate matrix addition and matrix dot multiplication, respectively.

[0112] Then, the image context information extracted in the previous step is extracted layer by layer through convolution kernels of different sizes and stacked (such as Figure 4 As shown in the figure), key contextual information in the image can be obtained.

[0113] Finally, the feature map passed through the HDA (HaDamard Attention) module is transmitted to the standard decoupled detection head to perform classification and regression operations respectively.

[0114] In another possible embodiment, Figure 6 As shown, a preferred implementation process of the method provided by the present invention is as follows:

[0115] First, a T-shaped backbone network is constructed by improving the image feature resolution and fusing shallow features with deep features; then a four-branch enhanced feature extraction network is constructed based on the output features of the backbone network; finally, an HDA attention module is constructed and introduced into a standard decoupled detection head. The overall process of this embodiment is as follows: Figure 5 As shown, the first to twelfth steps in the following steps are the model training process.

[0116] Step 1: Use the VisDrone2019 dataset and the SHWD (Safety Helmet Wearing Dataset) dataset, convert them into the format of the PASCAL VOC dataset, and train them according to the original training set, test set, and validation set division.

[0117] Step 2: Set the training method: If a GPU is available, use the GPU for training; if no GPU is available, use the CPU for training.

[0118] Step 3: Set the training parameters: fix the random number seed to 1; use mixed precision training; input image size is 1024×1024; total training generations are 300; batch_size is 8; do not use pre-trained weights; use adaptive learning rate with a maximum learning rate of 0.01 and a minimum learning rate of 0.0001; use the stochastic gradient descent SGD (Stochastic Gradient Descent) optimizer with a momentum of 0.937 and a weight decay of 0.0005.

[0119] Step 4: In the first 70% of epochs, there is a 50% probability of performing mosaic data enhancement, and there is a 50% probability of performing mixed data enhancement while using mosaic data enhancement.

[0120] Step 5: Initialize the small target detection model, determine whether to use the pre-trained model weights, and clear the optimizer gradient.

[0121] Step 6: Get the resized image and label.

[0122] Step 7: Perform forward propagation: Switch the small target detection model to training mode. The image obtains four image feature layers of different sizes through the backbone network. Then, the output of the backbone network is enhanced through the enhanced feature extraction network to extract richer features. Finally, the image feature layer is transmitted to the detection head to classify and locate the target.

[0123] Step 8: Calculate the loss: Use the BCEWithLogitsLoss loss function to calculate the target multi-classification loss, and use the iou loss function to calculate the distance between the predicted box and the true box. The sum of the two loss functions can better make the predicted box fit the true box.

[0124] Step 9: Update the gradient features through backpropagation of the loss value calculated in step 8, and adaptively obtain the optimizer learning rate to further update the parameters through gradient descent.

[0125] Step 10: Switch the small object detection model to verification mode and execute steps 7 and 8 to obtain the loss value of the model on the verification set. If the error value on the verification set is the smallest at this time, save the model.

[0126] Step 11: Repeat steps 6, 7, 8, 9 and 10 until the total number of training generations set in step 6 is met.

[0127] Step 12: End the training and obtain a converged small target detection model.

[0128] The present invention also provides an embodiment for demonstrating the detection effect of executing the method provided by the present invention.

[0129] Small target: area ≤ 32×32 pixels; Medium target: 32×32 pixels < area < 96×96 pixels; Large target: area ≥ 96×96 pixels

[0130] In the VisDrone dataset, the ratio of training set, validation set and test set is 6471:1610:548. This dataset contains ten categories, namely pedestrian, person, car, van, bus, truck, motorcycle, bicycle, awning-tricycle and tricycle. Figure 7 and Figure 8As shown in the figure, the number of cars and pedestrians in this dataset is much larger than other categories, and the target categories of motorcycles (30×30 pixels), people (10×10 pixels) and pedestrians (15×15 pixels) are small targets, while the target categories of trucks (80×80 pixels), vans (60×60 pixels) and buses (80×80 pixels) are medium and large targets. Due to the huge difference in the number of categories, this will cause the model to be more inclined to learn the characteristics of cars during training, and even cause the model to consider awning-tricycles and tricycles as negative samples, thereby giving little weight to the characteristics of these areas. In addition, due to the different areas occupied by target instances of each category in the dataset, it will make it easy for the model to learn the characteristics of trucks, vans and buses, but difficult to learn the characteristics of motorcycles, people and pedestrians. These problems will make it difficult for the target detection model to adapt to the VisDrone dataset.

[0131] The SHWD dataset is a dataset used to identify the wearing of hard hats on construction sites. The ratio of the training set, validation set, and test set is 15887:4641:2261. The dataset contains two categories: hard hats and heads. There are 62282 instances of hard hats and 128266 instances of heads. Fig. 9 As shown in the figure, in the SHWD dataset, most of the targets in the head category are small targets, most of the targets in the helmet category are small and medium-sized targets, and only a very small number of target instances in the entire dataset are large targets. This also means that the application of target detection methods on this dataset will be extremely challenging.

[0132] In the summary of this embodiment, a t-shaped backbone network is first constructed, that is, by upsampling in the deep space of the backbone network, it is ensured that the features of the small target area are retained in the image features. In addition, the shallow features are reused in the t-shaped backbone network, which are fused with the deep features and output four feature maps with different scales, which can maximize the use of computing resources while alleviating the feature loss in the backbone network; then a four-branch enhanced feature extraction network is constructed, and the four feature layers output by the backbone network are subjected to feature fusion through continuous upsampling, downsampling and stacking operations in the four-branch enhanced feature extraction network to mine the target information hidden in the image, so that each feature layer output contains both the feature information of small targets and the feature information of large targets; finally, an HDA decoupling detection head is constructed, and the feature layer extracted by the four-branch enhanced feature extraction network constructs the global relationship in the feature map through the HDA attention module, and performs classification and regression operations through the decoupling head. Through the above method, the present invention alleviates the problem of information loss that occurs in the neural network as the network deepens, and achieves better detection results in the VisDrone2019 dataset and the SHWD dataset. The evaluation results of the present invention on the VisDrone2019 dataset under the MS COCO dataset evaluation standard are shown in Table 1, and the evaluation results on the SHWD dataset are shown in Table 2.

[0133] Table 1 Evaluation results of the method provided by the present invention in the VisDrone dataset

[0134]

[0135] Note: AP is the average precision of IOU threshold from 0.5 to 0.95, with each value of 0.05. 50 is the average precision when IOU is 0.5, AP 75 The average precision of IOU is 0.75, AP S Represents the average precision of small targets, AP m Represents the average precision of the target, AP l Represents the average accuracy of large targets

[0136] Table 2 Evaluation results of the method provided by the present invention in the SHWD dataset

[0137]

[0138] In summary, the present invention provides a small target detection method based on image feature fusion, constructs a T-shaped backbone network based on feature reuse, upsamples the lowest feature layer of the initial backbone network, and fuses shallow features with deep features, and finally outputs four feature layers, which retains more small target area features while alleviating the problem of feature information loss that occurs as the network deepens; the present invention constructs a four-branch enhanced feature extraction network according to the four feature maps output by the T-shaped backbone network, and continuously upsamples and downsamples feature maps of different scales and fuses them to mine more potential features of small targets; the present invention constructs the global relationship in the image through the HDA attention module, thereby enriching the global position information of the target instance in the image, and finally improving the detection accuracy of small targets.

[0139] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0140] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0142] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A small target detection method based on image feature fusion, characterized in that: include: S1 uses the dataset to train the small target detection model by setting the training method and training parameters; S2 uses mosaic data augmentation operations on the dataset as it is traversed; S3 initializes the trained small target detection model, determines whether to use the pre-trained model weights, and clears the optimizer gradient; S4 obtains the resized image and the corresponding label; S5 inputs the resized image into the small target detection model that has executed step S3, obtains image feature layers of different sizes through the T-shaped backbone network of the small target detection model, extracts features from the image feature layers of different sizes through the four-branch enhanced feature extraction network of the small target detection model, and transmits the output of the four-branch enhanced feature extraction network to the HDA decoupling detection head of the small target detection model to classify, identify and locate the small targets in the image; S6 calculates the multi-classification loss value of the small target obtained in step S5 through the BCEWithLogitsLoss loss function, and uses the iou loss function to calculate the distance between the predicted box and the real box of the small target; S7 inputs the multi-classification loss value obtained by executing step S6 into the small target detection model for back propagation to update the gradient features, and adaptively obtains the optimizer learning rate of the small target detection model, and performs a model parameter update operation through the gradient descent of the small target detection model; S8 uses the validation set in the data set to obtain a multi-classification loss value of the target obtained using the validation set by executing steps S5 and S6; if the multi-classification loss value is less than a preset threshold, the small target detection model of this step is saved; S9, repeatedly executing steps S4 to S8 until a preset number of iterations is met, and a converged small target detection model is obtained; S10 performs small target detection on image data through a converged small target detection model.

2. The method according to claim 1, characterized in that The T-shaped backbone network of the small target detection model includes an initial network and a fusion operation unit, wherein the fusion operation unit is used to perform a matrix addition operation on the output of the initial network to obtain four image feature layers that are downsampled by 4 times, 8 times, 16 times and 32 times respectively; The four-branch enhanced feature extraction network of the small target detection model is specifically used for: Extracting features, upsampling and stacking operations are performed on the image feature layers with downsampling ratios of 8 times, 16 times and 32 times output by the T-shaped backbone network; Downsampling, feature extraction and stacking operations are performed on the feature layers with downsampling ratios of 4 times, 8 times and 16 times output by the T-shaped backbone network; Based on the operation results of the above two processes, an enhanced feature map is obtained; The small target detection model also has an HDA decoupling detection head for: Based on the enhanced feature map, constructing a global relationship of features of the enhanced feature map through three set linear layers; The K, Q and V matrices are obtained by setting the convolution kernels to 3×3 and 5×5; the K, Q and V matrices are obtained by the formula K = C onv 2D kernel=3( Li near( L ayer N orm( X )w1)+ b 1) (1) Q= C onv 2D kernel=5( Li near( L ayer N orm( X )w2)+ b 2) (2) V = C onv 2D kernel=3( Li near( L ayer N orm( X )w3)+ b 3) (3) express; Perform matrix addition operation on K matrix and Q matrix, perform matrix dot multiplication operation on V matrix, and perform residual link with input features; By setting the calculation formula of the HDA attention module HDA=MLP(LayerNorm((((Q+K)☉V+X)))+((Q+K)☉V+X) Get all the context information of the enhanced feature map; where Conv2Dkerne l＝1 (·) indicates a 2D convolution with a convolution kernel of 1×1, Conv2Dkerne l＝3 (·) indicates a 2D convolution with a convolution kernel of 3×3, Conv2Dkerne l＝5 (·) indicates a 2D convolution with a kernel of 5×5, Linear(·) indicates a linear layer, LayerNorm(·) indicates layer normalization, X indicates input features, w1, w2, and w3 are weights of the linear layer, b1, b2, and b3 are biases of the linear layer, + and ⊙ indicate matrix addition and matrix dot multiplication, respectively, kernel indicates the convolution kernel, kernel=1 indicates a convolution kernel of 1, kernel=3 indicates a convolution kernel of 3, and kernel=5 indicates a convolution kernel of 5; The multi-layer perceptron is set up to extract features layer by layer from all the context information of the enhanced feature map, and stack the information to obtain the key context information of the enhanced feature map. The key context information of the enhanced feature map is used to input into the standard decoupled detection head of the small object detection model for classification and regression operations.

3. The method according to claim 2, characterized in that The processing process of the four-branch enhanced feature extraction network of the small target detection model specifically includes: For the four image feature layers feat1, feat2, feat3 and feat4 output by the T-shaped backbone network, the following three sub-steps are performed: The features of the E1 image feature layer fe4 are adjusted and upsampled for the first time and then stacked with the features of the image feature layer fe3; The features of the E2 image feature layer feat4 and the image feature layer feat3 are stacked, and the features are adjusted for the second time and stacked with the features of the image feature layer feat2 after upsampling; E3: The features of the stacked image feature layer feat4, image feature layer feat3 and image feature layer feat2 are adjusted for the third time and stacked with the features of the image feature layer feat1 after upsampling; Based on the features after stacking the execution results of sub-steps E1 to E4, it is first directly output as an enhanced feature map feat1', then the first downsampling and feature channel adjustment are performed and the output is an enhanced feature map feat2', then the second downsampling and feature channel adjustment are performed and the output is an enhanced feature map feat3', and finally the third downsampling and feature channel adjustment are performed and the output is an enhanced feature map feat4'.

4. The method according to claim 2, characterized in that: The dataset is a combination of the VisDrone2019 dataset and the SHWD (Safety Helmet Wearing Dataset) dataset, and has a training set, a test set, and a validation set.

5. The method according to any one of claims 1 to 4, characterized in that: In step S1, the process of setting the training method includes: determining whether there is an available GPU, if so, using the GPU, otherwise, using the CPU; The process of setting training parameters includes: Fixed random number seed to 1; Use mixed precision training; The input image size is 1024×1024; The total number of training epochs is 300; batch_size is 8; No pre-trained weights are used; Use adaptive learning rate with a maximum learning rate of 0.01 and a minimum learning rate of 0.0001; The stochastic gradient descent (SGD) optimizer is used with a momentum of 0.937 and a weight decay of 0.0005.

Images