A micro-target adaptive detection method based on data enhancement and feature fusion

Abstract

The present application relates to a kind of micro target adaptive detection methods based on data enhancement and feature fusion, comprising the following steps: S1, all targets in the dataset are processed by pixel screening method;S2, backbone network is constructed;S3, adaptive image pyramid network is constructed;S4, multi-scale fusion feature information is extracted;S5, detection prediction network is constructed.The present application successfully increases the number of small target samples without increasing the amount of image data by using the screening resampling processing method for target objects, the method is more simple and direct, and the detection performance is obviously improved;By using convolutional neural network in deep learning to extract multi-scale fusion feature information of image, it is more adaptable, and the detection effect is better.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology in deep learning, and in particular to an adaptive detection method for small targets based on data augmentation and feature fusion. Background Technology

[0002] Small object detection has numerous applications in real life. In autonomous driving, safe driving requires detecting distant road signs and pedestrians from images captured by car cameras. In medical lesion detection imaging, the initial lesion area is small, making early detection of lumps and tumors crucial for accurate diagnosis and treatment. In intelligent security, it's necessary to detect whether workers at distant locations are correctly wearing safety helmets; however, workers scattered across various locations appear as only a few dozen or even a few pixels in the captured images, making small object detection a critical problem that needs to be solved in intelligent security. Currently, many experts have proposed object detection algorithms based on deep learning, but due to the insufficient number of small object samples in datasets and incomplete feature extraction, the performance of small object detection remains unsatisfactory.

[0003] Object detection algorithms based on convolutional neural networks can be broadly categorized into two-stage and one-stage algorithms. Two-stage algorithms, such as the R-CNN series, R-FCN, and AAAI, utilize selective search algorithms to select a large number of candidate regions, which are then used as input to the subsequent convolutional neural network to acquire features. Finally, the detection results are obtained by classifying and regressing the candidate regions. One-stage algorithms, such as SSD (Single Shot MultiBox Detector) and the YOLO (You Only Look Once) series, directly predict the target's location and category information by inputting the image into the convolutional neural network.

[0004] Some algorithms locate regions by determining the contextual information of the target to avoid losing key location information of small targets during feature extraction. Lin used the multi-scale fusion strategy of FPN to extract and fuse features at different scales to obtain deep semantic information and shallow location information (Lin TY, Dollar P, Girshick R, et al. Feature pyramid networks for object detection[C] / / Proceedings of the IEEE conferenceon computer vision and pattern recognition.2017:2117-2125.). Chen et al. used different feature extraction methods for deep and shallow features to improve the detection effect of small targets (CHEN HJ, WANG QQ, YANG GW, et al. SSD object detection algorithm with multi-scale convolution feature fusion[J]. Journal of Frontiers of Computer Science and Technology,2019,13(6):1049-1061.). Zhai et al. proposed a multi-scale feature fusion mechanism to fuse shallow positional features with deep semantic features in the network structure (ZHAI S, SHANG D, WANG S, et al. DF-SSD: An improved SSD object detection algorithm based on DenseNet and feature fusion[J]. IEEE Access, 2020, 8: 24344-24357.). Huang et al. proposed a cross-scale feature fusion method to enhance contextual connections, but cross-scale feature maps have semantic differences, and direct cross-layer fusion may lead to positional shifts and aliasing effects (HUANG W, LI G, CHEN Q, et al. Cf2pn: A cross-scale feature fusion pyramid network based remote sensing target detection[J]. Remote Sensing, 2021, 13(5): 847.).

[0005] To address the above issues, an attention mechanism is proposed to improve feature information extraction capabilities. AFF adds local channel context information to the global channel through attention feature fusion to overcome the problems of semantic differences and scale inconsistencies between input features. Liang et al. used a dense connection structure to improve feature information extraction capabilities and enhance the contextual semantic information of small targets in shallow features (LIANG YANYU, LI JINBAO. Small target detection algorithm for multi-scale non-local attention networks[J]. Computer Science and Exploration,2020,14(10):1744-1753.). RFB-Net constructs a multi-branch convolutional layer receptive field module and uses dilated convolution to expand the receptive field of the feature map, which improves the detection capability of small targets to a certain extent.

[0006] Some existing algorithms replace sparse coding labels with dense coding labels based on angle classification; approximate any rotated rectangle as a two-dimensional Gaussian distribution; and extract constant-speed rotation features by merging a constant-speed rotation network with a detector, in order to solve the problem of multi-angle detection difficulties caused by the complexity of small target position information.

[0007] CenterNet, a simple and highly scalable single-stage object detection algorithm, can detect and classify objects using only a fully convolutional network, without the need for anchor boxes and non-maximum suppression (NMS), thus offering some benefits for addressing the aforementioned issues. However, CenterNet also has some drawbacks, such as relying solely on the features of the last layer for prediction, which leads to the loss of some object feature information and results in poor performance when detecting small-scale objects. Summary of the Invention

[0008] To address the technical problems existing in the prior art, this invention provides an adaptive detection method for small targets based on data augmentation and feature fusion. By utilizing a selective resampling process for the target object, the number of small target samples is successfully increased without increasing the amount of image data. The method is simpler and more direct, and the detection performance is significantly improved. Furthermore, by using a convolutional neural network in deep learning to extract multi-scale fusion feature information from the image, the method is more adaptable and achieves better detection results.

[0009] This invention employs the following technical solution: an adaptive detection method for small targets based on data augmentation and feature fusion, comprising the following steps:

[0010] S1. Pixel-filtering processing of all targets in the dataset: By selecting a public dataset of helmet wearing, a selective small target copying algorithm is used to perform pixel-filtering processing on all targets in the dataset images, and the processed target boxes are put back into the corresponding original images to expand the image dataset.

[0011] S2. Constructing the backbone network: Downsample the dataset images obtained in step S1 to obtain deep feature maps;

[0012] S3. Construct an adaptive image pyramid network: Decouple the detection of each target object and assign more suitable feature levels to small targets with high-resolution feature mappings;

[0013] S4. Extract multi-scale fusion feature information: fuse the feature information of the backbone network and the adaptive image pyramid network. By connecting the backbone network and the adaptive image pyramid network laterally, multi-scale fusion feature information is obtained. Construct a bottom-up and top-down fusion path to fuse deep semantic information into shallow positional information.

[0014] S5. Construct a detection and prediction network: After generating the mapping map in step S4, output the fused features for prediction, generate key point heatmaps, bounding box scale predictions and key point deviation predictions, optimize the loss function until the loss function converges; use the predicted center key point coordinates to estimate the target position, and complete the training of the multi-scale detection model for small target objects with multi-semantic feature fusion.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0016] 1. This invention utilizes a screening-based resampling method for target objects to successfully increase the number of small target samples without increasing the amount of image data. The method is simpler and more direct, and the detection performance is significantly improved.

[0017] 2. This invention extracts multi-scale fusion feature information of images by using convolutional neural networks in deep learning, which is more adaptable and has better detection effect. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention;

[0019] Figure 2 This is the overall flowchart of the selective small target replication algorithm;

[0020] Figure 3 This is a schematic diagram of the composition structure of the adaptive feature extraction module;

[0021] Figure 4This is a diagram comparing the number of target samples and the difference in the dataset with and without using the selective small target replication algorithm;

[0022] Figure 5 (a) is a schematic diagram of the data image of Example 1 after processing by the Selective Small Target Replication Algorithm;

[0023] Figure 5 (b) is a schematic diagram of the data image of Example 2 after processing by the Selective Small Target Replication Algorithm;

[0024] Figure 5 (c) is a schematic diagram of the data image of Example 3 after processing by the Selective Small Target Replication Algorithm;

[0025] Figure 5 (d) is a schematic diagram of the data image of Example 4 after processing by the Selective Small Target Replication Algorithm;

[0026] Figure 5 (e) is a schematic diagram of the data image of Example 5 after processing by the Selective Small Target Replication Algorithm;

[0027] Figure 5 (f) is a schematic diagram of the data image of Example 6 after processing by the Selective Small Target Replication Algorithm;

[0028] Figure 6 (a) is a schematic diagram of the visualization detection results of the original method;

[0029] Figure 6 (b) is a schematic diagram of the visual detection result 1 of the method proposed in this invention;

[0030] Figure 6 (c) is a schematic diagram of the visualization results of the original method;

[0031] Figure 6 (d) is a schematic diagram of the visualization detection results of the method proposed in this invention.

[0032] Figure 6 (e) is a schematic diagram of the visualization results of the original method;

[0033] Figure 6 (f) is a schematic diagram of the visualization detection results of the method proposed in this invention.

[0034] Figure 6 (g) is a schematic diagram of the visualization results of the original method;

[0035] Figure 6 (h) is a schematic diagram of the visualization detection results of the method proposed in this invention.

[0036] Figure 6(i) is a schematic diagram of the visualization results of the original method;

[0037] Figure 6 (j) is a schematic diagram of the visualization detection results of the method proposed in this invention.

[0038] Figure 6 (k) is a schematic diagram of the visualization results of the original method;

[0039] Figure 6 (l) is a schematic diagram of the visualization detection results of the method proposed in this invention;

[0040] Figure 7 The method proposed in this invention differs from the original method in terms of Log-Average Miss Rate (MR). -2 A diagram showing the comparison of results in the indicators. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0042] Example

[0043] like Figure 1 As shown in the figure, this embodiment presents an adaptive detection method for small targets based on data augmentation and feature fusion, which includes the following steps:

[0044] S1. Pixel-filtering processing of all targets in the dataset: By selecting a public dataset of helmet wearing, a selective small target copying algorithm is used to perform pixel-filtering processing on all targets in the dataset images, and the processed target boxes are put back into the corresponding original images to expand the image dataset.

[0045] S2. Constructing the backbone network: Downsample the dataset images obtained in step S1 to obtain deep feature maps;

[0046] S3. Construct an adaptive image pyramid network: Decouple the detection of each target object and assign more suitable feature levels to small targets with high-resolution feature mappings;

[0047] S4. Extract multi-scale fusion feature information: fuse the feature information of the backbone network and the adaptive image pyramid network. By connecting the backbone network and the adaptive image pyramid network laterally, multi-scale fusion feature information is obtained. Construct a bottom-up and top-down fusion path to fuse deep semantic information into shallow positional information.

[0048] S5. Construct a detection and prediction network: After generating the mapping map in step S4, output the fused features for prediction, generate key point heatmaps, bounding box scale predictions and key point deviation predictions, optimize the loss function until the loss function converges; use the predicted center key point coordinates to estimate the target position, and complete the training of the multi-scale detection model for small target objects with multi-semantic feature fusion.

[0049] Specifically, in this embodiment, the specific process of step S1 is as follows:

[0050] S11. Select a public dataset of helmet wearing, and use a selective small target object copying algorithm, such as... Figure 2 As shown, a filtering resampling is performed on all target boxes in the dataset to filter out all targets whose pixel values ​​are within the threshold and treat them as small targets. All small targets are extracted and their corresponding original images are found.

[0051] S12. The extracted small target is copied twice, and the original small target image is horizontally flipped once during the second copy.

[0052] S13. After copying, put these small target objects back into the original images of their respective datasets according to their initial extracted sizes. Add the relevant information of these small target objects to the XML file corresponding to the images to obtain the expanded image dataset.

[0053] Specifically, in this embodiment, the specific process of step S2 is as follows:

[0054] S21. Construct the ST-CenterNet backbone network, using a standard residual network.

[0055] S22. A four-layer feature layer is used for target downsampling to obtain deep semantic information while maximizing the preservation of the positional information of shallow small targets. The first layer uses a convolutional layer with a stride of 2 to obtain a feature map with a resolution of 1 / 2 of the original image. The second layer uses a convolutional layer with a stride of 2 to obtain a feature map with a resolution of 1 / 4 of the original image. The third and fourth layers all use convolutional layers with a stride of 2 to extract features. The final output feature map has a resolution of 1 / 16 of the original image.

[0056] S23. Represent the feature map extracted by the residual network as C:

[0057] C = {C2, C3, C4, C5}

[0058] Where c2, c3, c4, and C5 represent the extracted layers of the ResNet-50 network, respectively.

[0059] Specifically, in this embodiment, the specific process of step S3 is as follows:

[0060] S31. Use shallow feature layers to extract feature information of small target objects to obtain location region information;

[0061] S32. Using the feature map obtained from downsampling in step S2 as input, construct its pyramid set P:

[0062] p = {P2, P3, P4, P5}

[0063] Wherein, P2, P3, P4, and P5 represent the levels of the extracted image pyramid, which correspond to the number of levels of the residual network in step S2.

[0064] Specifically, such as Figure 3 As shown, in this embodiment, the specific process of step S4 is as follows:

[0065] S41. Perform pixel-by-pixel addition and fusion processing on the deep feature layer Ci after double upsampling and the adjacent feature layer Pi:

[0066] P i =P i+1 ↑ 2× +C i

[0067] Among them, C i P represents the feature of the last residual block corresponding to the i-th layer; i P represents the final feature corresponding to the i-th layer; i+1 This represents the final feature corresponding to the (i+1)th layer; ↑ 2× This indicates that upsampling is performed using nearest neighbor interpolation;

[0068] S42. To address the gradient explosion problem that occurs as the number of network layers increases, a linear combination of network layers is performed on the basis of the original network:

[0069]

[0070] Where W is the output vector after linear combination processing of the feature layer; This represents multiple convolutional layers, where Let L be the sigmoid function, γ be the input vector of the feature layer, and L be the input vector of the feature layer. i This is the weight layer; the square matrix α is used for specific linear operations;

[0071] S43. Each feature map is increased to the same number of channels using convolutional kernels, enhancing the feature information of each predicted feature map. Lateral connections are then used to fuse these features with shallow features. These lateral connections are analogous to skip connections in the residual structure. Features obtained through upsampling and downsampling are passed to an activation gate composed of multiple convolutional kernels and activation functions, generating a lateral feature fusion factor that fuses deep semantic features with shallow positional features.

[0072] A F =P(S) i (P i ,f,s))

[0073] Among them, A F This refers to the feature information that fuses deep semantic features and shallow location features; β is the activation function; S i The convolution is at layer i, where i takes values ​​of {2, 3, 4, 5}; f represents the lateral connections; s = 2. i-1 The convolution stride;

[0074] S44. Use convolution to eliminate the aliasing effect of upsampling and generate a new feature map that fits the target:

[0075]

[0076] Where i takes the value {2,3}; w and h are the width and height of the target box, respectively.

[0077] Specifically, in this embodiment, during the construction of the detection and prediction network in step S5, the generated loss is divided into three parts: heatmap loss, bounding box width and height loss, and center key point bias loss; the total loss function is expressed as:

[0078] L SHL =L H +λ SIZE L SIZE +λ OFF L OFF

[0079] Among them, L H This represents the heatmap loss value, which borrows from focal loss to address the issue of class imbalance in detection; L SIZE This represents the loss value for the width and height of the bounding box, set to λ. SIZE 0.1; L OFF This represents the bias loss value at the center point, and λ is set. OFF The value is set to 1 to improve the accuracy of center point coordinate prediction.

[0080] The following detailed description of the effects of the present invention is based on experimental results, comparative charts, and examples:

[0081] like Figure 4 The image shows a comparison of the number of target samples and the difference in the dataset under the conditions of using and not using the selective small target replication algorithm on the helmet wearing dataset, based on the present invention. Figure 4 As can be seen, after using the selective small target copying algorithm to perform secondary copying and flipping of small targets, the number of samples for each type of target is increased.

[0082] like Figure 5 As shown in (a), 5(b), 5(c), 5(d), 5(e), and 5(f), it can be seen from the figures that the target box with a pixel value less than or equal to 32×32 is copied twice, and the second copy is horizontally flipped. When the copied target is put back into the original image, it does not overlap with the original target box, which further enriches the position information of the small target.

[0083] like Figure 6 As shown in (a), 6(b), 6(c), 6(d), 6(e), 6(f), 6(g), 6(h), 6(i), 6(j), 6(k), and 6(l), the qualitative detection results comparison in the figure shows that the present invention is significantly better than the original algorithm in detecting small targets in the dataset images. Small-scale targets that are difficult to detect and small targets that are occluded can be detected more and more accurately, which further proves that the improved method adopted in the present invention effectively improves the model's detection performance for small targets.

[0084] like Figure 7 The figure shown is a comparison of the results of the proposed method and the original algorithm in the Log-Average Miss Rate index. Figure 7 As can be seen, the Log-Average Miss Rate of the method proposed in this invention is lower than that of the original algorithm for various targets, demonstrating that the detection performance of this invention is significantly improved.

[0085] To verify the effectiveness of the proposed adaptive detection method for small targets based on data augmentation and feature fusion in detecting small objects, experiments were conducted on the Safety Helmet Wearing Dataset. First, the performance of the proposed algorithm was compared with commonly used algorithms, as shown in Table 1, which compares the detection performance of various target detection algorithms with that of the proposed algorithm on the safety helmet wearing detection dataset. The superior detection performance of the proposed algorithm is evident; specifically, Table 1 is shown below:

[0086]

[0087] Table 1

[0088] Secondly, in the ablation experiments, the contribution of each module to the model performance and the specific reasons for the effects of these modules were analyzed using the Safety Helmet dataset. Specifically, the effect of each module in the proposed method was analyzed by applying these modules step by step, and the contribution of each module is recorded in Table 2; specifically, Table 2 is shown below:

[0089]

[0090] Table 2

[0091] As shown in Table 2, the proposed algorithm, utilizing a selective small target object replication algorithm, improves the detection performance by 13.90% in mAP compared to the original algorithm. This indicates that increasing the number of small target samples by replicating small target objects is effective in improving target detection results. The target-adaptive feature extraction module fuses deep and shallow features, achieving a 13.22% improvement in mAP compared to the original algorithm. This demonstrates that the module can effectively fuse shallow detail information with deep semantic information, enhancing the semantic information and feature expressive power of the shallow feature map. Finally, with both modules incorporated (i.e., the present invention), the detection performance is improved by 18.08% in mAP compared to the original algorithm.

[0092] This invention proposes a small target data augmentation and detection algorithm to address the problems of insufficient shallow feature information extraction and insufficient small target sample size in the dataset required for small target detection. The algorithm first utilizes a selective resampling method for target objects, successfully increasing the number of small target samples without increasing the image data volume. Then, it proposes a target-adaptive feature extraction module, which enhances the ability to extract shallow semantic information from the image by combining a residual network with an adaptive feature pyramid network, thereby extracting more complete feature semantic information of the target. Experiments on the safety helmet public dataset show that, compared to the original dataset, the dataset after copying and flipping small targets using a selective small target object replication algorithm improves the model's mAP by 13.9%; the target-adaptive feature extraction module enhances the ability to extract target features, improving the model's mAP by 13.22% compared to the original model; and compared to the original algorithm, the proposed algorithm ST-CenterNet improves the mAP by 18.08%.

[0093] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for micro object adaptive detection based on data enhancement and feature fusion, characterized in that, The method comprises the following steps: S1, pixel screening all targets in the data set: by selecting a safety helmet wearing public data set, using a selection type small target copy algorithm to process all targets in the data set image, and putting the processed target frame back to the corresponding original image to expand the image data set; S2, constructing a backbone network: the data set image obtained by step S1 is down-sampled to obtain a deep feature map; S3, constructing an adaptive image pyramid network: decoupling the detection of each target object, and assigning a more suitable feature level to the small target with high resolution feature mapping; S4, extracting multi-scale fusion feature information: fusing the feature information of the backbone network and the adaptive image pyramid network, obtaining multi-scale fusion feature information by horizontally connecting the backbone network and the adaptive image pyramid network, constructing a bottom-up and top-down fusion path, and fusing deep semantic information into shallow position information; S5, constructing a detection prediction network: after generating the mapping diagram in step S4, output the fusion features for prediction, generate the key point heat map, the scale prediction of the bounding box and the deviation prediction of the key point, optimize the loss function until the loss function converges; the position of the target is estimated by using the center key point coordinates obtained by prediction, and the training of the multi-scale detection model of the small target object with multi-semantic feature fusion is completed; The specific process of step S1 is as follows: S11, selecting a safety helmet wearing public data set, using a selection type small target object copy algorithm, and performing a screening type repeated sampling on all target frames in the data set, screening out all targets with pixel values within the threshold and regarding them as small targets, and extracting all small targets and finding the corresponding original images respectively; S12, duplicating the extracted small targets again, and performing a horizontal flip processing on the original small target image during the second duplication; S13, after duplication, put these small target objects back into the original images of the data set according to the initial size extracted, add the related information of these small target objects in the xml file corresponding to the image, and obtain the expanded image data set; The specific process of step S2 is as follows: S21, constructing an ST-CenterNet backbone network, and adopting a standard residual network; S22, using four layers of feature layers for target down-sampling processing; the first level uses a convolution layer with a step of 2 to obtain a feature map with a resolution of 1 / 2 of the original image; The second level uses a convolution layer with a step of 2 to obtain a feature map with a resolution of 1 / 4 of the original image; the third to fourth levels all use convolution layers with a step of 2 to extract features, and finally output a feature map with a resolution of 1 / 16 of the original image; S23, the feature map extracted by the residual network is represented as C: C={C2,C3,C4,C5} Wherein, C2, C3, C4, C5 respectively represent each level of the extracted ResNet-50 network; The specific process of step S4 is as follows: S41, the deep feature layer after twice up-sampling is pixel by pixel added and fused with the adjacent feature layer: P i = P i+1 ↑ 2× + C i where C i denotes the last residual block feature corresponding to the i-th layer; P i denotes the final feature corresponding to the i-th layer; P i+1 denotes the final feature corresponding to the i+1-th layer; ↑ 2× denotes a two times up-sampling process by nearest neighbor interpolation; S42, the network layers are linearly combined on the basis of the original network: wherein W is an output vector after linear combination processing of feature layers; denotes a plurality of convolutional layers, wherein is a sigmoid function, γ is an input vector of a feature layer, L i is a weight layer; a square matrix α is used for specific linear operations; S43, each feature map is improved to the same number of channels through a convolution kernel, enhancing the feature information of each prediction feature map; the feature map is fused with the shallow feature before and after by using horizontal connection, the horizontal connection is the same as the skip connection in the residual structure, and each layer feature obtained by up-sampling is transmitted to an activation gate composed of multiple convolution kernels and activation functions before and after combination, a feature horizontal fusion factor is generated, and the deep semantic feature information and the shallow position feature information are fused: A F = β(S i (P i ,f,s)) wherein A F is the feature information fused by the deep semantic feature information and the shallow location feature information; β is an activation function; S i is the convolution of the i-th layer, i respectively takes {2, 3, 4, 5}; f is a lateral connection; s = 2 i-1 is a convolution step length; S44, convolution is used to eliminate the aliasing effect of up-sampling, and a new feature map suitable for the target is generated: Where i is {2,3}; w and h are the width and height of the target frame respectively.

2. The micro-object adaptive detection method based on data enhancement and feature fusion according to claim 1, characterized in that, The specific process of step S3 is as follows: S31, the position region information is obtained by using the shallow feature layer to extract small target object feature information; S32, the feature mapping obtained by down-sampling in step S2 is taken as input to construct a pyramid set P: P={P2, P3, P4, P5} Where P2, P3, P4, P5 represent each level of the extracted image pyramid respectively.

3. The micro-object adaptive detection method based on data enhancement and feature fusion according to claim 1, characterized in that, In the process of constructing the detection prediction network in step S5, the generated loss is divided into three parts: heat map loss, loss of bounding box width and height, and bias loss of center key point; the total loss function is represented as: L SHL = L H + λ SIZE L SIZE + λ OFF L OFF wherein, L H represents a heat map loss value; L SIZE represents a bounding box width-height loss value, and λ SIZE is set to 0.1; L OFF represents a center point bias loss value, and λ OFF is set to 1.

Images