An improved SSD target detection method based on data enhancement and feature fusion

Abstract

The application discloses an improved SSD target detection method based on data enhancement and feature fusion, relates to the field of machine vision, and first transforms and expands input images through various data enhancement methods, reduces overfitting of a model to training samples, and improves generalization capability of the model on unseen samples; a main feature extraction network VGG16 of an original SSD model is replaced by ResNet-50 which has stronger representation capability; a new three-way fusion convolution is used to replace an original convolution layer, a self deep-to-shallow recursive reverse path is added to transmit deep semantic information back to a shallow layer for feature fusion; richer feature information can be acquired, more context information is fused, the expression capability of features is improved, the perception range is improved, and the precision and robustness of target detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, in particular to an improved SSD target detection method based on data enhancement and feature fusion. BACKGROUND

[0002] SSD (Single Shot MultiBox Detector) is a One-stage target detector same as YOLO series, which has become the SOTA model at that time, and is much better than YOLO in accuracy and speed at that time, and the two most important reasons for being superior to other models are: first, SSD uses different scale feature maps for detection, shallow feature maps for small objects and deep feature maps for large objects; second, SSD uses different scale and aspect ratio of prior boxes by referring to the Anchors mechanism in Faster R-CNN, these optimizations make SSD overcome the common shortcomings of other models to a certain extent, such as difficulty in detecting small targets and inaccurate positioning. But because SSD needs to perform convolution operation on multiple feature maps of different sizes, it needs to consume a lot of computing resources and has high computational complexity, in some cases, SSD may miss some small size objects or detect wrong objects, resulting in low detection accuracy, and SSD is sensitive to the shape and occlusion of objects when detecting objects, when the object is deformed or occluded, the detection effect of the algorithm will be greatly affected, and SSD algorithm needs a large amount of labeled data set during training, and the performance of the algorithm is often affected by the quality and size of the data set used, so the adaptability and generalization ability of the algorithm are relatively weak.

[0003] In the prior art, for example, document 1 (FSSD), Li, Z., & Zhou, F. (2017). FSSD: Feature Fusion Single Shot Multibox Detector. ArXiv, abs / 1712.00960. The original SSD is difficult to fuse the features on the feature maps of different scales, and the FSSD is an improved scheme based on the feature pyramid network, which mainly optimizes the problems of the SSD algorithm in processing small targets; FSSD can better utilize the features of different scales in the feature pyramid network to improve the detection accuracy of small targets by introducing a new branch network, document 2 (DSSD), Fu, C., Liu, W., Ranga, A., Tyagi, A., & Berg, A. C. (2017). DSSD: Deconvolutional Single Shot Detector. ArXiv, abs / 1701.06659. The VGG-16 backbone feature extraction network in the SSD algorithm is replaced by the residual network Residual-101, and a deconvolution module is introduced, which improves the detection accuracy, but the DSSD network structure is complex, which leads to slow network training speed, therefore, an improved SSD target detection method based on data enhancement and feature fusion is needed to solve the above problems. SUMMARY

[0004] The application provides an improved SSD target detection method based on data enhancement and feature fusion, which uses other data enhancement methods to expand the training data under the condition of using the same VOC07+12 data set, enhances the generalization ability and robustness of the trained model, adds three fusion convolution and feature fusion modules, and improves the small target detection effect of the model.

[0005] To achieve the above purpose, the application provides the following technical scheme: an improved SSD target detection method based on data enhancement and feature fusion, comprising the following steps:

[0006] S1, based on the input requirements of the SSD algorithm, adaptively process the data set; and randomly select a part of images in each training batch for data enhancement to generate enhanced training samples;

[0007] S2, improve the SSD model:

[0008] Replace the VGG16 model with the ResNet-50 model;

[0009] Replace the convolution layer with three fusion convolution;

[0010] Add a recursive reverse path from deep to shallow to transmit deep semantic information back to the shallow layer for feature fusion;

[0011] S3, input the training sample into the improved SSD model for iterative training:

[0012] S4, using the trained improved SSD model to detect the target of the test set in the data set.

[0013] According to the preferred embodiment of the present application, in step S1, the images and annotation files in the VOC data set are loaded into the memory, and the images are processed to meet the input requirements of the SSD algorithm, such as size adjustment, normalization, etc.; and the annotation files are parsed to obtain the corresponding information, such as the category of the target, the boundary box coordinates, etc.

[0014] According to the preferred embodiment of the present application, in step S1, the data augmentation means includes Mosaic, MixUp and CopyPaste, wherein the data augmentation means is turned off in the last few tens of training.

[0015] According to the preferred embodiment of the present application, in step S2, the VGG16 model is replaced by a ResNet-50 model, specifically:

[0016] The model architecture of VGG16 is replaced by the model architecture of ResNet-50;

[0017] According to the pre-training weight of ResNet-50, the parameters in the network are initialized;

[0018] Adjust the size of the input data to adapt to the input requirements of the ResNet-50 model;

[0019] Extract image features through convolution and pooling layers, and classify through fully connected layers and Softmax layers.

[0020] According to the preferred embodiment of the present application, in step S2, the three-way fusion convolution is configured to perform maximum pooling, dilated convolution and ordinary convolution operations on the input features to realize multi-scale feature extraction and information fusion.

[0021] Further, the output feature maps of the maximum pooling, dilated convolution and ordinary convolution are spliced in the channel dimension to form a new feature map.

[0022] According to the preferred embodiment of the present application, in step S2, different scale feature maps are fused by deconvolution and weight addition.

[0023] According to the preferred embodiment of the present application, in step S3, during the training process, the loss of target classification and regression is calculated using the loss function, and gradient backpropagation and parameter update are performed; until the model converges or reaches the predetermined number of training rounds.

[0024] According to the content of the application, preferably, in step S4, the target frame and category information output by the improved SSD model are evaluated based on evaluation indexes including detection accuracy, recall rate and average precision, and the model is optimized and improved according to the evaluation results.

[0025] Compared with the prior art, the application has the following beneficial effects:

[0026] In the application, the training data is expanded by data enhancement means, the new samples introduce certain noise and uncertainty, which helps to reduce the overfitting of the model to the training samples and improve the generalization ability of the model on unseen samples; in the generation process of the mixed samples, the target label is also linearly interpolated, so that the model has better learning and discrimination ability for the boundary region between different categories in the training process; which helps to improve the robustness of the model to noise, interference and fuzzy boundaries between categories, making the model more robust to changes in input data and reducing the overfitting of the model to training samples.

[0027] In the application, the SSD model is improved to improve the small target detection effect of the model, wherein the VGG16 model is replaced by the ResNet-50 model, which can better capture the semantic information of the image and improve the expression ability of the model, and the ResNet-50 introduces a residual block to solve the problems of gradient disappearance and model degradation, and accelerate the training speed and convergence speed; ResNet-50 has lower parameter quantity and computational complexity than VGG16, which improves the efficiency and adaptability of the model.

[0028] By replacing the convolution layer with a three-way fusion convolution, the main features are extracted by the max pooling operation, the receptive field is increased by the dilated convolution operation, and the local details are captured by the ordinary convolution operation, which can obtain more rich feature information; and this feature fusion strategy can fully utilize the global context information to improve the accuracy and robustness of target detection.

[0029] By using the recursive reverse path, the fusion of feature maps with different scales is realized by using the method of deconvolution and weight addition. Since the deconvolution operation introduces more detailed information in the upsampling process, by fusing the upsampled feature map with the original feature map, the detailed information of the original feature map can be preserved, while more context information is fused, the expression ability of the feature is improved, the perception range is improved, and the accuracy and robustness of target detection are improved. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, which together with the embodiments of the application, is used to explain the application, and does not constitute a limitation on the application.

[0031] In the drawings:

[0032] Figure 1 is the improved SSD network structure diagram of the application;

[0033] Figure 2 is the three-way fusion convolution (TFC) structure diagram of the application;

[0034] Figure 3 is the feature fusion module (SUM) structure diagram of the application. DETAILED DESCRIPTION

[0035] The preferred embodiments of the application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0036] Embodiment: An improved SSD target detection method based on data enhancement and feature fusion, comprising the following steps:

[0037] S1, pre-processing the data

[0038] 1.1, load the images and annotation files in the VOC dataset into the memory;

[0039] 1.2, perform pre-processing operations on the images, such as size adjustment, normalization, etc., to make them suitable for the input requirements of the SSD algorithm;

[0040] 1.3, parse the annotation files to extract information such as the class of the target, the boundary box coordinates, etc.

[0041] 1.4, randomly select a part of the images in each training batch for data enhancement operations;

[0042] Among them, in addition to the pixel content transformation, spatial geometric transformation and coordinate conversion, scaling and mean value reduction contained in baseline, the application also adds Mosaic, MixUp and CopyPaste means, the way Mosaic increases samples is to randomly select 4 pictures to splice, and mix the labels of the 4 pictures together as the label of the newly generated picture, this method is verified by Yolov5, and it is proved that it can bring significant effect; The way Mixup increases samples is to linearly interpolate the training samples, so that two pictures mix the RGB values in a certain proportion, and MixUp is currently a stable point strategy in various competitions and various target detection; The way CopyPaste increases samples is to copy and paste the instances in one picture to another picture to get new training samples during training, and it is found that the use of this method can significantly improve the effect compared with the original SOTA method, but it should be noted that in the last few times of training or even dozens of times, these data enhancement means should be turned off, because the pictures generated by Mosaic+MixUp+CopyPaste are greatly deviated from the real distribution of natural pictures, and a large number of cutting operations will cause many inaccurate marking boxes;

[0043] 1.5、According to the enhanced image and the label information, generate an enhanced training sample.

[0044] S2, improve the SSD model, refer to Figure 1 The, the improved SSD network structure diagram of the application; wherein:

[0045] 2.1, take the original SSD model as the benchmark model, improve on the basis of this model, the source code of the model is open source in GitHub, including feature extraction network and prediction network, use convolutional neural network structure to extract the feature map of the input image, the prediction network carries out target classification and boundary box regression on different scale feature maps;

[0046] 2.2, replace the VGG16 model with the ResNet-50 model;

[0047] 1) Model architecture replacement: replace the model architecture of VGG16 with the model architecture of ResNet-50, including convolutional layer, batch normalization layer, residual block, etc.;

[0048] 2) Parameter initialization: according to the pre-training weight of ResNet-50, initialize the parameters in the network to ensure that the network has good feature extraction capability;

[0049] 3) Input adaptation: adjust the size of the input data to adapt to the input requirements of the ResNet-50 model;

[0050] 4) Feature extraction and classification: image features are extracted through convolution and pooling layers, and classification is performed through fully connected layers and Softmax layers;

[0051] 5) Parameter fine-tuning: according to the requirements, the ResNet-50 model is selected for parameter fine-tuning to improve the performance of the model;

[0052] 2.3, replace the convolutional layer with a three-way fusion convolution, as shown in Figure 2 , which is a three-way fusion convolution (TFC) structure diagram. Through maximum pooling, dilated convolution and ordinary convolution operations on the input features, using the Mish activation function, the output feature maps of the three paths are spliced in the channel dimension to form a new feature map, and finally a 1x1 convolution is used to fuse the three features;

[0053] 1) Input features: assume that the input features are a three-dimensional tensor with size HxWxC, where H and W represent the height and width of the feature map, respectively, and C represents the number of channels of the feature map;

[0054] 2) Max pooling: maximum pooling operation is performed on the input features, which can extract the main features of the feature map and reduce the size of the feature map; the maximum pooling operation usually uses a fixed-size pooling window, and selects the maximum value in the window as the output; assuming that the maximum pooling operation uses a pooling window with size kxk, the output feature map size of the maximum pooling path is (H / k)x(W / k)x C;

[0055] 3) Dilated convolution: dilated convolution is performed on the input features, which can increase the receptive field of the convolution operation and extract more global context information; dilated convolution uses a dilation rate parameter to represent the spacing between sampling points within the convolution kernel; assuming that the dilated convolution operation uses a 3x3 convolution kernel and a dilation rate of d, the output feature map size of the dilated convolution path is the same as the input feature map, i.e. HxWxC;

[0056] 4) Standard convolution: standard convolution is performed on the input features, which can capture local details and edge information; standard convolution operation uses a fixed-size convolution kernel, which slides on the input feature map and performs convolution calculation; assuming that the standard convolution operation uses a 3x3 convolution kernel, the output feature map size of the standard convolution path is the same as the input feature map, i.e. HxWxC;

[0057] 5) Feature fusion: The output feature maps of the three paths are spliced in the channel dimension to form a new feature map; the size of the spliced feature map is HxWx(3C), where 3C represents that the channel number is three times the original, and finally a 1x1 convolution is used to fuse the three features;

[0058] 6) Output feature: After feature fusion, a feature map with a size of HxWx(3C) is finally obtained as the output;

[0059] 2.4, a self-deep-to-shallow recursive reverse path is added to transmit deep semantic information back to the shallow layer for feature fusion; reference Figure 3 is a feature fusion module (SUM) structure diagram, and the deep feature map Conv_2 is deconvoluted to be upsampled to the same scale as the feature map Conv_1; the deconvolution operation uses a fixed-size deconvolution kernel, and the deconvolution calculation is performed on the feature map B to expand its scale to the same scale as the feature map A, and then two 3x3 convolution layers are used to learn better feature fusion, and then normalized according to the proportions of 10 and 20, and the two feature maps are weighted and fused by defining a set of weight parameters, and finally the fused feature map Fusion_conv is obtained as the output; details:

[0060] 1) Input feature map: assuming that there are two input feature maps with different scales, which are feature map A and feature map B; the size of the feature map A is Ha x Wa x Ca, and the size of the feature map B is Hb x Wb x Cb, wherein Ha, Wa and Ca represent the height, width and channel number of the feature map A, and Hb, Wb and Cb represent the height, width and channel number of the feature map B;

[0061] 2) Deconvolution (Transpose Convolution): The feature map B is deconvoluted to be upsampled to the same scale as the feature map A; the deconvolution operation uses a fixed-size deconvolution kernel, and the deconvolution calculation is performed on the feature map B to expand its scale to the same scale as the feature map A; assuming that the deconvolution operation uses a deconvolution kernel with a size of kxk, then the size of the deconvoluted feature map B is (Hbxs) x (Wbxs) x (Cb / s^2), wherein s represents the upsampled multiple;

[0062] 3) Weight addition: the deconvoluted feature map B and the feature map A are weighted and added; the two feature maps are weighted and fused by defining a set of weight parameters; assuming that the feature map A and the deconvoluted feature map B are Fa and Fb respectively, and the weight parameters are a and β, then the fused feature map is fusion_conv = a*Fa + β*Fb;

[0063] 4), output feature: the fused feature map fusion_conv is obtained as the output;

[0064] S3, model training:

[0065] 3.1, input the data enhanced training sample into the improved SSD model for training;

[0066] 3.2, use the loss function to calculate the loss of target classification and regression, and carry out gradient back propagation and parameter update;

[0067] 3.3, iterative training process until the model converges or reaches the predetermined training round;

[0068] S4, model evaluation:

[0069] 4.1, using the trained improved SSD model to detect the test set in VOC data set;

[0070] 4.2, according to the target frame and category information output by the model, calculate the detection precision, recall rate, average precision and other indicators;

[0071] 4.3, the model can be optimized and improved according to the evaluation results.

[0072] Finally, it should be pointed out that: the above only for the preferred examples of the present application, and not for limiting the present application, although the present application is described in detail with reference to the foregoing examples, for those skilled in the art, it still can modify the technical scheme recorded in the foregoing examples, or make equivalent replacement for part of the technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An improved SSD target detection method based on data augmentation and feature fusion, characterized in that, Includes the following steps: S1. Based on the input requirements of the SSD algorithm, the dataset is adaptively processed; and in each training batch, a portion of images are randomly selected for data augmentation to generate augmented training samples. S2. Improvements to the SSD model: Replace the VGG16 model with the ResNet-50 model; Replace the convolutional layers with three-way fused convolutions; Add a recursive reverse path from deep to shallow to transmit deep semantic information back to the shallow layer for feature fusion; S3. Input the training samples into the improved SSD model for iterative training: S4. Use the trained, improved SSD model to perform object detection on the test set in the dataset. In step S2, the three-way fusion convolution is configured to perform max pooling, dilated convolution, and ordinary convolution operations on the input features to achieve multi-scale feature extraction and information fusion. The output feature maps of the max pooling path, dilated convolution path, and ordinary convolution path are concatenated along the channel dimension to form a new feature map.

2. The improved SSD target detection method based on data augmentation and feature fusion according to claim 1, characterized in that: In step S1, the images and annotation files in the VOC dataset are loaded into memory, and the images are processed to meet the input requirements of the SSD algorithm. And to parse the annotation files to obtain the corresponding information.

3. The improved SSD target detection method based on data augmentation and feature fusion according to claim 1, characterized in that: In step S1, the data augmentation methods include Mosaic, MixUp, and CopyPaste, with the data augmentation methods turned off in the last few dozen training iterations.

4. The improved SSD target detection method based on data augmentation and feature fusion according to claim 1, characterized in that: In step S2, the VGG16 model is replaced with the ResNet-50 model, specifically as follows: Replace the VGG16 model architecture with the ResNet-50 model architecture; The parameters in the network are initialized based on the pre-trained weights of ResNet-50; Adjust the size of the input data to fit the input requirements of the ResNet-50 model; Image features are extracted using convolutional and pooling layers, and classification is performed using fully connected and softmax layers.

5. The improved SSD target detection method based on data augmentation and feature fusion according to claim 1, characterized in that: In step S2, feature maps of different scales are fused by deconvolution and weight addition.

6. The improved SSD target detection method based on data augmentation and feature fusion according to claim 1, characterized in that: In step S3, during the training process, the loss function is used to calculate the loss for target classification and regression, and gradient backpropagation and parameter updates are performed until the model converges or reaches the predetermined number of training rounds.

7. The improved SSD target detection method based on data augmentation and feature fusion according to claim 1, characterized in that: In step S4, the target bounding boxes and category information output by the improved SSD model are evaluated, and the evaluation metrics include detection precision, recall rate and average precision.

Images