Construction method of multi-scale lightweight densely connected object detection network

Abstract

The invention discloses a method for constructing a multi-scale lightweight densely connected target detection network, data preprocessing; constructing a multi-scale lightweight target detection neural network; using the processing result to train and test the constructed neural network; the invention uses the PASCAL VOC 2012 data set to verify the reliability of the network, and the mAP of the final network is almost the same as YoloV4, only reduced by 1.5%, and improved by 5% compared with YoloV3. The parameter amount of the network model trained by the invention is compressed to 11.61M, which is reduced to 1 / 6 of YoloV4; the total calculation amount Total Flops is reduced to 1.13G, which is reduced to 1 / 26 of the original; the total memory usage is reduced to 135.46M, which is reduced to 1 / 5 of YoloV4, and the real-time speed is improved by 32 times, which increases the possibility of deploying the target detection method in embedded devices and mobile devices with limited computing resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and artificial intelligence, and relates to a method for constructing a multi-scale lightweight densely connected target detection network. Background Art

[0002] The task of object detection is to find all the objects of interest in an image and determine their categories and locations. It is one of the core issues in the field of computer vision. In recent years, convolutional neural networks have achieved excellent performance in image recognition, object detection, semantic segmentation, etc. However, traditional CNNs usually require a large number of parameters and floating-point operations to maintain performance. Although they have extremely high accuracy, they also have relatively high requirements for computing resources. Such convolutional neural networks are not suitable for deployment on embedded devices or mobile devices with limited computing resources, which greatly limits the deployment and application of object detection methods in complex environments. Therefore, lightweighting the object detection network is a necessary step to increase the generalization ability of object detection methods for various deployment environments.

[0003] Aiming at the problems of large number of parameters, slow inference speed, high storage space occupation, and difficulty in deployment on embedded platforms with limited computing resources in traditional deep learning methods, the present invention designs a multi-scale lightweight target detection network suitable for deployment on embedded platforms, and conducts research around lightweighting and de-redundancy of target detection networks, including data preprocessing, lightweight feature extraction structure, lightweight feature fusion model structure, and target classification. The lightweight target detection network is realized by using four key technologies, which makes up for the high computational complexity of the network model of the existing target detection method and the inability to be deployed on embedded platforms and mobile platforms with limited computing resources, and is of great significance for the realization of technologies such as video surveillance, medical image processing, face recognition, remote sensing imaging, and real-time target tracking. Summary of the invention

[0004] The purpose of the present invention is to provide a method for constructing a multi-scale lightweight densely connected target detection network. The constructed target detection network solves the problems of existing target detection and classification methods, such as large number of network model parameters, high computational complexity, and difficulty in deployment on embedded devices and mobile devices with limited computing resources.

[0005] The technical solution adopted by the present invention is a method for constructing a multi-scale lightweight densely connected target detection network. The main part of the target detection network is divided into two stages. The shallow stage uses a multi-scale structure to simultaneously obtain coarse-scale and fine-scale feature information. The deep stage uses a lightweight densely connected module to achieve feature reuse and redundancy removal. In the feature fusion stage, the network is lightweighted through deep separable convolution and multi-scale input. The specific implementation is as follows:

[0006] Step 1, data preprocessing;

[0007] Step 2: construct a multi-scale lightweight object detection neural network;

[0008] Step 3: Use the results processed in step 1 to train and test the neural network constructed in step 2.

[0009] The present invention is also characterized in that:

[0010] The data preprocessing in step 1 is specifically implemented in the following steps:

[0011] Step 1.1, use 90% of the images of PASCAL VOC2012 as training + validation data sets, 10% as test data sets, 90% of the training data sets and 10% as validation data sets during training, and perform data augmentation on the training data images;

[0012] Step 1.2, perform corresponding mosaic data enhancement on the dataset;

[0013] Step 1.3, obtain the coordinates and category information of the real box of the training target according to the label information and data set information;

[0014] The multi-scale lightweight target detection neural network in step 2 is implemented in the following steps:

[0015] Step 2.1, construct a multi-scale lightweight feature extraction module MSM;

[0016] Step 2.2, construct a lightweight densely connected module FD-CDM;

[0017] Step 2.3, construct a fast downsampling backbone network;

[0018] Step 2.4, construct a lightweight feature fusion module;

[0019] Step 2.5, construct a classification prediction module;

[0020] The construction of the multi-scale lightweight feature extraction module MSM in step 2.1 is specifically implemented in the following steps:

[0021] Step 2.1.1, save the input feature tensor as three identical tensors x, y, z, which are used as the starting point of three scales;

[0022] Step 2.1.2: No operation is performed on the x scale, so that the original feature information is retained and the feature is supplemented as the residual edge as follows:

[0023] Step 2.1.3, use 3x3 depthwise convolution, BatchNorm2d, ReLU6 operation and 1x1 pointwise convolution, BatchNorm2d, ReLU6 operation to form a complete depthwise separable convolution, and use two depthwise separable convolution modules on the y scale to obtain fine-scale feature information.

[0024] Step 2.1.4: Use depth-wise separable dilated convolutions with dilation rates of 5 and 3 to form a dilated convolution group. Apply the dilated convolution group on the z scale to obtain coarse-scale feature information.

[0025] Step 2.1.5, Stack vertically, and then use the channel attention mechanism to filter the stacked information;

[0026] The lightweight dense connection module FD-CDM for feature reuse in step 2.2 is implemented in the following steps:

[0027] Step 2.2.1, construct a GC module consisting of 3x3 group convolutions, which groups different feature maps of the input layer and then uses different convolution kernels to generate intermediate features;

[0028] Step 2.2.2, construct an LGC module composed of 1x1 convolutions. This module uses 1x1 convolutions to learn channel information and filter out more important channel information;

[0029] Step 2.2.3, construct a feature reactivation SFR module, which adds an index layer to different channels of the input feature and selects the feature layer to be retained through the index layer;

[0030] Step 2.2.4, first use the LGC module and the GC module to extract and refine the input features, then save the features as two tensors s and t, the t feature tensor enters the SFR module for screening, the s tensor continues to be passed, and the results of the two branches are spliced ​​to achieve the screening of dense connections;

[0031] The construction of a lightweight backbone network for fast downsampling in step 2.3 is specifically implemented in the following steps:

[0032] Step 2.3.1, use two MSM modules to extract shallow coarse-scale feature information;

[0033] Step 2.3.2, use three FD-CDM modules to extract deep feature information and obtain sufficiently refined data to serve high-level target detection tasks;

[0034] Step 2.3.3, use the modules mentioned in 2.3.1 and 2.3.2 to form a backbone network, and use the idea of ​​fast downsampling to enhance network feature extraction;

[0035] like Figure 5 As shown, the construction of the lightweight feature fusion module in step 2.4 is specifically implemented in the following steps:

[0036] Step 2.4.1: PANet is lightweighted by replacing 90% of ordinary convolutions with depthwise separable convolutions, reducing the number of parameters in the feature fusion part to 1 / 9 of the original number.

[0037] Step 2.4.2, strengthen the input to PANet, combine the features of six scales, so that the feature fusion part obtains more sufficient feature information;

[0038] In step 2.5, the classification prediction module is constructed to classify the target according to the following steps:

[0039] Step 2.5.1, use 3x3 convolution and 1x1 convolution to build the prediction head of the network;

[0040] Step 2.5.2, for the multi-scale output of the feature fusion part, design the corresponding prediction heads respectively;

[0041] In step 3, the training and testing of the neural network constructed in step 2 using the results of step 1 are specifically implemented in the following steps:

[0042] Step 3.1, obtain the label of the training set of the dataset;

[0043] Step 3.2, obtain the labels of the training and validation sets;

[0044] Step 3.3, the network training is divided into two stages: freezing training and thawing training. The freezing training refers to freezing the backbone of the network and only training the feature fusion and prediction parts. The thawing training is to train the entire network model. The initial learning rate is set to 1e-3 in the freezing training part and adjusted to 1e-4 in the thawing training part. Stop training when the loss tends to be stable to obtain the network model.

[0045] In step 3.4, each frame of the test set image is passed into the network model, the image is recognized and detected, and the mean average precision (mAP) is calculated.

[0046] The beneficial effects of the present invention are:

[0047] In the method of the present invention, data preprocessing does not participate in model training. The backbone network of the present invention performs feature extraction, and uses the SFR (feature reactivation) module to selectively screen and remove redundancy of feature information in dense connections, greatly reducing the amount of calculation, and then further screens features through the channel attention module, so that the network can complete high-level tasks such as target detection with very few parameters.

[0048] The multi-scale lightweight densely connected target detection network model of the present invention has only 11.6M parameters, which can fully meet the target detection requirements of various mobile devices or other embedded devices. The present invention also adopts a multi-scale feature extraction design to make full use of the features of various scales in the image to enhance the generalization ability of the network for different scenes;

[0049] The reliability of the network is verified by using the PASCAL VOC 2012 data set. The final average precision mean mAP of the network is improved by 5% compared with YoloV3 and decreased by 1.5% compared with YoloV4. The network parameters are compressed to 11.61M, which is 1 / 6 of YoloV4. The total calculation amount Total Flops is reduced to 1.13G, which is 1 / 26 of the original. The total memory usage is reduced to 135.46M, which is 1 / 5 of YoloV4. The real-time speed is improved by 32 times, and the portability on mobile terminals or embedded devices with limited resources is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of mosaic data enhancement used in the method for constructing a multi-scale lightweight densely connected target detection network of the present invention;

[0051] Figure 2 It is a schematic diagram of a multi-stage lightweight backbone network structure in the method for constructing a multi-scale lightweight densely connected target detection network of the present invention;

[0052] Figure 3 It is a schematic diagram of the structure of a multi-scale feature extraction module in the method for constructing a multi-scale lightweight densely connected target detection network of the present invention;

[0053] Figure 4 It is a structural diagram of a lightweight dense connection module in the method for constructing a multi-scale lightweight densely connected target detection network of the present invention;

[0054] Figure 5 It is a lightweight feature fusion module in the method for constructing a multi-scale lightweight densely connected target detection network of the present invention;

[0055] Figure 6 It is the overall network structure of the method for constructing a multi-scale lightweight densely connected target detection network of the present invention;

[0056] Figure 7 It is an effect diagram of network detection using a network model constructed by the method for constructing a multi-scale lightweight densely connected target detection network of the present invention. DETAILED DESCRIPTION

[0057] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] The present invention is to design a multi-stage backbone network, such as Figure 6 As shown in the figure, a method for constructing a multi-scale lightweight densely connected target detection network is provided. A multi-scale module is used in the shallow layer of the backbone network to extract coarse-scale and fine-scale features at the same time, and a densely connected module is used in the deep layer of the backbone network to reuse features and remove redundancy. The specific implementation is as follows:

[0059] Step 1: perform data preprocessing to enrich the information of the data set. The present invention divides the data set into three parts: training set, training validation set, and test set, and obtains the label information corresponding to the data set before training. The specific implementation steps are as follows:

[0060] Step 1.1, use 90% of the images of PASCAL VOC2012 as training + validation data sets, and 10% as test data sets. In the training + validation data sets, 90% of the images are used as training data sets, and 10% are used as validation data sets during training. The training data images are augmented by random horizontal flipping, random rotation, random cropping, etc. For image flipping, the training set is first input into the network according to a preset batch, and then the random data set selects a size ratio for each image in the range of [0.7, 1.3] for scaling, and then all images are flipped left and right with a probability of 0.5, and then the images are adjusted in color. Finally, the size of the training images is uniformly cropped to the set size, which makes the image size and shape of the training samples richer and more random, which can avoid overfitting to a certain extent;

[0061] Step 1.2, perform mosaic data enhancement on the dataset. Mosaic data enhancement first removes a batch of data from the dataset, randomly selects 4 images from it each time, crops and splices them at random positions, and synthesizes new images. The specific process is as follows: Figure 1 As shown, repeat batch size times, and finally obtain a batch of new data of batch size images after mosaic data enhancement, and then transmit it to the neural network for training;

[0062] Step 1.3, obtain the coordinates and category information of the real box of the training target according to the label information and data set information;

[0063] Step 2: Build a lightweight target detection neural network, obtain multi-scale information through multi-scale feature modules and dense connection modules, reuse features and remove redundancy, complete feature integration through lightweight feature fusion modules, and finally use ordinary convolution to classify targets;

[0064] The backbone network includes a multi-scale feature extraction module and a lightweight dense connection module. The feature fusion part reduces parameters through multi-scale input and depth-separable convolution. The classification prediction module includes 3x3 convolution and 1x1 convolution.

[0065] Step 2 includes building a multi-scale module, a lightweight dense connection module, a fast downsampling backbone network, a lightweight feature fusion module, and a classification prediction module. The specific implementation steps are as follows:

[0066] Step 2.1, the multi-scale feature extraction module MSM is implemented in the following steps:

[0067] Step 2.1.1, save the input feature tensor as three identical tensors x, y, z, which are used as the starting point of three scales;

[0068] Step 2.1.2, use the x scale to retain the original feature information and use it as the residual edge to supplement the feature expression as

[0069] Step 2.1.3, use 3x3 depthwise convolution, BatchNorm2d, ReLU6 operation and 1x1 pointwise convolution, BatchNorm2d, ReLU6 operation to form a complete depthwise separable convolution; use two depthwise separable convolution modules on the y scale to obtain fine-scale feature information

[0070] Step 2.1.4: Use depth-wise separable dilated convolutions with dilation rates of 5 and 3 to form a dilated convolution group. Apply the dilated convolution group on the z scale to obtain coarse-scale feature information.

[0071] Step 2.1.5, The structure of the multi-scale lightweight feature extraction module MSM is as follows: Figure 3 As shown;

[0072] The multi-scale extraction module is used to improve the network's shallow feature extraction capability. In the multi-scale feature extraction module, we use three different scales; the first scale retains the original feature information; the second scale uses two groups of depthwise separable convolutions for fine-scale feature extraction; the third scale uses a group of depthwise separable dilated convolutions with dilation rates of 5 and 3 for coarse-scale feature extraction; after completing the feature extraction of the three scales, they are stacked and merged to form features with global spatial information, and the channel attention module is used to integrate global information, effectively screening features while eliminating redundant features with low relevance.

[0073] like Figure 4 As shown, the lightweight dense connection module in step 2.2 is implemented in the following steps:

[0074] Step 2.2.1, construct a GC (Group Convolution) module consisting of 3x3 group convolutions, which groups different feature maps of the input layer and then uses different convolution kernels to generate intermediate features;

[0075] Step 2.2.2: construct a LGC (Learning Group Convolution) module consisting of 3x3 convolutions, and then use 1x1 convolutions to learn channel information and filter out more important channel information.

[0076] Step 2.2.3, construct a feature reactivation SFR module, which adds an index layer to different channels of the input feature and selects the feature layer to be retained through the index layer;

[0077] Step 2.2.4, use the LGC module and the GC module to extract the input features, and save the features as two tensors, s and t. The t tensor enters the feature reactivation SFR module for screening, and the s tensor continues to be passed backward. Then the results of the two branches are spliced ​​to achieve dense connection screening;

[0078] In order to make the network lightweight and efficient, the present invention believes that it is not necessary to reactivate all shallow features during the network learning process. The lightweight dense connection network uses a feature reactivation module SFR, which learns the input features and screens out the features that were considered outdated in the early stage of the network for reactivation, thereby making up for the defect of inefficient features caused by long-distance connections in dense connections. In addition, the SFR module in the lightweight dense connection module can be converted into an effective group convolution during testing, and will not cause too much delay cost. Lightweight dense connections can not only reduce computing costs and computing delays, but also remove redundant features in dense connections, making the network more lightweight;

[0079] like Figure 2As shown, step 2.3 for constructing the backbone network comprises the modules mentioned in 2.1.1 and 2.1.2 to form the backbone network, and the specific implementation steps are as follows:

[0080] Step 2.3.1, use two MSM modules to extract shallow coarse-scale feature information;

[0081] Step 2.3.2, use three FD-CDM modules to extract deep feature information;

[0082] Step 2.3.3, use the modules mentioned in 2.3.1 and 2.3.2 to form a backbone network, and use the idea of ​​fast downsampling to change the dimension;

[0083] Step 2.4 builds a lightweight feature fusion module and is implemented in the following steps:

[0084] Step 2.4.1: PANet is lightweighted and the number of parameters is reduced to 1 / 9 of the original through depthwise separable convolution.

[0085] Step 2.4.2, input six scale information into the feature fusion module;

[0086] The present invention obtains shallow features through multi-scale structure and channel attention mechanism; in order to make the network approximation effect better, the present invention uses a lightweight dense connection structure to abstract and filter features in the deep stage of the feature extraction network;

[0087] The prediction in step 2.5 is implemented as follows:

[0088] Step 2.5.1, use 3x3 convolution and 1x1 convolution to build the prediction head of the network;

[0089] Step 2.5.2: For the multi-scale output of the feature fusion part, use the prediction head with the corresponding number of channels respectively;

[0090] The specific process of network training and testing in step 3 is to input the training data into the network to train the network model, input the test image into the network to obtain the target detection result, and calculate the mAP of the network under the current data set:

[0091] Step 3.1, obtain the label of the training set of the dataset;

[0092] Step 3.2, obtain the labels of the training and validation sets;

[0093] Step 3.3, the network training is divided into two stages: frozen training and unfrozen training. The frozen training only trains the feature fusion and prediction parts, while the unfrozen training trains the entire network model. The initial learning rate is set to 1e-3 during the frozen training, and the initial learning rate is adjusted to 1e-4 during the unfrozen training. When the loss tends to be stable, the training is stopped to obtain the network model.

[0094] Step 3.4: pass each frame of the test set image into the network model, identify and detect the image, and calculate the mean average precision (mAP);

[0095] When the input size is 416x416, the model parameter volume is 11.6M and the model size is 40M, which can meet the real-time requirements. In terms of prediction accuracy, the mAP of the target detection network proposed in the present invention has surpassed most lightweight target detection networks, and the difference with the YoloV4 network is within 2%. The present invention takes into account both accuracy and speed, and can achieve high-precision real-time target detection, meeting the conditions for deployment on embedded devices. The detection effect is as follows: Figure 7 As shown, it can be seen that the target detection network proposed in the present invention has a good recognition rate and high accuracy for targets in images; the resource occupancy comparison between the present invention and other target detection networks is shown in Table 1.

[0096] Table 1. Comparison of network resource usage

[0097]

[0098] It can be seen from Table 1 that the memory resources occupied by the present invention are 5 times smaller than those of the YoloV4 network, the total computational complexity is 26 times smaller than that of YoloV4, and the real-time inference speed on CPUi5-9300H is 32 times that of YoloV4. The performance comparison with other target detection networks is shown in Table 2.

[0099] Table 2 Overall network performance parameters

[0100]

[0101]

[0102] It can be seen from Table 2 that the accuracy of the present invention exceeds that of the Yolov3 series and other lightweight Yolov4 series. On the VOC dataset, when the intersection-over-union ratio is 0.5, the average accuracy is only 1.5% lower than that of Yolov4.

Claims

1. A method for constructing a multi-scale lightweight densely connected target detection network, characterized in that: Please follow the steps below to implement: Step 1, data preprocessing; Step 2: construct a multi-scale lightweight object detection neural network; The multi-scale lightweight target detection neural network in step 2 is specifically implemented according to the following steps: Step 2.1, construct a multi-scale lightweight feature extraction module MSM; The construction of the multi-scale lightweight feature extraction module MSM in step 2.1 is specifically implemented by the following steps: Step 2.1.1, save the input feature tensor as three identical tensors x, y, z, which are used as the starting point of three scales; Step 2.1.2: No operation is performed on the x scale, so that the original feature information is retained and the feature is supplemented as the residual edge as follows: ; Step 2.1.3, use 3x3 depthwise convolution, BatchNorm2d, ReLU6 operation and 1x1 pointwise convolution, BatchNorm2d, ReLU6 operation to form a complete depthwise separable convolution, and use two depthwise separable convolution modules on the y scale to obtain fine-scale feature information. ; Step 2.1.4: Use depth-wise separable dilated convolutions with dilation rates of 5 and 3 to form a dilated convolution group. Apply the dilated convolution group on the z scale to obtain coarse-scale feature information. ; Step 2.1.5, Stack vertically, and then use the channel attention mechanism to filter the stacked information; Step 2.2, construct a lightweight densely connected module FD-CDM; The lightweight dense connection module FD-CDM for feature reuse in step 2.2 is specifically implemented in the following steps: Step 2.2.1, construct a GC module consisting of 3x3 group convolutions, which groups different feature maps of the input layer and then uses different convolution kernels to generate intermediate features; Step 2.2.2, construct an LGC module composed of 1x1 convolutions. This module uses 1x1 convolutions to learn channel information and filter out more important channel information; Step 2.2.3, construct a feature reactivation SFR module, which adds an index layer to different channels of the input feature and selects the feature layer to be retained through the index layer; Step 2.2.4, first use the LGC module and the GC module to extract and refine the input features, then save the features as two tensors s and t, the t feature tensor enters the SFR module for screening, the s tensor continues to be passed, and the results of the two branches are spliced ​​to achieve the screening of dense connections; Step 2.3, construct a fast downsampling backbone network; The construction of a lightweight backbone network for fast downsampling in step 2.3 is specifically implemented in the following steps: Step 2.3.1, use two MSM modules to extract shallow coarse-scale feature information; Step 2.3.2, use three FD-CDM modules to extract deep feature information and obtain sufficiently refined data to serve high-level target detection tasks; Step 2.3.3, use the modules mentioned in 2.3.1 and 2.3.2 to form a backbone network, and use the idea of ​​fast downsampling to enhance network feature extraction; Step 2.4, construct a lightweight feature fusion module; The construction of the lightweight feature fusion module in step 2.4 is specifically implemented in the following steps: Step 2.4.1: PANet is lightweighted by replacing 90% of ordinary convolutions with depthwise separable convolutions, reducing the number of parameters in the feature fusion part to 1 / 9 of the original number. Step 2.4.2, strengthen the input to PANet, combine the features of six scales, so that the feature fusion part obtains more sufficient feature information; Step 2.5, construct a classification prediction module; The classification prediction module constructed in step 2.5 is specifically implemented by the following steps to classify the target: Step 2.5.1, use 3x3 convolution and 1x1 convolution to build the prediction head of the network; Step 2.5.2, for the multi-scale output of the feature fusion part, design the corresponding prediction heads respectively; Step 3, using the results processed in step 1 to train and test the neural network constructed in step 2; Please follow the steps below to implement: Step 3.1, obtain the label of the training set of the dataset; Step 3.2, obtain the labels of the training and validation sets; Step 3.3, the network training is divided into two stages: freezing training and thawing training. The freezing training refers to freezing the backbone of the network and only training the feature fusion and prediction parts. The thawing training is to train the entire network model. The initial learning rate is set to 1e-3 in the freezing training part and adjusted to 1e-4 in the thawing training part. Stop training when the loss tends to be stable to obtain the network model. In step 3.4, each frame of the test set image is passed into the network model, the image is recognized and detected, and the mean average precision (mAP) is calculated.

2. The method for constructing a multi-scale lightweight densely connected target detection network according to claim 1, characterized in that: The data preprocessing in step 1 is specifically implemented according to the following steps: Step 1.1, use 90% of the images of PASCAL VOC2012 as training + validation data sets, 10% as test data sets, 90% of the training data sets and 10% as validation data sets during training, and perform data expansion on the training data images; Step 1.2, perform corresponding mosaic data enhancement on the dataset; Step 1.3, obtain the coordinates and category information of the real box of the training target based on the label information and dataset information.

Images