Target detection method based on lightweight multi-scale feature fusion
By designing a lightweight multi-scale feature fusion module in the object detection algorithm, using partial convolution and multi-scale pooling kernel to extract features, and through the target space attention module, the problem of feature information loss and redundancy in calculation amount is solved, and the problem of feature information is achieved. High precision and high speed target detection.
Patent Information
- Application Number
- CN202410936613.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-26
AI Technical Summary
Existing object detection algorithms are prone to losing information when extracting features, resulting in a decrease in detection accuracy. The calculation amount increases when traditional methods improve detection accuracy, especially when deploying edge devices.
A lightweight multi-scale feature fusion module (LMFFM) is designed to reduce the number of channels participating in convolution operations through partial convolution (PConv). Multi-scale pooling kernels and convolution kernels are superimposed to extract features, and pay attention to the target space. The force module improves the efficiency of information extraction.
It effectively reduces the loss of target information, improves the target detection accuracy, and makes the model lighter, reduces the amount of calculation and parameters, and improves the detection speed.
Smart Images

Figure CN119027642A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning and computer vision, and in particular to a target detection algorithm based on lightweight multi-scale feature fusion. Background Art
[0002] Since the information of the target is lost when extracting features, its positioning and recognition remains a challenging problem. First, because the size of the target is small, it is easy to be occluded or lost in the image, and it is difficult to be accurately recognized by the machine learning model. Secondly, the targets usually have a high degree of spatial relationship, that is, the relative position relationship between them may have an important impact on the recognition results, so how to make full use of this relationship is an important issue that needs attention. The targets in the image usually have weak features, so it is difficult to identify them using traditional target recognition methods. In the YOLO series, in order to improve the detection accuracy of the target, the resolution of the image is usually increased, but this method usually increases the amount of calculation, especially when deploying edge devices. At the same time, when the backbone network is used to extract features, more target information will be lost, resulting in a decrease in the accuracy of target detection.
[0003] Tao Kong et al. proposed the HyperNet algorithm. The core idea of the HyperNet algorithm is to fuse information at different feature levels through a multi-branch network to improve the accuracy of target detection. Specifically, the HyperNet algorithm first extracts shallow features and deep features through the basic network. Then, the feature information at different levels is fused to obtain a more accurate fusion network feature representation. The fusion network adopts a multi-branch design, and each branch fuses different feature levels. Finally, the target is detected and located through the classification network and regression network. Lin et al. proposed a feature pyramid network structure. Through a top-down feature extraction network and a bottom-up feature fusion network, feature pyramids of different sizes can be obtained. In this process, the high-level feature map is upsampled to obtain the same scale as the low-level feature map, and then fused with the low-level feature map to obtain a feature map with more semantic information. Finally, the target is detected and located through the detection network. Whether it is to increase the image resolution or to use multi-scale feature extraction, it will bring a certain amount of calculation, which is not very friendly to the deployment of the model.
[0004] In view of the problem that the current model is difficult to extract effective target information and the amount of computation is redundant, the LMFFM (Lightweight Multi-scale Feature Fusion Module) proposed in this invention uses the redundancy of input channels during feature extraction to reduce the number of channels involved in the convolution operation, thereby making the model lightweight. At the same time, in order for the model to effectively extract target information, a multi-scale feature extraction is designed, which improves the model's extraction of target information by superimposing convolution kernels with different pooling kernels and different receptive fields, and improves the efficiency of information extraction by a specific target space attention module. The designed module reduces the loss of target information, improves the accuracy of target detection, and also makes the model lightweight. Summary of the invention
[0005] The purpose of the present invention is to design a lightweight multi-scale fusion target detection method that can detect targets in images with high detection accuracy and speed.
[0006] The present invention provides a target detection network based on lightweight multi-scale feature fusion, which mainly includes a backbone network for extracting features of an input image for subsequent network training; a LMFFM module for processing the features extracted by the backbone network so that the network can learn more target information; and a detection head for outputting the final detection result of the model. The lightweight principle and LMFFM construction process are as follows:
[0007] In order to design fast neural networks, the number of floating point operations (FLOPs) is reduced. However, a reduction in FLOPs does not necessarily lead to a similar reduction in latency. This is mainly due to the inefficiency of floating point operations per second (FLOPS). The relationship between Latency, FLOPS, and FLOPs is as follows:
[0008]
[0009] From formula (1), we can know that to improve the speed of the model, we need to consider both FLOPs and FLOPS. Further research found that reducing FLOPS is related to the frequency of model access to memory. Reducing the memory access of the model can effectively reduce FLOPS. Assume that the input I∈R c×h×w ,W∈R k×k , the memory access amount of conventional convolution is calculated as follows:
[0010] h×w×2c+k 2 ×c 2 ≈h×w×2c (2)
[0011] In partial convolution (PConv), only some channels are used for convolution feature extraction, and the remaining channels are not calculated. Assume that the number of channels for convolution is c p , then the memory access amount of the partial convolution is:
[0012]
[0013] If part of The memory access amount of partial convolution is At the same time, the FLOPs of the partial convolution are:
[0014]
[0015] Formula (4) shows that when When the FLOPs of partial convolution is only 1 / 2 of that of regular convolution Therefore, partial convolution reduces both FLOPS and FLOPs, improving the inference speed of the model. The construction of lightweight multi-scale feature fusion module LMFFM includes the following steps:
[0016] Step 1: The input features are first subjected to a partial convolution PConv (Partial Convolution). Partial convolution is to perform convolution operation on only a part of the input feature channels, and the remaining input channels are spliced later. In PConv, the channel is split into two parts using the Split method, one of which is input into the MSPM module, and the other is input into the MSSF module to further extract multi-scale information;
[0017] Step 2: The output features of the MSSF module and the MSPM module are concatted and input into a 1×1 convolution with a step size of 1, and then pass through the BatchNorm layer, and finally pass through the ReLU activation function to obtain the processed feature map. The processed feature map is then subjected to a 1×1 convolution.
[0018] Step 3: Add the input channel in step 1 and the features after multi-scale fusion by the MSSF module and the MSPM module in step 2 to obtain the final refined target information features.
[0019] The multi-scale feature fusion module is mainly composed of three branches. The first branch is designed based on the residual network. It directly inputs the input channel into the subsequent concat to enhance the feature information of the target. The second branch uses multiple Maxpool and Avgpool cascade operations, which is equivalent to using different pooling kernels to extract target features of different scales without introducing parameters. The third branch uses two cascaded convolutions to obtain multi-scale features, and then performs spatial attention on the multi-scale features. The construction of the multi-scale feature extraction module includes the following steps:
[0020] Step 1: First, the input features in the lightweight module are divided into three branches. The first branch is used for residual connection to connect the input features C×H×W to the subsequent addition; Split is used to divide the input channel into two parts, one part is input into a cascaded Maxpool and Avgpool structure; the other part is input into a cascaded 3×3 convolution kernel;
[0021] Step 2: Input the feature channel into the cascaded Maxpool and cascaded Avgpool structures. First, a 1×1 convolution with a step size of 1 is required to reduce the dimension of the input channel. Then input the output into a Maxpool and Avgpool respectively. In this way, three pooling kernels of 5×5 are cascaded. The step size calculation formula of each pooling kernel is k / 2. There are three feature maps of different scales. The feature maps of the same scale obtained by Maxpool and Avgpool are added, and then the input C×H×W and the outputs of the three addition operations are concat. In order to make the input channel and output channel the same, the output needs to be convolved with a 1×1 convolution, and then the refined target feature map is obtained through BatchNorm and ReLU activation functions;
[0022] Step 3: Construct multi-scale information of convolution stacking, and input the feature channel into a cascaded convolution operation. Two 3×3 convolutions with a step size of 1 are used for cascading. On the one hand, multi-scale information can be obtained. On the other hand, through the knowledge of convolution kernel stacking, it can be known that the receptive field of two 3×3 convolution cascades is equivalent to a 5×5 convolution. Therefore, this design not only obtains multi-scale information but also constructs a large receptive field, which is conducive to the construction of long-distance relationships in images. The features Feature1 and Feature2 of the two branches are added together and then subjected to a 7×7 convolution for contextual interaction, and then input into the Sigmoid activation function to obtain the spatial attention matrix;
[0023] Step 4: Multiply the spatial attention matrix obtained in step 3 with features Feature1 and Feature2, then add the two multiplied features to obtain the fused multi-scale features, and multiply the fused multi-scale features with the input Input to obtain the output Output;
[0024] Step 5: Concat the multi-scale information in step 3 and step 4, and then pass it through an MLP layer consisting of two 1×1 convolutions and BN and ReLU to obtain the final multi-scale feature information.
[0025] Step 6: Add the multi-scale information obtained in step 5 and the input C×H×W to obtain the final multi-scale features.
[0026] The present invention is mainly divided into the following steps:
[0027] 1. Obtain relevant annotated data sets and convert the label format of the data sets into the YOLO label format;
[0028] 2. Use the YOLOv8 series model to build a target detection network based on lightweight multi-scale feature fusion. It mainly includes: backbone network, lightweight network, multi-scale feature fusion module (LMFFM) and target detection module;
[0029] 3. The constructed target detection model is trained on the prepared YOLO format data set, and the trained model weight file is saved;
[0030] 4. Call the saved model weight file for the validation set in the dataset to test the performance of the model detection target.
[0031] The lightweight multi-scale feature fusion target detection network proposed in the present invention is improved for the YOLOv8 detection model, and a newly designed LMFFM module is added to the C2f module. The number of input channels involved in the convolution operation is reduced, so that the number of parameters and the amount of calculation of the model are reduced. The core modules of the lightweight and multi-scale feature fusion LMFFM are the MSSF module and the MSPM module, which will enter the detection head for detection after the fusion is completed.
[0032] Due to the adoption of the above technical solution, the present invention has the following advantages:
[0033] 1. A lightweight module is used to reduce the amount of model calculation and improve the model's reasoning speed.
[0034] 2. The present invention designs a multi-scale feature fusion module. The serious problem of target information loss is solved by extracting multi-scale feature information. The target information is fully retained by extracting feature information through different receptive fields. At the same time, the spatial attention designed for the target is added to more effectively fuse multi-scale features, so that the network can fully learn the target's texture, edge and other detailed information. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:
[0036] Figure 1 It is a summary figure of the detection network based on lightweight multi-scale feature fusion of the present invention.
[0037] Figure 2 It is a schematic diagram of the structure of the lightweight multi-scale feature fusion module (LMFFM) of the present invention.
[0038] Figure 3 It is a schematic diagram of the structure of the LMFFM submodule MSPM (Multi-scale Pooling Module) of the present invention.
[0039] Figure 4 It is a schematic diagram of the structure of the LMFFM submodule MSSF (Multi-scale Spatial Fusion) of the present invention.
[0040] Figure 5 It is a schematic diagram of the C2f_LMFFM structure of the present invention.
[0041] Figure 6 It is a detection network diagram based on lightweight multi-scale feature fusion of the present invention. Specific implementation plan
[0042] The present invention is further described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention. The present invention proposes a target detection method based on lightweight and multi-scale feature fusion, and the specific implementation includes the following contents:
[0043] Step 1. Obtain a target image dataset of a certain scene and convert it into a dataset in YOLO format (for example, the VisDrone target dataset from the perspective of drone aerial photography).
[0044] Step 2. Use YOLOv8s Backbone as the backbone network, YOLOv8 PANet network as the neck, and YOLOv8's 20×20×512, 40×40×512, 80×80×256 resolution decoupling heads as the detection head. Improve the C2f module for feature extraction in the entire network, improve the Bottleneck module in C2f, and add the lightweight multi-scale feature fusion module LMFFM of the present invention.
[0045] Step 3. Construct a lightweight multi-scale feature (LMFFM) module.
[0046] In the attached Figure 2 In the LMFFM module designed for the present invention, firstly, split can be used to divide the input features C×H×W according to the ratio. Because it can be known from channel visualization that there is a high degree of information redundancy in the input channel, convolving all channels by conventional convolution will cause a waste of computing resources. Therefore, in the present invention, the input channel C is divided into two parts, and the specific allocation ratio can be adjusted according to the ratio factor n_div. The value of n_div in the target data set is 2, so the number of channels input to MSSF and MSPM is C / 2. Then the channels of the two parts are respectively input into the MSSF and MSPM modules of the present invention. Then the channels of the two parts are concat out and then pass through two 1×1 convolution Conv2d and the corresponding BN and ReLU layers, and finally added to the input Input to obtain the final feature map.
[0047] Step 3-1. The construction of the MSPM part in LMFFM includes the following:
[0048] The construction of MSPM module is based on the attached Figure 3 It can be seen that first, a 1×1 convolution is performed on the input to reduce the dimension, and the output features after dimension reduction are input into a module consisting of three Maxpool cascades and three Avgpool cascades. The size of the pooling kernel is 5, and different padding sizes are set to be equivalent to three different pooling kernels. Figure 2, after passing through the first Maxpool and Avgpool of size 5×5, the two outputs are added, and the padding calculation formula is k / 2. After passing through the second Maxpool and Avgpool, the two outputs are added, and the size can be equivalent to 9×9. Finally, after passing through a Maxpool and Avgpool of size equivalent to 13×13, the two outputs are added. In order to obtain multi-scale feature information, the output after each cascade of Maxpool and Avgpool is concat with the original input C / 2×H×W, and finally feature maps of different scales of 5×5, 9×9 and 13×13 can be obtained. Finally, a 1×1 convolution is performed to restore the original channel to obtain the final output feature map C×H×W.
[0049] Step 3-2. The construction of the MSSF part in LMFFM includes the following:
[0050] Combined with Figure 4 First, the input feature Input is input into two cascaded 3×3 convolutions to obtain features Feature1 and Feature2 of different scales. The two features are added and then input into a spatial attention designed for the target. Specifically, it first passes through a Maxpool and then a 7×7 convolution for context interaction. Finally, a Sigmoid activation function is used to obtain the spatial attention matrix. The obtained attention matrix is multiplied with the two branch features Feature1 and Feature2 respectively, and then added to obtain the fused feature. The fused feature is multiplied with the input Input to obtain the final output Output.
[0051] Step 4. Add the MSPM and MSSF constructed in step 3-1 and step 3-2 to LMFFM. LMFFM is the lightweight and multi-scale feature fusion module proposed in the present invention.
[0052] Step 5. Improve the C2f module in YOLOv8s. The specific construction includes the following contents:
[0053] C2f is a new module proposed by the YOLOv8 series. It is lightweight and can effectively extract feature information. In order to further improve the model's detection performance on the target and make the model lightweight, the Bottleneck module in C2f is replaced by the LMFFM module of the present invention. Figure 5 The newly designed module is named C2f LMFFM.
[0054] Step 6. Attach Figure 6 In order to finally build a good detection network, the constructed target detection network based on lightweight multi-scale feature fusion is trained multiple times on the target training set, and the parameters are adjusted to obtain the optimal detection model.
Claims
1. A target detection method based on lightweight multi-scale features, characterized in that: Using the YOLOv8 network model as the basic network, the network is optimized according to the characteristics of the target image. The target detection network mainly includes the following steps: Step 1: Obtain relevant scene target datasets and convert them into YOLO format datasets; Step 2: Optimize the YOLOv8 model, replace the C2f module with the optimized C2f_LMFFM module, and extract multi-scale features from shallow to deep; Step 3: Construct a lightweight multi-scale feature fusion module C2f_LMFFM. C2f_LMFFM is divided into two sub-modules, a multi-scale pooling module MSPM and a multi-scale spatial fusion module MSSF. The MSPM module is composed of the addition of a multi-scale Maxpool and a multi-scale Avgpool. The MSSF is composed of a specific multi-scale spatial attention module. The input channel is divided into two parts, one part is input into MSPM, and the other part is input into MSSF. Multi-scale features are extracted through the multi-scale feature extraction module, and multi-scale information is effectively fused through spatial attention. Finally, the two branches are spliced together through a concat, and the spliced feature vector is input into BN and ReLU, passed through the convolution layer Conv2d, and then input into BN and ReLU, and then input into BN and ReLU through Conv2d and added to the input to obtain the output; Step 4: Train the constructed lightweight multi-scale fusion-based object detection network on the corresponding training set for several rounds, and adjust the parameters to obtain the optimal model; Step 5: Get the trained model weight file for testing the corresponding dataset.
2. The target detection method based on lightweight multi-scale features according to claim 1, characterized in that: The C2f_LMFFM module uses two submodules, MSSF and MSPM, to extract multi-scale information. Constructing the MSSF module includes the following steps: Step 1: Input the input feature Input into two cascaded 3×3 convolutions to obtain multi-scale features Feature1 and Feature2; Step 2: Add the multi-scale features in step 1 to obtain the fused information Feature_sum, input the fused information into Maxpool, and then input it into a 7×7 convolution for context interaction. After a Sigmoid activation function, the spatial attention matrix is obtained. The spatial attention matrix is multiplied by the two branch features Feature1 and Feature2, and then added to obtain the final multi-scale features; Step 3: Multiply the multi-scale features in step 2 with the input Input to get the final output Output.
3. The target detection method based on lightweight multi-scale features according to claim 2, characterized in that: The C2f_LMFFM module uses two submodules, MSSF and MSPM, to extract multi-scale information and construct the MSPM module, including the following steps: Step 1: The input feature is first subjected to a 1×1 convolution, and then processed through BatchNorm and ReLU to obtain the processed feature map; Step 2: Input the feature maps obtained in step 1 into a module consisting of 3 Maxpool cascades and 3 Avgpool cascades respectively. The padding of each Maxpool and Avgpool is equal to k / 2, k is the pooling kernel size, and both are 5. Add the feature maps of the same scale and concat them with the output in step 1 to obtain the multi-scale features of the target; Step 3: Input the concat feature information in step 2 into a 1×1 convolution, and then pass through BatchNorm and ReLU to obtain the final output feature map of the MSPM submodule.
Citation Information
Patent Citations
Lightweight multi-scale remote sensing image rotating target detection method and system
CN116030360A
Road target detection method and system based on improved YOLOv8
CN117037119A
Lightweight flame detection method fused with BiFPN
CN117409358A
Improved YOLOv8 unmanned aerial vehicle aerial target detection method
CN117557922A
Method and System For Accelerating Rapid Class Augmentation for Object Detection in Deep Neural Networks
US20230010033A1
Cited By
Multidimensional information parallel targeting extraction method based on multiphase flow droplet / bubble spatial characteristics
CN120126115A
A multi-dimensional information parallel targeting extraction method based on spatial characteristics of droplets / bubbles of multiphase flow
CN120126115B
