A feature extraction method based on multi-scale lightweight backbone network
By introducing a lightweight inverse bottleneck residual module based on channel recombination into the YOLOv5 backbone backbone network, a multi-scale lightweight backbone feature extraction network MSL_backbone is built, which solves the problem of difficult to improve the target detection accuracy in road scenarios, and achieves better extraction of multi-scale features, with a detection accuracy of 0.35%.
Patent Information
- Application Number
- CN202310492350.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2025-04-11
- Estimated Expiration
- 2043-05-05
AI Technical Summary
In the prior art, when detecting targets in road scenarios, there are problems such as poor robustness, large calculation amount and poor real-time performance, especially the poor detection effect of small-scale targets, which makes it difficult to improve the overall detection accuracy.
The method based on multi-scale lightweight backbone feature extraction network MSL_backbone is adopted. By improving the YOLOv5 backbone feature extraction network, the lightweight reverse bottleneck residual module shuffle_res2net_s based on channel recombination is introduced to enhance the network's ability to extract multi-scale features.
The ability to extract multi-scale features has been improved, and the detection accuracy has been improved by 0.35% compared with the original method model, which can better deal with the difficulties in multi-scale object detection in road scenarios.
Smart Images
Figure CN116563558B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of road scene target detection, and in particular to a target detection method based on a multi-scale lightweight backbone feature extraction network. Background Art
[0002] Road scene object detection refers to the use of computer vision technology to identify and extract various traffic signs, vehicles, pedestrians and other targets in road scenes. This technology is widely used in the fields of intelligent transportation and autonomous driving. When performing road scene object detection, it is necessary to perform background analysis on the image, that is, to separate the background and foreground information from the image in order to more accurately identify the foreground target.
[0003] Traditional target detection methods are not used in road scenes because of their poor robustness, large amount of calculation, low real-time performance and other problems. Although several popular target detection methods can achieve relatively good accuracy when applied to road scenes, a careful analysis of the detection results shows that the detection effect of small-scale targets is poor, resulting in the inability to further improve the overall detection accuracy.
[0004] In addition, there are problems with the inconsistency of the size and type of targets in road scenes. For example, the scale of cars is relatively large, while the scale of pedestrians and bicycles is relatively small. In addition to the different scales of the targets themselves, the scale of the target will also change when the distance of the target from the camera is different. Therefore, in the same road scene, the same type of target is at different distances from the camera, which will also lead to different scales and large differences in features. This scale inconsistency problem poses a higher challenge to the performance of the target detection method. In addition, by observing the pictures collected by the camera, it can be found that in the same scene, the frequency of vehicle-type targets is very high, while the frequency of pedestrians and other types of targets is much lower. This problem of unbalanced number of target categories in road scenes will also affect the accuracy of the target detection method. Summary of the invention
[0005] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: to improve the ability of the method model to extract multi-scale features.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solution: a feature extraction method based on a multi-scale lightweight backbone network, comprising the following steps:
[0007] S1: Obtain an image and preprocess it. A preprocessed image is a training sample.
[0008] S2: Construct the MSL_backbone model, which uses the backbone feature extraction network of YOLOv5 as the basic model, and replaces the residual structure in CSP_BottleNeck in the backbone feature extraction network of YOLOv5 with the lightweight inverse bottleneck residual module shuffle_res2net_s based on channel reorganization;
[0009] S3: Input the training samples into the MSL_backbone model to train it, and save the model parameters after optimal convergence to obtain the optimal MSL_backbone;
[0010] S4: Call the optimal MSL_backbone, input the test data to get the result and output it by extracting the last feature map of the network.
[0011] Preferably, the preprocessing in S1 refers to: resizing the image to 32×32, and then randomly flipping it to obtain multiple images, and then performing Tensor conversion on the multiple images to change the grayscale range from 0-255 to 0-1, and finally performing a normalization operation on the image.
[0012] Preferably, the lightweight inverse bottleneck residual module based on channel reorganization in S2 includes:
[0013] The input feature map of shuffle_res2net_s is X∈R C×H×W , X represents the input feature map, C is the input channel, H is the feature map height, and W is the feature map width;
[0014] After the convolution with a convolution kernel size of 1×1, the channel dimension is increased, and the channel dimension becomes C′=C×W / 64×S, and the feature map is obtained as X′∈R C′×H×W , where X′ represents the feature map obtained after the convolution operation, and C′ represents the changed channel dimension;
[0015] X′ is segmented at the channel level C′ into n feature subgraphs with equal channel sizes, and X i Represents, where i∈(1,2,3...n), X i The size of the space is consistent with that of X′, and the number of channels C i is 1 / n of X′;
[0016] The last branch is not filtered by the 3×3 group convolution kernel, and the remaining three branches are filtered by the 3×3 group convolution. The input of each branch after the first branch contains the output of the previous branch.
[0017] In the branch containing grouped convolution, after the input feature subgraph undergoes grouped convolution, the extracted feature subgraph is reorganized in the channel dimension;
[0018] The outputs of the four branches are concatenated at the channel level, and a convolution with a kernel size of 1×1 is used to perform feature fusion and dimensionality reduction, and a skip connection is used to perform element-wise addition with the input.
[0019]
[0020]
[0021] Among them, GCONV3(.) represents the grouped convolution operation with a convolution kernel size of 3×3; shuffle represents the channel reorganization operation; Conv 1×1 represents a convolution operation with a convolution kernel size of 1×1; X i Represents the input of each branch; y i represents the output of each branch; Y represents the final output of the module.
[0022] Preferably, the channel reorganization operation specifically includes:
[0023] The extracted feature sub-graph is transformed into i Divide the above into g groups of equal size;
[0024] Reshape the g groups into a matrix with g rows and h columns, where h = C i / g;
[0025] The obtained matrix is first transposed (Transpose) and then flattened into one dimension by Fallten operation.
[0026] Compared with the prior art, the present invention has at least the following advantages:
[0027] The present invention improves the YOLOv5 backbone feature extraction network backbone as the basic framework (baseline), proposes a lightweight inverse bottleneck residual structure shuffle_res2net_s based on channel reorganization, integrates it with the YOLOv5 backbone network, and proposes a multi-scale lightweight backbone feature extraction network MSL_backbone. The experimental verification results show that the method of the present invention improves the ability to extract multi-scale features, with an accuracy improvement of 0.35% compared with the original method model. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 The figure is a flow chart of the method.
[0029] Figure 2This is the structure diagram of the lightweight inverse bottleneck residual module based on channel reorganization, shuffle_res2net_s structure s=4.
[0030] Figure 3 This is the difference between ordinary convolution and grouped convolution.
[0031] Figure 4 Schematic diagram of channel reorganization process.
[0032] Figure 5 This is the structure diagram of CSP_Bottle2Neck (with jump connection).
[0033] Figure 6 The figure shows the fusion strategy approach.
[0034] Figure 7 Schematic diagram of CIFAR-10 categories.
[0035] Figure 8 This is a graph showing the heat map visualization results. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0037] In the object detection method, a particularly important part of the structure is the backbone network used for feature extraction, which is the backbone of CNN. The feature extraction backbone network first generates feature maps by extracting features layer by layer from the input image. These feature maps are then fused through the Neck network and finally handed over to the detection head for regression detection.
[0038] Although the Neck network can directly fuse (join) the features extracted from different layers of the backbone network to achieve the effect of multi-scale information fusion, the backbone feature network may have design problems, which may cause the extracted features to contain missing target information in the image, so that direct fusion does not fully utilize the advantages of multi-scale information fusion.
[0039] Therefore, the quality of the feature extraction backbone network design determines the accuracy of the final target detection method to a large extent. In the backbone network, the deeper the layer, the more abstract the image features extracted (the semantic information of the object), and conversely, it mainly contains shallow information (color, edge, etc.). Due to the small size of the small target, after multi-layer feature extraction, the information of the final target may be lost, resulting in the low detection accuracy of the final target detection method (especially the small target detection accuracy). The present invention mainly improves the feature extraction backbone network of the YOLOv5 target detection method, increases its multi-scale feature extraction capability, and enables it to better cope with the multi-scale target detection difficulties in road scenes.
[0040] See also Figure 1 , a feature extraction method based on a multi-scale lightweight backbone network, comprising the following steps:
[0041] S1: Obtain an image and preprocess it. A preprocessed image is a training sample.
[0042] S2: Construct the MSL_backbone model, which uses the backbone feature extraction network of YOLOv5 as the basic model, and replaces the residual structure in CSP_BottleNeck in the backbone feature extraction network of YOLOv5 with the lightweight inverse bottleneck residual module shuffle_res2net_s based on channel reorganization;
[0043] S3: Input the training samples into the MSL_backbone model to train it, and save the model parameters after optimal convergence to obtain the optimal MSL_backbone;
[0044] S4: Call the optimal MSL_backbone, input the test data to get the result and output it by extracting the last feature map of the network (before the global mean pooling layer).
[0045] Specifically, the preprocessing in S1 refers to: resizing the image to 32×32, then randomly flipping it to obtain multiple images, then performing Tensor conversion on the multiple images, changing the grayscale range from 0-255 to 0-1, and finally normalizing the image.
[0046] Specifically, refer to Figure 2 , the lightweight inverse bottleneck residual module based on channel reorganization in S2 includes:
[0047] The input feature map of shuffle_res2net_s is X∈R C×H×W , X represents the input feature map, C is the input channel, H is the feature map height, and W is the feature map width;
[0048] After the convolution with a convolution kernel size of 1×1, the channel dimension is increased, and the channel dimension becomes C′=C×W / 64×S, and the feature map is obtained as X′∈R C′×H×W , where X′ represents the feature map obtained after the convolution operation, and C′ represents the changed channel dimension;
[0049] X′ is segmented at the channel level C′ into n feature subgraphs with equal channel sizes, and X i Represents, where i∈(1,2,3...n), X i The size of the space is consistent with the size of X′, and channel C i is 1 / n of X′;
[0050] The last branch is not filtered by the 3×3 group convolution kernel, and the remaining three branches are filtered by the 3×3 group convolution. The input of each branch after the first branch contains the output of the previous branch.
[0051] In the branch containing grouped convolution, after the input feature subgraph undergoes grouped convolution, the extracted feature subgraph is reorganized in the channel dimension;
[0052] The outputs of the four branches are concatenated at the channel level, and a convolution with a kernel size of 1×1 is used to perform feature fusion and dimensionality reduction, and a skip connection is used to perform element-wise addition with the input.
[0053]
[0054]
[0055] Among them, GCONV3(.) represents the grouped convolution operation with a convolution kernel size of 3×3; shuffle represents the channel reorganization operation; Conv 1×1 represents a convolution operation with a convolution kernel size of 1×1; X i Represents the input of each branch; y i represents the output of each branch; Y represents the final output of the module.
[0056] refer to Figure 3 , the difference between ordinary convolution and grouped convolution. When the input feature map is The output feature map is The size of each set of input feature maps is C1 / g×H×W, and there are g sets in total. The size of each set of a single convolution kernel is C2 / g×H×W. A convolution kernel is divided into g groups. The number of parameters for ordinary convolution is K×K×C1×C2, and the number of parameters for grouped convolution is K 2 ×C1 / g×C2 / g×g=K 2×C1C2 / g. It can be seen that the number of parameters of grouped convolution is 1 / g of that of conventional convolution, where g is the number of groups.
[0057] Grouped convolution also has its flaws, that is, the independence of channels between groups and the lack of interactivity. After adding grouped convolution, although the number of parameters has decreased, the accuracy has also decreased. Therefore, in order to solve this problem, in the branch containing grouped convolution, after the input feature subgraph has undergone grouped convolution, the extracted feature subgraph is shuffled and redistributed in the channel dimension, so that the information between each group is shuffled to achieve information exchange.
[0058] refer to Figure 4 , the channel reorganization operation specifically includes:
[0059] The extracted feature sub-graph is placed in channel C i Divide the above into g groups of equal size.
[0060] Reshape the g groups into a matrix with g rows and h columns, where h = C i / g.
[0061] The obtained matrix is first transposed and then flattened into one dimension by Fallten operation to realize the information flow between groups.
[0062] The difference between CSP_Bottle2Neck and CSP_BottleNeck is the part marked by the dotted rectangle. The CBS convolution block contains the convolution layer, the normalization layer BN, and the SILU activation function. C×H×W It is divided into two branches. Both branches first pass through the CBS convolution block. Then the upper branch passes through n lightweight inverse bottleneck residual modules based on channel reorganization to extract multi-scale features. Finally, the outputs of the two branches are concatenated at the channel level and then pass through the CBS convolution block to obtain the output feature map Y∈R C×H×W Compared with the original CSP_BottleNeck, CSP_Bottle2Neck is lighter and has the ability to extract multi-scale features.
[0063] refer to Figure 6, which is a diagram showing the fusion strategy. The fusion strategy of CSP_Bottle2Neck and the backbone network of YOLOv5: replace all the previous CSP1_BottleNeck(x) in the backbone network with CSP1_Bottle2Neck(x), and the last layer CSP2_BottleNeck(x) remains unchanged. CSP1 and CSP2 mean that the module named with CSP1 contains jump connections, and the module named with CSP2 does not contain jump connections. Here, x represents the number of stacked bottleneck residual modules in the structure. The dotted rectangle represents the replacement position. The improved backbone feature extraction network is named: MultiscaleLightweight backbone, MSL_backbone; the name, number of channels, and number of parameters of each layer are shown in Table 1. In order to achieve the effect of controlling variables, the fusion strategy only considers the added position, and the rest remains unchanged (such as the number of stacked modules). The scale S parameter and width W parameter of the lightweight inverse bottleneck residual module based on channel reorganization are 4 and 26 respectively.
[0064] Table 1 Comparison of parameters of each backbone network
[0065]
[0066] Experimental verification:
[0067] The backbone feature extraction network of the YOLOv5 method is used as the baseline model, and global average pooling and fully connected layers are added at the end of the network to adapt to the CIFAR-10 dataset. The following groups of comparative experiments are carried out on the proposed feature extraction backbone network that integrates a lightweight inverse bottleneck residual structure based on channel reorganization: the first group is a comparative experiment between the original YOLOv5 backbone network (baseline) and the improved feature extraction backbone network; the second group is a comparative experiment on the influence of different scale parameters S and width parameters W on the feature extraction performance of the network model.
[0068] 1. Experimental Dataset
[0069] In order to demonstrate the feature extraction capability of the improved backbone feature extraction network, as well as the impact of the scale parameter s and width parameter w in the shuffle_res2net_s module on the feature extraction capability of the backbone network, the CIFAR-10, a commonly used benchmark dataset in the field of image classification, is used as the dataset required for the experiment. The CIFAR-10 dataset has a total of 60,000 images. It contains 10 common categories in life, each with 6,000 images. The dataset is divided into 50,000 images for the training set and 10,000 images for the test set. The dataset category diagram is shown below. Figure 7 shown.
[0070] 2. Evaluation Criteria
[0071] Accuracy is used as the evaluation standard of the experiment. The number of paired samples is divided by the total number of samples, that is: Accuracy (classification) rate = number of correctly predicted positive and negative examples / total number. Accuracy is generally used to evaluate the global accuracy of the model. It is also an overall evaluation of the accuracy of the model's predictions. The commonly used accuracy calculation formula is as follows:
[0072]
[0073] 3. Experimental Setup
[0074] The experiment of the present invention adopts the CIFAR-10 dataset. For all models, the experimental settings are: the image input size is 32×32, the batch_size is set to 128, the training cycle (epoch) is 100, the loss function is the cross entropy loss function (Cross Entropy Loss), the initial learning rate learning rate=0.01, the optimizer adopts SGD, the momentum momentum=0.9, the weight decay term weight_decay=5e-4, and the cosine annealing is used to adjust the learning rate in an orderly manner.
[0075] 4. Experimental results analysis
[0076] The first set of experiments: The YOLOv5 feature extraction backbone network model (baseline) and the improved feature extraction backbone network are compared horizontally to compare the accuracy performance of various models on the CIFAR-10 dataset. In order to control the variables, the number of stacked modules in CSP_BottleNeck and CSP_Bottle2Neck is kept the same; the scale parameter S and width parameter W of shuffle_res2net_s are 4 and 26 respectively. When the accuracy is similar, the parameters of the models are compared. The training accuracy, test accuracy and parameter amount index results of each model are shown in Table 2.
[0077] Table 2 Performance indicators of each model in the first group of experiments
[0078]
[0079] The second group of experiments: The backbone feature extraction network proposed in this paper does not focus on improving the complexity of the model. The comparative experiments are carried out under the premise of keeping the parameters of each model roughly the same. The selection of the scale parameter S in this section refers to the selection data S∈{2,4,6,8} of the scale parameter S in Res2Net. Figure 5The replacement strategy in shuffle_res2net_s is used to replace the residual structure in CSP_BottleNeck in the backbone of YOLOv5. To ensure the same parameter volume as the YOLOv5 backbone network, the combination of scale parameter S and width parameter W is calculated to be {(2,48), (4,26), (6,18), (8,14)}, and the parameter volume is about 4.2M. This set of parameter combinations is then used in the proposed shuffle_res2net_s. The improved backbone network parameter volume is about 3.8M after calculation.
[0080] Then, in order to verify the influence of scale parameter and width parameter on the feature extraction backbone network, the backbone models with different scale parameters S and width parameters W are compared horizontally, and the accuracy performance of various models on the CIFAR-10 dataset is compared. The training accuracy, test accuracy and parameter quantity index results of each model are shown in Table 3.
[0081] Table 3 Performance indicators of each model in the second group of experiments
[0082]
[0083] From the comparative experimental results, it can be seen that when the parameters are basically the same, (S=4, W=26) has a higher test accuracy. Therefore, considering the accuracy and parameter factors, MSL_backbone (S=4, W=26) is selected as the final multi-scale lightweight backbone feature extraction network.
[0084] For the backbone (baseline) model of the multi-scale lightweight backbone feature extraction network MSL_backbone and the original YOLOv5, the last feature map output of the backbone network (before the global mean pooling layer) is extracted, and the grad-CAM method is used to visualize the weight heat map of the test image. The smaller the weight, the lighter the color, which means that the model does not pay much attention to the feature information of this part. The larger the weight, the darker the color, which means that the model pays more attention to the feature information of this part. The heat map is as follows Figure 8As shown in the figure, the first row is the result of the original YOLOv5 backbone network, and the second row is the result of MSL_backbone. From left to right, the categories are ship, truck, airplane, and automobile. It can be seen from the picture that MSL_backbone can better focus on the representative parts of the target itself and better extract the feature information of the target than the original feature extraction network. For example, pictures (a) and (e) represent ships: the improved backbone feature extraction network pays more attention to the bottom of the ship; pictures (b) and (f) represent trucks: the improved backbone feature extraction network pays more attention to the front and body of the truck; pictures (c) and (g) represent airplanes: the improved backbone feature extraction network pays more attention to the wings and tail of the aircraft; pictures (d) and (h) represent automobiles: the improved backbone feature extraction network pays more attention to the body and tail of the car, which is more in line with human intuitive thinking.
[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
Claims
1. A feature extraction method based on a multi-scale lightweight backbone network, characterized in that: The steps include: S1: Obtain an image and preprocess it. A preprocessed image is a training sample. S2: Construct the MSL_backbone model, which uses the backbone feature extraction network of YOLOv5 as the basic model, and replaces the residual structure in CSP_BottleNeck in the backbone feature extraction network of YOLOv5 with the lightweight inverse bottleneck residual module shuffle_res2net_s based on channel reorganization; The lightweight inverse bottleneck residual module based on channel reorganization includes: The input feature map of shuffle_res2net_s is X∈R C×H×W , X represents the input feature map, C is the input channel, H is the feature map height, and W is the feature map width; After the convolution with a convolution kernel size of 1×1, the channel dimension is changed to C'=C×W / 64×S, and the feature map is obtained as X'∈R C′×H×W , where X' represents the feature map obtained after the convolution operation, and C' represents the changed channel dimension; X' is segmented at the channel level C' into n feature subgraphs with equal channel sizes, and X i Represents, where i∈(1,2,3...n), X i The size of the space is consistent with that of X', and the number of channels C i is 1 / n of X'; The last branch is not filtered by the 3×3 group convolution kernel, and the remaining three branches are filtered by the 3×3 group convolution. The input of each branch after the first branch contains the output of the previous branch. In the branch containing grouped convolution, after the input feature subgraph undergoes grouped convolution, the extracted feature subgraph is reorganized in the channel dimension; The outputs of the four branches are concatenated at the channel level, and a convolution with a kernel size of 1×1 is used to perform feature fusion and dimensionality reduction, and a skip connection is used to perform element-wise addition with the input. Among them, GCONV3(.) represents the grouped convolution operation with a convolution kernel size of 3×3; shuffle represents the channel reorganization operation; Conv 1×1 represents a convolution operation with a convolution kernel size of 1×1; X i Represents the input of each branch; y i Represents the output of each branch; Y represents the final output of the module; S3: Input the training samples into the MSL_backbone model to train it, and save the model parameters after optimal convergence to obtain the optimal MSL_backbone; S4: Call the optimal MSL_backbone, input the test data to get the result and output it by extracting the last feature map of the network.
2. The feature extraction method based on a multi-scale lightweight backbone network according to claim 1, characterized in that: The preprocessing in S1 refers to: resizing the image to 32×32, then randomly flipping it to get multiple images, then performing Tensor conversion on the multiple images, changing the grayscale range from 0-255 to 0-1, and finally normalizing the image.
3. The feature extraction method based on a multi-scale lightweight backbone network according to claim 1, characterized in that: The channel reorganization operation specifically includes: The extracted feature sub-graph is transformed into i Divide the above into g groups of equal size; Reshape the g groups into a matrix with g rows and h columns, where h = C i / g; The obtained matrix is first transposed and then flattened into one dimension by Fallten operation.