A small target detection method based on multi-scale information fusion

Abstract

This invention relates to a small target detection method based on multi-scale information fusion, comprising the following steps: acquiring an existing dataset and preprocessing the dataset, with all preprocessed images constituting a training set; constructing a small target detection model, which uses the YOLOv5 backbone feature extraction network as the base model, replacing several modules within it; introducing data resampling operations on the input dataset and training it, saving the model parameters after optimal convergence to obtain the optimal small target detection model; calling the optimal small target detection model, inputting the test image for detection, and outputting the recognition result. Experimental results demonstrate that this method improves the ability to extract multi-scale features and effectively improves the detection performance of small-scale targets, with test accuracy improved compared to existing methods.

Description

Technical Field

[0001] This invention relates to the field of road scene target detection technology, and in particular to a method for detecting small targets in road scenes based on multi-scale information fusion. Background Technology

[0002] Object detection in road scenes is a core technology for achieving environmental perception in advanced driver assistance systems (ADAS). Collecting road information via onboard cameras and then applying road scene object detection methods to quickly and accurately identify road targets is a crucial prerequisite for ADAS to implement lane keeping and collision warning functions.

[0003] Scale inconsistency negatively impacts target detection performance in road scenes. If a model can only adapt to targets at a specific scale, targets at different scales will fail to be correctly identified. Furthermore, due to variations in the scale of vehicles, pedestrians, and other objects, numerous targets with inconsistent scales exist in road scenes, making scale inconsistency a significant challenge in target detection. To address this issue, in 2015, He et al. designed a novel module, Spatial Pyramid Pooling (SPP). SPP partitions the input data at different scales, performs max pooling on each of the resulting blocks, and finally merges the results to obtain a fixed input size for the fully connected layer. This solves the problem of requiring a fixed-size input data.

[0004] The YOLOv5 framework employs spatial pyramid pooling, aiming to fuse max-pooling operations of various sizes to enhance the correlation between local and global features, further expanding the representational power of the final feature map and obtaining more useful semantic information. It also enhances the method's robustness to spatial layout and object deformation, thereby improving detection accuracy. Before the Neck network, the YOLOv5 method adds an SPP structure to the later part of the backbone feature extraction network.

[0005] While the YOLOv5 method uses the SPP module to extract features using pooling layers of different sizes, enabling the fusion of features at different scales and enhancing the expressive power of the final features, the pooling layer operation itself has shortcomings. While increasing the receptive field, it also reduces the resolution, leading to the loss of some detailed information in the image and an inability to fully restore details. This results in poor detection performance for small objects, particularly noticeable in road scene object detection. For example, when an object in front of a vehicle is far away, its scale is small, occupying fewer pixels in the image, and providing less feature information to the model, leading to poor detection performance for such small-scale objects. Summary of the Invention

[0006] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to improve the accuracy of small-scale target detection in road scene target detection.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a small target detection method based on multi-scale information fusion, comprising the following steps:

[0008] S1: Obtain the existing dataset and preprocess it. All the images after preprocessing constitute the training set.

[0009] S2: Construct a small target detection model. The small target detection model uses the YOLOv5 backbone feature extraction network as the basic model. The lightweight inverse bottleneck residual module shuffle_res2net_s based on channel recombination is replaced with the residual structure in CSP_BottleNeck in the YOLOv5 backbone feature extraction network and defined as the MSL_backbone model.

[0010] The improved multi-scale void spatial pyramid pooling module Imp_ASPP replaces the spatial pyramid pooling module SPP in YOLOv5;

[0011] The improved CONV_NS module, designed to enhance small target detection performance, will replace the downsampling module in the YOLOv5 network.

[0012] S3: Introduce data resampling operation to the input dataset and train it. After optimal convergence, save the model parameters to obtain the optimal small target detection model.

[0013] S4: Call the optimal small object detection model, input the test image for detection, and output the recognition result.

[0014] As a preferred option, preprocessing in S1 involves: removing the Misc and DontCare categories from the existing data; merging the three categories of Van, Truck, and Tram, which have a small number of samples, into the Car category; merging the Person_sitting category into the Pedestrain category; and performing standardization on all images in the dataset to finally obtain images in the three major categories of Car, Cyclist, and Pedestrain.

[0015] Preferably, the lightweight inverse bottleneck residual module based on channel recombination in S2 includes:

[0016] The input feature map of the shuffle_res2net_s is X∈R C×H×W X represents the input feature map, C is the input channel, H is the feature map height, and W is the feature map width;

[0017] After performing a 1×1 convolution to increase the dimensionality of the channels, the channel dimension becomes C′=C×W / 64×S, resulting in the feature map X′∈R. C′×H×W Where X′ represents the feature map obtained after the convolution operation, and C′ represents the changed channel dimension;

[0018] Segment X′ at the channel level C′, dividing it into n feature sub-images of equal channel size, and use X... i Let X represent the expression where i∈(1,2,3...n), and X i The size of the space is the same as the size of X′, and the channel C i It is 1 / n of X′;

[0019] The last branch is not filtered by the 3×3 grouped convolution kernel, while the other three branches are filtered by the 3×3 grouped convolution kernel. The input of each branch after the first branch contains the output of the previous branch.

[0020] In the branch containing grouped convolution, after the input feature map is processed by grouped convolution, the extracted feature map is reorganized in the channel dimension;

[0021] The outputs of the four branches are concatenated at the channel level, and a 1×1 convolution kernel is used to perform feature fusion and dimensionality reduction. Skip connections are then used to add the outputs to the input element-wise.

[0022]

[0023]

[0024] Where GCONV3(.) represents a grouped convolution operation with a kernel size of 3×3; shuffle represents a channel rearrangement operation; Conv 1×1 This represents a convolution operation with a kernel size of 1×1; X i This represents the input for each branch; y i Y represents the output of each branch; Y represents the final output of the module.

[0025] Preferably, the channel reconfiguration operation specifically includes:

[0026] The extracted feature submap is displayed in channel C. i Divide the data into g groups of equal size;

[0027] Reshape the g groups into a matrix of g rows and h columns, where h = C i / g;

[0028] The obtained matrix is ​​first transposed, and then flattened into one dimension using the Fallten operation.

[0029] Preferably, the improved multi-scale void space pyramid pooling module in S2 includes

[0030] The input feature map of Imp_ASPP is X∈R C×H×W ;

[0031] The input feature map is divided into 5 branches. The first branch performs a convolution operation with a kernel size of 1×1.

[0032] For branches 2-4, image context information at different scales is obtained through parallel sampling using depthwise separable dilated convolutions at different sampling rates. A channel shuffle operation is performed after each depthwise separable dilated convolution. For the input of a branch, the output of the previous branch is concatenated with the input of the current branch. Similarly, for the output of a branch, the output of the previous branch is concatenated with the output of the current branch.

[0033] The fifth branch is first passed through an adaptive mean pooling, then through a 1×1 convolutional layer, and finally the feature map is upsampled from the 1×1 size back to the original size;

[0034] Output from 5 branches Where i = (1, 2... 5) are concatenated at the channel level, and then Conv(1×1) is used to reduce the dimensionality and restore the original number of channels, finally obtaining the output feature map Y∈R. C×H×W .

[0035] Preferably, the improved CONV_NS module in S2 for improving small target detection performance includes:

[0036] The input feature map of CONV_NS is X∈R C×H×W ;

[0037] Feature extraction is performed using a 3×3 convolution kernel with a stride of 1. Without changing the spatial size of the input feature map, the channel dimension is first reduced to half its original value, becoming C / 2, to obtain the features. Figure X ′=R C / 2×H×W ;

[0038] Features Figure X ′=R C / 2×H×W Perform a reshaping operation. The feature map becomes X”=R C / 2×new_h×scale×new_w×scale new_h and new_w are the height and width of the newly generated feature map, and scale is the scaling factor;

[0039] Perform a transpose operation on X” to make X”′=R C / 2×scale×scale×new_h×new_w ;

[0040] After transposing, a reshaping operation is performed to obtain Y∈R. 2C×new_h×new_w By concatenating the channels, the channel dimension is doubled, while the spatial size of the feature map is reduced to half of its original size.

[0041] Preferably, the specific steps of the resampling operation in S3 are as follows:

[0042] Assign a learning weight to each category in the training set;

[0043] Calculate the number of each category in each image in the training set, and then calculate the sampling weight of each image based on the learning weight of each category.

[0044] Based on the sampling weight of each image, an image index is generated using the random.choices() function, and images are loaded according to each index each time data is loaded.

[0045] Compared with the prior art, the present invention has at least the following advantages:

[0046] This invention improves upon the YOLOv5 backbone feature extraction network, proposing a lightweight inverse bottleneck residual structure, shuffle_res2net_s, based on channel recombination. This structure is then fused with the YOLOv5 backbone to create a multi-scale lightweight backbone feature extraction network, MSL_backbone. Furthermore, an ASPP module is introduced and improved, resulting in Imp_ASPP, which replaces the SPP module in YOLOv5. To improve small object detection performance, a module (CONV_NS) is designed to replace the downsampling module in YOLOv5. Data resampling is introduced on the KITTI dataset to address class imbalance in road scenes. Experimental results demonstrate that MS-YOLOv5 improves multi-scale feature extraction capabilities while effectively enhancing small-scale object detection performance, achieving a 4.19% improvement in accuracy compared to YOLOv5, and improvements of 7.96% and 5.28% in Cyclist and Pedestrain, respectively.

[0047] This enables road scene target detection methods to achieve higher detection accuracy for small-scale targets and targets with small sample sizes while maintaining overall target detection accuracy. Attached Figure Description

[0048] Figure 1 This is a diagram of the MS-YOLOv5 network framework.

[0049] Figure 2 This is a structural diagram of a lightweight inverse bottleneck residual module based on channel recombination.

[0050] Figure 3 This diagram illustrates the difference between regular convolution and grouped convolution.

[0051] Figure 4 This is a schematic diagram of the channel reconfiguration process.

[0052] Figure 5 This is a structure diagram of CSP_Bottle2Neck (with jump connections).

[0053] Figure 6 This diagram illustrates the integration strategy.

[0054] Figure 7 An improved multi-scale void space pyramid pooling (Imp_ASPP) structure diagram.

[0055] Figure 8 This diagram illustrates the computational process of depth-separable dilated convolution.

[0056] Figure 9To improve the structure diagram of the small target detection module (CONV_NS).

[0057] Figure 10 This is a diagram illustrating the CONV_NS fusion strategy.

[0058] Figure 11 This is a visualization of the test results and heat map. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] See Figure 1 A small target detection method based on multi-scale information fusion includes the following steps:

[0061] S1: Obtain the existing dataset and preprocess it. All the images after preprocessing constitute the training set.

[0062] A small target detection model is constructed. The small target detection model uses the YOLOv5 backbone feature extraction network as the basic model. The lightweight inverse bottleneck residual module shuffle_res2net_s based on channel recombination is replaced with the residual structure in CSP_BottleNeck in the YOLOv5 backbone feature extraction network and defined as the MSL_backbone model.

[0063] The improved multi-scale void spatial pyramid pooling module Imp_ASPP replaces the spatial pyramid pooling module SPP in YOLOv5;

[0064] The improved CONV_NS module, designed to enhance small target detection performance, will replace the downsampling module in the YOLOv5 network.

[0065] S3: Introduce data resampling operation to the input dataset and train it. After optimal convergence, save the model parameters to obtain the optimal small target detection model.

[0066] S4: Call the optimal small object detection model, input the test image for detection, and output the recognition result.

[0067] Specifically, preprocessing in S1 involves removing the Misc and DontCare categories from the existing data, merging the Van, Truck, and Tram categories (which have very few samples) into the Car category, merging the Person_sitting category into the Pedestrain category, and standardizing all images in the dataset to obtain images in the three main categories of Car, Cyclist, and Pedestrain. This invention uses the widely used traffic scene dataset KITTI, which includes nine categories: Car, Van, Truck, Tram, Cyclist, Pedestrain, Person_sitting, Misc, and DontCare.

[0068] For details, please refer to Figure 2 The lightweight inverse bottleneck residual module based on channel recombination in S2 specifically includes:

[0069] The input feature map of the shuffle_res2net_s is X∈R C×H×W X represents the input feature map, C is the input channel, H is the feature map height, and W is the feature map width;

[0070] After performing a 1×1 convolution to increase the dimensionality of the channels, the channel dimension becomes C′=C×W / 64×S, resulting in the feature map X′∈R. C′×H×W Where X′ represents the feature map obtained after the convolution operation, and C′ represents the changed channel dimension;

[0071] Segment X′ at the channel level C′, dividing it into n feature sub-images of equal channel size, and use X... i Let X represent the expression where i∈(1,2,3...n), and X i The size of the space is the same as the size of X′, and the channel C i It is 1 / n of X′;

[0072] The last branch is not filtered by the 3×3 grouped convolution kernel, while the other three branches are filtered by the 3×3 grouped convolution kernel. The input of each branch after the first branch contains the output of the previous branch.

[0073] In the branch containing grouped convolution, after the input feature map is processed by grouped convolution, the extracted feature map is reorganized in the channel dimension;

[0074] The outputs of the four branches are concatenated at the channel level, and a 1×1 convolution kernel is used to perform feature fusion and dimensionality reduction. Skip connections are then used to add the outputs to the input element-wise.

[0075]

[0076]

[0077] Where GCONV3(.) represents a grouped convolution operation with a kernel size of 3×3; shuffle represents a channel rearrangement operation; Conv 1×1 This represents a convolution operation with a kernel size of 1×1; X i This represents the input for each branch; y i Y represents the output of each branch; Y represents the final output of the module.

[0078] refer to Figure 3 A diagram illustrating the difference between regular convolution and grouped convolution. When the input feature map is... The output feature map is The input feature maps for each group have a size of C1 / g × H × W, and there are a total of g groups. Each convolutional kernel has a size of C2 / g × H × W, and one kernel is divided into g groups. The number of parameters for a regular convolution is K × K × C1 × C2, and the number of parameters for a grouped convolution is K. 2 ×C1 / g×C2 / g×g=K 2 ×C1C2 / g. It can be seen that the number of parameters in grouped convolution is 1 / g of that in regular convolution, where g is the number of groups.

[0079] Grouped convolution also has its drawbacks, namely the lack of independence of channels between groups and the absence of interactivity. Although the number of parameters decreases after adding grouped convolution, the accuracy also decreases. Therefore, in order to solve this problem, in the branch containing grouped convolution, after the input feature submap is processed by grouped convolution, the extracted feature submap is shuffled and redistributed in the channel dimension, so that the information between each group is shuffled and can achieve information exchange.

[0080] refer to Figure 4 The channel reconfiguration operation specifically includes:

[0081] The extracted feature submap is displayed in channel C. i Divide the data into g groups of equal size.

[0082] Reshape the g groups into a matrix of g rows and h columns, where h = C i / g.

[0083] The resulting matrix is ​​first transposed, then flattened into one dimension using a falloff operation. This enables information flow between groups.

[0084] The difference between CSP_Bottle2Neck and CSP_BottleNeck lies in the part marked by the dashed rectangle. The CBS convolutional block contains convolutional layers, a normalization layer (BN), and a SILU activation function. The input... Figure X ∈R C×H×W The algorithm is divided into two branches. Both branches first pass through CBS convolutional blocks. The upper branch then extracts multi-scale features through n lightweight inverse bottleneck residual modules based on channel recombination. Finally, the outputs of the two branches are concatenated at the channel level (Concat), and then passed through a CBS convolutional block to obtain the output feature map Y∈R. C×H×W Compared to the original CSP_BottleNeck, CSP_Bottle2Neck is more lightweight and has the ability to extract features at multiple scales.

[0085] refer to Figure 6 The diagram illustrates the fusion strategy. The fusion strategy of CSP_Bottle2Neck and the YOLOv5 backbone network is as follows: all the preceding CSP1_BottleNeck(x) in the backbone network are replaced with CSP1_Bottle2Neck(x), while the last layer CSP2_BottleNeck(x) remains unchanged. CSP1 and CSP2 mean that modules named CSP1 contain skip connections, while modules named CSP2 do not. Here, x represents the number of stacked bottleneck residual modules in the structure. The dashed rectangles represent the replacement locations. The improved backbone feature extraction network is named: MultiscaleLightweight backbone, MSL_backbone; the names, number of channels, and number of parameters for each layer are shown in Table 1. To achieve the effect of controlling variables, the fusion strategy only considers the addition locations, while keeping the rest unchanged (such as the number of stacked modules). The scale S-parameter and width W-parameter of the lightweight inverse bottleneck residual module based on channel recombination are 4 and 26, respectively.

[0086] Table 1 Comparison of parameters of each backbone network

[0087]

[0088] See Figure 7 The improved multi-scale void space pyramid pooling module in S2 includes

[0089] The input feature map of Imp_ASPP is X∈R C×H×W ;

[0090] The input feature map is divided into 5 branches. The first branch performs a convolution operation with a kernel size of 1×1.

[0091] For branches 2-4, image context information at different scales is obtained through parallel sampling using depthwise separable dilated convolutions at different sampling rates. A channel shuffle operation is performed after each depthwise separable dilated convolution. For the input of a branch, the output of the previous branch is concatenated with the input of the current branch. Similarly, for the output of a branch, the output of the previous branch is concatenated with the output of the current branch.

[0092] The fifth branch is first passed through an adaptive mean pooling, then through a 1×1 convolutional layer, and finally the feature map is upsampled from the 1×1 size back to the original size;

[0093] The output X′ of the 5 branches i =R C×H×W Where i = (1, 2... 5) are concatenated at the channel level, and then Conv(1×1) is used to reduce the dimensionality to restore the original number of channels, finally obtaining the output feature map Y∈R. C×H×W .

[0094] The expression for Imp_ASPP is as follows:

[0095]

[0096]

[0097] All convolution operations in the formula include a BN layer and an activation function, as shown in the reference. Figure 7 As shown in the dashed rectangle. Where X... i Represents the i-th branch of the input, Conv 1×1 (.) indicates a 1×1 kernel convolution, which halves the number of input channels entering the depthwise separable dilated convolution to prevent the model from becoming too wide. DWATConv(.) indicates a 3-kernel depthwise separable dilated convolution, shuffle indicates a channel shuffling operation, concat(.) indicates concatenation at the channel level, pool(.) indicates adaptive mean pooling for the 5th branch, and up_sample(.) indicates an upsampling operation.

[0098] The original ASPP module had 2,644,480 parameters, while the improved multi-scale void spatial pyramid pooling Imp_ASPP module has only 1,576,448 parameters, a reduction of approximately 40%. Subsequent comparative experiments verified that, despite the reduced parameter count, Imp_ASPP delivered higher detection accuracy than the original ASPP.

[0099] refer to Figure 8Depthwise separable dilated convolution consists of channel-wise dilated convolution (DAConv) and pointwise convolution (PWConv). The difference between DAConv and depthwise separable convolution is that the first part of the convolution is replaced by dilated convolution. This breaks down the one-step operation of ordinary dilated convolution into two steps, reducing the number of parameters by two-thirds and significantly saving storage space.

[0100] See Figure 9 The improved CONV_NS module in S2, used to improve the performance of small target detection, includes:

[0101] The input feature map of CONV_NS is X∈R C×H×W ;

[0102] Feature extraction is performed using a 3×3 convolution kernel with a stride of 1. Without changing the spatial size of the input feature map, the channel dimension is first reduced to half its original value, becoming C / 2, to obtain the features. Figure X ′=R C / 2×H×W ;

[0103] Features Figure X ′=R C / 2×H×W Perform a reshaping operation. The feature map becomes X”=R C / 2×new_h×scale×new_w×scale new_h and new_w are the height and width of the newly generated feature map, and scale is the scaling factor. Here, scale = 2, which is equivalent to reducing the spatial size of the feature map to 1 / 2 of its original size.

[0104] Perform a transpose operation on X” to make X” = R C / 2×scale×scale×new_h×new_w ;

[0105] After transposing, a reshaping operation is performed to obtain Y∈R. 2C×new_h×new_w By concatenating the channels, the channel dimension is doubled, while the spatial size of the feature map is reduced to half of its original size.

[0106] The expression for CONV_NS is as follows:

[0107] Y=reshape(transpose(reshape(Conv 3x3 (X))))(5)

[0108] The CONV_NS module can perform a downsampling convolution operation similar to stride=2 without losing information in the image. Subsequent experiments demonstrate that replacing the downsampling convolution module in the YOLOv5 method framework with the CONV_NS module significantly improves the accuracy of small target detection in road-based scenes.

[0109] refer to Figure 10 This diagram illustrates the CONV_NS fusion strategy. The YOLOv5 object detection method uses downsampling convolutions in two structures: the feature extraction backbone network and the feature fusion network. The downsampling in these two structures serves different purposes. Downsampling in the backbone feature extraction network is for feature extraction, reducing computational cost, preventing overfitting, and increasing the receptive field size to enhance the representation of global information. Downsampling in the feature fusion network is for fusing feature maps of different scales.

[0110] Therefore, there are two strategies for combining the CONV_NS module with the YOLOv5 method: First, replace the four downsampling convolutional modules in the feature extraction backbone of the YOLOv5 method with CONV_NS modules. Second, based on the first method, replace the two downsampling convolutional modules in the Neck network of the YOLOv5 method with CONV_NS modules as well. The fusion strategies are described in the reference. Figure 10 As shown.

[0111] Experimental verification:

[0112] The experiments in this invention use the 2D object detection subset of the KITTI dataset. By conducting comparative ablation experiments on this dataset, the effectiveness of the proposed improved module is verified, and the detection results are visualized using a weighted heatmap.

[0113] The method of this invention is abbreviated as MS-YOLOv5.

[0114] 1. Experimental Dataset

[0115] To demonstrate whether the MS-YOLOv5 object detection method, which integrates various improved modules, improves upon the original YOLOv5 method in traffic and road scenarios, the widely used traffic and road scenario dataset KITTI was adopted.

[0116] The scenes in the KITTI dataset are real-world traffic scenarios (rural areas, highways, urban areas), collected by vehicles equipped with various sensors. This dataset is commonly used for tasks such as 2D object detection, 3D object detection, and 3D tracking. This study uses a subset of the 2D object detection dataset from KITTI. This subset includes 7481 labeled training images and 7518 unlabeled test images. This invention uses the 7481 labeled images for experiments. The number of samples for each category in KITTI after data preprocessing is shown in Table 2.

[0117] Table 2. Number of samples in each category for KITTI

[0118] category Car Pedestrain Cyclist Total quantity 32750 4709 1627 39086

[0119] 2. Evaluation Criteria

[0120] Since the experimental dataset of this invention is based on the VOC dataset format, the evaluation index of the VOC dataset is used to evaluate the effectiveness of the designed object detection method model.

[0121] IOU (Intersection over Union): refers to the ratio of the overlapping area between the detection box and the ground truth box to the area of ​​the union.

[0122] TP (True Positive): When the actual value is a positive sample, the object detection model also predicts a positive sample, which is denoted as TP.

[0123] FP (False Positive): When the object detection model predicts a positive sample, but the actual true value is a negative sample, it is denoted as FP. Also known as a false detection, it occurs when the IOU between the predicted bounding box and the ground truth is less than a threshold (localization error) or when the predicted type does not match the label type (classification error).

[0124] FN (False Negative): When the object detection model predicts a negative sample, but the actual true value is a positive sample, it is denoted as FN. Also known as a missed detection, it refers to a Ground Truth region that was not detected.

[0125] TN (True Negative): When the actual value is a negative sample, the object detection model predicts a negative sample. In object detection, TN is usually not considered.

[0126] Precision, also known as accuracy, refers to the percentage of target objects identified (TP). It represents the accuracy with which an object detection model predicts a certain category of objects.

[0127]

[0128] Recall, also known as recall rate, refers to the ratio of objects identified as correct to objects that are actually correct.

[0129]

[0130] AP (Average Precision) refers to the area enclosed by the PR curve, with recall on the horizontal axis and precision on the vertical axis.

[0131] MAP (Mean AP) refers to the average AP of all categories after applying different thresholds for object detection. Generally, object detection often uses MAP(@.5) and MAP(@.5:.95) as evaluation metrics. MAP(@.5) represents the MAP value when the IOU threshold is 0.5, and MAP(@.5:.95) represents the MAP value when the IOU threshold is in the discrete interval [0.5:0.05:0.95]. This invention uses MAP(@.5) as the metric for evaluating the performance of the object detection model.

[0132] 3. Experimental Setup

[0133] This invention uses the 2D object detection subset of the KITTI dataset. The dataset is divided into training, validation, and test sets in a train:val:test ratio of 8:2 and 9:val (9:1). The training dataset contains 5385 images, the val dataset contains 599 images, and the test dataset contains 1497 images. Since the images in the KITTI dataset are mostly extracted frame-by-frame from videos, scenes within a video are often similar. To prevent overfitting, the dataset partitioning avoids scenarios where similar scenes appear in both the training and test sets. For all models, the experimental settings were as follows: input image size 640×640, batch size 10, training epochs 100, loss function GIOU_Loss, initial learning rate 0.001, Adam optimizer with momentum 0.937 and weight decay term 5e-4. Cosine annealing (cosine) was used to adjust the learning rate. Mosica data augmentation was used for dataset preprocessing, enabled only in the first 70% of epochs (e.g., 70 out of 100 epochs). The last 30 epochs used standard data augmentation (random flipping, color gamut transformation, etc.). This is because the training images generated by Mosaic deviate significantly from the true distribution of natural images, and the images should be restored to their natural distribution near the end of training.

[0134] 4. Analysis of Experimental Results

[0135] To verify the performance of the improved module added to the YOLOv5 method in road scenarios, the following sets of comparative ablation experiments were conducted, as shown in the table below:

[0136] Table 3 Comparison of ablation experiments Note: Method 1: Data resampling; Model 1: MSL_backbone; Model 2: ASPP; Model 3: Imp_ASPP; Model 4: CONV_NS strategy 1; Model 5: CONV_NS strategy 2.

[0137] The validation set loss curves of the comparative ablation experiments in the table above are shown in the figure. As the number of training epochs increases, the loss curves eventually converge.

[0138] The first row in the table shows YOLOv5 as the baseline method for this invention. Firstly, regarding the datasets, Cyclist and Pedestrain have significantly fewer elements than Car, and these two categories are small-scale targets that are relatively difficult to identify. This causes the model trained on KITTI using the YOLOv5 object detection method to be more biased towards learning and predicting targets of the Car category, while ignoring the other two categories, ultimately resulting in a decrease in the overall accuracy of the model.

[0139] The second row in the table shows the experiments conducted after data resampling. By incorporating data resampling during dataset preprocessing, the number of images from each category fed into the model for training was balanced, ensuring the model was adequately trained on each target category and able to learn information about each target.

[0140] The third row of the table shows the experiments conducted when the MSL_backbone replaced the original backbone. The results show that the MSL_backbone can extract multi-scale information from the image more effectively.

[0141] The fourth and fifth rows of the table show the replacement of SPP in MSL_backbone with ASPP and Imp_ASPP, respectively. The results show that both modules improve performance, but Imp_ASPP provides a greater improvement. This is because the Imp_ASPP structure allows the backbone network to extract more multi-scale information, providing features with integrated multi-scale information for subsequent Neck network feature fusion.

[0142] Rows six and seven in the table will specify CONV_S according to the reference. Figure 10 Experiments were conducted using the fusion strategy described above. The results show that fusion strategy two achieves higher detection accuracy. This indicates that the proposed CONV_NS module can satisfy the downsampling requirements of both structures in YOLOv5. The accuracy on the test set improved by 0.58% compared to the previous experiment. This is because the CONV_NS module was used instead of the stride=2 convolutional structure, which does not lose detail information in the image and improves the detection of small-scale objects.

[0143] refer to Figure 11This section demonstrates the detection of test images using both the MS-YOLOv5 model and the original YOLOv5 (baseline) model, and visualizes the prediction results using heatmaps. Odd-numbered rows show the results for MS-YOLOv5, and even-numbered rows show the results for the original YOLOv5 (baseline). The first column shows the heatmap visualization results, and the second column shows the detection results. The comparison of the heatmaps reveals that MS-YOLOv5 focuses more on the detected objects themselves, as indicated by a darker color on the object in the heatmap, representing a higher weight. It also pays more attention to small-scale targets. The comparison of the detection results shows that MS-YOLOv5 performs better than the original YOLOv5 (baseline) in detecting targets with a small sample size and small-scale targets at a distance.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A small target detection method based on multi-scale information fusion, characterized in that, The method comprises the following steps: S1: obtaining an existing data set and preprocessing the data set, all pictures after preprocessing constituting a training set; S2: constructing a small target detection model, the small target detection model adopting a main feature extraction network of YOLOv5 as a basic model, replacing a residual structure in CSP_BottleNeck in the main feature extraction network of YOLOv5 with a light-weight inverse bottle-neck residual module shuffle_res2net_s based on channel reorganization, and defining as an MSL_backbone model; The light-weight inverse bottle-neck residual module based on channel reorganization in S2 comprises: The input feature map of the shuffle_res2net_s is , denotes the input feature map, is the input channel, is the feature map height, is the feature map width; After the dimension change on the channel by the convolution kernel with the size of 1*1, the channel dimension becomes , and the feature map is , wherein, represents the feature map obtained after the convolution operation, represents the changed channel dimension; To At the channel level is divided into n feature subgraphs of equal channel size, denoted as , where , The spatial size of is consistent with the size of Channel is 1 / n of The last branch is not filtered through a 3*3 grouped convolution kernel, and the remaining three branches are filtered through a 3*3 grouped convolution kernel, and the input of each branch after the first branch contains the output of the previous branch; In the branch containing the grouped convolution, the extracted feature subgraph is reorganized in the channel dimension after the input feature subgraph passes through the grouped convolution; The outputs of the four branches are spliced on the channel level, a convolution kernel with a size of 1*1 is used for feature fusion and dimension reduction, and a jump connection is used for element-level addition with the input; (1) (2) wherein denotes a convolution kernel size of 3 3 denotes a grouped convolution operation with a group size of 3; denotes a channel reorganization operation; denotes a convolution operation with a convolution kernel size of 1x1; denotes the input of each branch; denotes the output of each branch; denotes the final output of the module; An improved multi-scale hollow spatial pyramid pooling module Imp_ASPP is used to replace a spatial pyramid pooling module SPP in YOLOv5; The improved multi-scale hollow spatial pyramid pooling module in S2 comprises The input feature map of the Imp_ASPP is ; The input feature map is divided into five branches, and the first branch is subjected to a convolution operation through a convolution kernel with a size of 1*1; For the second to fourth branches, different sampling rates of depth separable hollow convolution are used to obtain different scale image context information in parallel, and a channel shuffling operation is used after each depth separable hollow convolution; for the input of the branch, the output of the previous branch is spliced with the input of the branch as the input of the branch; at the same time, for the output of the branch, the output of the previous branch is spliced with the output of the branch as the output of the branch; The fifth branch is first subjected to an adaptive mean pooling, then subjected to a 1*1 convolution layer, and finally the feature map is up-sampled from 1*1 to the original size; The output of the 5 branches is concatenated wherein Concat is performed at the channel level, and then dimensionality reduction is performed to restore the original number of channels, and finally the output feature map is obtained ; The improved module CONV_NS for improving small target detection performance is used to replace a down-sampling module in YOLOv5 network; S3: introducing a data resampling operation to the input data set data, and training, saving the model parameters of the optimal small target detection model after optimal convergence; S4: calling the optimal small target detection model, inputting a picture to be detected for detection, and outputting the recognition result.

2. The small target detection method based on multi-scale information fusion according to claim 1, characterized in that: The preprocessing in S1 refers to: removing the Misc and DontCare categories in the existing data, merging the three categories with small sample quantities, Van, Truck and Tram, into the Car category, merging the Person_sitting category into the Pedestrain category, performing a standardization operation on all pictures in the data set, and finally obtaining pictures of the three large categories of Car, Cyclist and Pedestrain.

3. The small target detection method based on multi-scale information fusion according to claim 2, characterized in that: The channel recombination operation specifically includes: The extracted feature sub-graphs are divided into g groups of equal size in channel order. The g groups are reshaped into a matrix of g rows and h columns, where h = g / g / g; The obtained matrix is first transposed and then flattened into one dimension by Fallten operation.

4. The small target detection method based on multi-scale information fusion according to claim 3, characterized in that: The CONV_NS module for improving small target detection performance after improvement in S2 includes: The input feature map of the CONV_NS is ; After feature extraction by a convolution kernel with a size of 3x3 and a stride of 1, the spatial size of the input feature map is not changed, the channel dimension is first reduced to 1 / 2 of the original, becoming C / 2, to obtain a feature map ; to the feature map a reshaping operation is performed; the feature map becomes , new_h, new_w are the height and width of the newly generated feature map, and scale is the scaling scale To Perform Transposition operation into ; Transpose operation is performed first and reshaping operation is performed later Concatenate in channel dimension, so that the channel dimension becomes twice the original, and the spatial size of the feature map is reduced to 1 / 2.

5. The small target detection method based on multi-scale information fusion according to claim 4, characterized in that: The specific steps of the resampling operation in S3 are: Assign a learning weight to each category in the training set; Calculate the number of each category in each picture in the training set, and then calculate the sampling weight of each picture according to the learning weight of each category; According to the sampling weight of each picture, generate picture indexes through the random.choices() function, and load pictures according to the indexes each time the data is loaded.

Images