Lightweight MP-YOLO dense road target detection method and system in complex environment

Abstract

The application relates to the technical field of automatic driving, and discloses a light MP-YOLO dense road target detection system in a complex environment. The system proposes two multi-scale feature fusion modules MSFB and HFF, enhances the fusion effect of different scale features without affecting the model precision, reduces the input and output channels, and reduces the number of parameters. Secondly, a small target detection head is added to the head network to enhance the feature extraction and small target detection capability. Then, the CIOU function used by YOLOv8n is replaced by a WIOU loss function to solve the problem of high road target overlap. Finally, in order to greatly reduce the size and parameter quantity of the model, an LAMP pruning method based on importance score is used to prune the improved YOLOv8 model. The MP-YOLO model is tested on the latest automatic driving dataset DAIR-V2X, and the results show that the performance of the proposed MP-YOLO exceeds that of the original model, mAP@0.5 is increased by 4.7%, mAP@0.5-0.95 is increased by 4.2%, and the parameter quantity is reduced from 3.2 million to 0.94 million.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a lightweight MP-YOLO dense road target detection method and system in complex environments. Background Technology

[0002] With the continuous advancement of artificial intelligence technologies such as deep learning and neural networks, computers have achieved significant breakthroughs in areas such as image recognition, speech recognition, and natural language processing, providing strong support for autonomous driving technology. Object detection technology plays a crucial role in the field of autonomous driving, effectively improving the safety, efficiency, and intelligence of autonomous vehicles by detecting obstacles on the road in real time, recognizing traffic lights and signs, assisting in path planning and autonomous driving, and providing accurate environmental perception data and intelligent decision support. This provides important technical support and guarantees for the commercial application and popularization of autonomous driving technology.

[0003] Currently, target detection technologies for autonomous driving mainly employ deep learning-based methods, such as the R-CNN series (e.g., Faster R-CNN, Mask R-CNN), the YOLO series (e.g., YOLOv3, YOLOv4), and SSD, utilizing deep learning networks to achieve end-to-end target detection and localization. However, current mainstream target detection technologies often require larger computational resources and perform poorly in recognizing small targets, occluded targets, and targets affected by lighting conditions, limiting their practical deployment. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a lightweight MP-YOLO dense road target detection method and system for complex environments. It has the advantages of fully utilizing a large amount of dense road target image data, automatically learning and recognizing dense and occluded vehicles and pedestrians, thereby achieving efficient, accurate and low-cost road target detection, and solving the aforementioned technical problems.

[0005] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0006] This invention provides a lightweight MP-YOLO dense road target detection method in complex environments, comprising:

[0007] The YOLOv8 model includes the following steps:

[0008] Step 1: Acquire several images of dense road targets;

[0009] Step 2: Improve the YOLOv8 model by adding a small target detection head; modify the neck network to add MSFB and HFF modules that fuse multi-scale features; replace the original CIOU loss with WIOU loss; and prune the model using the LAMP method to obtain the MP-YOLO model.

[0010] The specific steps of step 2 are as follows: a 160×160 feature map on the backbone is introduced into the neck network of the YOLOv8 model, and feature maps of different scales are fused through the MSFB module and HFF module. A small target detection head is added to the head network of the YOLOv8 model. The CIoU loss function in the YOLOv8 model is replaced with the WIoU loss function. After the replacement, the LAMP pruning method is used to prune the model to obtain the MP-YOLO model.

[0011] Step 3: Input several road target images into the MP-YOLO model for training to obtain the MP-YOLO dense road target detection model;

[0012] Step 4: Input the image to be identified into the MP-YOLO dense road target detection to obtain the dense road target detection result of the image to be identified.

[0013] As a preferred embodiment of the present invention, the images in step 1 all include motor vehicles, non-motor vehicles, and pedestrians.

[0014] As a preferred embodiment of the present invention, the MSFB module is used to process feature maps of different scales: its specific expression is as follows:

[0015]

[0016]

[0017] in, This represents the feature map data processed by a Gaussian filter kernel. Represents feature maps of different sizes from the input. Indicates width. Indicates high, Represents the natural constant. This represents the Gaussian filter function. This represents the standard deviation of the Gaussian filter.

[0018] As a preferred embodiment of the present invention, the expression processed by the HFF module is as follows:

[0019]

[0020] in, This represents the feature map output by the HFF module. These represent three different sizes of feature maps. This indicates feature map splicing.

[0021] As a preferred embodiment of the present invention, the expression for the WIoU loss function is as follows:

[0022]

[0023] in, Represents the WIoU loss function. This indicates a distance-focusing mechanism. Indicates the non-monotonic focusing coefficient. Represents the dynamic moving average. Represents the natural constant.

[0024] As a preferred technical solution of the present invention, the distance focusing mechanism Non-monotonic focusing coefficient The expression is as follows:

[0025]

[0026]

[0027] in, For the outlier measure of anchor box mass, This represents an exponential function with the natural constant as its base. , , , These represent the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. and The distribution represents the width and height of the ground truth box, with the superscript * indicating that... and Separate from the computational graph, Indicates outlier values. Represents the weighting function. Let r be the learning parameter, which satisfies the condition that r = 1 when β = δ.

[0028] As a preferred embodiment of the present invention, the LAMP pruning method employs an amplitude-based layer adaptive LAMP importance score method to prune the model, and its expression is as follows:

[0029]

[0030] in, In a neural network, the first... The weight of each connection, This indicates the distribution of weights across the entire layer. Indicates the first The weight tensor of each connection relative to the weight of each connection in the neural network. LAMP score.

[0031] The present invention also provides a lightweight MP-YOLO dense road target detection system in complex environments. Based on the above-mentioned lightweight MP-YOLO dense road target detection method in complex environments, it includes: an image acquisition module for acquiring several dense road target images;

[0032] The model improvement module is used to improve the YOLOv8 model by introducing the MSFB module, HFF module, small object detection head, WIoU loss function and LAMP pruning method to obtain the MP-YOLO model.

[0033] The model training module is used to input several dense road target images into the MP-YOLO model for training, thereby obtaining the MP-YOLO dense road target detection model.

[0034] The dense road target detection module is used to input the image to be identified into the MP-YOLO dense road target detection model to obtain the dense road target detection result of the image to be identified.

[0035] The beneficial effects of this invention are mainly reflected in:

[0036] This invention proposes two multi-scale feature fusion modules, MSFB and HFF, to enhance the fusion effect of features at different scales without affecting model accuracy, reducing input and output channels and the number of parameters. Secondly, it adds a small target detection head to the head network to enhance feature extraction and small target detection capabilities. Then, it replaces the CIOU function used in YOLOv8n with the WIOU loss function to address the problem of high road target overlap. Finally, to significantly reduce the model size and number of parameters, it employs the LAMP pruning method based on importance scores to prune the improved YOLOv8 model. The MP-YOLO model is tested on the latest autonomous driving dataset DAIR-V2X. The results show that the proposed MP-YOLO outperforms the original model, improving mAP@0.5 by 4.7% and mAP@0.5-0.95 by 4.2%, while reducing the number of parameters from 3.2 million to 940,000. It outperforms other classic detection models in both size and accuracy, meeting the requirements for edge device deployment. Attached Figure Description

[0037] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the YOLOv8 model structure of the present invention;

[0039] Figure 2 This is a schematic diagram of the MP-YOLO model structure of the present invention; the red line represents the schematic diagram of the model structure of the present invention with the addition of a small target detection head.

[0040] Figure 3 This is a schematic diagram of the module structure of the MSFB of the present invention;

[0041] Figure 4 This is a schematic diagram of the module structure of the HFF of the present invention;

[0042] Figure 5 This is a schematic diagram of the Lamp pruning method of the present invention;

[0043] Figure 6 This is a schematic diagram of the small target detection head structure of the present invention;

[0044] Figure 7 This is a comparison diagram of the channel before and after pruning according to the present invention, wherein the blue bar chart represents before pruning, the red bar chart represents after pruning, and the red and blue bar chart is a comparison diagram before and after pruning;

[0045] Figure 8 This is a comparison chart of the training accuracy of MP-YOLO and YOLOv8n in terms of mAP50, mAP@0.5-0.95, precision, and recall.

[0046] Figure 9 This diagram illustrates a comparison of the detection performance of YOLOv8n and MP-YOLO in dark scenes, target occlusion, and complex scenes.

[0047] Figure 10 This image shows the detection performance of MP-YOLO in complex scenes such as streets, roads, crowds, small figures, motion deformation, and occlusion in the DAIR-V2X dataset.

[0048] Figure 11 This is an example of the detection performance of MP-YOLO in complex scenes such as streets, roads, crowds, small figures, motion deformation, and occlusion in the SODA10M dataset.

[0049] Figure 12This is a schematic diagram illustrating the steps of the MP-YOLO dense road target detection method of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The technical solutions provided by various embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0051] To better understand the purpose of this invention, the invention will be described in further detail below.

[0052] The YOLOv8 detection model, proposed by Ultralytics in 2023, is an advanced computer vision model. Building upon YOLOv5, it incorporates the advantages of YOLOv7, YOLOX, and other vision models, achieving state-of-the-art (SOTA) status in object detection. The model's network structure is as follows: Figure 1As shown. It includes a backbone, neck, and head, emphasizing modular design to achieve a balance between flexibility and performance, comprehensively surpassing existing object detection and instance segmentation models. 1) Backbone: The YOLOv8 backbone network prototype is the Darknet53 network, with several improvements made to it. The backbone mainly consists of CBS, C2f, and SPPF modules, all using the CSP concept for connection and feature transfer. After inputting the image, it is downsampled five times sequentially to generate five features at different scales (P1-P5). After each downsampling, the features are transferred to the C2f module, and the CBS module performs convolution, batch normalization, and SiLU activation on the input information to obtain the output result. Finally, the Spatial Pyramid Pooling Fast (SPPF) module is used for pooling with three consecutive max-pooling layers with residual structures to obtain a fixed-size graph adaptive dimensionality output, which helps capture information at different scales. 2) Neck: YOLOv8 adopts a PAN-FPN neck network structure, fully drawing on the design ideas of PANet and improving upon the YOLOv8 model. The task of the neck network is to fuse multi-scale features to form a feature pyramid, thereby improving the performance of object detection. 3) Head: Compared to YOLOv5, the head design of YOLOv8 has undergone significant changes, changing from a coupled head to a decoupled head structure, separating the classification head and the detection head. The decoupled head structure means using two independent branches for object classification and bounding box regression prediction, with each branch using a different loss function. These improvements have enabled YOLOv8 to achieve significant performance improvements in object detection tasks.

[0053] This invention introduces feature maps from a small object detection layer into the neck network to enhance the combination of deep and shallow semantic information, thereby improving the accuracy of small object detection. It also adds a detection head corresponding to the small object detection layer. The number of channels in the neck structure is adjusted to reduce the model's computational load and memory consumption.

[0054] The invention employs the MSFB module, combined with other modules. MSFB enhances the neural network's ability to perceive and represent information at different levels through multi-scale feature fusion. This module includes two convolution operations and a custom feature fusion operation to facilitate more effective information integration across different scales.

[0055] The invention employs an HFF module to enhance the neural network's ability to express multi-scale information. This module uses an adaptive 3D feature fusion strategy to capture spatial and channel relationships in the input features. By introducing residual connections and global pooling, the receptive field and feature representation of the model are effectively increased.

[0056] This invention replaces the original CIoU (Complete Intersection over Union) loss function with the WIoU (Weighted Intersection over Union) loss function in the bounding box regression loss and TAL label assignment strategy. WIoU employs a dynamic non-monotonic mechanism to design a more reasonable gradient gain allocation strategy, effectively reducing gradient gain and enhancing the model's localization performance and generalization ability. A small object detection head is added to the head, combining shallow and deep feature fusion to enhance the perception of small objects.

[0057] A global pruning method based on importance score LAMP is used to prune the model. This method outperforms other methods in layer sparsity selection based on layer adaptive magnitude, and can reduce model complexity, improve inference speed, and reduce storage and computation costs in resource-constrained environments while maintaining model performance.

[0058] Please see Figure 1-12 A lightweight MP-YOLO dense road target detection method in complex environments, including a YOLOv8 model, includes the following steps:

[0059] Step 1: Obtain several dense road target images. The dense road target dataset covers road scenes with different lighting, different degrees of occlusion, and different backgrounds. Each image includes targets such as motor vehicles, non-motor vehicles, and pedestrians.

[0060] Step 2: Improve the YOLOv8 model by using the MSFB module, HFF module, small object detection head, WIoU loss function, and LAMP pruning method to obtain the MP-YOLO model. Specifically, this includes: introducing a 160×160 feature map on the backbone into the neck network of the YOLOv8 model, fusing feature maps of different scales through the MSFB and HFF modules, adding a small object detection head to the head network of the YOLOv8 model, replacing the CIoU loss function in the YOLOv8 model with the WIoU loss function, and then pruning the model using the LAMP pruning method after the replacement to obtain the MP-YOLO model.

[0061] Specifically, the MSFB module's principle originates from the SSFF module proposed in the ASF paper. Most existing network architectures only use summation or concatenation to fuse pyramid features when using a feature pyramid structure. This approach may lead to information loss or redundancy, especially when dealing with complex or small objects, and may not fully utilize the correlation between feature maps at different scales. Following the idea of ​​the SSFF module, this paper proposes the MSFB module, such as... Figure 3As shown. In the neck region, this invention uses two MSFB modules to process feature maps at three different scales, constructing a scale space for each. Along the scale axis of the image, feature maps of different scales are horizontally stacked to form a feature map sequence. Then, a three-dimensional convolution operation is used to process the stacked feature map sequence to extract scale sequence features. This three-dimensional feature map contains feature information at different scales, thereby improving the accuracy and robustness of object detection. Specifically, the feature maps in the scale space are generated by applying a series of convolutions to feature maps of different sizes using a 2D Gaussian filter Gσ(w, h). These generated images have different aspect ratios but the same resolution. Subsequently, the MSFB module can adjust the effective feature maps of different resolutions to the same resolution for cascading.

[0062] Its specific expression is as follows:

[0063]

[0064]

[0065] in, This represents the feature map data processed by a Gaussian filter kernel. Represents feature maps of different sizes from the input. Indicates width. Indicates high, Represents the natural constant. This represents the Gaussian filter function. The standard deviation of the Gaussian filter determines the smoothness of the filter.

[0066] Taking the detailed process of the first MSFB module as an example, the specific process is as follows:

[0067] (1) Three feature maps of different sizes are extracted from the backbone, denoted by L, M, and S respectively. These three feature maps are downsampled using a two-dimensional Gaussian filter as the convolution kernel. This kernel can calculate the mean and variance to make the Gaussian two-dimensional distribution between pixels in the image more uniform, thereby smoothing the image resolution. This results in feature maps of different sizes but the same resolution, allowing for better matching of contextual features at different scales. Applying this to the image yields a clearer and smoother image, while also removing noise.

[0068] (2) Use the nearest neighbor interpolation method to adjust the size of different feature maps. Find the size of the feature map with the most information among the three scales as the target size (generally the feature map with the largest scale). Represent the target size as (X,Y). Adjust the scale of the feature maps of the three scales to be consistent. Based on the previously obtained Convσ(w,h), the interpolation formula is expressed as follows:

[0069]

[0070] Formula explanation: where (xi, yi) represents the image coordinates, and (X, Y) represents the length and width of the reference.

[0071] (3) Add a depth dimension to the three processed feature maps above, and then stitch them together according to the depth dimension to obtain the 3D feature map as follows:

[0072]

[0073] Then, 3D convolution is used to process its features to obtain the dimension of the convolutional feature map, as shown in the following formula:

[0074]

[0075] Where Conv(C,W,H) represents the dimension of the feature map after 3D convolution.

[0076] (4) Use batch normalization and activation functions on the above results to extract scale sequence features to facilitate further extraction of deeper features;

[0077] The HFF module is inspired by the Triple Feature Encoding (TFE) module proposed in the ASF paper. YOLOv8's original neck network used Feature Pyramid Network (FPN), employing a top-down feature propagation approach. This mechanism downsamples high-level features, resulting in significant information loss in small targets and neglecting rich details in large-scale features. To address this issue, this invention proposes a Hierarchical Feature Fusion (HFF) module, aiming to effectively fuse features from different scales (i.e., large, medium, and small sizes). Figure 2As can be seen, two HFF modules are used in the neck network to fuse feature maps of different sizes in the backbone. The feature maps concatenated by the HFF modules integrate the local details of the shallow feature maps and the global semantic information of the deep feature maps in the backbone, and are then passed to the subsequent feature extraction modules. This design can improve the performance and generalization ability of the model. When a 640*640 image is input into the network, the backbone extracts feature maps from shallow to deep layers, which are divided into 160*160, 80*80, 40*40, and 20*20 pixels, and defined as P2, P3, P4, and P5. Then, two HFF modules, named HFFa and HFFb respectively, are designed to concatenate feature maps of different sizes. The formula is expressed as follows:

[0078]

[0079] in, This represents the feature map output by the HFF module. These represent three different sizes of feature maps. This indicates feature map splicing;

[0080] The specific workflow is as follows: (1) The HFFa module denotes the feature maps of three different scales, 80*80, 40*40, and 20*20, as follows: , , Set the mid-layer feature map. Adjust as target size and The feature map represents the target size. Feature map processing: A hybrid structure of max pooling and average pooling is used to downsample large-scale feature maps, effectively preserving high-resolution features. Feature map processing: The nearest neighbor interpolation method is used to upsample the small-sized feature map to maintain the richness of local features.

[0081] Feature fusion: The three processed feature maps are concatenated along the channel dimensions to form the output feature map of the HFFa module, with a size of 40*40. The formula is as follows: F=Concat(FL,FM,FS), where F represents the feature map output by the HFF module. These represent feature maps of large, medium, and small sizes, respectively. F is... It is connected in series. The resolution of F and The same, but the number of channels is three times that of the original. Repeat steps (1) and (2) to obtain the output of HFFb by taking the spliced ​​feature map, the 80*80 and 160*160 feature maps of the backbone obtained by HFFa;

[0082] The original YOLOv8 network structure only has three detector heads. In practical engineering applications, the neck network lacks a small object detection layer (P2), resulting in poor performance in detecting small objects. This paper introduces feature maps for a small object detection layer from the P2 layer of the backbone network. Then, through a Concat and Add structure, this invention merges these features with the 3D feature maps processed by two MSFB modules, achieving the fusion of shallow and deep features, such as... Figure 1 As shown by the red line. After fusing the above features, this invention designs a small target detection head, DetectB, with a detection size of 160x160. Figure 1 The principles of DatectA, B, C, and D are the same. DetectB is a newly added small target detection head in this paper. It receives the backbone feature map output fused by the neck network. This feature map contains richer information, which helps the model improve its detection performance for small targets. The detection head uses two independent convolutional layer sequences, cv2 and cv3, to predict bounding box parameters and class probabilities, respectively. Figure 6 As shown, cv2 consists of two 3x3 convolutional layers and one 1x1 2D convolutional layer. First, a 3x3 convolutional kernel is used to extract features, and then a 1x1 convolutional kernel is used to predict bounding box parameters. cv3 has a similar structure to cv2, but it uses a 1x1 convolutional kernel to predict the probability of each target's class. The detection head then fuses the bounding box parameters and class probabilities and outputs the prediction results. The addition of a small target detection head enhances the model's ability to predict and regress small targets. Although it increases the model's computational complexity and memory usage, it significantly reduces the model's false negative and false positive rates.

[0083] In deep learning, the loss function plays a crucial role in training models, optimizing parameters, and evaluating performance. YOLOv8 uses the CIoU loss function for bounding boxes, which calculates the impact of the predicted box on the loss function based on the aspect ratio, center distance, and overlap area between the predicted box and the ground truth box. This makes it easier for the model to capture the precise shape of the target. Furthermore, the CIoU loss function considers the diagonal distance of the target box, making it more sensitive to the position prediction of the target box. The relevant formula for CIoU is shown below:

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] b and bgt represent the center points of the predicted bounding box and the ground truth bounding box, respectively; ρ represents the Euclidean distance between the two center points; c represents the diagonal length of the smallest enclosing rectangle of the ground truth bounding box and the predicted bounding box; α is a weighting function; and ν is used to measure the consistency of aspect ratio. The second formula above is the weighted penalty term in the CIoU loss function, used to penalize the positional and size differences between the predicted bounding box and the ground truth bounding box. However, when the aspect ratio of the predicted box meets certain conditions, such as when the aspect ratio of the predicted box is the same as that of the ground truth bounding box, the aspect ratio penalty term remains at 0, causing the CIoU penalty function to degenerate and hindering the convergence of the model. From the above formula, it can be observed that the gradients of w and h with respect to ν in CIoU are opposite, indicating that w and h cannot be increased or decreased simultaneously. Therefore, although CIoU solves the bounding box matching problem, it cannot effectively handle low-quality images and emphasizes the penalty term, resulting in a decrease in the model's generalization ability, leading to a significant impact of low-quality annotations on detection performance. Therefore, this invention introduces the Wise-IoU (WIoU) loss function. WIoU utilizes a dynamic non-monotonic focusing mechanism called "outlier values" to evaluate anchor box quality, rather than IoU, and provides a smart gradient gain allocation strategy. This allows WIoU to focus on anchor boxes of normal quality and improve the overall performance of the detector. There are three versions of WIoU in total. WIoU v1 constructs an attention-based bounding box loss. Then, WIoU v2 and v3 enhance this by introducing a method for calculating gradient gain (focusing coefficients) to increase the focusing mechanism. Considering that low-quality samples in the training data are penalized based on geometric metrics such as distance and aspect ratio, thus reducing the model's generalization performance, WIoUv1 constructs distance attention based on distance metrics, giving WIoUv1 a two-layer attention mechanism. The formula is shown below:

[0090]

[0091]

[0092] Distance Attention Coefficient Multiply by IoU loss We obtain the above equation, and thus... loss function In the above formula, x, y, xgt, and ygt represent the center point of the predicted bounding box and the center point of the ground truth box, respectively. Distance weights are calculated by determining the distance between these two center points and the width and height of the ground truth box. To prevent gradient hindrance in RWIoU, Wg and Hg are separated from the computation graph (indicated by the superscript *). WIoUv2 constructs monotonic focusing coefficients for LWIoU v1, enabling the model to focus on challenging samples and improve classification performance. The calculation formula for WIoUv2 is shown below:

[0093]

[0094] In the above formula, LIoU acts as a normalization factor to maintain the gradient gain at a high level, addressing the problem of slow convergence in the later stages of training. WIoUv3 introduces a dynamic non-monotonic focusing coefficient. The formula is as follows:

[0095]

[0096]

[0097] The formula defines the outlier value β to describe the quality of the anchor box; the smaller the β, the higher the quality of the anchor box. In the above formula, r represents the non-monotonic focusing coefficient constructed from β. This invention applies r to WIoUv1 to dynamically assign gradient gain to anchor boxes with different outlier values, effectively preventing low-quality samples from generating significant harmful gradients. In summary, this invention considers WIoUv3 to be a novel bounding box loss function because it may be more suitable for training data containing low-quality samples and requiring better generalization performance. WIoUv3 can effectively handle low-quality samples and dynamically adjust gradient gain, thereby improving the model's performance and generalization ability.

[0098] The final distance focusing mechanism Non-monotonic focusing coefficient The expression is as follows:

[0099]

[0100] in, For the outlier measure of anchor box mass, This represents an exponential function with the natural constant as its base. , , , These represent the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. and The distribution represents the width and height of the ground truth box, with the superscript * indicating that... and Separate from the computational graph, Indicates outlier values. Represents the weighting function. Let r be the learning parameter, which satisfies the condition that r = 1 when β = δ.

[0101] Deep neural networks have been widely used in the development of autonomous driving. However, these models typically require significant computational resources and storage space. Developing lightweight deep neural network models that can be deployed on mobile edge devices is crucial. Currently, researchers mainly employ two methods to develop lightweight models: one is to replace the backbone network and modules with lightweight components, and the other is to simplify the model. Among these methods, pruning provides a way to achieve lightweight models without sacrificing accuracy. This paper focuses on pruning as a method for developing lightweight models.

[0102] In neural networks, the weights of each connection are crucial to network performance. Importance-based pruning methods focus on the importance of convolutional kernels and intra-layer feature maps, and strive to maintain the integrity of the original network as much as possible during the pruning process. Amplitude-based layer adaptive LAMP scoring determines which connections to retain or prune by considering the relative importance of connection weights, helping to preserve important features and information in the model and avoiding performance degradation or failure caused by pruning. Therefore, this paper adopts an amplitude-based layer adaptive LAMP importance score method for model pruning.

[0103] The LAMP importance score is derived from the weight tensors of each network layer in the model. Each weight is assigned an index to that layer. All network layers are then sorted in ascending order according to the given index graph, resulting in a weight sequence W[i] <= W[j] (j > i), where W[i] is obtained by mapping the weight indices, and there are a total of j weights. The formula for obtaining the LAMP score in this invention is as follows:

[0104]

[0105] in, In a neural network, the first... The weight of each connection, This indicates the distribution of weights across the entire layer. Indicates the first The weight tensor of each connection relative to the weight of each connection in the neural network. LAMP score;

[0106] By comparing the numerator and denominator, LAMP scores accurately assess the relative importance of each connection within a layer. In summary, LAMP scores provide an effective pruning guide through detailed analysis of connection weights, promoting network sparsity to improve the efficiency and generalization ability of neural networks. Its simple computation method makes it easy to perform practical network pruning tasks, providing necessary guidance for model optimization and compression.

[0107] This invention encountered a problem during the pruning process. Because LAMP performs global pruning, pruning the detector head portion of the YOLOv8 model leads to a significant decrease in model accuracy. Therefore, this paper modifies the pruning strategy by setting up layer skipping for the detector head portion. A schematic diagram of model pruning and layer skipping is shown below. Figure 5 As shown;

[0108] The pruning process is as follows:

[0109] (1) Model training: First, the YOLOv8 basic model is improved using the previous modules. The MSFB and HFF modules are added to the neck network to add a detection head for small targets. Then, the WIOU loss function is used to improve the model. The integrated model is then trained until it converges.

[0110] (2) Model pruning: The LAMP score-based layer adaptive pruning method is used to prune the model. During the pruning process, the importance level of the channels is evaluated based on the LAMP score, and filters with low importance are removed. When encountering cv2 and cv3, which are part of the detection head,

[0111] The modules will be automatically skipped, and the final result is the lightweight MP-YOLO model.

[0112] Model fine-tuning is necessary because the accuracy of the pruned model drops significantly, requiring fine-turn training of the MP-YOLO model to restore its accuracy. The pruning algorithm is shown below.

[0113]

[0114] Step 3: Input several road target images into the MP-YOLO model for training to obtain the MP-YOLO dense road target detection model;

[0115] Step 4: Input the image to be identified into the MP-YOLO dense road target detection to obtain the dense road target detection result of the image to be identified.

[0116] The present invention also provides a lightweight MP-YOLO dense road target detection system in complex environments, based on the above-mentioned lightweight MP-YOLO dense road target detection method in complex environments, including: an image acquisition module for acquiring several dense road target images;

[0117] The model improvement module is used to improve the YOLOv8 model by introducing the MSFB module, HFF module, small object detection head, WIoU loss function and LAMP pruning method to obtain the MP-YOLO model.

[0118] The model training module is used to input several dense road target images into the MP-YOLO model for training, thereby obtaining the MP-YOLO dense road target detection model.

[0119] The dense road target detection module is used to input the image to be identified into the MP-YOLO dense road target detection model to obtain the dense road target detection result of the image to be identified.

[0120] Experimental verification:

[0121] The experimental environment configuration is as follows: the operating system is Windows Step 11 (Professional Edition), the CPU is an Intel Xeon(R) Bronze 3204, and the graphics card is a GeForce RTX 3080 GPU 10G. The development language is Python 3.10, the deep learning framework is PyTorch 1.13.1, the CUDA version is 12.3, and the cuDNN version is 10.0.

[0122] The parameters for this experiment are set as follows: input image size is 640×640, batch size is 16, initial learning rate is 0.01, decay factor is 0.0005, optimizer is SGD, and training epochs are 300. All other parameters are the default parameters from the official YOLOv8 documentation.

[0123] The dataset used in this experiment is the publicly accessible DAIR-V2X dataset. The DAIR-V2X dataset is the first large-scale, multi-modal, multi-view dataset used for research on cooperative autonomous driving. It includes four types of vehicles: cars, trucks, vans, and buses; three types of non-motorized vehicles: bicycles, tricycles, and motorcycles; and pedestrians. This invention extracted 10,768 2D-annotated images. The dataset was first divided into training, testing, and validation sets in a 7:2:1 ratio. Then, the poorly performing parts of the dataset were removed, resulting in a training set of 7,877 images, a testing set of 2,015 images, and a validation set of 876 images.

[0124] To verify the performance of the method proposed in this invention, precision, recall, average precision (mAP), floating-point operations (FLOPs), and inference speed are used as evaluation metrics. Precision is the proportion of correctly classified positive samples out of all detected positive samples, and recall is the proportion of detected positive samples. The specific formulas for precision and recall are as follows:

[0125]

[0126]

[0127] In the formula, TP represents the number of true positive samples, FP represents the number of false positive samples, and FN represents the number of false negative samples. The formula for calculating mAP is:

[0128]

[0129]

[0130] Where n is the number of classes. This represents the average precision of class i. In this invention, mAP @0.5 represents the average precision with an IoU threshold of 0.5, and mAP @0.5: 0.9 represents the average precision with an IoU threshold range of 0.5 to 0.95, with a step size of 0.05. Furthermore, for simplicity, this invention uses AP50 and AP to represent mAP @0.5 and mAP @0.5: 0.9, respectively.

[0131] The model was trained and validated using the experimental dataset described above. The experimental results comparing different models are shown in Table 1 below.

[0132]

[0133] The experiments above show that compared to the YOLOv3-Tiny model, AP50 and AP are improved by 21.6% and 16.4%, respectively. The number of model parameters is reduced from 12.1 million to 940,000, resulting in a speed increase of approximately 5 times. Compared to YOLOv5, MP-YOLO has only one-ninth of the parameters and more than twice the FPS. Compared to YOLOv5n, MP-YOLO's AP50 and AP are improved by 5.4% and 5.3%, respectively. Compared to YOLOv6-S, the method of this invention improves AP50 by 10.7% and AP by 10.9%, with the number of model parameters being 1 / 18 of YOLOv6-S. Compared to YOLOv7-Tiny, the method of this invention is 4.7% faster in AP50 and 7.6% faster in AP, approximately three times faster than YOLOv7-Tiny. Compared to YOLOv8s, MP-YOLO has only 1 / 11 of the parameters of YOLOv8s, but its AP value is similar to that of YOLOv8s. Compared to the base model YOLOv8n, MP-YOLO improves ap50 and AP by 4.7% and 4.2% respectively, while reducing the number of parameters by two-thirds. MP-YOLO also significantly improves the accuracy of detecting small targets. This invention replaces the original YOLOv8 backbone with two lightweight backbones. The v8n+MobileNetv3 model reduces the number of parameters by about one-third, but AP50 decreases by 4.2%, AP decreases by 4.7%, and FPS decreases by two-thirds. Similarly, the v8n+GhostNet model reduces the number of parameters by about one-third, but AP50 decreases by 5.5%, AP decreases by 6.1%, and FPS decreases by one-third. In contrast, with the same number of parameters, MP-YOLO's ap50 value is 8.4% and 10.2% higher, respectively. In conclusion, in autonomous driving scenarios, MP-YOLO demonstrates more balanced and superior performance in detecting complex and small objects on the road.

[0134] To verify the effectiveness of our model, we used the SODA10M dataset for comparative experiments. SODA10M is a dataset for autonomous driving jointly released by Huawei Noah's Ark Lab and Sun Yat-sen University. SODA10M is a large-scale two-dimensional dataset containing six categories of human-vehicle scenarios: pedestrians, bicycles, cars, trucks, trams, and tricycles. The segmented dataset includes 6964 training sets, 995 validation sets, and 1990 test sets. Using the same environment configuration, training parameters, and evaluation metrics as described above, we only compared the results of MP-YOLO and YOLOv8n on the SODA10M dataset, as shown in Table 2.

[0135]

[0136] Table 2 shows the comparative experiments on the SODA10M dataset.

[0137] As can be seen from the above experiments, compared with the basic model YOLOv8n of this invention, MP-YOLO's AP50 is improved by 4.1%, AP is improved by 2.9%, parameters are reduced by two-thirds, FLOPs value is reduced from 8.1G to 7G, and FPS is increased from 105.3 to 156.3, which improves the detection speed by about one-third.

[0138] To visually demonstrate the performance gap between the MP-YOLO and YOLOv8-n basic models, this invention... Figure 9 The paper visually illustrates the performance gap between the MP-YOLO and YOLOv8-n basic models. This invention... Figure 9 The image shows evaluation metrics for the two models, displaying their accuracy, recall, and mAP curves after 300 training iterations. The x-axis represents the number of iterations, and the y-axis represents the evaluation metrics. The blue curve represents the MP-YOLO training process, which is the result of retraining after pruning. Figure 7 As can be seen, the blue curve of MP-YOLO, representing recall and mAP, is consistently better than the orange curve representing the basic YOLOv8n model, indicating that the MP-YOLO model of this invention achieves higher detection accuracy.

[0139] To further verify its effectiveness, a series of ablation experiments were conducted to investigate the impact of each MP-YOLO module on model performance. The experimental results are shown in Table 3 below.

[0140]

[0141] Table 3

[0142] The above experiments show that (1) replacing CIoU with WIOU increases AP50 by 0.6% and AP by 0.2%, without requiring additional computation. (2) using MSFB and HFF modules in the neck increases AP50 by 0.4% and FLOPs by 8.5G. Although the computational cost increases slightly, the improvement in AP50 proves the effectiveness of these modules in feature fusion. (3) The small target detector increases AP50 by 5.7%, AP by 5.8%, and FLOPs by 12.2 compared to the baseline. Although FPS decreases slightly, the detection accuracy is significantly improved. (4) After pruning the model with LAMP, AP50 decreases by 1.5%, but the model size is reduced by about two-thirds. (5) replacing the CIoU loss function with WIOU and using MSFB and HFF modules in the neck increases AP50 by 1.3% and AP by 1.0. (6) replacing the CIoU loss function with WIOU and adding a small target detector increases AP50 by 5.6% and AP by 6.0%. Compared with the improved (3), AP increased by 0.2%, indicating that the improved loss function is effective. (7) In the improvement process, the MSFB and HFF modules were combined with the small target detection head, reducing the number of channels. AP50 increased by 5.4%, AP increased by 5.8%, and the model size was reduced by one-sixth. (8) Based on the combination of MSFB and HFF modules and the small target detection head, this invention uses the WIOU loss function to increase AP50 and AP by 5.9% and 6.0% respectively, while reducing the model size from 6M to 5.1M. That is, the detection accuracy is improved and the model size is slightly reduced;

[0143] Finally, based on the previous improvements, this invention uses the LAMP pruning method to prune the model, proposing the MP-YOLO model. Compared to the basic YOLOv8-n model in AP50, the MP-YOLO model improves performance by 4.7% and AP by 4.2%, while its size is only one-third that of the basic model. It achieves 294.12 FPS on an RTX 3080, meeting the real-time detection speed requirements of real-world roads. The significant reduction in model size, relative to the slight loss in accuracy, allows the model to be embedded in edge devices with smaller storage spaces, contributing to the development of intelligent transportation and autonomous driving.

[0144] To verify the improvement effect of MP-YOLO, this invention performs visualization analysis on the model's detection results, demonstrating the detection performance of MP-YOLO on different datasets and in different environments, such as... Figure 7 , Figure 8As shown; to verify the detection performance of the MP-YOLO and YOLOv8n algorithms in real-world scenarios, this experiment compares the visualization effects of YOLOv8n and MP-YOLO algorithms in street scenes during both daytime and nighttime, including dark scenes affected by lighting, target occlusion, and other complex background scenes, such as... Figure 10 As shown.

[0145] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations. The above-described embodiments of this invention do not constitute a limitation on the scope of protection of this invention.

Claims

1. A lightweight MP-YOLO dense road target detection method in complex environments, including a YOLOv8 model, characterized in that: Includes the following steps: Step 1: Acquire several images of dense road targets; Step 2: Improve the YOLOv8 model by adding a small target detection head, modifying the neck network to add MSFB and HFF modules that fuse multi-scale features, and replacing the original CIOU loss with WIOU loss; use the LAMP method to prune the model to obtain the MP-YOLO model. Step 3: Input several road target images into the MP-YOLO model for training to obtain the MP-YOLO dense road target detection model; Step 4: Input the image to be identified into the MP-YOLO dense road target detection system to obtain the dense road target detection results of the image to be identified; The specific steps of step 2 are as follows: a 160×160 feature map on the backbone is introduced into the neck network of the YOLOv8 model, and feature maps of different scales are fused through the MSFB module and HFF module. A small target detection head with a detection size of 160*160 is added to the head network of the YOLOv8 model. The CIoU loss function in the YOLOv8 model is replaced with the WIoU loss function. After the replacement, the LAMP pruning method is used to prune the model to obtain the MP-YOLO model. The MP-YOLO model consists of two MSFB modules and two HFF modules. The first HFF module first processes feature maps of three different scales: 80*80, 40*40, and 20*20. , and Set with mid-layer feature map Adjust as target size and The feature map is the target size, and a hybrid structure of max pooling and average pooling is used to process the feature map. Downsampling is performed, and the nearest neighbor interpolation method is used to modify the feature map. Upsampling is performed; then the three processed feature maps are stitched together along the channel size to form the output feature map of the first HFF module, which is 40*40 in size; The second HFF module repeats the processing steps of the first HFF module with the output feature map of the first HFF module and the 80*80 and 160*160 feature maps on the backbone to obtain the output feature map of the second HFF module, which is 80*80 in size. The first MSFB module first extracts feature maps of three different scales (80*80, 40*40, and 20*20) from the backbone. These feature maps are then downsampled using a two-dimensional Gaussian filter as the convolution kernel. Next, nearest neighbor interpolation is used to adjust the size of the three feature maps. A depth dimension is then added to the three feature maps, and they are concatenated along this depth dimension to obtain a 3D feature map. 3D convolution is then used to process the 3D feature map, resulting in the dimension of the convolutional feature map. Finally, batch normalization and activation functions are used to extract scale sequence features, forming the output feature map of the first MSFB module. The second MSFB module extracts the feature map resulting from the concatenation of the output feature map of the second HFF module and the output feature map of the first MSFB module, the output feature map of the first HFF module, and the 160*160 feature map on the backbone. The processing steps of the first MSFB module are repeated to obtain the output feature map of the second MSFB module. The feature map resulting from the concatenation of the output feature map of the second HFF module and the output feature map of the first MSFB module is fused with the 160*160 feature map on the backbone, and then concatenated with the output feature map of the second MSFB module as the input of the small target detection head with a detection size of 160*160.

2. The lightweight MP-YOLO dense road target detection method in complex environments according to claim 1, characterized in that: The images in step 1 all include motor vehicles, non-motor vehicles, and pedestrians.

3. The lightweight MP-YOLO dense road target detection method in complex environments according to claim 1, characterized in that: The MSFB module is used to process feature maps at different scales; its specific expression is as follows: in, This represents the feature map data processed by the Gaussian filter kernel. Representing feature maps of different sizes from the input, Indicates width. Indicates high, Represents the natural constant. This represents the Gaussian filter function. This represents the standard deviation of the Gaussian filter.

4. The lightweight MP-YOLO dense road target detection method in complex environments according to claim 1, characterized in that: The expression processed by the HFF module is as follows: in, This represents the feature map output by the HFF module. These represent three different sizes of feature maps. This indicates feature map splicing.

5. The lightweight MP-YOLO dense road target detection method in complex environments according to claim 1, characterized in that: The expression for the WIoU loss function is as follows: Among them, L WIoU Let R represent the WIoU loss function. WIoU This indicates a distance-focusing mechanism. Indicates the non-monotonic focusing coefficient. Represents the dynamic moving average. Represents the natural constant.

6. The lightweight MP-YOLO dense road target detection method in complex environments according to claim 5, characterized in that: The distance focusing mechanism R WIoU Non-monotonic focusing coefficient The expression is as follows: in, For the outlier measure of anchor box mass, This represents an exponential function with the natural constant as its base. , , , These represent the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. and The distribution represents the width and height of the ground truth box, with the superscript * indicating that... and Separate from the computational graph, Indicates outlier values. Represents the weighting function. Indicates the learning parameters.

7. The lightweight MP-YOLO dense road target detection method in complex environments according to claim 1, characterized in that: The LAMP pruning method described above uses an amplitude-based layer adaptive LAMP importance score method to prune the model, and its expression is as follows: in, In a neural network, the first... The weight of each connection, This indicates the distribution of weights across the entire layer. Indicates the first The weight tensor of each connection relative to the weight of each connection in the neural network. LAMP score.

8. A lightweight MP-YOLO dense road target detection system in complex environments, based on the lightweight MP-YOLO dense road target detection method in complex environments as described in any one of claims 1-7, characterized in that: include: The image acquisition module is used to acquire several images of dense road targets; The model improvement module is used to improve the YOLOv8 model by introducing the MSFB module, HFF module, small object detection head, WIoU loss function and LAMP pruning method to obtain the MP-YOLO model. The model training module is used to input several dense road target images into the MP-YOLO model for training, thereby obtaining the MP-YOLO dense road target detection model. The dense road target detection module is used to input the image to be identified into the MP-YOLO dense road target detection model to obtain the dense road target detection result of the image to be identified.

Images