Traffic scene target detection method based on lightweight YOLOv5

Abstract

The invention provides a traffic scene target detection method based on lightweight YOLOv5, and the method comprises the steps: constructing a traffic scene data set, carrying out the preprocessing and data set division of the traffic scene data set, obtaining a training set, a verification set and a test set, carrying out the lightweight processing of a YOLOv5 network, and constructing a traffic scene target detection model based on the lightweight YOLOv5 network. And inputting the training set, the verification set and the test set into a traffic scene target detection model for training, testing and verification to obtain a final traffic scene target detection model, and inputting a to-be-detected image into the final traffic scene target detection model to obtain a detection result. By improving the model structure and applying the optimization strategy, the calculation amount and the parameter amount of the model are reduced, the real-time detection performance in the resource-constrained environment is improved, and meanwhile high detection precision is kept.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a traffic scene target detection method based on lightweight YOLOv5. Background Art

[0002] With the rapid development of autonomous driving, intelligent transportation and smart cities, object detection in traffic scenes has become increasingly important, such as the detection of vehicles, pedestrians, bicycles, etc. The accuracy and real-time performance of these object detection tasks are directly related to traffic safety and system effectiveness.

[0003] YOLO (You Only Look Once) series algorithms have been widely used in the field of computer vision due to their efficient detection speed and good detection accuracy. YOLOv5 is a version of the YOLO series algorithm. The YOLOv5 model has a large number of parameters and high computational complexity. When deployed on resource-constrained embedded devices or mobile devices, there are problems such as high memory usage and insufficient inference speed. Therefore, how to make lightweight improvements to YOLOv5 so that it can adapt to low-resource application scenarios while ensuring detection accuracy is a technical problem that needs to be solved urgently. Summary of the invention

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A traffic scene target detection method based on lightweight YOLOv5, comprising:

[0006] Construct a traffic scene dataset;

[0007] Preprocess the traffic scene dataset and divide the dataset into training set, validation set and test set;

[0008] Lightweight the YOLOv5 network and build a traffic scene target detection model based on the lightweight YOLOv5 network;

[0009] Inputting the training set, the validation set and the test set into a traffic scene object detection model for training, testing and verification to obtain a final traffic scene object detection model;

[0010] The image to be detected is input into the final traffic scene target detection model to obtain the detection result.

[0011] Preferably, a traffic scene dataset is constructed, specifically:

[0012] The KITTI dataset, PASCAL VOC dataset, and COCO dataset are obtained. Some images are filtered and extracted from the three datasets according to six categories: car, person, bus, bicycle, train, and motorcycle, to obtain a traffic scene dataset.

[0013] Preferably, the traffic scene data set is preprocessed and divided into a training set, a validation set and a test set, specifically:

[0014] Perform data cleaning on traffic scene datasets;

[0015] Perform image enhancement on the traffic scene dataset after data cleaning;

[0016] The traffic scene dataset after image enhancement is divided into training set, validation set and test set according to the preset ratio.

[0017] Preferably, data cleaning is performed on the traffic scene dataset, specifically:

[0018] Get all the images in the traffic scene dataset and classify them into six categories: car, person, bus, bicycle, train, and motorcycle.

[0019] Train an encoder based on the preprocessed image, and extract a feature vector of the image based on the trained encoder;

[0020] Cluster the images based on the feature vectors and calculate the similarity between the images of the same type one by one;

[0021] Remove similar images from the same category based on the similarity between the images.

[0022] Preferably, the YOLOv5 network is lightweight, specifically:

[0023] Lightweight improvements to the backbone network of the YOLOv5 network;

[0024] The neck network of the YOLOv5 network is lightened by improving it. Specifically, BiFPN is introduced into the neck network of the YOLOv5 network to replace the splicing module.

[0025] The attention mechanism is introduced into the lightweight and improved YOLOv5 network.

[0026] Preferably, the backbone network of the YOLOv5 network is lightweight improved, specifically:

[0027] Combine MobileNetV3-Small with the C3 module of the backbone network of the YOLOv5 network into the MobileC3 module, and make lightweight improvements to the backbone network of the YOLOv5 network.

[0028] Preferably, the backbone network of the YOLOv5 network is lightweight improved, specifically:

[0029] The C3 module of the GhostNetV2 and YOLOv5 backbone networks is combined into the C3GhostV2 module to make lightweight improvements to the backbone network of the YOLOv5 network.

[0030] Preferably, the backbone network of the YOLOv5 network is lightweight improved, specifically:

[0031] The Swin Transformer is combined with the C3 module of the backbone network of the YOLOv5 network into the C3STR module to make lightweight improvements to the backbone network of the YOLOv5 network.

[0032] Preferably, the attention mechanism is introduced into the lightweight improved YOLOv5 network, specifically:

[0033] The CA attention mechanism is introduced into the lightweight and improved YOLOv5 network.

[0034] Preferably, the attention mechanism is introduced into the lightweight improved YOLOv5 network, specifically:

[0035] The EMA attention mechanism is introduced into the lightweight and improved YOLOv5 network.

[0036] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0037] The present invention provides a traffic scene target detection method based on lightweight YOLOv5, the method comprising: constructing a traffic scene data set, preprocessing and dividing the traffic scene data set to obtain a training set, a verification set and a test set, lightweight processing the YOLOv5 network, and constructing a traffic scene target detection model based on the lightweight YOLOv5 network, inputting the training set, the verification set and the test set into the traffic scene target detection model for training, testing and verification, obtaining a final traffic scene target detection model, and inputting an image to be detected into the final traffic scene target detection model to obtain a detection result. The present invention reduces the amount of calculation and the amount of parameters of the model by improving the model structure and applying the optimization strategy, improves the real-time detection performance in a resource-constrained environment, and maintains a high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the MobileNetV3 network;

[0040] Figure 3 Schematic diagram of standard convolution and depth-separable convolution structure;

[0041] Figure 4 Schematic diagram of the deep convolution process;

[0042] Figure 5 Schematic diagram of the point-by-point convolution process;

[0043] Figure 6 Schematic diagram of depth-wise separable convolution;

[0044] Figure 7 It is a schematic diagram of a linear bottleneck structure;

[0045] Figure 8 This is the schematic diagram of the MobileC3 module;

[0046] Fig. 9 YOLOv5s framework diagram of introducing MobileC3 into the backbone network;

[0047] Fig.10 This is a schematic diagram of the Ghost Module operation process;

[0048] Fig.11 This is a schematic diagram of the GhostNetV2 Bottleneck structure;

[0049] Fig.12 This is the schematic diagram of the C3GhostV2 module;

[0050] Fig.13 Schematic diagram of YOLOv5s framework with C3GhostV2 introduced as the backbone network;

[0051] Fig.14 This is a schematic diagram of the overall architecture of Swin Transformer;

[0052] Fig.15 Schematic diagram of image block segmentation operation;

[0053] Fig.16 It is the schematic diagram of SW-MSA module;

[0054] Fig.17 It is a schematic diagram of the C3STR module structure;

[0055] Fig.18 Schematic diagram of YOLOv5s framework with C3STR introduced as the backbone network;

[0056] Fig.19 Schematic diagram of the structure for fusion of multiple multi-scale features;

[0057] Fig. 20 Schematic diagram of multiple BiFPN modules;

[0058] Fig.21 Schematic diagram of YOLOv5s framework introducing BiFPN for Neck network;

[0059] Fig. 22 Schematic diagram of the CA attention module;

[0060] Fig.23 Schematic diagram of EMA attention module;

[0061] Fig.24 Schematic diagram of the YOLOv5s network framework with CA attention introduced;

[0062] Fig.25 Schematic diagram of the YOLOv5s network framework that introduces EMA attention;

[0063] Fig.26 This is a schematic diagram of some dataset images;

[0064] Fig. 27 This is a schematic diagram for comparing detection results of some traffic scene images. DETAILED DESCRIPTION

[0065] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a traffic scene target detection method based on lightweight YOLOv5, comprising:

[0066] Step 100: Construct a traffic scene dataset;

[0067] Step 200: preprocessing and dividing the traffic scene data set to obtain a training set, a validation set, and a test set;

[0068] Step 300: Lightweight the YOLOv5 network, and build a traffic scene target detection model based on the lightweight YOLOv5 network;

[0069] Step 400: inputting the training set, the validation set and the test set into a traffic scene object detection model for training, testing and verification to obtain a final traffic scene object detection model;

[0070] Step 500: Input the image to be detected into the final traffic scene target detection model to obtain the detection result.

[0071] The YOLOv5 target detection network is an existing technology, so it will not be introduced in detail here.

[0072] In step 100, a traffic scene dataset is constructed, specifically: a KITTI dataset, a PASCAL VOC dataset, and a COCO dataset are obtained, and partial images are filtered and extracted from the three datasets according to six categories of car, person, bus, bicycle, train, and motorcycle to obtain a traffic scene dataset.

[0073] Step 200, preprocessing and dividing the traffic scene data set to obtain a training set, a validation set, and a test set, specifically:

[0074] Step 201: perform data cleaning on the traffic scene data set; Step 202: perform image enhancement on the traffic scene data set after data cleaning; Step 203: divide the traffic scene data set after image enhancement according to a preset ratio to obtain a training set, a validation set and a test set.

[0075] Step 201, data cleaning is performed on the traffic scene data set, specifically: all images in the traffic scene data set are obtained, and they are classified into six categories: car, person, bus, bicycle, train, and motorcycle; an encoder is trained based on the preprocessed images, and a feature vector of the image is extracted based on the trained encoder, wherein the preprocessing refers to scaling the image size to M rows and N columns of pixels, where M and N are both positive integers, performing noise reduction on the image, rotating and cropping the image, and normalizing the image according to the following formula:

[0076]

[0077] Among them, p ij is the pixel value of the original image in row i and column j, x ij is the pixel value in row i and column j after normalization of the image, μ is the mean of the pixel values ​​of the original image, and σ is the variance of the pixel values ​​of the original image;

[0078] During the training process of the encoder, the labeled image samples are set as the input and output of the encoder, and the triplet loss is used as the loss function. The parameters of the encoder are obtained through iterative training. The trained encoder extracts the feature vector of the image by performing feature extraction on the image.

[0079] Cluster the images based on the feature vectors and calculate the similarity between the images of the same type one by one;

[0080] The clustering steps include compression of feature vectors, similarity calculation of compressed feature vectors, and clustering of original feature vectors;

[0081] When compressing the feature vector, the original D feature vector space is divided into the Cartesian product of m low-dimensional sub-vector spaces, where the dimension of each low-dimensional sub-vector is D / m;

[0082] The original feature vector D = 512 is divided into m = 4 groups, each group is a sub-vector of D / m = 512 / 4 = 128 dimensions;

[0083] Each sub-vector is clustered using the kmeans algorithm, where each sub-vector is clustered into k = 2 categories, and D = 512 feature vectors are compressed and stored as k m =2 4 = 16-dimensional vector;

[0084] The similarity measurement is performed on two vectors of image x and image y whose image feature vector dimension is D. The similarity calculation after feature vector compression adopts the following steps, and the Euclidean distance d(x, y) between x and y is defined as:

[0085]

[0086] Among them, x i and i are the i-th component of the feature vector obtained after feature extraction of image x and image y, and the Euclidean distance d(x, y) between x and y after quantization is expressed as the similarity distance Approximately, the similar distance for:

[0087]

[0088] Among them, C(x) i and C(y) i The i-th component of the feature vectors extracted from the images x and y after compression respectively; the original feature vectors are clustered using the kmeans method based on the similarity calculation results after feature vector compression;

[0089] Use the Euclidean distance or cosine distance of feature vectors to calculate the similarity between images;

[0090] Remove similar images from the same category based on the similarity between the images.

[0091] In step 300, the YOLOv5 network is lightweighted, specifically:

[0092] Step 301: Perform lightweight improvement on the backbone network of the YOLOv5 network;

[0093] Step 302: performing lightweight improvement on the neck network of the YOLOv5 network, specifically introducing BiFPN to replace the splicing module in the neck network of the YOLOv5 network;

[0094] Step 303: Introduce the attention mechanism into the lightweight improved YOLOv5 network.

[0095] In step 301, the backbone network of the YOLOv5 network is light-weighted, including three improvement methods, which are introduced respectively:

[0096] (I) Lightweight improvement of the backbone network based on MobileNetV3

[0097] The structure of MobileNetV3 includes basic module, inverted residual module and SE (Squeeze-and-Excitation) module, such as Figure 2 As shown in the figure, the basic module is the basic building block of MobileNetV3, which is used to extract features. It consists of more efficient depthwise separable convolution and linear bottleneck structure, which can reduce the number of parameters and calculations of the model. The inverted residual module is the innovation of MobileNetV3. It increases the depth and width of the network by inverting the residual connection, thereby improving the performance of the network. The SE module is used to enhance the feature expression ability of the network. It adjusts the weight of the feature map by learning the relationship between feature channels. MobileNetV3 is divided into MobileNetV3-Small and MobileNetV3-Large according to the dependence on computing and storage performance.

[0098] Depthwise separable convolution is implemented by combining two stages: depthwise convolution and pointwise convolution. These two layers of convolution are usually followed by batch normalization and ReLU activation function to enhance the nonlinear expression ability of the model and accelerate the training process. The structure of standard convolution and depthwise separable convolution is shown in Figure 1. Figure 3 shown.

[0099] Depthwise Convolution (DW) is different from standard convolution. The convolution kernel of deep convolution is single-channel mode, and one convolution kernel is responsible for only one channel. The output feature map obtained after deep convolution has the same number of channels as the input feature map. The deep convolution process is as follows: Figure 4 As shown in Figure 2, the number of channels in the output feature map after the deep convolution is completed is not expanded, so the feature information of different channels at the same spatial position is not effectively utilized.

[0100] The operation of pointwise convolution (PW) is very similar to the standard convolution operation. Its convolution kernel size is 1×1×M (M is the dimension of the input feature map), and one pixel area is convolved each time. The number of pointwise convolutions matches the number of input channels of the next layer, and the number of output feature maps is directly controlled by the number of convolution kernels used. This convolution method has less computational complexity and can achieve dimensionality change. The pointwise convolution process is as follows Figure 5 shown.

[0101] The number of parameters F used to perform depthwise convolution s And the parameter F of point-by-point convolution d , as shown below:

[0102] F s =D k ×D k ×M

[0103] F d =1×1×M×N

[0104] Among them, D k ×D k is the convolution kernel size, M is the number of input channels, and N is the number of output channels;

[0105] To sum up, the schematic diagram of depth-separable convolution is as follows Figure 6 As shown;

[0106] From the above formula, the parameter quantity of depth-separable convolution is:

[0107] parameter = D k ×D k ×+1×1×M×N

[0108] In the formula, parameter is the parameter quantity, D k ×D k is the convolution kernel size, M is the number of input channels, and N is the number of output channels.

[0109] The linear bottleneck structure consists of two point-by-point convolutional layers with a convolution kernel size of 1×1 and a depth-wise convolutional layer with a convolution kernel size of 3×3. Figure 7 shown.

[0110] The linear bottleneck structure effectively reduces the number of parameters in the deep convolution layer, and uses the point-by-point convolution layer to perform dimensionality increase operations, optimizing the network in terms of space and time. Compared with standard convolution, it can reduce the number of parameters and the amount of convolution calculation. Therefore, the first method of the present invention is to use MobileNetV3-Small to introduce

[0111] The backbone network of YOLOv5s is lightweight and improved.

[0112] The structure of MobileNetV3-Small is shown in Table 1.

[0113] Table 1 MobileNetV3-Small structure table

[0114]

[0115]

[0116] Among them, Input represents the feature information size of the input image, Operator is the operation module performed by this layer, bneck represents a linear bottleneck structure, the 3×3 in the suffix represents the convolution kernel size used in the deep convolution in the bneck module, 16 in size represents that the 1×1 convolution in the bneck module needs to increase the channel dimension to 16, out represents the number of channels of the output image, SE represents whether the SE attention mechanism is introduced, NL represents the type of activation function used, and s is the step size of the convolution operation.

[0117] The C3 module in the YOLOv5s backbone network is relatively complex, which greatly increases the network calculation amount and slows down the detection speed, making it difficult to apply to some mobile or embedded devices. Therefore, the present invention combines MobileNetV3-Small with C3 into a MobileC3 module for lightweight improvement, and redesigns the backbone network. The MobileC3 structure is as follows Figure 8 shown.

[0118] The backbone network introduces the YOLOv5s framework of the MobileC3 module as follows Fig. 9 shown.

[0119] (II) Lightweight improvement of the backbone network based on GhostNetV2

[0120] The core idea of ​​the GhostNet series is to perform simple linear operations (Cheap Operation) and feature concatenation (Identity) on feature maps to generate more feature maps with fewer parameters. The core module of GhostNetV2 is GhostModule. By stacking Ghost Modules, the Ghost bottleneck structure is obtained, and then the entire network is built. GhostModule can generate the same number of feature maps as the standard convolutional layer, and ensure a comprehensive understanding of the input data with low-cost calculations, thereby reducing the computational cost. The Ghost Module operation process is as follows Fig.10 shown.

[0121] In the Ghost Module, there are two groups of convolution operations. First, a convolution operation is performed to obtain a feature map with a small number of channels. Second, a small number of feature maps are linearly transformed to generate a "Ghost" feature map. Finally, the two feature maps are concatenated to obtain a feature map of the same size as the conventional convolution output. In practical applications, to improve efficiency, conventional convolution can use 1×1 point-by-point convolution to process features across channels, and linear transformation can use deep convolution to process spatial information channel by channel.

[0122] The operating mechanism of Ghost Module is to use a set of small convolution kernels to extract a set of relatively sparse feature maps from the input data. Assuming that the number of channels of these feature maps is set to m, a linear transformation operation is applied to these m initial feature maps to generate n new feature maps. It is assumed that each initial feature map undergoes s such linear transformations, and the mathematical relationship between m and n is shown in the following formula.

[0123] n=m×s

[0124] It is worth noting that since the output feature map contains an "identity map" that has not been transformed (i.e., a feature map that is not transformed), the number of linear transformation operations actually involved in the calculation is s-1. The number of linear changes obtained based on the above formula, i.e., the number of Ghost feature maps, is:

[0125]

[0126] Assuming the input size is H×W×C, the output size is H×W×n, the convolution kernel size is k×k, and the GhostModule linear transformation convolution size is d×d, in standard convolution, the required computational effort F B As shown in the following formula, in GhostModule, the required calculation amount F G As shown in the following formula.

[0127] F B =H′×W′×k×k×n×C

[0128]

[0129] The Ghost bottleneck structure is mainly composed of two stacked Ghost Modules. The first Ghost Module is used to increase the number of channels. The second Ghost Module is used to reduce the number of channels. GhostNetV2 introduces the attention mechanism into the Ghost bottleneck structure, which is divided into two combinations according to the step size of 1 or 2, such as Fig.11As shown in the figure, the attention mechanism branch is parallel to the first Ghost Module to enhance the extended features, and then the enhanced features are passed to the second GhostModule to improve the representation ability of the model. The network configuration parameters of GhostNetV2 are shown in Table 2.

[0130] Table 2 GhostNetV2 network configuration parameter table

[0131]

[0132]

[0133] In the table, Output represents the output size, Operator represents the execution operation, Conv3×3 represents the convolution operation with a convolution kernel size of 3×3, Block represents the stacked Ghost Module, Cheap refers to the cheap computing operation for generating the "Ghost" feature map, #out represents the number of output channels, and Stride represents the stride.

[0134] The present invention combines GhostNetV2 and C3 modules into C3GhostV2 module to perform lightweight design on the backbone network of YOLOv5s. The module structure of C3GhostV2 is as follows: Fig.12 shown.

[0135] The backbone network introduces the YOLOv5s framework of the C3GhostV2 module as follows Fig.13 shown.

[0136] (III) Lightweight improvement of backbone network based on SwinTransformer

[0137] Swin Transformer is a network proposed by Microsoft Research Asia in 2021. It introduces a sliding window mechanism that allows the model to learn cross-window information. It also uses a hierarchical feature map construction method and a special downsampling strategy to help the network extract more advanced features, saving computational effort and focusing on global and local information. Fig.14 shown.

[0138] Swin Transformer first divides the input RGB image into non-overlapping image blocks through the image block partition module, and then constructs feature maps of different sizes through four stages. A downsampling operation is usually implemented during the construction of each stage, and this process is performed by a patch merging layer, such as Fig.15As shown in Figure 2, after passing the image block merging layer, the height and width of the feature map will be halved, and the depth will be doubled.

[0139] In the Swin Transformer architecture, the SW-MSA (Shifted Windows Multi-Head Self-Attention) module is an innovative self-attention mechanism that introduces the concept of shifted windows to achieve information interaction between different windows. The model performs MSA calculations on pixels in the offset window, while promoting information interaction between pixels in different windows, thereby indirectly improving the model's ability to capture global information. The SW-MSA module structure is shown in the figure below: Fig.16 shown.

[0140] The present invention combines the Swin Transformer with the C3 module of the backbone network of the YOLOv5 network into a C3STR module, and performs lightweight improvement on the backbone network of the YOLOv5 network. The structure of C3STR is as follows: Fig.17 shown.

[0141] The backbone network introduces the YOLOv5s framework of C3STR as follows Fig.18 shown.

[0142] Step 302, lightweight improvement is performed on the neck network of the YOLOv5 network, specifically, BiFPN is introduced into the neck network of the YOLOv5 network to replace the splicing module, which is specifically introduced as follows:

[0143] The neck network is the part that connects the backbone network and the prediction head, and is responsible for feature fusion and processing. The neck network of YOLOv5s combines the FPN and PAN structures. The FPN downsampling process causes feature information to be lost during the transmission process because of the dimension increase and decrease. At the same time, FPN only has a top-down process, which makes it difficult for the last layer to obtain low-level information. However, a single node in PAN obtains very little information, and the number of model parameters increases, which can easily lead to training and learning deviations and affect the accuracy of detection.

[0144] To address the above problems, the present invention starts with the feature fusion of the neck network and introduces a weighted bidirectional feature pyramid (BiFPN) to solve the problems of excessive computational complexity and insufficient feature information when processing high-resolution images. It can also better utilize the underlying information of the feature map and fuse the feature maps of different scales, thereby enhancing the feature information.

[0145] BiFPN was proposed in the EfficientDet paper and is mainly used for target detection tasks. BiFPN adopts the idea of ​​multi-scale feature fusion of FPN and improves it on the basis of PANet. Its full name is Bi-directional Feature Pyramid Network. Several common multi-scale feature fusion structures are as follows Fig.19 shown.

[0146] Fig.19 a is the structure of FPN, the traditional top-down aggregation of multi-scale feature maps, Fig.19 b is PANet, which is an improvement on FPN and adds bottom-up transmission. Fig.19 c is BiFPN. BiFPN removes nodes with only one input edge based on PANet, adds direct jump connections from input nodes to output nodes, and repeats this structure several times. Each BiFPN module contains two branches: upsampling and downsampling branches and lateral connection branches. Among them, the upsampling and downsampling branches use bilinear interpolation for upsampling or downsampling, and the lateral connection branches are used for feature fusion. The optimized cross-scale connection idea proposed by BiFPN for PANet is to regard each bidirectional path (top-down + down-top) as a feature network layer (corresponding to Fig.19 The BiFPN module in , and is repeated multiple times to enable higher-level feature fusion, such as Fig. 20 shown.

[0147] In general, BiFPN introduces contextual information by constructing a bidirectional channel to achieve cross-scale connection to enhance feature expression capabilities; it adopts a simple and efficient weighted feature fusion mechanism to enable the network to learn the importance of different input features to balance the contribution of features at different levels; at the same time, it repeatedly applies top-down and bottom-up multi-scale feature fusion to make the network more adaptable to multi-scale targets, so that the fused feature network has higher recognition efficiency and accuracy.

[0148] Combined with the above analysis, the present invention introduces BiFPN to replace the splicing module in the neck network of YOLOv5s, and the neck network introduces BiFPN into the YOLOv5s framework as shown in Fig.21 shown.

[0149] In step 30, the attention mechanism is introduced into the lightweight improved YOLOv5 network, specifically:

[0150] Although lightweight improvements can reduce the number of network parameters and computational complexity, they may sometimes have a certain adverse effect on the detection accuracy of the network. In order to enhance the feature learning and expression capabilities of lightweight networks, many networks often adopt the method of adding attention mechanisms to the backbone network, such as SE, CBAM, etc. However, SE attention only considers the information between the encoding channels during the encoding process and ignores the importance of position information. Although the CBAM attention mechanism uses spatial attention to extract position information, it can only capture local relationships and cannot capture long-distance dependencies. The present invention intends to introduce CA (Coordinate Attention, CA) and EMA (Efficient Multi-Scale Attention, EMA) attention mechanisms respectively to enable the network to better focus on important features, enhance the ability to extract important features through specific strategies, and effectively suppress the interference of non-critical information to improve the detection accuracy of the network model. CA attention consists of two steps: coordinate information embedding and coordinated attention generation. In the coordinate information embedding step, the position information is embedded into the input feature map. In the coordinate attention generation step, the attention map is generated using the position information and applied to the input feature map to emphasize the expression of important information. Fig. 22 This is the structural diagram of the CA module.

[0151] (1) Coordinate information embedding: In order to enable the attention module to obtain a comprehensive expression of features at a specific position, the coordinate information embedding method is to decompose the global pooling and use two pooling kernels of size (H, 1) and (1, W) to aggregate the features of each channel in the horizontal and vertical directions respectively. Therefore, the output of the cth channel at height h and the output at width w can be expressed as:

[0152]

[0153] Among them, x c (i, j) is the value of the position (i, j) of the c-th channel feature map, represents the average value of the c-th channel feature map at height h, It represents the average value of the c-th channel feature map at width w, where H and w are the height and width of the feature map respectively. The output is a pair of direction-aware feature maps of shape C×H×1 and C×1×W, which enables the attention module to focus on global information while retaining precise position information, thereby locating important features more accurately.

[0154] (2) Coordinated attention generation: In order to make better use of the above coordinate information, CA proposes a simple, efficient, accurate positioning method that can effectively capture the relationship between channels. First, the output results are concatenated and the result is obtained through a shared 1×1 convolution transformation function F1, as shown in the following formula:

[0155] f=δ(F1[z h ,z w ])

[0156] Where [,] is the concatenation operation along the spatial dimension, δ represents a nonlinear activation function, and f is used to generate an intermediate feature map encoded in two directions. Then f is decomposed into two separate vectors f h ∈R C / r×H and f w ∈R C / r×W Then use two 1×1 convolution transformations F h and F u They are converted into tensors with the same number of channels, as shown below:

[0157] g h =σ(F h (f h ))

[0158] g w =σ(F w (f w ))

[0159] Where σ is the Sigmoid activation function, g h and g' represents the attention weight coefficients in two dimensions. Finally, the output of the coordinated attention block is obtained by the following formula:

[0160]

[0161] Among them, y c (i, j) is the weight value of the position (i, j) in the feature map. Each weight of the CA module integrates the relationship between channels, horizontal space, and vertical space information. This integration gives the network stronger positioning accuracy, allowing the model to more accurately identify and locate targets in the image.

[0162] The EMA attention module is an efficient multi-scale attention module. This module adopts a multi-branch network architecture, combined with multi-scale fusion and feature grouping strategies, aiming to reduce computational overhead and retain the key information of each channel. The structure of the EMA module is as follows Fig.23 shown.

[0163] The EMA module processes the input through different branches: one branch performs one-dimensional global pooling, and the other performs feature extraction through 3×3 convolution. The output features of the two branches are then processed by the sigmoid function and normalization operation, and finally merged through the cross-dimensional interaction module to capture the pairwise relationship at the pixel level.

[0164] The EMA module proposes a new cross-space learning method and designs a multi-scale parallel subnetwork in combination with the parallel subnetwork of CA to establish long-short dependencies. Its main innovation is to reshape part of the channel dimension into a batch dimension to avoid a form of dimensionality reduction through universal convolution, thereby improving the model's ability to process features; in addition to building local cross-channel interactions in each parallel subnetwork without channel dimensionality reduction, it also fuses the output feature maps of two parallel subnetworks through a cross-space learning method; the channel dimension is grouped into multiple sub-features so that the spatial semantic features are evenly distributed in each feature group to retain the information of each channel and reduce computational overhead. The EMA module enhances the ability of feature representation by encoding global information to recalibrate the channel weights in each parallel branch and capturing pixel-level relationships through cross-dimensional interactions.

[0165] Based on the YOLOv5s network, the present invention selects three lightweight structures to replace the C3 module of the backbone network respectively, thereby completing the lightweight design of the backbone network. The lightweight design of the neck network is achieved by introducing the BiFPN module into the neck network. However, since the improved lightweight network often loses some detection accuracy, the CA attention module and EMA attention module introduced above are respectively introduced into the end of the backbone network to enhance the feature learning ability of the network. The YOLOv5s network framework with CA attention is shown in Figure 1. Fig.24 As shown, the YOLOv5s network framework that introduces EMA attention is as follows Fig.25 The present invention will subsequently conduct an experimental comparison on the effects of the two attention mechanisms in YOLOv5s target detection.

[0166] The YOLOv5 loss function is composed of the positioning loss L bax , classification loss L cls and confidence loss L oby It consists of three parts, and the loss value formula is as follows.

[0167]

[0168] Among them, Loss is the expression of the loss function, λ1, λ2, and λ3 correspond to three weight coefficients of three different losses, and N is the number of detection layers.

[0169] L box Use GoU_Loss to calculate. The GoU loss function is calculated as follows:

[0170]

[0171] Among them, A and B represent the real box and the predicted box respectively, and C represents the minimum rectangular box surrounding A and B.

[0172] The classification loss and confidence loss of YOLOv5 are both calculated using the Binary Cross-Entropy (BCE) loss function, which is designed to measure the difference between the predicted probability and the actual label. Its specific form is shown in the following formula.

[0173] L BCE =-[ylog(p)+(1-y)log(1-p)]

[0174] Among them, y represents the true category, which takes the value of 1 or 0, and p represents the probability of being predicted as the positive class, which is obtained by processing the predicted value with the Sigmoid activation function and takes the value between 0 and 1.

[0175] The loss function is used to evaluate the degree to which the predicted value of the model is different from the true value. The present invention chooses to use SIoU to replace the loss function of the original network. SIoU specifically includes four parts: angle loss (Angle cost), distance loss (Distancecost), shape loss (Shape cost), and IoU loss (IoU cost).

[0176] (1) The definition of angle loss is as follows:

[0177]

[0178] Among them, c h is the height difference between the center point of the real box and the predicted box, σ is the distance between the center point of the real box and the predicted box, and is the center coordinate of the real frame, and is the center coordinate of the prediction box.

[0179] (2) Distance loss, defined as follows:

[0180]

[0181] in:

[0182]

[0183] In the formula, and is the center coordinate of the real frame, and is the center coordinate of the prediction box, C w and C h It is the width and height of the minimum enclosing rectangle of the real box and the predicted box.

[0184] (3) Shape loss, defined as follows:

[0185]

[0186] in:

[0187]

[0188] Where w, h, w gt ,h gt are the width and height of the predicted box and the true box respectively, and the θ parameter controls the degree of attention to shape loss.

[0189] In summary, the final SIoU loss function is defined as follows:

[0190]

[0191] The experimental results of the present invention are analyzed: the data set of the present invention finally includes 7489 training sets, 1872 validation sets, and 1872 test sets. Fig.26 Some images of the dataset.

[0192] All experiments in this paper were conducted in Python 3.8, Pytorch 1.9.0, and Cuda 11.1 environments, and on a cloud server equipped with a GeForce RTX 3090 and 24GB of video memory. The input image size was 640×640, and the network training parameters were as follows:

[0193] Table 4 Network training parameters

[0194]

[0195]

[0196] (1) Analysis of neck network lightweight results

[0197] The lightweight improvement of the YOLOv5s network in this paper is mainly concentrated in two aspects: one is the lightweight design of the main network, and the other is the lightweight optimization of the neck network. First, a lightweight experiment is carried out on the neck network, and the performance of the improved YOLOv5s is analyzed by five indicators: mAP@0.5, Paremeters, GFLOPs, inference time, and network memory usage. Table 5 shows the quantization results of the improved neck network using BiFPN.

[0198] Table 5 Neck network lightweight results analysis table

[0199]

[0200] As can be seen from Table 5, after the neck network is lightweight, the network mAP is improved by 0.4%, but the number of parameters and the network memory occupied are slightly increased, and the inference time is shortened to 2.0ms. The lightweight improvement of the neck network in the present invention reduces the number of floating-point operations while improving the model detection accuracy and inference time, but the number of parameters and the model memory occupied still have room for improvement. Therefore, the present invention completes the lightweight design of the YOLOv5s backbone network on this basis.

[0201] (2) Analysis of backbone network lightweight results

[0202] The present invention replaces the C3 module in the backbone network with MobileC3, GhostV2C3 and C3STR respectively, and evaluates it through five indicators such as mAP@0.5, Paremeters, and GFLOPs. The experimental results are shown in Table 6:

[0203] Table 6 Evaluation results of backbone network module replacement in YOLOv5s

[0204]

[0205] Table 7 Detection effect table of each target category

[0206]

[0207] As can be seen from Table 6, compared with YOLOv5s-B, the network inference time of the MobileC3 module is the best, which is 0.1ms higher than YOLOv5s-B, the mAP value is reduced by 5.1%, the number of parameters is reduced by 28%, and the number of floating-point operations is reduced by 29%. The mAP value of the network introduced by the GhostV2C3 module is reduced by 0.4%, the number of parameters is reduced by 22%, and the number of floating-point operations is reduced by 23%. Table 7 shows the detection accuracy of each network for each target category in the dataset. Compared with YOLOv5s-B, except for the method that introduces the GhostV2C3 module, the detection accuracy of the other two methods on each target type has dropped significantly. In the method that introduces the GhostV2C3 module, the AP values ​​of the person and motorcycle categories are slightly improved compared with YOLOv5s-B, and the AP value loss of other categories is smaller.

[0208] Through the above analysis, it can be concluded that the improvement of the GhostV2C3 module has achieved the best balance among various indicators. However, due to the reduction of network parameters, the network's feature extraction ability in complex traffic scenarios will also decline to a certain extent. Therefore, in the subsequent ablation experiment, the present invention introduces the attention mechanism and the improved loss function in the YOLOv5s network model that introduces the GhostV2C3 module to enhance the network's ability to extract target features, and explore the specific contribution of each module to the network performance.

[0209] (3) Ablation experiment

[0210] The ablation experiment of the present invention gradually adds the CA module, EMA module and loss function improvement to the YOLOv5s-B model according to the improvement method described above to verify the effectiveness of the attention mechanism and loss function for model improvement. The ablation experiment results are shown in Table 8.

[0211] Table 8 Ablation experiment results

[0212]

[0213] It can be seen from the table that the use of attention mechanism and improved loss function effectively compensates for the negative impact of lightweight. After introducing the SIoU loss function, the network that uses GhostV2C3 module and BiFPN to complete lightweight improvement and combines EMA attention has achieved the best effect. The mAP of the method proposed in the present invention is increased from 52.7% to 53.8%, an increase of 1.1% over the original YOLOv5s model. The number of floating-point operations is reduced from 15.8 to 12.2, a reduction of 22% over the original model. And the final network parameter amount, inference time and network memory usage are better than the original model.

[0214] Analysis of target detection results: The present invention makes lightweight improvements to the YOLOv5s network and compares it with the single-stage target detection network SSD and the two-stage target detection network Faster R-CNN. The detailed experimental results are shown in Table 9.

[0215] Table 9 Evaluation results of different target detection networks

[0216]

[0217] Through in-depth analysis of the experimental results, both SSD and Faster-RCNN have certain deficiencies in target detection accuracy. Compared with the above networks, the lightweight YOLOv5s model proposed in this invention is the best in all evaluation indicators, with an mAP of 53.8%. This value effectively verifies the efficiency and practicality of the lightweight YOLOv5s traffic scene target detection model with EMA attention proposed in this invention.

[0218] The present invention visualizes the detection results of YOLOv5s and the improved YOLOv5s network on the test set. Fig. 27 The following is a comparison of the detection results of some traffic scene images. The detection results in the figure show that the improved network can effectively alleviate the problems of false detection and missed detection compared with YOLOv5s, and can achieve better detection results.

Claims

1. A traffic scene target detection method based on lightweight YOLOv5, characterized in that: include: Construct a traffic scene dataset; Preprocess the traffic scene dataset and divide the dataset into training set, validation set and test set; Lightweight the YOLOv5 network and build a traffic scene target detection model based on the lightweight YOLOv5 network; Inputting the training set, the validation set and the test set into a traffic scene object detection model for training, testing and verification to obtain a final traffic scene object detection model; The image to be detected is input into the final traffic scene target detection model to obtain the detection result.

2. The traffic scene target detection method based on lightweight YOLOv5 according to claim 1, characterized in that: Construct a traffic scene dataset, specifically: The KITTI dataset, PASCAL VOC dataset, and COCO dataset are obtained. Some images are filtered and extracted from the three datasets according to six categories: car, person, bus, bicycle, train, and motorcycle, to obtain a traffic scene dataset.

3. The traffic scene target detection method based on lightweight YOLOv5 according to claim 2 is characterized in that: The traffic scene dataset is preprocessed and divided into training set, validation set and test set, which are as follows: Perform data cleaning on traffic scene datasets; Perform image enhancement on the traffic scene dataset after data cleaning; The traffic scene dataset after image enhancement is divided into training set, validation set and test set according to the preset ratio.

4. The traffic scene target detection method based on lightweight YOLOv5 according to claim 3 is characterized in that: The traffic scene dataset is cleaned as follows: Get all the images in the traffic scene dataset and classify them into six categories: car, person, bus, bicycle, train, and motorcycle. Train an encoder based on the preprocessed image, and extract a feature vector of the image based on the trained encoder; Cluster the images based on the feature vectors and calculate the similarity between the images of the same type one by one; Remove similar images from the same category based on the similarity between the images.

5. The traffic scene target detection method based on lightweight YOLOv5 according to claim 1, characterized in that: The YOLOv5 network is lightweight, specifically: Lightweight improvements to the backbone network of the YOLOv5 network; The neck network of the YOLOv5 network is lightened by improving it. Specifically, BiFPN is introduced into the neck network of the YOLOv5 network to replace the splicing module. The attention mechanism is introduced into the lightweight and improved YOLOv5 network.

6. The traffic scene target detection method based on lightweight YOLOv5 according to claim 5 is characterized in that: The backbone network of the YOLOv5 network is light-weighted and improved, specifically: Combine MobileNetV3-Small with the C3 module of the backbone network of the YOLOv5 network into the MobileC3 module, and make lightweight improvements to the backbone network of the YOLOv5 network.

7. The traffic scene target detection method based on lightweight YOLOv5 according to claim 5 is characterized in that: The backbone network of the YOLOv5 network is light-weighted and improved, specifically: The C3 module of the GhostNetV2 and YOLOv5 backbone networks is combined into the C3GhostV2 module to make lightweight improvements to the backbone network of the YOLOv5 network.

8. The traffic scene target detection method based on lightweight YOLOv5 according to claim 5, characterized in that: The backbone network of the YOLOv5 network is light-weighted and improved, specifically: The Swin Transformer is combined with the C3 module of the backbone network of the YOLOv5 network into the C3STR module to make lightweight improvements to the backbone network of the YOLOv5 network.

9. The traffic scene target detection method based on lightweight YOLOv5 according to claim 5, characterized in that: The attention mechanism is introduced into the lightweight and improved YOLOv5 network, specifically: The CA attention mechanism is introduced into the lightweight and improved YOLOv5 network.

10. The traffic scene target detection method based on lightweight YOLOv5 according to claim 5, characterized in that: The attention mechanism is introduced into the lightweight and improved YOLOv5 network, specifically: The EMA attention mechanism is introduced into the lightweight and improved YOLOv5 network.

Images