An Improved Road Obstacle Detection Method Based on YOLOv4

Abstract

This invention provides a road obstacle detection method based on an improved version of YOLOv4. Improvements have been made to YOLOv4, making the network more lightweight, enhancing its robustness and ability to handle complex environments, and enabling more accurate obstacle detection, thus improving the detection performance for small targets. The method includes: using the lightweight MobileNetV3 as the backbone network to provide feature extraction operations; employing a linear bottleneck inverse residual structure combined with depthwise separable convolution and SE attention mechanisms; improving the feature fusion network; using an efficient channel attention module in the model, which avoids dimensionality reduction and effectively captures cross-channel interaction information, maintaining the number of channel layers after global pooling, and using a non-dimensionality-reduced local cross-channel interaction strategy; focusing on the relationships between different channels through band matrix learning, selectively learning different information in different channels; and proposing a new DBR module, which consists of DepthwiseConv2D, BatchNormalization, and ReLU6.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a road obstacle detection method based on an improvement of YOLOv4. Background Technology

[0002] As of the end of March 2022, the number of motor vehicles in China reached 402 million, and the number of motor vehicle drivers reached 487 million, including 450 million car drivers. By the end of 2020, the total length of highways in China reached 5.198 million kilometers. In 2021, a total of 244,937 road traffic accidents occurred nationwide, resulting in 63,194 deaths, 258,532 injuries, and direct property losses of 1.38 billion yuan. If vehicles could detect hazards in a timely manner while driving, it would greatly protect people's lives and property. To alleviate the enormous pressure of traffic problems on urban traffic and reduce traffic accidents, intelligent driving methods have become a hot research topic both domestically and internationally in recent years. Assisted driving, as a crucial component of intelligent driving, can safely and effectively detect obstacles on the road during driving, reducing the safety hazards posed by road obstacles to drivers.

[0003] In recent years, with the development of science and technology, deep learning has been widely applied in society. The term "deep learning" was first proposed in the 1980s, and it is a very important branch of machine learning. Early deep learning faltered due to its inability to classify nonlinear problems. Later, the emergence of the backpropagation algorithm enabled neural networks to learn nonlinearly, but as the number of network layers increased, a series of problems arose, preventing an overall improvement in the network's learning ability and hindering the development of neural networks again. It wasn't until the concept of deep belief networks was proposed in 2006 that the technical problems of neural networks were solved. This not only greatly improved the training speed of neural networks but also effectively solved the gradient vanishing problem. Deep learning entered a stage of rapid development, with various network frameworks being proposed, among which convolutional neural networks have become a prominent model.

[0004] The rapid improvement of computer hardware performance and the development of cloud computing and distributed computing systems have provided sufficient computing power for deep learning. Applying deep learning to the field of computer vision has become an inevitable trend in today's technological development. In object detection, traditional algorithms can only detect images with obvious features and simple backgrounds, struggling with complex situations and performing poorly on objects with abstract features. Furthermore, their time complexity is too high, the sliding window used is redundant, and manual feature design is required. Compared to traditional methods, deep learning can extract rich features and information from the same target, using large amounts of data to train the model, making it more robust and perfect. Deep learning is not only stable and robust, but it can also be applied in many fields, with each sample in the data being independent and easily expandable. The application of deep learning is also a huge advancement for object detection. Traditional methods cannot effectively detect road conditions in real time when cars are traveling at high speeds. However, deep learning can perfectly solve the problem of real-time detection of cars traveling at high speeds. It can extract features from the identified images through a well-built network model, thereby achieving the desired effect. Summary of the Invention

[0005] To address the technical problems raised in the background, this invention provides a road obstacle detection method based on an improved YOLOv4. The improved YOLOv4 network is more lightweight, enhancing its robustness and ability to handle complex environments. This allows the network to more accurately detect obstacle locations and improves the detection performance for small targets.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] A road obstacle detection method based on an improved YOLOv4, wherein the method establishes a neural network model as follows:

[0008] Step 1: The method uses the lightweight MobileNetV3 as the backbone network to provide feature extraction operations for the model; the backbone network uses an inverse residual structure with a linear bottleneck, combined with depthwise separable convolution and SE attention mechanism; this structure forms a very compact representation at the input and output, while extending the internal structure to a higher-dimensional feature space, increasing the expressiveness of channel transformation; at the same time, a new activation function h-swish(x) is used, which enables the model to maintain good accuracy while improving running speed;

[0009] Step 2: Improve the feature fusion network: In the feature extraction stage of the network, shallower networks extract shallow features with more detailed information, clear texture lines, and higher feature resolution; in deeper networks, the features extracted by the network have a larger receptive field and obtain richer semantic information.

[0010] Step 3: Use an efficient channel attention module in the model. This module avoids dimensionality reduction and effectively captures cross-channel interaction information. It maintains the number of channel layers after global pooling and uses a local cross-channel interaction strategy without dimensionality reduction. By learning the band matrix, it focuses on the relationship between different channels and learns different information in different channels in a focused manner. In local channel interactions, the module is both lightweight and ensures the effectiveness of local cross-channel interactions.

[0011] Step 4: A new DBR module is proposed, which consists of DepthwiseConv2D, BatchNormalization, and ReLU6. Due to the channel-wise and point-wise convolution operations in depthwise separable convolution, the computational parameters of the model are greatly reduced, thus reducing the model size. At the same time, in order to obtain better numerical resolution and avoid accuracy loss under the low precision of float16 on mobile devices, ReLU6 is used as the activation function in this module.

[0012] Furthermore, in the first step, the new activation function h-swish(x) in H-Swish selects ReLU6 to eliminate some numerical precision loss caused by approximating sigmoid, and the optimized implementation function of ReLU6 can be used in most hardware and software frameworks. The specific function is as follows:

[0013] Swish(x) = x·sigmoid(βx)

[0014]

[0015] sigmoid and Relu6 are functions, where x is the input value and β is a constant coefficient.

[0016] The bneck module in the network uses depthwise separable convolution and SE channel attention mechanism in the residual structure, and the module as a whole first increases the dimensionality and then decreases it.

[0017] Furthermore, the bneck module in the network uses depthwise separable convolutions and SE channel attention mechanisms in the residual structure. The overall module first increases the dimensionality and then reduces it, specifically including the following:

[0018] 1) The module uses 1*1 convolution to map the low-dimensional space to the high-dimensional space, and after the 1*1 convolution, BatchNormalization is used to process the data format and add an activation function.

[0019] 2) After the dimensionality increase, there is a 3*3 depthwise separable convolution, which is the same as the previous 1*1 convolution. After the depthwise separable convolution, there is also Batch Normalization and activation function processing. After extracting enough information through the depthwise separable convolution, the information enters the SE attention mechanism to learn important features.

[0020] 3) Finally, after the output of the SE module, 1*1 convolution is used again for dimensionality reduction, and only Batch Normalization is used without using a non-linear activation function. However, there is another route in the Bneck module, which adds the features input to Bneck and the features output under the same channel conditions. This operation reduces the computational load of the network and does not cause information loss.

[0021] Furthermore, in the second step, to address the issue of poor detection performance for small targets, the feature information at different levels in the network layers is studied, and effective information fusion is achieved to improve detection performance. The improvement in the second step is that a shallower layer of feature maps in the backbone network is extracted during the feature fusion stage. During the bottom-up propagation process, an upsampling method is used to fuse these features with the semantically rich features in the original PAN structure, enabling the network to obtain higher resolution features and more subtle information. During the top-down process, a downsampling method is used to combine them. The model obtains more detailed information while also solving the problem of insufficient semantic information in the new feature layer.

[0022] Furthermore, in the third step, to improve the overall accuracy of the model, this invention incorporates an ECA module in the feature fusion stage. This allows the model to learn features with a focus, effectively improving the overall accuracy of the model. When features are output from the global average pooling layer and enter the 1D convolution, the 1D convolution does not perform dimensionality reduction, maintaining the original channel size. The focus of each channel is learned using the following formula:

[0023] ω=σ(Wy)

[0024] Where W is a C×C parameter matrix and σ is the Sigmoid function, the band matrix W is used in the ECA module. k To learn the focus of the channels, capture local cross-channel interactions, and ensure efficiency and effectiveness; y is the input value, and ω is the weight of each channel.

[0025] The specific formula is as follows:

[0026]

[0027] y i The weights only need to consider y i The interaction between its k neighboring channels is used for computation, and the weights are shared in 1D convolution, reducing the number of parameters from k*C to k. Indicates y i The set of adjacent channels; j is one of the k adjacent channels, and i is a channel from a different layer. This strategy is implemented using a fast convolution with a kernel size of k, as shown in the following formula:

[0028] ω=σ(C1D k (y))

[0029] Here, C1D represents a 1D convolution; the above method is called by the ECA module, which only involves k parameters.

[0030] Furthermore, in the third step, the module achieves both lightweighting and ensures the effectiveness of local cross-channel interactions in local channel interactions. Specifically, the ECA module automatically determines the coverage of the interaction and adaptively adjusts the feature response values ​​between each channel. The interaction magnitude between channels is proportional to the channel dimension C, and there is a mapping φ between K and C:

[0031]

[0032] in For a linear function:

[0033]

[0034] Because the channel size is set to a power of 2, this is achieved in ECA by extending the linear function. This problem can be solved by introducing it into a nonlinear function:

[0035]

[0036] After determining C, the size of the convolution kernel k is automatically determined according to the following formula:

[0037]

[0038] Where |t| odd This represents the nearest odd number to t. γ and b are set to 2 and 1 respectively.

[0039] Furthermore, in the fourth step, to make the model more lightweight overall, the CBL module in the feature fusion stage is improved. To further reduce model weight, a different convolution method is adopted, and the LeakyReLU function is also modified. A new module, the DBR module, is proposed, which is composed of depthwise separable convolution, Batch Normalization, and ReLU6; specifically as follows:

[0040] 1) Since depthwise separable convolution is composed of channel-wise convolution and pointwise convolution, each channel in channel-wise convolution corresponds to a specific convolution kernel, and the number of channels does not change after channel-wise convolution; pointwise convolution uses 1*1 convolution to fuse the output of channel-wise convolution, and changes the number of output channels with the number of convolution kernels; therefore, in the new DBR module, the use of depthwise separable convolution greatly reduces the amount of computation brought by ordinary convolution.

[0041] 2) Batch Normalization is used after convolution to prevent network overfitting and gradient vanishing, and to speed up convergence.

[0042] 3) Finally, ReLU6 is used as the activation function. ReLU6 limits the maximum ReLU value to 6, as shown in the following formula:

[0043] Relu = max(0, x)

[0044] Relu6 = min(6, max(0, x))

[0045] When the value of x is greater than 6, its derivative is 0; this is done to obtain better numerical resolution in the low precision of the lightweight model Float16; if not restricted, the output range can reach positive infinity, making the numerical value impossible to describe precisely.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] 1) The network model of the present invention is robust to road obstacles: the model can obtain a good detection result for problems such as occlusion, congestion and bad weather in the data, which is a great improvement over the original model.

[0048] 2) This invention uses a lightweight backbone network and proposes a DBR module, effectively utilizing the characteristics of depthwise separable convolution and reasonably combining activation functions, residual structures, and other operations. This invention effectively reduces the computational parameters of the model, decreases the model's complexity, and improves the model's detection speed.

[0049] 3) This invention employs a highly efficient channel attention model, avoiding dimensionality reduction and effectively capturing cross-channel interaction information. Inserting this model into the interface of the feature fusion network allows the model to selectively learn feature information from different channels, effectively improving the overall detection accuracy of the model.

[0050] 4) This invention proposes a new feature fusion method that effectively combines feature maps at various levels, making full use of the semantic and spatial information of feature maps at different stages, thus improving the problem of poor small target detection performance. Attached Figure Description

[0051] Figure 1 This is a general structural diagram of the present invention;

[0052] Figure 2 This is a diagram of the lightweight backbone network structure in this invention;

[0053] Figure 3 This is a diagram of the ECA module used in this invention;

[0054] Figure 4 This is the novel feature fusion map proposed in this invention;

[0055] Figure 5 This is a diagram of the DBR module proposed in this invention;

[0056] Figure 6 This is a structural diagram of the depthwise separable convolution used in this invention. Detailed Implementation

[0057] The specific embodiments provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0058] This invention proposes an improved road obstacle detection algorithm for YOLOv4. It utilizes a lightweight backbone network and introduces a DBR module, making the overall model lightweight and easily adaptable for mobile devices. Improvements in the feature fusion stage enhance the network's detection performance for small targets. The inclusion of an ECA module improves the overall detection accuracy of the model. Ultimately, the model achieves satisfactory detection results.

[0059] like Figures 1-6 As shown, it includes the following:

[0060] Step 1: To address the issue of poor performance on mobile devices, this model employs a lightweight backbone network for feature extraction, which has low computational cost and short inference time, making it highly suitable for road obstacle detection. The network uses a linear bottleneck inverse residual structure, combined with depthwise separable convolutions and SE attention mechanisms. This structure forms a very compact representation at both the input and output, while extending the internal structure to a higher-dimensional feature space, increasing the expressiveness of channel transformations. Furthermore, a novel activation function, h-swish(x), is used, allowing the model to maintain good accuracy while improving running speed.

[0061] Step Two: To address the issue of poor performance in detecting small targets, this invention improves the feature fusion network. In the feature extraction stage, shallower networks extract shallow features with more detailed information, clearer texture lines, and higher feature resolution. However, deeper networks extract features with a larger receptive field, yielding richer semantic information. The main function of the PAN structure in the original model is to use features from different feature layers, fusing them together and transferring features and information from different levels, thus improving the model's ability to detect targets of different sizes. Furthermore, it shortens the information transmission path and effectively avoids information loss caused by an excessive number of network layers.

[0062] Step 3: To improve the overall accuracy of the model, this invention employs a highly efficient channel attention module. This module avoids dimensionality reduction and effectively captures cross-channel interaction information. It maintains the number of channel layers after global pooling and uses a local cross-channel interaction strategy without dimensionality reduction. By learning the band matrix, it focuses on the relationships between different channels, selectively learning different information within each channel. In local channel interactions, the module achieves both lightweight design and ensures the effectiveness of local cross-channel interactions, thereby improving the accuracy of road obstacle detection.

[0063] Step 4: To address the issue of excessive model computation, this invention proposes a novel DBR module. The module consists of DepthwiseConv2D, BatchNormalization, and ReLU6. Due to the channel-wise and point-wise convolution operations in depthwise separable convolution, the computational parameters of the model are significantly reduced, effectively decreasing the model size. To achieve good numerical resolution even with the low precision of float16 on mobile devices and avoid accuracy loss, ReLU6 is used as the activation function step in the module.

[0064] The details are as follows:

[0065] In the first step, this invention uses MobileNetV3 as the backbone network. ReLU6 is chosen in H-Swish to eliminate some numerical accuracy loss caused by approximating sigmoid in a special mode. Furthermore, the optimizations of ReLU6 enable the functions to be used in most hardware and software frameworks.

[0066] Swish(x) = x·sigmoid(βx)

[0067]

[0068] Compared to other nonlinear functions, the swish activation function can reduce the number of filters while maintaining higher accuracy, reducing the time by about 3 milliseconds and the computation parameters by 10 million MAdds. H-Swish has a higher computation speed than swish.

[0069] The Bneck module in the network uses depthwise separable convolutions and SE (Sequential Enlargement) channel attention in the residual structure. The module as a whole first increases dimensionality and then decreases it. The module uses 1x1 convolutions to map the low-dimensional space to the high-dimensional space, and after the 1x1 convolution, Batch Normalization processes the data format and adds an activation function. After the dimensionality increase, a 3x3 depthwise separable convolution is used, similar to the previous 1x1 convolution, and also undergoes Batch Normalization and activation function processing. After extracting sufficient information through the depthwise separable convolutions, the information enters the SE attention mechanism to learn important features. Finally, after the output of the SE module, a 1x1 convolution is used again for dimensionality reduction, using only Batch Normalization without a non-linear activation function. However, there is another approach in the Bneck module: adding the features input to the Bneck and the output features under the same channel condition. This operation reduces the computational cost of the network without causing information loss.

[0070] In the second step, addressing the issue of poor detection performance for small targets, this invention primarily studies the feature information at different levels within the network layers and effectively achieves information fusion to improve detection performance. The main improvement in this paper is the extraction of feature maps from a shallower layer in the backbone network during the feature fusion stage. During the bottom-up propagation process, an upsampling method is used to fuse these features with semantically rich features from the original PAN structure, resulting in higher-resolution features and more nuanced information. In the top-down process, downsampling is used to combine these features, allowing the model to obtain more detailed information while also addressing the problem of insufficient semantic information in the new feature layers.

[0071] In the third step, to improve the overall accuracy of the model, this invention incorporates an ECA module in the feature fusion stage. This allows the model to learn features with a focus, effectively improving the overall accuracy. When features are output from the global average pooling layer and enter the 1D convolution, the 1D convolution does not perform dimensionality reduction, maintaining the original channel size. The focus of each channel is learned using the following formula:

[0072] ω=σ(Wy)

[0073] Where W is a C×C parameter matrix and σ is the Sigmoid function. The band matrix W is used in the ECA module. k This approach focuses on the key aspects of the learning channels, capturing localized cross-channel interactions while ensuring efficiency and effectiveness. The specific formula is as follows:

[0074]

[0075] y i The weights only need to consider y i The interaction between the channel and its k neighboring channels is used for computation. In 1D convolution, the weights are shared, and the number of parameters is reduced from k*C to k. Indicates y i The set of adjacent channels. This strategy is implemented using a fast convolution with a kernel size of k, as shown in the following formula:

[0076] ω=σ(C1D k (y))

[0077] Here, C1D represents a 1D convolution. The above method is called by the ECA module, which only involves k parameters.

[0078] In local channel interactions, the module achieves both lightweight design and ensures the effectiveness of local cross-channel interactions. The ECA module automatically determines the coverage of the interaction and adaptively adjusts the feature response values ​​between channels. The magnitude of the interaction between channels is proportional to the channel dimension C, and a mapping φ exists between K and C:

[0079]

[0080] in For a linear function:

[0081]

[0082] Because the channel size is set to a power of 2, this is achieved in ECA by extending the linear function. This problem can be solved by introducing it into a nonlinear function:

[0083]

[0084] After determining C, the size of the convolution kernel k is automatically determined according to the following formula:

[0085]

[0086] Where |t| odd This represents the nearest odd number to t. γ and b are set to 2 and 1 respectively.

[0087] In the fourth step, to make the overall model more lightweight, this invention improves the CBL module in the feature fusion stage. The original CBL module was composed of DarknetConv2D, Batch Normalization, and LeakyReLU. To further reduce model weight, this invention adopts a different convolution method and modifies the LeakyReLU function, proposing a new module: the DBR module. The proposed DBR module is composed of depthwise separable convolution, BatchNormalization, and ReLU6. Since depthwise separable convolution consists of channel-wise convolution and pointwise convolution, each channel in channel-wise convolution corresponds to a specific convolution kernel, and the number of channels remains unchanged after channel-wise convolution. Pointwise convolution uses 1*1 convolution to fuse the outputs of channel-wise convolution, and the number of output channels changes with the number of convolution kernels. Therefore, in the new DBR module, the use of depthwise separable convolution significantly reduces the computational cost of ordinary convolution. After convolution, Batch Normalization is used to prevent overfitting and gradient vanishing, and to accelerate convergence. Finally, ReLU6 is used as the activation function. In essence, Relu6 simply limits the maximum Relu value to 6, as shown in the following formula:

[0088] Relu = max(0, x)

[0089] Relu6 = min(6, max(0, x))

[0090] When the value of x is greater than 6, its derivative is 0. The main purpose of this is to obtain better numerical resolution in the low precision of the lightweight Float16 model. If not restricted, the output range can reach positive infinity, making the numerical value impossible to describe precisely. Therefore, ReLU6 effectively solves the problem of precision loss. Specific implementation examples:

[0092] The specific implementation process of this invention is as follows:

[0093] Step 1: Download the YOLOv4 pre-trained model file yolo4_voc_weights.h5 from the relevant website and place it in the model_data folder of the YOLOv4 model. Use it as the network's initialization parameter model.

[0094] Step 2: Place the self-made road obstacle dataset and related detection box annotation files into the Annotations, ImageSets, and JPEGImages folders under the VOC2007 folder.

[0095] Step 3: To make the model more robust and have stronger generalization ability, this invention will perform real-time Mosaic data augmentation. Four different images are stitched together, greatly enriching the background information of the detected objects.

[0096] Step 4: The parameters for model training in this invention are set as follows: The model uses a frozen training method, freezing the backbone network to make network training more efficient. The first stage is training with the backbone network frozen, with a maximum learning rate set to 1e-3, a batch size of 16, and the Adam optimizer selected for model optimization. The weight decay parameter `weight_decy` is set to 5e-4. The model trains for 40 epochs in this stage. The second stage is training after the backbone network is unfrozen, with a maximum learning rate set to 1e-4, a batch size of 4, and the Adam optimizer selected for model optimization. This stage consists of 60 epochs. The total number of training epochs is 100.

[0097] Step 5: After the entire network training is complete, the network will generate a weight model for each training epoch. The generated model will be named according to the current training epoch, the loss value obtained in the current epoch, and the validation error in the current epoch, and the generated files will be saved in .h5 format. Finally, the network performance will be evaluated using the model with the best training performance.

[0098] Step 6: Figure 1 This is the overall structure diagram of the present invention, which uses the MobilenetV3 backbone network on the basis of YOLOv4; ECA attention mechanism module; improved feature fusion network; and lightweight DBR convolution module.

[0099] Step 7: Figure 2This is the MobileNetV3 architecture used in this invention, all aimed at addressing the problem of slow detection speed due to model complexity. First, the network begins with a 2D convolution operation on the input image. Then, it enters a series of Bneck modules, where a linear bottleneck inverse residual structure processes the data, gradually increasing the number of data channels. In MobileNetV3, the network is largely composed of Bneck modules. Finally, the ending section consists of a 1*1 2D Conv2D with Batch Normalization, a 7*7 global average pooling, and two 1*1 Conv2D modules without Batch Normalization. The penultimate 1*1 Conv2D module introduces the h-swish activation function to improve model efficiency.

[0100] Step 8: Figure 3 This is a structural diagram of the ECA module inserted in this invention, all aimed at addressing the issue of insufficient overall model accuracy. The ECA attention mechanism consists of Global Average Pooling, 1D convolution, and the Sigmoid function. When the feature map enters the fusion network, an image of size H*W*C is input. After passing through a global average pooling layer, a feature of size 1*1*C is obtained. Then, a 1D convolution with kernel k is used to achieve local cross-channel interaction without dimensionality reduction, resulting in significant improvements while reducing model parameters. Finally, the Sigmoid function is used to make the network non-linear. This improves detection accuracy with minimal increase in model complexity.

[0101] Step 9: Figure 4 This invention proposes a novel feature fusion map. Based on the original model, a new feature layer is added. Features of a shallower layer (104*104) are extracted from the backbone network and convolved. The already fused features in PAN are upsampled, and the extracted 104*104 feature map is convolved and fused. After five convolutions in the CBL module, downsampling begins, and a top-down fusion operation is started. Then, it is fused with the 52*52 feature map before upsampling, and a downward convolution operation is performed. The original model fused features of three different scales (13*13, 26*26, and 52*52) in Cat(P) layers. 13 P 26 P 52 In this paper, a 104*104 feature map from a shallower layer in the backbone network is added for fusion operation Cat(P). 13 P 26 P 52 P 104After using four feature layers, the model obtains more detailed feature information. The fusion of three feature layers is increased to four, and the fusion is performed along the channel dimension. This fully utilizes four feature layers of different scales for upward and downward fusion, making feature fusion more complete and enhancing the exchange of information.

[0102] Step 10: Figure 5 This invention presents the novel DBR module diagram and its insertion location diagram. The module is composed of depthwise separable convolution, Batch Normalization, and ReLU6. While inserting the DBR module into the feature fusion structure makes the model lighter overall, excessive use can negatively impact accuracy. Therefore, this invention replaces part of the CBL module, inserting it at the extraction points of four different scale feature maps and before upsampling of feature maps with scales of 13*13 and 26*26. Finally, this paper finds that this approach not only makes the model lighter but also improves overall detection accuracy.

[0103] Step 11: Figure 6 This is a diagram of the depthwise separable convolution structure used in this invention. Unlike ordinary convolution, depthwise separable convolution includes channel-wise convolution and pointwise convolution. In channel-wise convolution, each channel corresponds to a specific convolution kernel and can only be convolved by one kernel; the number of channels remains unchanged after channel-wise convolution. When the feature map enters pointwise convolution, pointwise convolution uses a 1×1 convolution to fuse the output of the channel-wise convolution, and the number of output channels can change with the number of convolution kernels. Depthwise separable convolution has very high computational efficiency, which can greatly make the model lightweight.

[0104] Step 12: Finally, the entire training process is completed, learning from the data and enabling obstacle detection.

[0105] As can be seen from the above technical solutions, this invention addresses the problems faced in road obstacle detection, such as usage on mobile devices, occlusion and congested environments, small target detection, and complex backgrounds. To reduce network computation and make the model more lightweight, this invention uses MobileNetv3 instead of the original CSPDarkNet53 backbone network for feature extraction, making the overall model more compact. In the feature fusion network, this paper incorporates the ECA attention mechanism, allowing the model to focus on the relationships between channels and selectively learn useful features, thus improving overall accuracy. Finally, fusion features are added to the fusion network, extracting feature maps from relatively early convolutional layers and fusing them with the original structure to enrich the fusion information and improve small target detection performance. Furthermore, an improved CBL model is incorporated into the feature fusion network, significantly reducing the overall computational parameters. The improved model offers improved accuracy and is easier to embed into mobile devices.

[0106] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0111] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A road obstacle detection method based on an improved YOLOv4, characterized in that, The method for establishing a neural network model includes the following: Step 1: The method uses the lightweight MobileNetV3 as the backbone network to provide feature extraction operations for the model; the backbone network uses an inverse residual structure with a linear bottleneck, combined with depthwise separable convolution and SE attention mechanism; this structure forms a very compact representation at the input and output, while extending the internal structure to a higher-dimensional feature space, increasing the expressiveness of channel transformation; at the same time, a new activation function h-swish(x) is used, which enables the model to maintain good accuracy while improving running speed; Step 2: Improve the feature fusion network: In the feature extraction stage of the network, the shallow features extracted by the shallow network have more detailed information, clear texture line features, and higher feature resolution. In deeper networks, the features extracted by the network have a larger receptive field, resulting in richer semantic information; In the second step, to address the issue of poor detection performance for small targets, the feature information at different levels in the network layers is studied, and effective information fusion is achieved to improve detection performance. The improvement in the second step involves extracting feature maps from a shallower layer in the backbone network during the feature fusion stage. During the bottom-up propagation process, upsampling is used to fuse these features with semantically rich features from the original PAN structure, resulting in higher resolution features and more subtle information. Downsampling is used during the top-down process to combine these features, allowing the model to obtain more detailed information while also addressing the problem of insufficient semantic information in the new feature layers. Based on the original model, a new feature layer is added. Features from shallower layers of the backbone network are extracted and convolutional operations are performed. Features that have already been fused in PAN are upsampled, and the extracted feature maps are convolved and then fused. After performing 5 convolutions in the CBL module, downsampling is performed, and a top-down fusion operation is started. Then, after feature fusion with the feature map before upsampling, a downward convolution operation is performed. By adding a shallower feature map from the backbone network for fusion, the model obtains more detailed feature information after using four feature layers. The fusion of three feature layers is increased to four, and they are fused along the channel dimension. This fully utilizes four feature layers of different scales for upward and downward fusion, making feature fusion more complete and enhancing the exchange of information. Step 3: Use an efficient channel attention module in the model. This module avoids dimensionality reduction and effectively captures cross-channel interaction information. It maintains the number of channel layers after global pooling and uses a local cross-channel interaction strategy without dimensionality reduction. By learning the band matrix, it focuses on the relationship between different channels and learns different information in different channels in a focused manner. In local channel interactions, the module is both lightweight and ensures the effectiveness of local cross-channel interactions. In the third step, to improve the overall accuracy of the model, this invention incorporates an ECA module in the feature fusion stage. This allows the model to learn features with a focus, effectively improving the overall accuracy of the model. When features are output from the global average pooling layer and enter the 1D convolution, the 1D convolution does not perform dimensionality reduction, maintaining the original channel size. The focus of each channel is learned using the following formula: Where W is a C×C parameter matrix, σ is the sigmoid function, and the band matrix is ​​used in the ECA module. This allows us to focus on the channels, capture local cross-channel interactions, and ensure efficiency and effectiveness; y input value, These are the weights of each channel; The specific formula is as follows: The weights only need to be considered The interaction between its k neighboring channels is used for computation, and the weights are shared in 1D convolution, reducing the number of parameters from k*C to k. express The set of adjacent channels, σ is the sigmoid function; j is one of the k adjacent channels, and i is a channel from a different layer; this strategy is implemented using a fast convolution with a kernel size of k, as shown in the following formula: Wherein, C1D represents a 1D convolution; the above method is called by the ECA module, which only involves k parameters; In the third step, the module achieves both lightweighting and effectiveness in local cross-channel interactions. Specifically, the ECA module automatically determines the coverage of the interaction and adaptively adjusts the feature response values ​​between each channel. The interaction size between channels is proportional to the channel dimension C, and there is a mapping φ between K and C. in For a linear function: Because the channel size is set to a power of 2, this is achieved in ECA by extending the linear function. We can solve this problem by introducing it into a nonlinear function: After determining C, the size of the convolution kernel k is automatically determined according to the following formula: in This represents the odd number closest to t; and γ and b are set to 2 and 1 respectively; Step 4: A new DBR module is proposed, which consists of DepthwiseConv2D, BatchNormalization, and ReLU6. Due to the channel-wise and point-wise convolution operations in depthwise separable convolution, the computational parameters of the model are greatly reduced, thus reducing the model size. At the same time, in order to obtain better numerical resolution and avoid accuracy loss under the low precision of float16 on mobile devices, ReLU6 is used as the activation function in this module. To make the model more lightweight, the CBL module in the feature fusion stage was improved. To further reduce model weight, a different convolutional method was adopted, and the LeakyReLU function was also modified. A new module, the DBR module, was proposed, which is composed of depthwise separable convolution, batch normalization, and ReLU6. Details are as follows: 1) Since depthwise separable convolution is composed of channel-wise convolution and pointwise convolution, each channel in channel-wise convolution corresponds to a specific convolution kernel, and the number of channels does not change after channel-wise convolution; pointwise convolution uses 1*1 convolution to fuse the output of channel-wise convolution, and the number of output channels changes with the number of convolution kernels; therefore, in the new DBR module, the use of depthwise separable convolution greatly reduces the amount of computation brought by ordinary convolution. 2) Batch Normalization is used after convolution to prevent overfitting and gradient vanishing, and to speed up convergence. 3) Finally, ReLU6 is used as the activation function. ReLU6 limits the maximum ReLU value to 6, as shown in the following formula: When the value of x is greater than 6, its derivative is 0; this is done to obtain better numerical resolution in the low precision of the lightweight model Float16; if not restricted, the output range can reach positive infinity, making the numerical value impossible to describe precisely.

2. The road obstacle detection method based on YOLOv4 as described in claim 1, characterized in that, In the first step, the new activation function h-swish(x) selects ReLU6 in H-Swish to eliminate some numerical precision loss caused by approximating sigmoid. Moreover, the optimized implementation function of ReLU6 can be used in most hardware and software frameworks. The specific function is as follows: sigmoid and ReLU6 are functions, where x is the input value. These are constant coefficients; The bneck module in the network uses depthwise separable convolution and SE channel attention mechanism in the residual structure, and the module as a whole first increases the dimensionality and then decreases it.

3. The road obstacle detection method based on YOLOv4 as described in claim 2, characterized in that, The bneck module in the network uses depthwise separable convolutions and SE channel attention mechanisms in the residual structure. The overall module first increases the dimensionality and then reduces it, specifically including the following: 1) The module uses 1*1 convolution to map the low-dimensional space to the high-dimensional space, and after the 1*1 convolution, BatchNormalization is used to process the data format and add an activation function. 2) After the dimensionality increase, there is a 3*3 depthwise separable convolution, which is the same as the previous 1*1 convolution. After the depthwise separable convolution, there is also Batch Normalization and activation function processing. After extracting enough information through the depthwise separable convolution, the information enters the SE attention mechanism to learn important features. 3) Finally, after the output of the SE module, 1*1 convolution is used again for dimensionality reduction, and only Batch Normalization is used without using a non-linear activation function. However, there is another route in the Bneck module, which adds the features input to Bneck and the features output under the same channel conditions. This operation reduces the computational load of the network and does not cause information loss.

Images