Automatic driving infrared image target detection method based on YOLOv5

Abstract

The invention relates to an automatic driving infrared image target detection method based on YOLOv5, relates to the technical field of target detection, and aims to solve the problems of less infrared image feature information, system complexity reduction, target and background confusion and the like. According to the method, a scheme based on YOLOv5 is adopted, and in order to extract effective feature information from an infrared image with less feature information, a novel multi-scale multi-branch fusion (CSP-MSMBF) module is designed to replace an original feature extraction module. In order to solve the problem of target and background blurring, a diagonal coordinate attention mechanism (DCA) is designed, and the model expressive force is enhanced. A sparse dynamic receptive field convolution up-sampling (SDRFCU) module is designed to solve the problem that a target is not prominent and realize more accurate feature reconstruction. The method is high in adaptability, can quickly and accurately detect the target in all weathers, and is more suitable for target detection tasks of the automatic driving automobile under various weather conditions.

Description

Technical Field

[0001] The present invention relates to an infrared image target detection method for autonomous driving based on YOLOv5, and belongs to the field of target detection. Background Art

[0002] Autonomous driving refers to the use of a series of advanced technologies and systems to enable cars to drive themselves without human intervention. It mainly relies on sensors, computer vision, deep learning, artificial intelligence and other technologies to complete the driving task from the starting point to the end point through perception, decision-making, control and other links. The goal of autonomous driving is to improve road safety, improve traffic efficiency, and ultimately achieve a fully automated driving experience. Environmental perception is the basis and prerequisite for autonomous driving, and target detection is the basis of environmental perception. Therefore, only when the car has the ability to accurately locate and identify surrounding targets can it ensure that autonomous driving can perform tasks correctly.

[0003] At present, the main method for automobiles to achieve environmental perception is to perceive the environmental information around the vehicle through cameras, laser radars, ultrasonic sensors, etc., to identify obstacles, pedestrians, lane lines, traffic signs, etc. Although laser radars can provide more accurate environmental information, they are expensive and easily affected by weather. Some high-performance laser radars are large in size and heavy in weight, which may affect the design of the vehicle or increase energy consumption. In addition, the collected data requires efficient calculations and algorithms to process and analyze, which increases the complexity of the system. Although ultrasonic sensors are low in cost and small in size, their detection distance is short and they are affected by the surface material of the object, which will affect the detection effect. At the same time, they will also cause inaccurate measurements or data loss under harsh environmental conditions. Visible light cameras are widely used in autonomous driving and provide rich environmental information. They still face challenges in terms of lighting, weather, depth perception, etc. Compared with the above technologies, infrared imaging does not rely on visible light, but uses the infrared heat radiated by objects to form images, so it can provide clear images even in completely dark environments. Infrared imaging has less interference with visible light. For example, in environments such as strong sunlight, backlight, reflection or glare, infrared imaging can still provide stable imaging. It has better penetration ability in adverse weather conditions (such as fog, snow, rain, etc.). Unlike visible light cameras, infrared imaging is not easily affected by particulate matter such as haze and rain, so it can provide clear images in these environments. Therefore, infrared imaging can provide reliable data all day long, and the cost is low, which can effectively reduce the cost of self-driving cars.

[0004] Although infrared imaging has many advantages in autonomous driving, it also has some disadvantages and limitations. The resolution of infrared imaging is usually lower than that of visible light cameras. Their resolution may not be enough to clearly display details at a distance in some low-end devices, especially when detecting small objects or distant targets, which may not provide enough details for accurate recognition. Infrared imaging can only sense the temperature distribution of objects, but cannot provide color and texture information of objects. Although it can effectively detect heat sources and distinguish different objects, it cannot provide rich details like visible light cameras. Current object detection technologies are mainly divided into two categories, namely CNNs-based methods and Transformer-based methods, among which Transformer-based methods often require a large time cost. In autonomous driving systems, especially in real-time scenarios, computational efficiency and response time are very important. Compared with Transformers, CNNs have less computational complexity and convolution operations are relatively efficient, and their computational complexity is usually smaller than that of Transformers. In the field of autonomous driving, CNNs have been widely used, and there are many proven model architectures that perform well in tasks.

[0005] Based on this, in order to solve the problems of less feature information in infrared images and reduce system complexity, the mature and relatively simple single-stage detection network YOLOv5 is selected as the basic network. The feature extraction capability of the original network backbone is limited. In order to extract effective feature information in infrared images with less feature information, a new multi-scale multi-branch fusion (CSP-MSMBF) module is designed to replace the original feature extraction module. In order to solve the low resolution of infrared images and the blurring of targets and backgrounds, the diagonal coordinate attention mechanism (DCA) is designed to be added to the backbone to enhance the model's expressiveness. In order to solve the problem of the target not being prominent and the loss of target feature information during feature reconstruction, the sparse dynamic receptive field convolution upsampling (SDRFCU) module is designed to replace the original network neck upsampling module. The target weight is dynamically calculated during upsampling, and the non-target weight is reduced, making the target feature more prominent and improving the model's recognition ability. Summary of the invention

[0006] The purpose of this invention is to provide an infrared image target detection method for autonomous driving based on YOLOv5. It effectively solves the problems of insufficient feature information of infrared images, difficulty in extracting effective features, and confusion between targets and backgrounds in target detection. The technical solution is as follows:

[0007] The objective of the present invention is to achieve the following technical solution: a method for infrared image target detection in autonomous driving based on YOLOv5, which comprises the following steps:

[0008] (1) The existing publicly available vehicle driving infrared image datasets were collected and organized online, and the data was preprocessed using the mosaic data enhancement method provided by YOLOv5 to improve the model stability and generalization.

[0009] (2) A new multi-scale multi-branch fusion (CSP-MSMBF) module is designed in the backbone network part of the YOLOv5 network to replace the original feature extraction module, so that the backbone network can fully extract the feature information in the infrared image.

[0010] (3) The diagonal coordinate attention mechanism (DCA) module is introduced to capture image context information in multiple directions, enabling the entire network to more comprehensively understand the spatial relationships in the image and improve the expressiveness of the model.

[0011] (4) A sparse dynamic receptive field convolution upsampling (SDRFCU) module is designed in the neck part of the network to replace the original network neck upsampling module, dynamically calculate the target weight, reduce the non-target weight, achieve more accurate feature reconstruction, and improve the model recognition ability.

[0012] (5) The processed images are sent to the modified network for training until the model converges.

[0013] (6) Input the image to be predicted into the trained model for detection and obtain the detection result.

[0014] Furthermore, the step (1) specifically includes the following steps:

[0015] Step (11) collects and organizes the existing publicly available vehicle driving infrared image datasets on the Internet, and pre-processes the data using the mosaic data enhancement method provided by YOLOv5 to improve the model stability and generalization.

[0016] Further, the step (2) specifically includes the following steps:

[0017] Step (21) A new multi-scale multi-branch fusion (CSP-MSMBF) module is designed in the backbone network part of the YOLOv5 network to replace the original feature extraction module. The module adopts a multi-scale and multi-branch structure design, so that the backbone network can fully extract the feature information in the infrared image.

[0018] Further, the step (3) specifically includes the following steps:

[0019] Step (31) introduces the diagonal coordinate attention mechanism (DCA) module, which pools the features in the horizontal, vertical, main diagonal and anti-diagonal directions and calculates the attention weights, so that the entire network can capture the image context information in multiple directions, and can more comprehensively understand the spatial relationship in the image and improve the expressiveness of the model.

[0020] Further, the step (4) specifically includes the following steps:

[0021] Step (41) designs a sparse dynamic receptive field convolution upsampling (SDRFCU) module in the neck part of the network to replace the original network neck upsampling module. The sparse upsampling module enables the network to focus on the target area, dynamically calculate the target weight, reduce the non-target weight, and perform feature reconstruction through dynamic receptive field convolution recognition to achieve more accurate feature reconstruction and improve model recognition ability.

[0022] Further, step (5) specifically includes the following steps:

[0023] Step (51) sends the processed image to the modified network for training until the model converges.

[0024] Further, step (6) specifically includes the following steps:

[0025] Step (61) inputs the image to be predicted into the trained model for detection to obtain the detection result.

[0026] Compared with the existing driver status perception method, the advantages of the present invention are:

[0027] (1) Compared with mainstream autonomous driving environment perception equipment, the invention has a relatively low cost, can work around the clock, has low system complexity, is easy to train, and has good portability.

[0028] (2) This invention uses the single-stage detector YOLOv5 as the basic framework, and improves the accuracy of autonomous driving target detection by changing the bottleneck structure and adding an attention mechanism.

[0029] (3) The present invention integrates a multi-branch multi-scale feature extraction module into the YOLOv5 network structure to achieve adaptability to different target object sizes. At the same time, sparse upsampling is used to focus on the target area, improving the problem of small target recognition and the occlusion problem between objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Improve the overall network framework of YOLOv5

[0031] Figure 2 Multi-scale multi-branch fusion (CSP-MSMBF) module structure diagram

[0032] Figure 3 Diagonal Coordinate Attention (DCA) module structure diagram

[0033] Figure 4 Sparse Dynamic Receptive Field Convolution Upsampling (SDRFCU) module structure diagram DETAILED DESCRIPTION

[0034] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings.

[0035] A method for detecting infrared images of autonomous driving based on YOLOv5, comprising the following steps:

[0036] (1) The existing publicly available vehicle driving infrared image datasets were collected and organized online, and the data was preprocessed using the mosaic data enhancement method provided by YOLOv5 to improve the model stability and generalization.

[0037] (2) A new multi-scale multi-branch fusion (CSP-MSMBF) module is designed in the backbone network part of the YOLOv5 network to replace the original feature extraction module, so that the backbone network can fully extract the feature information in the infrared image.

[0038] (3) The diagonal coordinate attention mechanism (DCA) module is introduced to capture image context information in multiple directions, enabling the entire network to more comprehensively understand the spatial relationships in the image and improve the expressiveness of the model.

[0039] (4) A sparse dynamic receptive field convolution upsampling (SDRFCU) module is designed in the neck part of the network to replace the original network neck upsampling module, dynamically calculate the target weight, reduce the non-target weight, achieve more accurate feature reconstruction, and improve the model recognition ability.

[0040] (5) The processed images are sent to the modified network for training until the model converges.

[0041] (6) Input the image to be predicted into the trained model for detection and obtain the detection result.

[0042] Further, step (1) specifically includes the following steps:

[0043] Step (11) collects and organizes the existing public vehicle driving infrared image datasets on the Internet, and preprocesses the data using the mosaic data enhancement method provided by YOLOv5. The idea of ​​the mosaic data enhancement method is to first randomly select four images from the dataset, then scale them, randomly crop them, and splice them together in a mosaic mode. The specific implementation steps are as follows:

[0044] 1) Initialize the entire background image.

[0045] 2) Randomly select a center point.

[0046] 3) Based on the center point, the four images are placed in the upper left, upper right, lower left, and lower right respectively. This part may be because the center point is smaller than the width and height of the four images.

[0047] 4) When stitching, the images may be cropped and the offset of the label border may be recalculated.

[0048] Further, step (2) specifically includes the following steps:

[0049] Step (21) A new multi-scale multi-branch fusion (CSP-MSMBF) module is designed in the backbone network part of the YOLOv5 network to replace the original feature extraction module. The structure of the multi-scale multi-branch fusion (CSP-MSMBF) module is as follows: Figure 2 As shown in the figure. This module adopts a multi-branch structure, which consists of 1×1 convolution, 3×3 convolution, and 5×5 convolution to form the initial three branches. The 1×1 convolution is used to retain the original feature information. The 3×3 convolution and 5×5 convolution are followed by n multi-branch feature extraction modules (MBFEM) of corresponding convolution kernel sizes, which are used to extract features of infrared images at different scales to extract as much feature information as possible. The multi-branch feature extraction module (MBFEM) also adopts a multi-branch structure design, which consists of 1×K convolution and K×1 convolution, where K is the convolution kernel size of the current branch, and is intended to extract feature information from height or width respectively. Finally, each branch performs feature fusion to obtain feature information of each scale and each branch.

[0050] Further, step (3) specifically includes the following steps:

[0051] Step (31) introduces the diagonal coordinate attention mechanism (DCA) module, such as Figure 3 As shown in the figure, the input feature map is pooled in the horizontal, vertical, main diagonal and anti-diagonal directions to help the model pay attention to spatial information of different scales, making the model more robust when processing images with complex structures. The pooled features are then concatenated in the height direction to obtain feature y, which is then convolved, batch normalized and activated. Feature y is segmented to obtain features x in each direction. h , x w , x main_diag , x anti_diag The features in each direction are convolved and sigmoid operated to obtain the corresponding attention weights. Finally, the input feature x is multiplied by the attention mechanism in each direction to obtain the final output.

[0052] Furthermore, step (4) specifically includes the following steps:

[0053] Step (41) designs a sparse dynamic receptive field convolution upsampling (SDRFCU) module in the neck part of the network to replace the original network neck upsampling module. First, 1×1 convolution is used for cross-channel interaction, and then a sparse upsampling module (SUM) is used. The sparse upsampling module (SUM) uses an interpolation method to upsample the input x, generates an "importance region" by calculating the absolute value, and then generates a sparse mask through a threshold. Finally, the important region is retained based on the sparse mask and other regions are ignored. After the sparse upsampling module, a dynamic receptive field convolution (DRFConv) is used to generate features with a dynamic receptive field, which are then fused with the input, and finally the number of output channels is adjusted through a 1×1 convolution. This module dynamically calculates the target weight, reduces the non-target weight, achieves more accurate feature reconstruction, and improves the model recognition capability.

[0054] Further, step (5) specifically includes the following steps:

[0055] Step (51) inputs the data set into the target detection network model, configures the corresponding training environment, sets the corresponding training parameters, and carries out the model training task.

[0056] In step (52), when configuring the training environment, the software parameters are: the operating system version is Ubuntu 22.04, the deep learning framework version is PyTorch 2.3.0, the programming language is Python 3.12, and the hardware parameters are: GPU: RTX3090 (24G), CPU: Intel (R) Xeon (R) Platinum 8358P CPU @ 2.60GHz, and the memory is 80G.

[0057] In step (53), in the configuration of training parameters, the optimizer uses SGD, the training rounds are 300, the batch size is set to 4, and the remaining parameters are default configurations.

[0058] Further, step (6) specifically includes the following steps:

[0059] Step (61) loads the trained weight file into the target detection network, randomly selects images from the test data set for detection, and returns labeled images with target and category information.

[0060] It will be apparent to those skilled in the relevant art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the appended claims rather than the above description, and therefore it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A method for infrared image target detection in autonomous driving based on YOLOv5, which is characterized by comprising the following steps: (1) The existing publicly available vehicle driving infrared image datasets were collected and organized online, and the data was preprocessed using the mosaic data enhancement method provided by YOLOv5 to improve the model stability and generalization. (2) A new multi-scale multi-branch fusion (CSP-MSMBF) module is designed in the backbone network part of the YOLOv5 network to replace the original feature extraction module, so that the backbone network can fully extract the feature information in the infrared image. (3) The diagonal coordinate attention mechanism (DCA) module is introduced to capture image context information in multiple directions, enabling the entire network to more comprehensively understand the spatial relationships in the image and improve the expressiveness of the model. (4) A sparse dynamic receptive field convolution upsampling (SDRFCU) module is designed in the neck part of the network to replace the original network neck upsampling module, dynamically calculate the target weight, reduce the non-target weight, achieve more accurate feature reconstruction, and improve the model recognition ability. (5) The processed images are sent to the network modified by the multi-scale multi-branch fusion (CSP-MSMBF) module, the diagonal coordinate attention mechanism (DCA) module, and the sparse dynamic receptive field convolution upsampling (SDRFCU) module for training until the model converges. (6) Load the trained weight file into the target detection network, randomly select images from the test data set for detection, return the labeled images with target and category information, and obtain the detection accuracy of each category in the infrared image data set and the overall average detection accuracy.

2. According to claim 1, in step (2), a new multi-scale multi-branch fusion (CSP-MSMBF) module is designed in the backbone network part of the YOLOv5 network to replace the original feature extraction module. The module adopts a multi-branch structure, and is composed of 1×1 convolution, 3×3 convolution, and 5×5 convolution to form the initial three branches, wherein the 1×1 convolution is used to retain the original feature information, and the 3×3 convolution and 5×5 convolution are followed by n multi-branch feature extraction modules (MBFEM) of corresponding convolution kernel sizes, which are used to extract features of infrared images at different scales, so as to extract as much feature information as possible. The process can be expressed as: out=Conv1(Concat(Conv1(x),Concat(MBFEM(Conv3(x)),MBFEM(Conv5(x))))) in, Convk(·) represents a convolution operation with a kernel size of k, Concat(·) represents a feature concatenation operation, MBFEM(·) represents a multi-branch feature extraction operation, x represents the input feature map, and out represents the output feature map. The multi-branch feature extraction module (MBFEM) also adopts a multi-branch structure design, which consists of 1×K convolution and K×1 convolution, where K is the convolution kernel size of the current branch, which is intended to extract feature information from the height or width respectively. The first branch is composed of 1×K convolution and K×1 convolution, the second branch is composed of K×1 convolution and 1×K convolution, and the third and fourth branches are composed of 1×K convolution and K×1 convolution respectively. Finally, the fusion and addition are used to obtain the output using the SiLU activation function.

3. According to claim 1, step (3) introduces a diagonal coordinate attention mechanism (DCA) module to perform pooling operations on the input feature map in the horizontal, vertical, main diagonal and anti-diagonal directions, helping the model to pay attention to spatial information of different scales, making the model more robust when processing images with complex structures. The pooled features are then concatenated in the height direction to obtain feature y, which is then convolved, batch normalized and activated. Feature y is segmented to obtain features x in each direction. h , x w , x main_diag , x anti_diag . Perform convolution and sigmoid operations on the features in each direction to obtain the corresponding attention weights, and finally multiply the input feature x with the attention mechanism in each direction to obtain the final result. The specific process is as follows: For the input image x∈R N×C×H×W Perform adaptive average pooling in the vertical (height) direction: x h =pool h (x)∈R N×C×H×1 in, pool h It is the adaptive average pooling in the vertical direction. Perform adaptive average pooling in the horizontal (width) direction on the input image: x w =pool w (x)∈R N×C×1×W Among them, pool w It is the adaptive average pooling in the horizontal direction. Extract features along the main diagonal from the input image: x main-diag =extract_main_diag(x)∈R N×C×1×1 Among them, extract_main_diag(x) means extracting the main diagonal features of the image. Extract features along the anti-diagonal line from the input image: x anti-diag =extract_anti_diag(x)∈R N×C×1×1 Among them, extract_anti_diag(x) means extracting the anti-diagonal features of the image. Then the pooled features are concatenated in the height direction to obtain feature y, followed by convolution, batch normalization and activation function. Feature y is segmented to obtain features x in each direction. h , x w , x main_diag , x anti_diag . Perform convolution and sigmoid operations on the features in each direction to obtain the corresponding attention weights, which can be expressed as: a h =Sigmoid(Conv h (x h ))∈R N×C×1×1 a w =Sigmoid(Conv w (x w ))∈R N×C×1×1 a main_diag =Sigmoid(Conv main_diag (x main_diag ))∈R N×C×1×1 a anti_diag =Sigmoid(Donv anti_diag (x anti_diag ))∈R N×C×1×1 Finally, the input feature x is multiplied by the attention mechanism in each direction to get the final output: out=x×a h ×a w ×a main_diag ×a anti_diag Among them, the multiplication is element-by-element (i.e., attention is weighted by element).

4. According to claim 1, in step (4), a sparse dynamic receptive field convolution upsampling (SDRFCU) module is designed in the neck part of the network to replace the original network neck upsampling module. First, 1×1 convolution is used for cross-channel interaction, and then a sparse upsampling module (SUM) is used. Specifically, it can be expressed as: x=Conv 1×1 (x)∈R B×C×H×W x up =SUM(x) in, The sparse upsampling module (SUM) consists of the following steps: Upsample the input x using interpolation: upsample x =interpolate(x) The "important region" is generated by calculating the absolute value, and then a sparse mask is generated by the threshold. Finally, the important region is retained according to the sparse mask, and other regions are ignored. The specific process can be expressed as: importance_map=|upsample_x| sprse_mask=importance_map>threshold sparse_x=upsample_x×spares_mask After the sparse upsampling module, the dynamic receptive field convolution (DRFConv) is used to generate features with dynamic receptive fields, which are then fused with the input features, and finally the number of output channels is adjusted through 1×1 convolution. This module dynamically calculates the target weights, reduces the non-target weights, achieves more accurate feature reconstruction, and improves the model recognition ability.

Images