Infrared target detection method for multiple types of traffic participants

Abstract

The invention discloses a multi-category traffic participant infrared target detection method. The method comprises the following steps: acquiring infrared image data of multi-category traffic participants in a traffic environment; training an infrared target detection model by using the infrared image data to obtain a trained detection model; and inputting to-be-detected infrared images of multiple types of traffic participants into the trained detection model, and outputting an infrared target detection result. According to the invention, accurate, real-time and stable detection can be carried out on multiple types of traffic participants in various complex infrared scenes.

Description

Technical Field

[0001] The invention relates to the field of infrared detection, and in particular to a method for detecting infrared targets of multiple categories of traffic participants. Background Art

[0002] With the continuous advancement of global urbanization, the number and categories of traffic participants (pedestrians, motor vehicles, non-motor vehicles, etc.) are increasing, and in real traffic environments, they often present high dynamics and multi-category mixing, which makes traffic safety issues more prominent. Unlike the principle of visible light imaging, thermal infrared imaging technology relies on target energy radiation to generate images and is not affected by changes in light conditions. Therefore, in traffic environments with extreme light such as dark nights, strong light, haze, or rain and snow, traffic monitoring systems and intelligent connected vehicles can still perform effective detection and identification.

[0003] However, infrared images often have shortcomings such as low signal-to-noise ratio, low resolution, and insufficient target texture details. In addition, most traffic embedded detection devices are limited by power consumption and storage, which leads to detection models that are prone to missed detection, false detection, and low real-time performance in complex background environments. In recent years, YOLO has achieved a better balance between detection accuracy and speed, and has therefore been applied by many scholars to the detection of traffic and pedestrian infrared targets. However, although the existing YOLO-based detection technology has achieved certain results in infrared target detection of a single traffic category, there are few studies on the simultaneous detection of multiple categories of traffic participants, especially for the complex urban traffic scenes that require accurate, real-time, and lightweight characteristics.

[0004] Therefore, there is an urgent need for an infrared target detection method for multiple categories of traffic participants, which can accurately, real-time and stably detect multiple categories of traffic participants in various complex infrared scenes. Summary of the invention

[0005] In view of this, the purpose of the present invention is to overcome the defects in the prior art and provide an infrared target detection method for multiple categories of traffic participants, which can accurately, real-time and stably detect multiple categories of traffic participants in various complex infrared scenes.

[0006] The infrared target detection method for multiple categories of traffic participants of the present invention comprises:

[0007] Collect infrared image data of multiple categories of traffic participants in traffic environments;

[0008] Using the infrared image data to train an infrared target detection model to obtain a trained detection model;

[0009] The infrared images of multiple categories of traffic participants to be tested are input into the trained detection model, and the infrared target detection results are output.

[0010] Furthermore, the infrared target detection model uses Yolov9m as a baseline model to integrate and improve the characteristics of traffic infrared targets.

[0011] Furthermore, the infrared target detection model uses a lightweight CE-MobileNetv3 network to replace the original backbone network of the baseline model;

[0012] The CE-MobileNetv3 network integrates the generalized efficient layer aggregation C-ELAN module into different feature layers of the MobileNetv3 network; the C-ELAN module combines CSPNet and ELAN, two neural network architecture modules designed with gradient path planning;

[0013] The C-ELAN module is used to obtain complete information in terms of information capture, achieve more efficient parameter usage through conventional convolution, and make up for false detection and missed detection caused by the fixed size of the convolution kernel.

[0014] Furthermore, the infrared target detection model uses a lightweight depth-separable convolution DSConv to replace the conventional convolution in the neck feature fusion layer and the CBL module of the baseline model;

[0015] The DSConv consists of two parts: channel-by-channel convolution and 1x1 point convolution. The channel-by-channel convolution performs independent convolution on each information channel; the 1x1 point convolution interacts and integrates feature information across channels and generates a new feature map.

[0016] Furthermore, the infrared target detection model introduces a Swin Transformer module in the feature fusion stage; wherein, residual connectivity and scale normalization of learnable parameters are performed before weighted summation of the Swin Transformer;

[0017] The architecture of the Swin Transformer consists of three parts: a local prior knowledge module, a self-attention module, and a global posterior detection knowledge module; wherein the local prior knowledge module is used to construct a hierarchical feature map; the self-attention module is used to perform self-attention calculations on the input sequence to capture long-distance dependencies in the input sequence; the global posterior detection knowledge module is used to capture global feature information in the sequence to improve the detection accuracy of small targets at a distance and occluded targets.

[0018] Furthermore, the infrared target detection model sets a small target detection head with a feature map size of 160x160 at the detection end of the baseline model Yolov9m to enhance the model's perception of local feature details of distant small targets.

[0019] The beneficial effects of the present invention are as follows: a multi-category traffic participant infrared target detection method disclosed in the present invention constructs a traffic participant infrared target lightweight detection model CNDS-YOLO that improves Yolov9m, and by designing a lightweight CE-MobileNetv3 network and replacing the entire backbone, the detection accuracy is improved while the model parameters are greatly reduced; the conventional convolution in the neck and CBL modules is replaced by a deep separable convolution (DSConv) to increase the real-time and lightweight deployment of the model detection; the normalized improved N-Swin Transformer module is integrated into the neck to efficiently enhance the capture and fusion capabilities of infrared target features; a small target detection head with a size of 160×160 is developed at the detection end to improve the detection sensitivity to small targets and occluded targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0021] Figure 1 It is a schematic diagram of the infrared target detection method of the present invention;

[0022] Figure 2 It is a schematic diagram of the CNDS-YOLO network model structure of the present invention;

[0023] Figure 3 A comparison diagram of the C-ELAN of the present invention and related architectures;

[0024] Figure 4 Schematic diagram of the structure of the depth-wise separable convolution of the present invention;

[0025] Figure 5 (a) is a schematic diagram of the Swin Transformer module principle of the present invention;

[0026] Figure 5 (b) is a schematic diagram of the shift window in the Swin Transformer module of the present invention;

[0027] Figure 5 (c) is a schematic diagram of the operation of two consecutive Swin Transformer modules of the present invention;

[0028] Figure 6 It is a schematic diagram of the improved detection layer structure of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings, as shown in the drawings:

[0030] This embodiment discloses a method for detecting infrared targets of multiple categories of traffic participants, including the following steps:

[0031] Collect infrared image data of multiple categories of traffic participants in traffic environments;

[0032] Using the infrared image data to train an infrared target detection model to obtain a trained detection model;

[0033] The infrared images of multiple categories of traffic participants to be tested are input into the trained detection model, and the infrared target detection results are output.

[0034] Aiming at the problems of missed detection, false detection, poor real-time and deployability in infrared target detection of multiple categories of traffic participants in intelligent connected vehicles and traffic monitoring systems, the present invention proposes a lightweight infrared target detection model CNDS-YOLO for traffic participants based on the fusion of Swin Transformer and Yolov9m. In terms of model improvement, firstly, the feature extraction backbone network of Yolov9m is replaced by the lightweight CE-MobileNetv3 network fused with C-ELAN, and DSConv is used to replace the conventional convolution in the neck feature fusion layer and CBL module, which greatly reduces the number of parameters and calculation of the model, and significantly improves the real-time and lightweight deployability of the detection. Secondly, the normalized improved N-SwinTransformer module is integrated into the feature fusion layer, and a small target detection head is developed at the detection end, which effectively utilizes the information input and semantic mapping from different detection layers, so that the model has excellent accurate detection performance, especially for infrared blurred small targets.

[0035] Yolov9m is a single-stage target detection algorithm with programmable gradient information based on YOLOv8. Like the previous YOLO series, the Yolov9m structure also has four main parts: input layer (Input), backbone feature extraction layer (backbone), neck feature fusion layer (neck) and detection layer (head). The difference is that in order to break through the information bottleneck limitation of CNN in the data feedforward process, Yolov9m designed a programmable gradient information layer (PGI) as the fifth part of the structure. PGI mainly consists of three parts: the main branch for process reasoning, the auxiliary reversible branch for receiving reliable gradient information, and the multi-level auxiliary information for aggregating all target object gradient information. It can effectively avoid the semantic loss that may be caused in the traditional multi-path feature fusion process, solve the problem of unreliable gradients caused by the deep information bottleneck generated by CNN, and thus eliminate the problem of error accumulation. The reasoning process of PGI only uses the backbone branch, so no additional reasoning is required. This paper uses Yolov9m as the baseline model to build the CNDS-YOLO network model for infrared target detection, and performs multiple fusions and improvements on the characteristics of traffic infrared targets. The detailed structure and location are as follows: Figure 2 shown.

[0036] In this embodiment, traffic participants often show high mobility and diversity in urban space operation, and infrared imaging has the characteristics of low contrast and unclear feature edge distinction. In order to avoid potential traffic hazards in time, the detection model needs to have good real-time and accuracy. To this end, the present invention uses a lightweight improved CE-MobileNetv3 network structure to replace the original backbone network of the baseline model.

[0037] MobileNetv3 is a feature extraction network architecture composed of inverted residual basic modules, Mobileet Block. While maintaining almost the same accuracy as traditional convolutional neural networks, it greatly reduces the amount of calculation and parameters, making the structure more lightweight. Compared with MobileNetv2, MobileNetv3 is mainly optimized in the following two aspects: First, 1x1 convolution is used in the feature generation layer to replace the network layer before average pooling, thereby removing the projection layer and filter layer in the bottleneck layer while retaining high-dimensional features, reducing the computational complexity. Secondly, MobileNetv3 uses the hardswish (h-swish) activation function to replace part of the swish nonlinear activation function, making the model calculation more efficient and the detection time on the mobile terminal shorter. The expression formula of the h-swish activation function is as follows:

[0038]

[0039] Where: x is the input feature value; ReLU6 is the activation function, and ReLU6=min[6,max(0,x)].

[0040] Since the detection model is mainly used for infrared target detection in complex traffic scenes, and the convolution kernel size in the MobileNet Block module is fixed, it is easy to cause confusion between the infrared target feature information and the background information, resulting in false detection and missed detection.

[0041] To this end, the present invention designs a C-ELAN module to address the above shortcomings. Figure 3 As shown in the figure, the C-ELAN module is a neural network architecture module designed by combining Cross Stage Partial Network (CSPNet) and ELAN, which are designed with gradient path planning. It can not only obtain complete information in terms of information capture and reduce feature pollution, but also achieve more efficient parameter utilization through conventional convolution, while avoiding the influence of inference stability due to changes in structural depth. The integration of the C-ELAN module can effectively make up for the false detection and missed detection caused by the fixed size of the convolution kernel.

[0042] By using the improved CE-MobileNetv3 architecture that integrates the C-ELAN module as the network backbone, the overall model becomes more lightweight, significantly improving the real-time detection and ease of deployment, while also improving detection accuracy.

[0043] In this embodiment, there are a large number of conventional convolutions in the neck feature fusion layer and CBL module of the baseline model Yolov9m. These convolutions reduce the model's reasoning speed due to the redundant number of parameters and hinder the deployment of the model on embedded devices. Therefore, the present invention uses a lightweight depthwise separable convolution DSConv to replace the conventional convolutions in the neck layer and CBL module. The specific location is as follows: Figure 2 shown.

[0044] Compared with conventional convolution, DSConv significantly reduces the number of parameters while achieving the same detection accuracy. Figure 4 As shown in Figure 1, DSConv mainly consists of two parts: channel-by-channel convolution and 1x1 point convolution. Channel-by-channel convolution performs independent convolution on each information channel, significantly reducing the number of parameters while ensuring the feature extraction capability remains unchanged; 1x1 point convolution interacts and integrates feature information across channels and generates new feature maps. In the process of depth-wise separable convolution, let the number of feature map inputs and outputs be C1 and C2, the height and width be H and W, and the convolution kernel size be K. The comparison of the amount of computation and the amount of parameters of conventional convolution and depth-wise separable convolution is shown in Table 1.

[0045] Table 1

[0046]

[0047] From the parameters in Table 1, we can see that the ratio of the calculation amount between the two is:

[0048]

[0049] The ratio of the parameters is:

[0050]

[0051] In summary, the ratio of the amount of computation and the amount of parameters of DSConv to conventional convolution is less than 1. Therefore, the application of DSConv not only improves the lightweight characteristics of the improved model, but also significantly improves the real-time performance of infrared detection of traffic participants and the convenience of model deployment on mobile embedded devices while ensuring accuracy.

[0052] In this embodiment, due to the limitations of the infrared imaging system, the infrared targets of multiple categories of traffic participants often show low contrast and resolution characteristics, which leads to the detection model being prone to cumulative deviations of feature information during data transmission. Therefore, the present invention introduces the Swin Transformer module in the model feature fusion stage to capture long-range dependencies in the input data and enhance the representation capability of the detection target feature information mapping. At the same time, a normalization process with learnable parameters and scale scaling is performed in the original residual structure of the Swin Transformer module to further achieve more efficient feature fusion.

[0053] Swin Transformer is a state-of-the-art computer vision task processing module based on the improved Transformer architecture. Its hierarchical attention mechanism breaks the limitations of traditional Transformer, which is high computational complexity and difficult to deploy on lightweight devices. The architecture of Swin Transformer mainly consists of three parts: local prior knowledge module, self-attention module and global post-inspection knowledge module.

[0054] The local prior knowledge module is used to construct a hierarchical feature map. The principle is as follows Figure 5 As shown in (a), it reduces the quadratic complexity of calculating self-attention in Transformer to linear complexity, thereby improving the detection rate of the model;

[0055] The self-attention module is used to perform self-attention calculation on the input sequence to capture the long-range dependencies in the input sequence, such as Figure 5As shown in (b), the shift window method it contains has lower latency than the previous sliding window-based method, which improves the model convergence speed;

[0056] The global post-detection knowledge module captures the global feature information in the sequence and improves the detection accuracy of the model for small targets at a long distance and occluded targets. Figure 5 As shown in (c), Swin Transformer processes the target image mainly by constructing a hierarchical feature map to divide the input image or video into different small visual block images. The small Transformer module of each layer processes the image information of the visual block window and exchanges information between layers to enhance the extraction and fusion of global information.

[0057] In order to ensure that Swin Transformer can accurately allocate weights of data streams of different paths, reduce the influence of redundant features, and promote adaptive learning of correlations between target features, the present invention performs normalization processing of residual connectivity and scale scaling of learnable parameters before weighted summation of Swin Transformer. The principle formula is as follows:

[0058]

[0059] Where: W i is the normalized parameter; X i is a non-standardized learning parameter.

[0060] This scaling normalization method simplifies the calculation while maintaining the relative proportionality of the original data, scaling the data to the range of [0,1] by dividing each value by the sum. In addition, the scaling is insensitive to outliers, and even the presence of abnormal data will not significantly affect the fluctuation of the results. The normalization of the residual network not only enhances the stability and generalization of the model, but also speeds up the convergence of the model during training.

[0061] The integration of the normalized and improved N-Swin Transformer module enables the model to accurately capture the long-distance dependencies in the input data, thereby achieving stable and efficient calculations. In many scenarios where multiple types of traffic participants are mixed in the city, the N-Swin Transformer module can effectively improve the model's efficiency in capturing the blurred detail features of infrared targets, thereby improving the overall accuracy and speed of the model detection.

[0062] In this embodiment, in a complex traffic scene where multiple types of traffic participants gather densely, targets often intersect and block each other. At the same time, due to the low resolution and severe noise characteristics of infrared targets, it is difficult for the model to detect the feature information of small distant targets, resulting in a large number of missed detections and false detections. The baseline model Yolov9m has three feature detection heads of different sizes: 20×20, 40×40, and 80×80 on the detection layer. Although the small-sized detection head has a wide receptive field and can obtain rich target feature information, it will lose a lot of local detail information during the downsampling process. Therefore, it is only suitable for the detection of larger targets. Despite the reversible branch information transmission of the PGI structure and the global performance enhancement of the N-SwinTransformer module, there is a lack of matching detection heads to fully apply the model performance to infrared extremely small targets.

[0063] Therefore, the present invention develops a small target detection head with a feature map size of 160x160 at the detection end of the baseline model Yolov9m, and the position is as follows Figure 2 As shown in the figure, the model can enhance its ability to perceive the local feature details of distant small targets. The improved detection layer structure is as follows Figure 6 In summary, the development of a large-scale detection head can effectively improve the model's sensitivity to target details, thereby enhancing the accuracy and reliability of the model's detection of blurred small targets and overlapping targets.

[0064] In this embodiment, the FLIR2 thermal imaging dataset collected and annotated by the Teledyne FLIR team can be used for model training and performance testing. The dataset contains 11,688 infrared target images taken from the driver's perspective and annotated, all of which are acquired synchronously by thermal imaging and visible light cameras installed on the vehicle. The scenes cover a variety of complex environments where visible light cannot be imaged normally, mainly including 7 extreme light scenes such as glare, extreme darkness and strong light; 9 extreme weather such as smoke, rain and dust; 16 types of traffic participants such as pedestrians, cars, and bicycles.

[0065] Most of the above scenes contain small distant targets, mutual occlusion, and cluttered backgrounds, which can fully train or test the detection performance of the model. 8356 thermal infrared images in the data set were selected and randomly divided into three parts: training set, validation set, and test set in a ratio of 6:2:2.

[0066] The equipment for model training and performance testing of the present invention adopts an existing computer. During the training process of the detection model of the present invention, the infrared image input resolution is set to 640×640, the training rounds are 300 rounds, the initial learning rate is 0.002, the warm-up learning rate is 3.0, the weight decay is 0.0005, and the batch size is 32.

[0067] Compared with the baseline model, the infrared target detection model (CNDS-YOLO model) of the present invention has significantly reduced the number of parameters and the amount of calculation, which are reduced by 31.0% and 35.5% in terms of the number of parameters and the number of floating-point operations (FLOPs), respectively, and the FPS and mAP@0.5 / % are increased by 24.2% and 7.0%, respectively; compared with the existing mainstream infrared target detection models, the CNDS-YOLO model has obvious advantages in comprehensive performance, is more suitable for real-time detection of embedded infrared devices, and can effectively meet the infrared real-time detection tasks of embedded devices in urban traffic.

[0068] Through the infrared target detection method of the present invention, the accuracy, real-time and lightweight deployment convenience of the overall model for infrared target detection of multiple categories of urban traffic participants in complex environments are improved, providing reliable technical support for intelligent safety detection and risk identification of intelligent connected vehicles and traffic monitoring systems.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting infrared targets of multiple categories of traffic participants, characterized by: include: Collect infrared image data of multiple categories of traffic participants in traffic environments; Using the infrared image data to train an infrared target detection model to obtain a trained detection model; The infrared images of multiple categories of traffic participants to be tested are input into the trained detection model, and the infrared target detection results are output.

2. The infrared target detection method for multiple categories of traffic participants according to claim 1 is characterized by: The infrared target detection model uses Yolov9m as the baseline model and integrates and improves the characteristics of traffic infrared targets.

3. The infrared target detection method for multiple categories of traffic participants according to claim 2 is characterized by: The infrared target detection model uses a lightweight CE-MobileNetv3 network to replace the original backbone network of the baseline model; The CE-MobileNetv3 network integrates the generalized efficient layer aggregation C-ELAN module into different feature layers of the MobileNetv3 network; the C-ELAN module combines CSPNet and ELAN, two neural network architecture modules designed with gradient path planning; The C-ELAN module is used to obtain complete information in terms of information capture, achieve more efficient parameter usage through conventional convolution, and make up for false detection and missed detection caused by the fixed size of the convolution kernel.

4. The infrared target detection method for multiple categories of traffic participants according to claim 2 is characterized by: The infrared target detection model uses a lightweight depth-separable convolution DSConv to replace the conventional convolution in the neck feature fusion layer and CBL module of the baseline model; The DSConv consists of two parts: channel-by-channel convolution and 1x1 point convolution. The channel-by-channel convolution performs independent convolution on each information channel; the 1x1 point convolution interacts and integrates feature information across channels and generates a new feature map.

5. The infrared target detection method for multiple categories of traffic participants according to claim 2 is characterized by: The infrared target detection model introduces a Swin Transformer module in the feature fusion stage; wherein, residual connectivity and scale normalization of learnable parameters are performed before the Swin Transformer weighted summation; The architecture of the Swin Transformer consists of three parts: a local prior knowledge module, a self-attention module, and a global posterior detection knowledge module; wherein the local prior knowledge module is used to construct a hierarchical feature map; the self-attention module is used to perform self-attention calculations on the input sequence to capture long-distance dependencies in the input sequence; the global posterior detection knowledge module is used to capture global feature information in the sequence to improve the detection accuracy of small targets at a distance and occluded targets.

6. The infrared target detection method for multiple categories of traffic participants according to claim 2 is characterized by: The infrared target detection model sets a small target detection head with a feature map size of 160x160 at the detection end of the baseline model Yolov9m to enhance the model's perception of local feature details of distant small targets.

Images