A visual navigation method for intelligent driving based on infrared target detection

Abstract

An intelligent driving visual navigation method based on infrared target detection, which obtains an infrared image of a driving environment and uses a detection model to extract features; an intelligent driving control system formulates a corresponding driving strategy according to the identified detection target position; the detection model backbone network part has multiple feature extraction modules; the feature extraction module includes a FasterNet module, a dynamic channel feature extraction module and multiple PCBS modules composed of partial convolution, batch normalization and activation functions; the dynamic channel feature extraction module includes a maximum pooling layer and a linear layer; the output of the FasterNet module is superimposed with the output of the PCBS module and then fused with the output of the dynamic channel feature extraction module as the output of the feature extraction module, so as to obtain richer deep and shallow feature information, enhance the model's dynamic attention to important features, thereby improving the feature extraction capability, accurately identifying infrared targets, and improving the reliability of intelligent driving visual navigation.

Description

Technical Field

[0001] The invention relates to an intelligent driving visual navigation method, in particular to an intelligent driving visual navigation method based on infrared target detection. Background Art

[0002] Intelligent driving is a technology that uses artificial intelligence to assist or replace human driving. It can break through the limitations of human perception of the surrounding environment and human reaction speed, bringing greater convenience and safety to driving. Visual navigation is a navigation technology that uses computer vision technology to perceive the vehicle's surrounding environment and uses it as the basis for intelligent driving decisions.

[0003] With the vigorous development of computer vision technology, image target detection has been widely used in the field of autonomous driving. However, under insufficient lighting conditions, visible light images are blurred and textures are missing, making it difficult to extract target features and inefficient to detect. In contrast, infrared imaging technology is not limited by lighting conditions and can obtain clear thermal distribution images, providing a new solution for target detection in low-light or dark environments. In recent years, deep learning technology has been widely used in the field of target detection, among which YOLO performs well in complex scenes and scenes with high real-time requirements, so it is often used in combination with infrared imaging technology for target detection. However, the capacity of the usual YOLO model is relatively large, making it difficult to balance detection accuracy and speed, and it is difficult to effectively extract depth and semantic information in complex scenes, resulting in problems such as false detection and missed detection, low detection efficiency, and affecting the reliability of intelligent driving visual navigation. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an intelligent driving visual navigation method based on infrared target detection to improve the reliability of intelligent driving visual navigation.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: an intelligent driving visual navigation method based on infrared target detection, which uses an infrared camera to obtain an infrared image of a driving environment, uses a detection model to extract features of the acquired infrared image, and identifies the detection target; the intelligent driving control system formulates a corresponding driving strategy according to the identified detection target position to achieve intelligent driving, the detection model uses YOLOv5s as a model framework, and its backbone network part has a plurality of feature extraction modules for basic feature extraction; the feature extraction module includes a FasterNet module, a dynamic channel feature extraction module and a plurality of PCBS modules composed of partial convolution, batch normalization and activation function; wherein the dynamic channel feature extraction module includes a maximum pooling layer and a linear layer arranged in sequence; the feature map of the input feature extraction module The feature map F is first processed by the first PCBS module to obtain the feature map F, which is divided into three paths. One path is sent to the dynamic channel feature extraction module for processing, one path is sent to the FasterNet module after being processed by the second PCBS module, and the other path is processed by the third PCBS module. The output of the FasterNet module is superimposed with the output of the third PCBS module and fused with the output of the dynamic channel feature extraction module as the output of the feature extraction module.

[0006] The FasterNet module includes a partial convolution layer, a point convolution layer and an inverted point convolution layer. The feature map input to the FasterNet module is processed by the partial convolution layer, and the point convolution layer performs point-by-point convolution on the output of the partial convolution layer. After batch normalization and activation, it passes through the inverted point convolution layer, and the output of the inverted point convolution layer forms a residual connection with the input of the FasterNet module.

[0007] The dynamic channel feature extraction module uses a maximum pooling layer to perform a maximum pooling operation on the input feature map F to extract the significant features of each channel, and then uses a linear layer to process the extracted significant features to generate dynamic correlation features to characterize the correlation between different channels.

[0008] The dynamic correlation features generated by the dynamic channel feature extraction module are used to dynamically weight the features extracted by the FasterNet module to obtain fused features.

[0009] The backbone network of the detection model has multiple feature extraction modules, and the feature extraction modules are used to replace the C3 module in the YOLOv5s backbone network.

[0010] The backbone network of the detection model has a cross-parallel fusion attention module, which uses parallel channel attention and spatial attention to extract features from feature maps.

[0011] The cross-parallel fusion attention module includes a channel attention module and a spatial attention module, and the feature map of the cross-parallel fusion attention module is input Split in one dimension and Two parts, Sent to the channel attention module, Sent to the spatial attention module, the outputs of the channel attention module and the spatial attention module are concatenated and re-aggregated in the original dimension.

[0012] The channel attention module uses average pooling and maximum pooling operations to aggregate feature maps Channel information of the , generating average aggregation features and the maximum aggregate feature , and then generate channel attention weights through the shared network MLP , the calculation formula is:

[0013] ;

[0014] in, The feature map The channel attention weights, is the Sigmoid function, Indicates a shared network. and is the shared weight of the two parts before and after MLP, represents average pooling, Represents maximum pooling.

[0015] The spatial attention module uses average pooling and maximum pooling operations to aggregate feature maps. The spatial information of and the maximum aggregate feature , and finally generate the two-dimensional spatial attention weights through the convolution layer , the calculation formula is:

[0016] ,in, The feature map The spatial attention weights, is the Sigmoid function, represents a convolution operation with a convolution kernel size of 7×7. represents average pooling, Represents maximum pooling.

[0017] The loss function of the detection model adopts the EIOU bounding box loss function.

[0018] The beneficial effects of the present invention are: by optimizing the infrared image detection model, infrared targets can be accurately identified, and the reliability of intelligent driving visual navigation can be improved. An improved feature extraction module is used in the backbone network of the detection model, which uses partial convolution to reduce the computational complexity and model volume, and a dynamic channel feature extraction module is used to extract the significant features of each channel to generate dynamic related features, and it is merged with the output of the FasterNet module, so that the network can obtain richer deep and shallow feature information during feature extraction, dynamically enhance the model's dynamic attention to important features, enhance generalization ability, and improve the model's expression ability, thereby improving feature extraction ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the detection model structure based on YOLOv5s in the present invention.

[0020] Figure 2 It is a structural schematic diagram of the feature extraction module of the present invention.

[0021] Figure 3 Schematic diagram of the structure of the cross-parallel fusion attention module of the present invention.

[0022] Figure 4 Visualize the results of embedding different attention mechanisms into the backbone network.

[0023] Figure 5 Training results for different bounding box loss functions. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings and specific implementation methods. The specific contents listed in the following embodiments are not limited to the technical features necessary for the technical problems to be solved by the technical solutions recorded in the claims. At the same time, the enumerated embodiments are only part of the present invention, not all embodiments.

[0025] The present invention is based on an intelligent driving visual navigation method for infrared target detection, and uses an infrared camera to obtain an infrared image of the driving environment, such as an image in front of a vehicle. The acquired infrared image is subjected to feature extraction using a trained detection model, and the region of interest is identified therefrom to obtain the detection target in the image. The intelligent driving control system determines the current situation according to the identified detection target position to make a driving strategy decision, for example, the detection target is located in the lane in front of the vehicle, or in the lane on the side of the vehicle, or the continuous detection results show that the detection target is approaching, and different driving strategies are made according to different situations. The decision of intelligent driving is usually made based on the detection results of multiple sensor fusions, such as combining millimeter wave radar to determine the target distance, combining laser radar to identify the detection target, or using two cameras to form a binocular system to determine the target position and distance. The main content of the present invention is target detection in infrared images. The method of combining other sensors and the specific decision-making method of driving strategy can refer to the prior art and will not be discussed in detail.

[0026] The present invention uses a detection model to extract features from the acquired infrared image and identify the detection target. The detection model uses YOLOv5s as the model framework. On the basis of the original YOLOv5s, the C3 module used for basic feature extraction in the backbone network of the original YOLOv5s is improved to form a new feature extraction module, which is named DPCC in the present invention, that is, dynamic partial convolution module. Figure 1 As shown, there are multiple improved DPCCs in the backbone network part.

[0027] like Figure 2 As shown, the feature extraction module of the present invention includes a FasterNet module (FasterNet Block), a dynamic channel feature extraction module and multiple PCBS modules. Among them, the PCBS module is composed of partial convolution, batch normalization and activation function. The dynamic channel feature extraction module includes a maximum pooling layer MaxPool and a linear layer Linear set in sequence. The feature map of the input feature extraction module First, the feature map F is processed by the first PCBS module. The feature map F is divided into three paths, one is sent to the dynamic channel feature extraction module for processing, one is sent to the FasterNet module after being processed by the second PCBS module, and the other is processed by the third PCBS module.

[0028] The present invention uses a PCBS module to process the input feature map, which can reduce the number of memory accesses during calculation through partial convolution, reduce the amount of calculation, reduce the model volume, and has a simple structure and good hardware compatibility. When using the PCBS module for feature extraction, all channel information in the second half can be retained, and the spatial information of the convolution in the first half can be integrated, avoiding the complexity of ordinary convolution calculation, model overfitting, and loss of detail information due to excessive receptive field.

[0029] The FasterNet module includes a partial convolution layer PConv, a point convolution layer Conv1×1 and an inverted point convolution layer Conv1×1. After the feature map F is processed by the second PCBS module, local spatial information is generated. , and then sent to the FasterNet module, which is composed of some convolutional layers PConv The feature map is segmented in the c channel dimension. Channel feature map Feature extraction is performed through convolution operation, and the rest Channel feature map The unprocessed and unchanged part is added to the part that has been convolved, so that the dimension of the feature map after partial convolution remains unchanged. After the point convolution layer Conv1×1 is attached to the partial convolution layer PConv, the features processed by partial convolution are convolved point by point to form a dual-stage feature aggregation of partial convolution and point convolution. , so that the key information is focused on the center. After batch normalization BN and nonlinear changes of the activation function RELU, it is finally connected to an inverted point convolution Conv1×1 to form an inverted residual block. The output of the inverted point convolution layer forms a residual connection with the input of the FasterNet module. The FasterNet module outputs the extracted feature results, which are superimposed and spliced ​​(ConCat) with the feature map processed by the third PCBS module to obtain the feature , which can reduce computational redundancy and improve computational efficiency. The calculation formula is as follows:

[0030] ;

[0031] ;

[0032] in is the SiLu function, is the ReLu function, is the PCBS standard convolution layer weight parameter, is the weight parameter of the partial convolution layer PConv, and They are the weight parameters of point convolution and inverted point convolution respectively.

[0033] The dynamic channel feature extraction module uses the maximum pooling layer to perform the maximum pooling operation on the input feature map F to extract the salient features of each channel. , and then use the linear layer to extract the salient features Processing, characterizing the correlation between different channels, generating dynamic correlation features The output of the FasterNet module is concatenated with the feature map processed by the third PCBS module. It is fused with the output of the dynamic channel feature extraction module, and the generated dynamic correlation features are used to further fuse the channel information of the features extracted at different stages through partial convolution and point convolution of the FasterNet module, and dynamically weight the fusion features at different levels to enhance the channel correlation features and effectively extract the semantic information DF of the feature map. The formula is as follows:

[0034] .

[0035] The partial convolution and point convolution in the FasterNet module form a T-shaped convolution, which can fuse the features of different parts before and after, but its structure is relatively fixed and difficult to adjust according to different task requirements, and the receptive field is mainly concentrated in the middle area, resulting in insufficient feature extraction of edges and corners. For tasks that require global features, it is not as effective as ordinary convolution. Moreover, the partial convolution only processes partial channels in this way and may not be able to fully utilize the information between different channels, thereby affecting the efficiency of feature fusion. In view of the problems such as poor flexibility of the partial convolution module, poor extraction ability of global feature extraction, and poor generalization, the present invention obtains a larger receptive field and richer features by stacking multiple PCBS layers and taking advantage of different operations. A dynamic channel feature extraction module is added, and T-shaped convolution is used in combination with it to dynamically enhance the dynamic attention of the model to important features, thereby improving the feature extraction capability.

[0036] like Figure 1 As shown in FIG. 1 , the present invention adds a cross-parallel fusion attention module CPFA (Cross-parallel fusion attention mechanism) to the backbone network of the detection model, and uses parallel channel attention and spatial attention to further extract feature information from the feature map in the channel dimension and the spatial dimension. The CPFA module can be Figure 1 As shown, it is added before the Neck network, so that the network can adaptively focus on the target channel and spatial information, suppress interference, and solve the problem of feature extraction in complex backgrounds.

[0037] like Figure 3 As shown in Figure 2, the cross-parallel fusion attention module includes a channel attention module and a spatial attention module. The size of the input cross-parallel fusion attention module is Feature Map The c-dimension is divided into and Two parts. Sent to the channel attention module, Sent to the spatial attention module.

[0038] The channel attention module uses average pooling and maximum pooling operations to aggregate feature maps Channel information of the , generating average aggregation features and the maximum aggregate feature , and then generate channel attention weights through the shared network MLP , the calculation formula is:

[0039] ;

[0040] in, The feature map The channel attention weights, is the Sigmoid function, Indicates a shared network. and is the shared weight of the two parts before and after MLP, represents average pooling, Represents maximum pooling.

[0041] The spatial attention module uses average pooling and maximum pooling operations to aggregate feature maps. The spatial information of and the maximum aggregate feature , and finally generate the two-dimensional spatial attention weights through the convolution layer , the calculation formula is:

[0042] ;

[0043] in, The feature map The spatial attention weights, is the Sigmoid function, represents a convolution operation with a convolution kernel size of 7×7. represents average pooling, Represents maximum pooling.

[0044] Finally, after cross-parallel processing by the channel attention module and the spatial attention module, they are spliced ​​and re-aggregated at the c dimension. The calculation process of the CPFA attention mechanism is as follows:

[0045] ;

[0046] ;

[0047] ;

[0048] in, and These are the two parts after the CPFA module divides the input feature map. is the output of the channel attention module, is the output of the spatial attention module.

[0049] The present invention divides the feature map through a cross-parallel fusion attention module, and directly retains rich channel information and spatial information through parallel processing of spatial attention and channel attention respectively. After splicing and fusion, the ability of spatial attention and channel attention to extract features is retained to the maximum extent, avoiding the problem that the features extracted during the usual serial processing of spatial attention and channel attention are reduced due to the superposition processing before and after. It can still effectively extract channel and spatial information, focus on the area of ​​interest, and suppress irrelevant features under low resolution and complex background conditions. For problems such as low resolution of infrared images, inability to display color information, and complex image environment, CPFA can more effectively focus on the area of ​​interest and suppress irrelevant information through spatial information and channel information.

[0050] In the present invention, the loss function of the detection model adopts the EIOU bounding box loss function. It not only considers the center point distance, but also the influence of the aspect ratio, splits the loss term of the aspect ratio, optimizes the width and height of the prediction box, and thus improves the performance of target detection, accelerates convergence, and improves regression accuracy.

[0051] The EIOU bounding box loss function is as follows:

[0052] ;

[0053] in, is the IoU loss, is the distance loss, is the edge length loss, , Represent the center points of the predicted box and the real box respectively, represents the Euclidean distance between two center points, c represents the diagonal distance of the minimum closure area that contains both the predicted box and the true box, , Represent the width of the predicted box and the real box respectively, is the width of the minimum bounding rectangle of the real box and the predicted box, , Represent the height of the predicted box and the real box respectively, It is the height of the minimum bounding rectangle of the real box and the predicted box.

[0054] IOU is used to measure the degree of match between the true bounding box and the predicted bounding box, and is calculated as follows: , where A represents the area of ​​the predicted box and B represents the area of ​​the real box.

[0055] The EIOU bounding box loss function directly minimizes the difference in width and height between the predicted box and the true box by introducing width and height losses, making the model converge faster and applicable to complex sample environments.

[0056] In order to verify the effects of different features of the present invention, the original YOLOv5s model is used as the basis for Experiment A. Experiment B uses the feature extraction module DPCC of the present invention to replace the C3 module in the backbone network of the YOLOv5s model on the basis of Experiment A to verify the effectiveness of the model; Experiment C adds the CPFA attention mechanism to the backbone network on the basis of Experiment B to verify the improvement effect of the CPFA attention mechanism on false detection and missed detection of infrared target detection. Experiment D verifies the ability of the EIOU bounding box loss function to improve the detection accuracy of the algorithm on the basis of Experiment C.

[0057] The experimental method and results are shown in the following table:

[0058]

[0059] Compared with experiment A, the precision rate P of experiment B increased by 1.7%, the mAP increased by 0.2%, and the number of parameters (Params) decreased by 0.4M. The precision rates P of experiments C and D increased by 3.2% and 5.4% respectively, the mAP values ​​increased by 4.9% and 9.8% respectively, and the number of parameters decreased by 0.4M and 0.6M. The effectiveness of the proposed method for infrared image target detection was verified.

[0060] Based on YOLOv5s, the present invention uses the feature extraction module DPCC module to replace the original C3 module, reconstructs the backbone network, and greatly reduces the number of model parameters; by adding the CPFA attention mechanism, the attention to relevant features is enhanced and irrelevant information is suppressed; the EIOU loss function is used when calculating the bounding box regression loss to accelerate network convergence and enhance the accuracy of target detection in complex environments.

[0061] In order to verify the superiority of the CPFA attention mechanism, SE, CA, CBAM and the CPFA attention of the present invention are embedded in the backbone network for comparative experiments. The detection accuracy and mPA of the CPFA attention of the present invention are better than SE, CA and CBAM. The heat map is used for visualization to compare the detection ability of the infrared image model embedded with different attention mechanisms. The experimental configuration and parameter settings remain unchanged. The experimental results are shown in Figure 2. Figure 4 As shown in the figure, the CPFA attention mechanism can accurately locate the region of interest for the spatial position of the target area, while also taking into account feature information of different sizes.

[0062] The original YOLOv5s algorithm model uses GIOU as the loss function in the bounding box loss function, which has certain limitations in the predicted bounding box. This paper analyzes the convergence of the model when the YOLOv5s model is trained using GIOU, CIOU, EIOU and DIOU bounding box loss functions. The experimental results are as follows: Figure 5 As shown in the figure, when EIOU is used as the bounding box loss function, the convergence speed is the fastest, gradually converges after 125 rounds, and the convergence loss value is the smallest compared to other loss functions.

[0063] In order to verify the effectiveness of the detection model of the present invention in different data sets, the LLVIP data set is used for verification based on the FLIR data set. The FLIR data set is an infrared image data set widely used in the field of computer vision. The data set contains about 14,000 pictures and a total of 17,504 labeled instances. The original data set contains labels for sixteen target object categories such as people, cars, bicycles, and motorcycles. The present invention randomly extracts 12,000 pictures for division, and uses 8,000 pictures as training sets, 3,000 pictures as test sets, and 1,000 pictures as verification sets. The LLVIP data set is an infrared pedestrian data set. The original data set contains labels for only pedestrian target object categories, including 15,488 pictures and 89,174 samples. The present invention randomly extracts 10,000 pictures for division, and uses 7,000 pictures as training sets, 2,000 pictures as test sets, and 1,000 pictures as verification sets. The experimental platform and parameter settings are all kept consistent.

[0064] The experimental results are shown in the following table:

[0065]

[0066] In the FLIR dataset, the detection model of the present invention performs best in detection accuracy (P) and average detection accuracy (mAP), and the parameter scale (Params) is smaller, which are 88.5%, 72.3%, and 6.6M respectively. In the LLVIP dataset, the detection model of the present invention performs best in detection accuracy (P) and average detection accuracy (mAP), and the parameter scale is smaller, which are 88.2%, 70.3%, and 6.6M respectively. The detection model of the present invention outperforms the YOLOv5s, YOLOv7, and YOLOv8 models in all indicators on the FLIR dataset and the LLVIP dataset, and shows excellent detection performance in different scenarios.

[0067] The above description of the specific implementation mode is only used to help understand the technical concept and core idea of ​​the present invention. Although the technical solution is described and illustrated using a specific preferred embodiment, it should not be understood as a limitation of the present invention itself. Those skilled in the art may make various changes in form and details without departing from the technical concept of the present invention. These easily conceived changes or substitutions should all be included in the protection scope of the present invention.

Claims

1. An intelligent driving visual navigation method based on infrared target detection, using an infrared camera to obtain an infrared image of the driving environment, extracting features from the acquired infrared image using a detection model, and identifying the detection target; an intelligent driving control system formulates a corresponding driving strategy according to the identified detection target position to achieve intelligent driving, characterized in that: The detection model adopts YOLOv5s as the model framework, and its backbone network part has multiple feature extraction modules for basic feature extraction; the feature extraction module includes a FasterNet module, a dynamic channel feature extraction module and multiple PCBS modules composed of partial convolution, batch normalization and activation function; wherein the dynamic channel feature extraction module includes a maximum pooling layer and a linear layer arranged in sequence; the feature map of the input feature extraction module The feature map F is first processed by the first PCBS module to obtain the feature map F, which is divided into three paths. One path is sent to the dynamic channel feature extraction module for processing, one path is sent to the FasterNet module after being processed by the second PCBS module, and the other path is processed by the third PCBS module. The output of the FasterNet module is superimposed with the output of the third PCBS module and fused with the output of the dynamic channel feature extraction module as the output of the feature extraction module.

2. The intelligent driving visual navigation method based on infrared target detection according to claim 1, characterized in that: The FasterNet module includes a partial convolution layer, a point convolution layer and an inverted point convolution layer. The feature map input to the FasterNet module is processed by the partial convolution layer, and the point convolution layer performs point-by-point convolution on the output of the partial convolution layer. After batch normalization and activation, it passes through the inverted point convolution layer, and the output of the inverted point convolution layer forms a residual connection with the input of the FasterNet module.

3. The intelligent driving visual navigation method based on infrared target detection according to claim 1, characterized in that: The dynamic channel feature extraction module uses a maximum pooling layer to perform a maximum pooling operation on the input feature map F to extract the significant features of each channel, and then uses a linear layer to process the extracted significant features to generate dynamic correlation features to characterize the correlation between different channels.

4. The intelligent driving visual navigation method based on infrared target detection according to claim 3, characterized in that: The dynamic correlation features generated by the dynamic channel feature extraction module are used to dynamically weight the features extracted by the FasterNet module to obtain fused features.

5. The intelligent driving visual navigation method based on infrared target detection according to claim 1, characterized in that: The backbone network of the detection model has multiple feature extraction modules, and the feature extraction modules are used to replace the C3 module in the YOLOv5s backbone network.

6. The intelligent driving visual navigation method based on infrared target detection according to claim 1, characterized in that: The backbone network of the detection model has a cross-parallel fusion attention module, which uses parallel channel attention and spatial attention to extract features from feature maps.

7. The intelligent driving visual navigation method based on infrared target detection according to claim 6, characterized in that: The cross-parallel fusion attention module includes a channel attention module and a spatial attention module, and the feature map of the cross-parallel fusion attention module is input Split in one dimension and Two parts, Sent to the channel attention module, Sent to the spatial attention module, the outputs of the channel attention module and the spatial attention module are concatenated and re-aggregated in the original dimension.

8. The intelligent driving visual navigation method based on infrared target detection according to claim 7, characterized in that: The channel attention module uses average pooling and maximum pooling operations to aggregate feature maps Channel information of the , generating average aggregation features and the maximum aggregate feature , and then generate channel attention weights through the shared network MLP , the calculation formula is: ； in, The feature map The channel attention weights, is the Sigmoid function, Indicates a shared network. and is the shared weight of the two parts before and after MLP, represents average pooling, Represents maximum pooling.

9. The intelligent driving visual navigation method based on infrared target detection according to claim 7, characterized in that: The spatial attention module uses average pooling and maximum pooling operations to aggregate feature maps. The spatial information of and the maximum aggregate feature , and finally generate the two-dimensional spatial attention weights through the convolution layer , the calculation formula is: ,in, The feature map The spatial attention weights, is the Sigmoid function, represents a convolution operation with a convolution kernel size of 7×7. represents average pooling, Represents maximum pooling.

10. The intelligent driving visual navigation method based on infrared target detection according to claim 1, characterized in that: The loss function of the detection model adopts the EIOU bounding box loss function.

Images