Automatic driving thermal auxiliary imaging method based on feature optimization and information enhancement

Abstract

The invention discloses an automatic driving thermal auxiliary imaging method based on feature optimization and information enhancement. The method comprises the following steps: step 1, acquiring and processing data in an HADAR database; step 2, providing an efficient residual feature representation module (ERes-FR), a local position attention enhancement module (LPAE) and a dynamic gradient descent module (DGD); 3, carrying out an ablation experiment, and verifying the effectiveness of the module; 4, carrying out an additional experiment to obtain an optimal pooling factor and a residual error depth; 5, training the model to obtain an improved model; and step 6, testing the training model and visualizing the training model. According to the method, an ERes-FR module, an LPAE module and a DGD module are innovatively provided on a TexNet model. And an ERes-FR module is provided, so that gradient feature extraction is more effective. And an LPAE module is provided, so that the local information processing capability of the model is enhanced. And a DGD module is provided, so that the feature learning and representation capability is further enhanced. And the effect of the model in an automatic driving thermal auxiliary imaging task is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of thermal-assisted imaging for autonomous driving, and in particular to a thermal-assisted imaging method for autonomous driving based on feature optimization and information enhancement. Background Art

[0002] With the rapid development of science and technology, autonomous driving technology has become a research hotspot. In the process of achieving reliable autonomous driving, accurate perception of the surrounding environment is crucial. Thermal imaging technology has attracted much attention in the field of autonomous driving due to its advantages in bad weather and low light environments.

[0003] Thermal imaging captures infrared radiation emitted by objects, allowing vehicles to detect obstacles and assess the surrounding environment in darkness, fog, rain, snow and other environments. However, thermal imaging technology faces many challenges. For example, the ghosting effect caused by the continuous emission and scattering of thermal radiation by objects and their surroundings seriously affects the accurate recognition and distinction of objects by autonomous vehicles, threatening the safety and reliability of the system.

[0004] Although existing thermal-assisted detection and ranging technologies have achieved certain results, they still have shortcomings. For example, although TeXNet uses deep learning technology to analyze thermal signals to enhance environmental perception, it needs to improve the accuracy of physical information decomposition and scene texture information recovery. At the same time, although there are many excellent methods in the field of semantic segmentation, they are still weak in dealing with the unique challenges of HADAR tasks.

[0005] Therefore, the current thermal-assisted imaging technology for autonomous driving still needs to be improved in terms of accuracy, reliability, and environmental adaptability. A new method is urgently needed to optimize feature representation and enhance information processing to meet the high requirements of autonomous driving.

[0006] The information disclosed in this background technology section is only intended to increase the understanding of the overall background of the invention and should not be regarded as an admission or suggestion in any form that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a thermal-assisted imaging method for autonomous driving based on feature optimization and information enhancement, so as to solve the problems faced by thermal imaging technology in autonomous driving, such as ghosting effect, complex data interpretation, and the shortcomings of existing technologies in terms of accuracy of physical information decomposition and restoration of scene texture information.

[0008] To achieve the above object, the present invention provides a thermal-assisted imaging method for autonomous driving based on feature optimization and information enhancement, comprising:

[0009] Step 1: Obtain and process data in the HADAR database;

[0010] Step 2: Improve the TexNet model and propose the efficient residual feature representation module (ERes-FR), the local position attention enhancement module (LPAE) and the dynamic gradient descent module (DGD);

[0011] Step 3: Conduct ablation experiments to verify the effectiveness of each module;

[0012] Step 4: Conduct additional experiments to obtain the optimal pooling factor and residual depth of the ERes-FR module;

[0013] Step 5: training the model to obtain an improved training model;

[0014] Step 6: Test the trained model and visualize it.

[0015] Preferably, in the above technical solution, step 1 obtains high-quality and authentic data from the HADAR database, which consists of 11 data sets covering scenes such as streets and highways.

[0016] Preferably, in the above technical solution, step 2 of improving the TexNet model includes:

[0017] (1) Replacing the Res module of the original TeXNet model can more effectively extract and fuse features and enhance the model's ability to capture key feature information in complex scenarios. The output of this module is:

[0018] Y(X)=F scale (σ(W fc (M -pool (X))),X)

[0019]

[0020] Where W fc represents the FC layer, σ(·) represents the Sigmoid function, F scale Indicates the Scale layer, M res represents the residual module, Represents addition of elements.

[0021] (2) Replace the spectral attention module of the original TeXNet model so that the model can focus on local details more effectively and accurately restore target boundaries and fine details. Especially when dealing with complex scenes or multi-scale targets, it can more accurately segment target boundaries and subtle structures. By fine-tuning the influence of high-level features on low-level features, the model can better focus on key local areas when restoring spatial resolution. The operation of this module can be expressed by the following mathematical formula:

[0022] Y W =σ(Gn (W 1d (F W (W c3 (X)))))

[0023] Y H =σ(G n (W 1d (F H (W c3 (X)))))

[0024]

[0025] Where X∈R H×W×C And X'∈R H'×W'×C' , Y W represents the output of the first branch of the LPAE module, and Y H represents the output of the third branch of the same module. In this process, W c3 represents a 3×3 convolution operation, F W Indicates that average pooling is applied along the W dimension, F H Indicates that average pooling is applied along the H dimension, W 1d represents the Conv1D operation, G n Represents a GroupNorm operation.

[0026] (3) The feature pyramid attention module of the original TeXNet model is replaced. By segmenting the feature map and realizing cross-stage local and hierarchical fusion, this module significantly improves the network's feature learning ability while reducing computational complexity. The operation of this module can be expressed by the following mathematical formula:

[0027]

[0028] Where X∈R H×W×C , W c1 represents 1×1 convolution, M DC Indicates the operation of the DC module, F C represents the fully connected layer, Represents addition of elements.

[0029] Preferably, in the above technical solution, step 3 performs an ablation experiment on the data of the HADAR database, combines and trains each module, obtains the IoU value and parameter quantity of each model, and compares the specific results.

[0030] Preferably, in the above technical solution, step 4 performs additional experiments to obtain the optimal pooling factor and residual depth of the efficient residual feature representation module (ERes-FR).

[0031] Preferably, in the above technical solution, step 5 trains the training set of the HADAR database to obtain an efficient training model of the improved method.

[0032] Preferably, in the above technical solution, step 6 tests the training model and visualizes the specific effect. Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) The present invention is a thermal-assisted imaging method for autonomous driving based on feature optimization and information enhancement. In order to overcome the difficulty of information parsing caused by the thermal radiation ghosting effect and efficiently extract more accurate and effective features, we proposed an efficient residual feature representation module (ERes-FR) to enable the model to decompose physical information more effectively. In addition, in order to enhance the model's attention to local detail information, we proposed a local position attention enhancement module (LPAE) to improve the feature expression of local areas. We also proposed a dynamic gradient descent module (DGD), which enables our method to improve the inference speed while maintaining high accuracy, greatly improving the efficiency of thermal imaging technology in decomposing physical signals in complex environments.

[0034] (2) The present invention is a thermal-assisted imaging method for autonomous driving based on feature optimization and information enhancement. Compared with traditional thermal imaging processing methods and existing advanced methods (such as TexNet, etc.), the present invention has a significant performance improvement, demonstrating its superiority in thermal-assisted imaging for autonomous driving based on feature optimization and information enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is an overall flow chart of the automatic driving thermal assisted imaging method based on feature optimization and information enhancement according to the present invention;

[0036] Figure 2 This is the improved ELDNet model framework diagram;

[0037] Figure 3 is the structural diagram of the proposed EFes-FR module;

[0038] Figure 4 It is a structural comparison diagram of the proposed LPAE module and the spectral attention module;

[0039] Figure 5 It is the structural diagram of the proposed DGD module;

[0040] Figure 6 It is the structure diagram of the double convolution (DC) module;

[0041] Figure 7 This is the visualization effect diagram of the improved model. DETAILED DESCRIPTION

[0042] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.

[0043] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0044] like Figures 1 to 7 As shown, according to a specific embodiment of the present invention, a thermal assisted imaging method for autonomous driving based on feature optimization and information enhancement includes the following steps:

[0045] Step 1: Get the dataset

[0046] Considering the existing problems faced in the thermal-assisted imaging task of autonomous driving, among all the existing datasets, we obtained high-quality and real data from the HADAR database. The HADAR database consists of 11 datasets covering scenes such as streets and highways. The HADAR database is a hyperspectral and stereo vision database focusing on the long-wave infrared (LWIR) field, which aims to promote the research and development of thermal navigation algorithms. This database not only supports the development of TeX decomposition algorithms to generate unique TeX vision, but also encourages researchers to explore cutting-edge technologies such as target detection, semantic / scene segmentation, optical flow / scene flow analysis, and stereo depth estimation based on TeX vision rather than traditional RGB images or thermal imaging. Step 2. Improve the TexNet model and propose an efficient residual feature representation module (ERes-FR), a local position attention enhancement module (LPAE), and a dynamic gradient descent module (DGD)

[0047] (1) The Res module of the original TeXNet model is replaced to enhance the feature extraction of multi-layer gradients, thereby improving the training efficiency and overall performance of the deep model. Specifically, the M-pool mechanism compresses the feature map of each channel into a single true value, which represents the global receptive field information of the channel. The FC layer then reduces the dimension of the channel descriptor from C to C / r, effectively reducing the computational cost and complexity of the model. Another FC layer restores the dimension to C and restores the original channel configuration. The Sigmoid function then generates weights for each channel, which are then applied to the original feature map through a scaling operation, thereby achieving feature recalibration and solving the problem of insufficient feature representation. The output of this module is:

[0048] Y(X)=F scale (σ(W fc (M -pool (X))),X)

[0049]

[0050] Where W fc represents the FC layer, σ(·) represents the Sigmoid function, F scale Indicates the Scale layer, M res represents the residual module, Represents addition of elements.

[0051] (2) Replace the spectral attention module of the original TeXNet model so that the model can focus on local details more effectively and accurately restore target boundaries and fine details. Especially when dealing with complex scenes or multi-scale targets, it can more accurately segment target boundaries and subtle structures. By fine-tuning the influence of high-level features on low-level features, the model can better focus on key local areas when restoring spatial resolution. The operation of this module can be expressed by the following mathematical formula:

[0052]

[0053]

[0054] Where X∈R H×W×C And X'∈R H'×W'×C' , Y W represents the output of the first branch of the LPAE module, and Y H represents the output of the third branch of the same module. In this process, W c3 represents a 3×3 convolution operation, F W Indicates that average pooling is applied along the W dimension, F H Indicates that average pooling is applied along the H dimension, W 1d represents the Conv1D operation, G n Represents a GroupNorm operation.

[0055] (3) The feature pyramid attention module of the original TeXNet model is replaced. By segmenting the feature map and realizing local and hierarchical fusion across stages, this module significantly improves the feature learning ability of the network while reducing the computational complexity. The core of the DGD module is a dual convolution (DC) module, which is specially designed to improve the performance of the model. It has a multi-branch configuration, including continuous convolution, multi-scale convolution, and average pooling operations. During the inference process, the DC module is dynamically converted into 3×3 convolution layers and 1×1 convolution layers, which simplifies the network structure in the inference stage while maintaining its expressive power. The operation of this module can be expressed by the mathematical formula as follows:

[0056]

[0057] Where X∈R H×W×C, W c1 represents 1×1 convolution, M DC Indicates the operation of the DC module, F C represents the fully connected layer, Represents addition of elements.

[0058] Step 3: Conduct ablation experiments to verify the effectiveness of each module

[0059] Ablation experiments were conducted on the three proposed modules to verify the effectiveness of each module and analyze the reasons why the module produced the effect. By comparing the ablation of each module, we can verify the effectiveness of each module. The result is that after adding the ERes-FR module, although the number of model parameters increased, the IoU index of the model was significantly improved. This is because the ERes-FR module effectively extracted features through operations such as M-pool, and realized feature recalibration, which enhanced the feature representation ability of the model. After adding the LPAE module, the IoU was significantly improved without a significant increase in the number of model parameters. After adding the DGD module, the model parameters were slightly reduced, and the IoU was improved. This is because the DGD module has a simple structure during reasoning, and at the same time improves the fusion and reuse rate of features, making the model more effective.

[0060] Step 4: Conduct additional experiments to obtain the optimal pooling factor and residual depth

[0061] We conducted additional experiments to confirm the effects of different pooling coefficients in the ERes-FR module and the effects of different residual depths in the network. We finally verified that the pooling coefficient of 0.7 and the network depth of ResNet50 achieved the best results.

[0062] Step 5: Train the improved model and get test results

[0063] The training set of the HADAR database is trained to obtain an efficient detection model after training of the improved method.

[0064] Step 6: Test and visualize the trained model

[0065] In order to comprehensively evaluate and verify the effectiveness of various modules in autonomous driving technology, we selected four highly representative scenes from the HADAR database and presented them through visualization.

[0066] It is particularly noteworthy that the ELDNet training model obtained by fully integrating the ERes-FR, LPAE and DGD modules into the TeXNet model has superior performance in these four autonomous driving scenarios.

[0067] This section not only visually demonstrates the high adaptability of ELDNet in complex driving environments, but also strongly supports the significant contribution of our designed modules in improving the efficacy of autonomous driving systems. Therefore, it further verifies the effectiveness and practicality of these modules in the field of autonomous driving.

[0068] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A thermal-assisted imaging method for autonomous driving based on feature optimization and information enhancement, characterized in that: include: Step 1: Obtain and process data in the HADAR database; Step 2: Improve the TexNet model and propose the efficient residual feature representation module (ERes-FR), the local position attention enhancement module (LPAE) and the dynamic gradient descent module (DGD); Step 3: Conduct ablation experiments to verify the effectiveness of each module; Step 4: Conduct additional experiments to obtain the optimal pooling factor and residual depth of the ERes-FR module; Step 5: Train the model to obtain an improved model; Step 6: Test and visualize the trained model.

2. The automatic driving thermal assisted imaging method based on feature optimization and information enhancement according to claim 1, characterized in that: Step 1: Obtain high-quality and authentic data from the HADAR database. The HADAR database consists of 11 data sets, covering scenes such as streets and highways.

3. The automatic driving thermal assisted imaging method based on feature optimization and information enhancement according to claim 1, characterized in that: Step 2 improves the TexNet model by: (1) Replace the Res module of the original TeXNet model to enable the model to extract and fuse features more effectively and enhance the model's ability to capture key feature information in complex scenarios. The output of this module is: Y(X)=F scale (σ(W fc (M -pool (X))),X) Where W fc represents the FC layer, σ(·) represents the Sigmoid function, F scale Indicates the Scale layer, M res represents the residual module; (2) Replace the spectral attention module of the original TeXNet model so that the model can focus on local details more effectively and accurately restore target boundaries and fine details. Especially when dealing with complex scenes or multi-scale targets, it can more accurately segment target boundaries and subtle structures. By fine-tuning the impact of high-level features on low-level features, the model can better focus on key local areas when restoring spatial resolution. The operation of this module can be expressed by the following mathematical formula: Y W =σ(G n (W 1d (F W (W c3 (X))))) Y H =σ(G n (W 1d (F H (W c3 (X))))) Where X∈R H×W×C And X'∈R H'×W'×C' , Y W represents the output of the first branch of the LPAE module, and Y H represents the output of the third branch of the same module. In this process, W c3 represents a 3×3 convolution operation, F W Indicates that average pooling is applied along the W dimension, F H Indicates that average pooling is applied along the H dimension, W 1d represents the Conv1D operation, G n Represents GroupNorm operation; (3) The feature pyramid attention module of the original TeXNet model is replaced. By segmenting the feature map and realizing cross-stage hierarchical fusion, this module significantly improves the network's feature learning ability while reducing the computational complexity. The operation of this module can be expressed by the following mathematical formula: Where X∈R H×W×C , W c1 represents 1×1 convolution, M DC Indicates the operation of the DC module, F C represents the fully connected layer, Represents addition of elements.

4. The automatic driving thermal assisted imaging method based on feature optimization and information enhancement according to claim 1, characterized in that: Step 3: Conduct an ablation experiment on the data from the HADAR database, combine and train each module, obtain the IoU value and parameter quantity of each model, and compare the specific results.

5. The automatic driving thermal assisted imaging method based on feature optimization and information enhancement according to claim 1, characterized in that: In step 4, additional experiments are performed to obtain the optimal pooling factor and residual depth of the efficient residual feature representation module (ERes-FR).

6. The automatic driving thermal assisted imaging method based on feature optimization and information enhancement according to claim 1, characterized in that: Step 5: Train the training set of the HADAR database to obtain a training model of the improved method.

7. The automatic driving thermal assisted imaging method based on feature optimization and information enhancement according to claim 1, characterized in that: Step 6 tests the trained model and visualizes the specific effects.

Images