A Driving Scene Segmentation Method Based on Thermal Infrared Attention Mechanism Neural Network

Abstract

This invention discloses a driving scene segmentation method based on a thermal infrared attention mechanism neural network. The method first acquires visible light and thermal infrared images of the same scene and resolution, and forms a composite image training dataset by labeling them. Then, a thermal infrared information attention network is constructed and trained using the composite image training set. After obtaining a good training model, the thermal infrared information attention network is used to segment driving images. It can stably segment driving environment images in different environments with high segmentation accuracy.

Description

Technical Field

[0001] This invention relates to the field of driving environment image segmentation technology, and in particular to a driving scene segmentation method based on a thermal infrared attention mechanism neural network. Background Technology

[0002] Drivers need to assess their surroundings while driving to ensure safe driving. In recent years, the application of automotive vision sensors has greatly aided drivers in judging complex driving environments. There are various types of automotive vision sensors. Rear vision sensors allow drivers to easily observe the situation behind the vehicle, assisting in reversing maneuvers. Side vision sensors help drivers understand blind spots to the sides, preventing traffic accidents. Furthermore, the rapid development of autonomous vehicles makes these vision sensors crucial for assessing the vehicle's surroundings. Therefore, rapid analysis of images captured by vision sensors can maximize their effectiveness, facilitate driver judgment, and reduce the occurrence of driving accidents.

[0003] Currently, vehicle vision sensor selection solutions mainly fall into two categories. One primarily uses LiDAR (Light Detection and Ranging) to create 3D maps and confirm the real-time three-dimensional scene around the vehicle for judgment. The other primarily uses camera-based devices to collect a large number of images, perform rapid analysis and processing, perceive drivable lanes and obstacles, and then make environmental judgments. The latter approach has a lower overall cost than the former and its hardware is more mature, attracting widespread attention from automakers.

[0004] Semantic segmentation technology for driving scenes based on visual sensors can perform pixel-level semantic classification of images generated by visual sensors, thereby quickly identifying objects in traffic scenes and facilitating subsequent judgments by the driving system. Its high speed and large information capacity have attracted much attention. However, due to the complexity of the driving environment, the large number of objects in the images, and the significant differences in image quality under different environments, especially at night when visible light images are almost useless compared to daytime, semantic segmentation of driving scenes is quite difficult. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by proposing a driving scene segmentation method based on a thermal infrared attention mechanism neural network, thus resolving the deficiencies in the existing technologies.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0007] A driving scene segmentation method based on a thermal infrared attention mechanism neural network includes the following steps:

[0008] Step 1: Obtain several pairs of visible light images and thermal infrared images of the same resolution and driving environment provided by the vehicle-mounted camera, perform category labeling, and obtain a composite image training dataset;

[0009] Step 2: Construct a thermal infrared information attention network;

[0010] Step 3: Train the thermal infrared information attention network based on the composite image training dataset to obtain the trained thermal infrared information attention network;

[0011] Step 4: Use the thermal infrared information attention network to segment the driving environment image. The driving environment image is a pair of visible light image and thermal infrared image with the same resolution as the composite image training dataset.

[0012] Furthermore, step 1 includes the following sub-steps:

[0013] Step 1.1: Acquire visible light and thermal infrared images of the same driving environment from the vehicle-mounted camera, both at the same resolution and depicting the same scene. The corresponding pixels in the two images should be identical. The visible light image is formed from the red, yellow, and blue spectral bands captured by a standard color camera, while the thermal infrared image is captured by a thermal infrared imager. The two images form a complex image pair. Several pairs of images at the same resolution need to be prepared, and these pairs should encompass various scenes and environments.

[0014] Step 1.2: Crop the complex image pairs into images of the same size, with the same orientation, length, and width, to form a composite image training dataset;

[0015] Furthermore, step 2 includes the following sub-steps:

[0016] Step 2.1: Determine the basic parameters of the thermal infrared information attention network based on factors such as the quality and quantity of complex image pairs;

[0017] Step 2.2: Construct a thermal infrared information attention network structure based on a deep learning environment, and set the network parameters according to the parameters determined in Step 2.1;

[0018] Furthermore, step 3 includes the following sub-steps:

[0019] Step 3.1: Set the number of training iterations for the thermal infrared information attention network based on the quality and quantity of the composite image training dataset;

[0020] Step 3.2: Train the thermal infrared information attention network using the composite image training dataset;

[0021] Step 3.3: Save the completed thermal infrared information. Note the network parameters.

[0022] Furthermore, step 4 includes the following sub-steps:

[0023] Step 4.1: Obtain composite image pairs of the driving environment provided by the vehicle camera with the same resolution as the composite image training dataset, and normalize the composite image pairs so that the length and width of the composite image pairs are the same as the image pairs in the training dataset.

[0024] Step 4.2: Input the normalized composite image pair to be segmented into the trained thermal infrared information attention network to obtain the network's segmentation result;

[0025] Step 4.3: Organize and save the thermal infrared information and note the vehicle scene segmentation results of the network.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] 1. This method proposes a systematic semantic segmentation approach for driving environments. This method leverages the advantages of high resolution and rich color information in visible light images, and the wide adaptability and high brightness of thermal infrared images, to determine the driving environment. Furthermore, the proposed driving environment segmentation system is applicable to various vehicles, and is low-cost, simple to operate, and produces good segmentation results.

[0028] 2. This method proposes a novel deep learning-based semantic segmentation network—the Thermal Infrared Information Attention Network. This network integrates features extracted from visible light images and thermal infrared images, and uses thermal infrared information attention blocks to enable the extracted thermal infrared features to supervise the network's learning. The weights of the visible light image in the segmentation process are determined based on basic driving environment information in the thermal infrared image. Compared with most semantic segmentation networks, the Thermal Infrared Information Attention Network has a simpler structure and can quickly process large amounts of images generated by vehicle sensors, improving segmentation efficiency. Attached Figure Description

[0029] Figure 1 This is a flowchart of the driving scene segmentation method based on thermal infrared attention mechanism neural network of the present invention;

[0030] Figure 2 This is an imaging schematic diagram of the visible light-thermal infrared binocular camera mentioned in this invention;

[0031] Figure 3 This is a diagram of the thermal infrared information attention network structure constructed in this invention;

[0032] Figure 4 This is a structural diagram of the thermal infrared information attention block designed in this invention;

[0033] Figure 5 This is a schematic diagram of the training of the thermal infrared information attention network designed in this invention;

[0034] Figure 6This is a schematic diagram of the thermal infrared information attention network segmentation in the driving environment designed in this invention;

[0035] Figure 7 This is a schematic diagram of the driving scene segmentation results using a publicly available dataset, as presented in this invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0037] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0038] like Figure 1 As shown, a driving scene segmentation method based on a thermal infrared attention mechanism neural network includes the following steps:

[0039] Step 1: Collect visible light images and thermal infrared images of the same scene and resolution to create a composite image training dataset;

[0040] Step 2: Construct a deep learning-based thermal infrared information attention network and set the network parameters;

[0041] Step 3: Train the thermal infrared information attention network using the composite image training dataset;

[0042] Step 4: Use thermal infrared information attention network to segment composite images of other driving environments.

[0043] Step 1 of this implementation plan includes the following steps:

[0044] Step 1.1: Acquire a composite image generated by a vehicle-mounted vision sensor with good image quality. This composite image consists of one visible light image and one thermal infrared image, forming a composite image pair. The two images in the composite image pair have the same resolution and depict the same driving scene. Generally, a visible light-thermal infrared binocular camera can be used to acquire the image pair, thus achieving the requirements of the same resolution and the same scene. A schematic diagram of a visible light-thermal infrared binocular camera imaging is shown below. Figure 2 As shown;

[0045] Step 1.2: Based on Step 1.1, perform pixel-by-pixel labeling on the composite image pairs, classifying them into seven categories (vehicles, pedestrians, bicycles, traffic markings, roadblocks, guardrails, and others). Labeling must be semantically labeled at the pixel level; each pixel in the image must have a corresponding semantic label.

[0046] Step 1.3: Based on steps 1.1 and 1.2, crop all image pairs to the same size, ensuring that each image pair has the same length and width. Given an image pair size of 256 pixels in length and 256 pixels in width, at least 200 pairs need to be labeled. The number and quality of labels directly affect the training effect of the deep learning network. Furthermore, the number of labeled objects of each category should be as close as possible to avoid class imbalance due to different training effects. The labeled images should cover as many weather conditions as possible (sunny, cloudy, rainy, snowy, foggy, etc.) and different time conditions (morning, noon, evening, night, etc.).

[0047] Step 2 of this implementation plan includes the following steps:

[0048] Step 2.1: Constructing a Deep Learning Network—Thermal Infrared Attention Network (TIAttNet). The Thermal Infrared Attention Network designed in this invention is a deep learning-based semantic segmentation model. Unlike most semantic segmentation networks, the Thermal Infrared Attention Network has two inputs: a visible light image and a thermal infrared image. The output is the segmentation result of the two corresponding images. The structure of the Thermal Infrared Attention Network is as follows: Figure 3 As shown, the thermal infrared information attention network adopts an "end-to-end" network structure and a basic "encoder-decoder" architecture. Structurally, the thermal infrared information attention network has a visible light image encoder, a composite image decoder, and a thermal infrared downsampling system.

[0049] The visible light image encoder consists of a series of downlink blocks. Each downlink block contains a convolution operation, a batch normalization operation, and an activation function operation. Except for the last downlink block, all other blocks contain a downsampling operation. Through this method, the network extracts shape and texture features from the visible light image. The operation formula for the downlink block of the visible light image encoder is as follows:

[0050]

[0051] In the formula, X in X is the input to the downlink block of the visible light image encoder. out2 This represents the output of the downlink block of the visible light image encoder. `Conv` represents a convolution operation with a kernel size of 3×3, and `BN` represents the batch normalization operation. The formula for the batch normalization operation is...

[0052]

[0053]

[0054]

[0055]

[0056] LeaklyReLU is a leaky linear rectification activation operation, and the formula is:

[0057] f(x) = max(0.01x, x) (6)

[0058] Leaking linear rectification activation prevents negative regions from being discarded by the network during computation, while AveragePooling indicates the use of average downsampling.

[0059] The thermal infrared downsampling system is not a standard encoder structure, but rather a downsampling feature extraction system. Its main function is to extract features from thermal infrared images at different sizes. The thermal infrared downsampling system consists of several thermal infrared downsampling blocks. Each downsampling block contains downsampling operations at different sizes, and includes a convolution operation, a batch normalization operation, and an activation function operation, as shown in the formula:

[0060]

[0061] In the formula, x in This represents the input to the thermal infrared downlink block, where the input is a thermal infrared image, x out2 This represents the output of the thermal infrared downlink block. MaxPooling represents the max pooling operation. Different downsampling sizes correspond to different downlink blocks. Conv represents a convolution operation with a kernel size of 3×3. BN represents batch normalization. LeaklyReLU represents leaky linear rectified activation.

[0062] The composite image decoder consists of a series of uplink blocks and a series of thermal infrared information attention blocks. The uplink blocks of the decoder are used to recover the image features and dimensions, and consist of one upsampling operation, two convolution operations, two batch normalization operations, and two activation functions. The operation formula for the uplink blocks of the decoder is as follows:

[0063]

[0064] In the formula, X in X' represents the multispectral remote sensing image features extracted from the downlink block of the corresponding visible light encoder. out This indicates the output of the thermal infrared information attention block preceding the uplink block (or 0 if none exists). Concatenate represents a matrix concatenation operation, and UpSampling represents an upsampling operation, which can expand the length and width of the feature to twice the input.

[0065] The thermal infrared information attention block in the decoder uses features extracted from the thermal infrared image as a reference, allowing the network to focus on the more prominent features in the thermal infrared image. The input to the thermal infrared information attention block consists of visible light features and thermal infrared image features of the same size. The thermal infrared information attention block first performs convolution and batch normalization operations on both types of features, then activates them using an activation function. Next, a gate operation is used for the visible light image features to determine the quality of the visible light image, thus determining the weight of the thermal infrared image features in the training process. For example, in a clear daytime environment, the quality of the visible light image is generally good, so the weight of the thermal infrared image in the training process is low, and the network mainly learns the feature representations from the visible light image. In cloudy or nighttime driving environments, the quality of the visible light image is generally poor, but due to the stability of the thermal infrared image under different environments, its quality is better than that of the visible light image, so the network mainly learns the feature representations from the thermal infrared image. Finally, the two features are added, activated, and convolved to obtain the output. The attention specification formula in the thermal infrared information attention block is as follows:

[0066]

[0067] In the formula, x in This indicates the output of the corresponding thermal infrared downlink block, X in This indicates the output of the preceding uplink block structure located in the thermal infrared information attention block, X. out2 This indicates the output of the thermal infrared information attention block, and Gate represents the gate operation. The formula is as follows:

[0068] G(X,x)=sigmoid(W1X+W2x) (10)

[0069] In the formula, W1 and W2 are weights. Sigmoid is the sigmoid function, and the formula is as follows:

[0070]

[0071] The specific structure of the thermal infrared information attention block is as follows: Figure 4 As shown;

[0072] Step 2.2: Based on the network structure and parameters of the composite image training dataset in Step 2.1, set the thermal infrared information. Note the network's input size, output size, loss function, optimizer function, learning rate, etc. Generally, the input size can be set to 256×256, the output size must be consistent with the input size, the loss function can be the cross-entropy loss function, the optimizer function can be the adaptive moment estimation function (Adam function), and the learning rate can be set to 0.001.

[0073] Step 3 of this implementation plan includes the following steps:

[0074] Step 3.1: Set the number of training iterations and train the thermal infrared information attention network using a composite image training dataset. The number of training iterations can be determined based on the quantity and quality of the composite image pairs. Generally, the more image pairs, the more training iterations are needed; the lower the quality of the image pairs, the more training iterations are needed. For a standard training dataset, 200 iterations can be set. During training, the thermal infrared information attention network aims to minimize the loss value of the loss function. Therefore, if the network's loss value does not decrease after multiple iterations, training can be stopped.

[0075] Step 3.2: During each training round, the composite image training dataset is input into the network in batches. The number of image pairs in each batch can be set according to the computer's performance; generally, the better the computer's performance, the more image pairs can be input in each batch. During each training round, the thermal infrared information attention network learns according to the direction that minimizes the loss value, thereby adjusting the network's own parameters. The network parameters that minimize the loss value should be saved during training.

[0076] Step 3.3: Repeat steps 3.1 and 3.2 to train the thermal infrared information attention network multiple times, compare the different training results, and save the set of network parameters that minimizes the loss value. A schematic diagram of the training of the thermal infrared information attention network is shown below. Figure 5 As shown.

[0077] Step 4 of this implementation plan includes the following sub-steps:

[0078] Step 4.1: Prepare the composite image test dataset. The composite image test dataset should have the same resolution and size as the training dataset, and the driving environment captured should be roughly the same. The images in the composite image test dataset should also be visible light and thermal infrared images of the same resolution and scene. Visible light-thermal infrared binocular cameras can be used for image acquisition.

[0079] Step 4.2: Input the image pairs from the composite image test dataset into the trained thermal infrared information attention network. The network outputs the driving environment segmentation result for each image pair. A schematic diagram of the thermal infrared information attention network segmentation in the driving environment is shown below. Figure 6 As shown. Figure 7 This image demonstrates the segmentation results generated by a thermal infrared information attention network for real-world driving image pairs. The left image is the visible light image, the middle image is the thermal infrared image, and the right image is the segmentation result. In the segmentation result image, black represents the background, and the other areas represent the segmented targets.

Claims

1. A method for segmenting driving scenes based on a neural network with a thermal infrared attention mechanism, characterized in that, Includes the following steps: Step 1: Acquire several pairs of visible light images and thermal infrared images of the same resolution and driving environment provided by the vehicle-mounted camera, perform category labeling, and obtain a composite image training dataset; Step 2: Construct a thermal infrared information attention network; Step 3: Train the thermal infrared information attention network based on the composite image training dataset to obtain the trained thermal infrared information attention network; Step 4: Use the thermal infrared information attention network to segment the driving environment image. The driving environment image is a pair of visible light image and thermal infrared image with the same resolution as the composite image training dataset. The thermal infrared information attention network has a visible light image encoder, a composite image decoder and a thermal infrared downsampling system; The visible light image encoder consists of a series of downlink blocks. Each downlink block contains a convolution operation, a batch normalization operation, and an activation function operation. Except for the last downlink block, each block contains a downsampling operation. Through this method, the network extracts shape and texture features from the visible light image. The operation formula for the downlink block of the visible light image encoder is as follows: ， In the formula, This is the input to the downlink block of the visible light image encoder. This is the output of the downlink block of the visible light image encoder. This indicates a convolution operation with a kernel size of 3×3. For batch normalization operations; For leaky linear rectification activation operation, This indicates the use of average downsampling operation; The thermal infrared downsampling system comprises several thermal infrared downsampling blocks. Each downsampling block contains downsampling operations of different sizes, and includes a convolution operation, a batch normalization operation, and an activation function operation, as shown in the formula. ， In the formula, This indicates the input to the thermal infrared downlink block; the input is a thermal infrared image. This indicates the output of the thermal infrared downlink block. This indicates a max-pooling operation, where the downsampling size varies for different downlink blocks. This indicates a convolution operation with a kernel size of 3×3. For batch normalization operations, This is for leaky linear rectification activation operation; The composite image decoder consists of a series of uplink blocks and a series of thermal infrared information attention blocks. The uplink blocks of the decoder are used to recover the features and size of the image, and consist of one upsampling operation, two convolution operations, two batch normalization operations, and two activation functions. The operation formula of the uplink blocks of the decoder is as follows: ， In the formula, This represents the multispectral remote sensing image features extracted from the downlink block of the corresponding visible light encoder. This indicates the output of the thermal infrared information attention block preceding the uplink block; if there is no such block, it is 0. This represents a matrix join operation. This indicates an upsampling operation, which expands the length and width of the feature to twice the size of the input; X out4 This indicates the features of the output multispectral remote sensing image.

2. The driving scene segmentation method based on thermal infrared attention mechanism neural network according to claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1: Obtain a visible light image and a thermal infrared image of the driving environment with the same resolution and the same scene provided by the vehicle-mounted camera. The content of corresponding pixels in the two images should be the same. The visible light image is an image formed by the red, yellow and blue spectrum bands taken by an ordinary color camera, and the thermal infrared image is an image taken by a thermal infrared imager. The two images form a complex image pair. Several pairs of image pairs with the same resolution need to be prepared, and they need to include multiple scenes and multiple environments. Step 1.2: Crop the complex image pairs to the same size, with the same orientation, length, and width, to form a composite image training dataset.

3. The driving scene segmentation method based on thermal infrared attention mechanism neural network according to claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Determine the basic parameters of the thermal infrared information attention network based on factors such as the quality and quantity of complex image pairs; Step 2.2: Construct a thermal infrared information attention network structure based on a deep learning environment, and set the network according to the parameters determined in Step 2.

1.

4. The driving scene segmentation method based on thermal infrared attention mechanism neural network according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: Set the number of training iterations for the thermal infrared information attention network based on the quality and quantity of the composite image training dataset; Step 3.2: Train the thermal infrared information attention network using the composite image training dataset; Step 3.3: Save the completed thermal infrared information. Note the network parameters of the network.

5. The driving scene segmentation method based on thermal infrared attention mechanism neural network according to claim 1, characterized in that, Step 4 includes the following sub-steps: Step 4.1: Obtain composite image pairs of the driving environment provided by the vehicle camera with the same resolution as the composite image training dataset, and normalize the composite image pairs so that the length and width of the composite image pairs are the same as the image pairs in the training dataset. Step 4.2: Input the normalized composite image pair to be segmented into the trained thermal infrared information attention network to obtain the network's segmentation result; Step 4.3: Organize and save the thermal infrared information and note the vehicle scene segmentation results of the network.

Images