Complete feature aggregation method based on target area demand guidance

Abstract

The invention provides a complete feature aggregation method based on target area demand guidance, and the method comprises the following steps: generating masks for a color image in an RGB mode and a thermal infrared image in a thermal infrared mode through employing a mask module through Gaussian distribution; using a mask to locate a foreground region of interest of the color image and a foreground region of interest of the thermal infrared image; performing feature difference alignment on the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image by using a cross-modal correction module; and a multi-modal fusion module is used for carrying out fusion processing on the differentially aligned features to obtain fusion complete features, so that finer semantic understanding is realized, complementary information among different modal images can be effectively utilized, and technical support is provided for operation of automatic driving. A construction method of the semantic segmentation network is further designed, and after complete features are obtained, a loss function is designed to supervise the obtained rich priori knowledge so as to guide the information flow to gradually refine the segmentation map.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a complete feature aggregation method guided by target area requirements. Background Art

[0002] Autonomous driving technology has become one of the focal areas of a new round of scientific and technological revolution and industrial transformation. The intelligent road scene understanding system is an important guarantee for the safe operation of autonomous driving. The semantic segmentation of road scenes aims to effectively segment and accurately identify scene elements at the pixel level (for example, lane lines, trees, signs, vehicles, and pedestrians, etc.), and provide accurate road condition information for intelligent vehicles. It is an important core technology of the road scene understanding system. With the diversification of the number of road scenes, the types of elements, and the sources of information, the complexity of road scenes is increasing dramatically. How to design a high-precision road scene semantic segmentation method has become an urgent problem to be solved in the field of autonomous driving.

[0003] So far, CNN-based RGB semantic segmentation methods have achieved outstanding results. However, unimodal sensory methods provide less information for model training and inaccurate prediction results in understanding scenes with complex backgrounds and dim lighting. To correct this, RGB-T introduces thermal sensors that are relatively robust to changes in lighting conditions. Unlike visual cameras that respond to the visible light spectrum, thermal sensors are able to capture infrared radiation emitted by objects, revealing subtle texture information about the surrounding environment even in harsh lighting conditions.

[0004] Based on the semantic segmentation of thermal sensors, the current main research direction is to design a multimodal fusion module to effectively combine the RGB modality and thermal modality information, thereby improving the accuracy of semantic segmentation. There are currently three typical cross-modal fusion paradigms, namely the encoder pre-fusion paradigm, the decoder post-fusion paradigm, and the intermediate fusion paradigm based on multi-level cross-modality between the encoding and decoding stages. Among them, encoder fusion means that the features are input into the decoder as a whole for processing in the feature extraction stage. Decoder fusion usually fuses the learned features in the inference stage. Intermediate fusion focuses on integrating information from different modalities at different levels. The above three fusion paradigms have greatly promoted the development of RGB-T semantic segmentation. However, their shortcomings are obvious. The paradigms of encoder fusion and decoder fusion may not make full use of the complementary information between different modalities. If the feature spaces of different modalities are very different, fusion may become difficult. The intermediate fusion paradigm may require a lot of experiments and adjustments to determine the best fusion point to avoid information loss and redundancy.

[0005] In practical applications, basic features often carry rich details and edge information, and these features vary in different modalities, and their importance will also change accordingly. It is worth noting that the foreground object to be segmented may only occupy a small proportion in the image. Taking the MFNet dataset as an example, the proportion of foreground pixels is only 7.862%. If the entire image is treated equally, it may lead to deviations in the segmentation results. Therefore, when designing a deep learning network to perform semantic segmentation tasks, it may not be appropriate to use a fixed feature fusion strategy. In order to achieve optimal performance, the feature fusion strategy should take into account the importance and distribution of features, as well as how to effectively locate the region of interest.

[0006] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a complete feature aggregation method guided by target area demand, so as to solve the problems of incomplete feature extraction and simple fusion of traditional network features, and provide a theoretical reference for realizing a high-performance autonomous driving road scene understanding system.

[0008] In order to achieve the above objectives, the present invention starts from the three steps of positioning, correction and fusion, and designs a complete feature aggregation method based on the guidance of target area requirements. First, the weight mask and inversion mask obtained by Gaussian distribution are used to guide the distribution balance of color images and thermal infrared images, and then, based on the rich prior knowledge obtained, the RGB image and TIR image information of the road scene are fused using an adaptive fusion method.

[0009] The technical solution of the present invention is: a complete feature aggregation method based on target area demand guidance, comprising the following steps:

[0010] A mask module is used to generate a mask by applying Gaussian distribution to the color image of the RGB modality and the thermal infrared image of the thermal infrared modality, and the mask is used to locate the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image;

[0011] The cross-modal correction module is used to align the feature differences between the foreground region of interest in the color image and the foreground region of interest in the thermal infrared image.

[0012] The multimodal fusion module is used to fuse the features of differential alignment to obtain fused complete features.

[0013] Preferably, the specific steps of using the cross-modal correction module to perform feature difference alignment on the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image are:

[0014] Feature map M of the foreground region of interest in color images based on channel attention mechanism RGB and the feature map M of the foreground region of interest in the thermal infrared image T Denoising is performed to obtain the RGB and M T Extract channel detail features, the formula is as follows:

[0015]

[0016] in, is the channel detail feature of the color image, is the channel detail feature of the thermal infrared image, and Channel attention maps for color images and thermal infrared images respectively;

[0017] Based on the spatial attention mechanism, the channel detail features are weighted and corrected to obtain fine features. The formula is as follows:

[0018]

[0019] in, and Representing the spatial attention maps of color images and thermal infrared images respectively; is the fine feature of the color image, It is the fine features of thermal infrared images;

[0020] The channel detail features and fine features of one image modality are used to correct the features of another image modality to obtain complete features of different image modalities:

[0021]

[0022] Among them, λ c and λ s is a pre-set hyperparameter value, ranging from 0 to 1; D RGB is the complete feature of the color image, D T It is a complete feature of thermal infrared image.

[0023] Preferably, the multimodal fusion module is used to fuse the features of the difference alignment, and the specific steps of obtaining the fused complete features are as follows:

[0024] Concatenate the complete features of the color image and the complete features of the thermal infrared image, and convert the feature size back to the original feature size;

[0025] Use a large kernel for depthwise separable convolution and re-weight the fused features through a simple Hadamard product, as follows:

[0026]

[0027] Among them, Linear( ) represents the linear layer function, f DWconv3×3 Indicates a convolution with a convolution kernel of 3×3, C fuse is the fusion of complete features; Q is the Hadamard product.

[0028] Furthermore, the present invention proposes an adaptive road scene semantic segmentation method based on target area demand guidance for autonomous driving road scenes, comprising the following steps: constructing an adaptive road scene semantic segmentation network, the adaptive road scene semantic segmentation network comprising a cross-modal correction module, a multimodal fusion module and a semantic prediction module, wherein the cross-modal correction module is used to perform feature difference alignment on the foreground area of ​​interest of the color image and the foreground area of ​​interest of the thermal infrared image; the multimodal fusion module is used to fuse the features of the difference alignment to obtain fused complete features; the semantic prediction module is used to perform final semantic prediction based on the fused complete features; the color image and thermal infrared image of the road scene are obtained; the mask module is used to apply Gaussian distribution to the color image and the thermal infrared image to generate a mask, and the mask is used to locate the foreground area of ​​interest of the color image and the foreground area of ​​interest of the thermal infrared image; the foreground area of ​​interest of the color image and the foreground area of ​​interest of the thermal infrared image are sent to the adaptive road scene semantic segmentation network, and the parameters of the adaptive road scene semantic segmentation network are trained based on the neutral cross entropy loss function to obtain the trained multimodal information guided adaptive road scene semantic segmentation network.

[0029] The beneficial effects of the present invention are as follows: the present invention studies the field of multimodal semantic segmentation methods, proposes a complete feature aggregation method guided by target area requirements, uses a mask module to apply Gaussian distribution to RGB modal color images and thermal infrared images of thermal infrared modalities to generate masks, and uses the mask to locate the foreground area of ​​interest of the color image and the foreground area of ​​interest of the thermal infrared image; uses a cross-modal correction module to perform feature difference alignment on the foreground area of ​​interest of the color image and the foreground area of ​​interest of the thermal infrared image; uses a multimodal fusion module to fuse the features of the difference alignment to obtain fused complete features. The above scheme ensures that the obtained features can accurately reflect the area of ​​interest of the input data and improves the segmentation accuracy, overcomes the problems of noise and incomplete feature extraction in feature extraction, provides an important reference for research in the field of semantic segmentation data processing, and further provides important auxiliary analysis tools for the fusion of multimodal features and semantic segmentation methods, providing technical support for practical applications.

[0030] Furthermore, the present invention proposes an adaptive road scene semantic segmentation method based on target area demand guidance, which can achieve high-precision semantic segmentation of road scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of a complete feature aggregation method guided by target area demand according to Example 1 of the present invention.

[0032] Figure 2 It is a flow chart of the cross-modal correction module in Example 1 of the present invention.

[0033] Figure 3 It is a flow chart of the multimodal fusion module in Example 1 of the present invention.

[0034] Figure 4 It is a flow chart of a complete feature aggregation method guided by target area demand according to Example 1 of the present invention.

[0035] Figure 5 It is a schematic diagram of a scene image segmented by using a complete feature fusion method for the data set MFNet in Example 1 of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0038] Example 1

[0039] This embodiment provides a complete feature aggregation method based on target area demand guidance, such as Figure 1 As shown, the following steps are included:

[0040] Step S1: Use the mask module to transform the color image RGB in = {R 1 ,R 2 ,R 3 ,R 4 Thermal infrared images TIR in ={T 1 ,T 2 ,T 3 ,T 4}Applying Gaussian distribution to generate a mask; after generating the mask, using the mask to locate the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image.

[0041] The specific steps of applying Gaussian distribution to generate a mask for a color image in RGB mode and a thermal infrared image in thermal infrared mode are as follows:

[0042] Calculate the average gray value of the three channels of the color image to obtain a gray image;

[0043] Apply a Gaussian filter to the gray image to obtain the weight mask M of the color image ask, used to indicate the importance of RGB information;

[0044] Subtract the weight mask M of the color image obtained in the previous step from the all-one matrix ask , generate the inverted mask IM ask , used to indicate the importance of each location of the thermal infrared image.

[0045] It should be noted that here the mask calculation is only performed once for the most original color image and thermal infrared image, and the mask calculation does not need to be performed again in the subsequent steps.

[0046] In some cases, the mask may be a mask matrix.

[0047] The specific steps of using a mask to locate the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image are as follows:

[0048] Using the weight mask M ask and inverted mask IM ask The color image and thermal infrared image are weighted by residual connection to obtain the feature maps of the foreground area of ​​interest in different modalities. The formula is as follows:

[0049] M RGB =f DWconv3×3 [M ask ; RGB in ]×f conv1×1 (RGB in )

[0050] M T =f DWconv3×3 [IM ask ;TIR in ]×f conv1×1 (TIR in )

[0051] Among them, f DWconv3×3 represents 3×3 convolution, f conv1×1 Represents 1×1 convolution, RGB in Represents the input color image, TIR in represents the input thermal infrared image, M RGB Represents the feature map of the foreground region of interest in RGB mode, M TIR Feature map representing the foreground region of interest for the thermal infrared modality.

[0052] Step S2, using a cross-modal correction module to perform feature difference alignment on the foreground region of interest in the color image and the foreground region of interest in the thermal infrared image.

[0053] like Figure 2 As shown, the specific steps are as follows:

[0054] Feature map M of the foreground region of interest in RGB mode based on channel attention mechanism RGB and the feature map of the foreground area of ​​interest in the thermal infrared modality M T Denoising is performed to obtain the RGB and M T Extract channel detail features, the formula is as follows:

[0055]

[0056] in, is the channel detail feature of the color image, is the channel detail feature of the thermal infrared image, and Channel attention maps for color images and thermal infrared images respectively; used to weight the calibration M by element-wise multiplication RGB and M T ;

[0057] Preferably, the channel detail features refer to texture features and color features.

[0058] It should be noted that through this step, texture features and color features can be extracted from the foreground regions of interest in both modalities while eliminating the inevitable camera noise.

[0059] Specifically, channel attention mapping and The steps to obtain are:

[0060] The feature map M of the foreground region of interest RGB and the feature map M of the foreground region of interest T Perform global maximum pooling processing, multi-layer perceptron MLP processing and sigmoid activation function operation respectively to obtain the corresponding channel attention mapping. The formula is as follows:

[0061]

[0062] f t c =δ(F MLP (MaxPool(M T )))

[0063] Among them, F MLP represents a multi-layer perceptron, δ() is a sigmoid activation function, and MaxPool() represents the global maximum pooling.

[0064] Based on the spatial attention mechanism, the channel detail features are weighted and corrected to obtain fine features. The formula is as follows:

[0065]

[0066] in, and f t s Representing the spatial attention maps of color images and thermal infrared images respectively; is the fine feature of the color image, It is the fine features of thermal infrared images;

[0067] Specifically, the steps for obtaining the spatial attention map are as follows:

[0068] The channel detail features of the RGB modality and the channel detail features of the thermal infrared modality obtained in the previous step are respectively subjected to 7×7 convolution and sigmoid activation function operations to obtain the corresponding spatial attention mapping. The formula is as follows:

[0069]

[0070] in, represents 7×7 convolution, δ() is the sigmoid activation function;

[0071] It can be understood that, through this step, the single-modal feature will contain more discriminative guidance information of other modalities, which can further reduce the difference between the RGB feature and the thermal infrared feature to a certain extent.

[0072] The channel detail features and fine features of one image modality are used to correct the features of another image modality to obtain complete features of different image modalities:

[0073]

[0074] Among them, λ c and λ s is a pre-set hyperparameter value, ranging from 0 to 1; D RGB is the complete feature of the color image, D T is the complete feature of the thermal infrared image; preferably, λ c and λ s The default value is 0.5.

[0075] Step S3, using a multimodal fusion module to fuse the differentially aligned features to obtain fused complete features.

[0076] The specific steps of using the multimodal fusion module to fuse the features of the difference alignment to obtain the fused complete features are as follows: Figure 3 As shown, including:

[0077] The specific steps for feature fusion processing of complete features of different modalities are as follows:

[0078] Concatenate the complete features of the color image and the complete features of the thermal infrared image, and convert the feature size back to the original feature size;

[0079] Use a large kernel for depthwise separable convolution and use the original features as guide weights to re-weight the fused features through a simple Hadamard product, as follows:

[0080]

[0081] Among them, Linear() is the linear layer function, f DWconv3×3 Indicates a convolution with a convolution kernel of 3×3, C fuse is the fusion of complete features; Q is the Hadamard product.

[0082] This is because the features after the large kernel depthwise separable convolution may deviate. For example, the large kernel can capture feature information in a larger range when extracting edge features, but it will also cause blurring of position information. Therefore, it is necessary to multiply the original features that have not been subjected to the large kernel depthwise separable convolution, that is, to use the convolution features as weights to modulate the value representation, so as to make better use of the large kernel convolution.

[0083] Example 2

[0084] This embodiment provides an adaptive road scene semantic segmentation method based on target area demand guidance, such as Figure 4 As shown, the following steps are included:

[0085] An adaptive road scene semantic segmentation network is constructed, the adaptive road scene semantic segmentation network comprising a mask module, a positioning module, a cross-modal correction module, a multimodal fusion module and a semantic prediction module, wherein the mask module is used to apply Gaussian distribution to the color image and the thermal infrared image to generate a mask, and the mask is used to locate the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image; the cross-modal correction module is used to perform feature difference alignment on the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image; the multimodal fusion module is used to fuse the features of the difference alignment to obtain fused complete features; the semantic prediction module is used to perform final semantic prediction based on the fused complete features;

[0086] Acquire color images and thermal infrared images of road scenes, send the color images and thermal infrared images to the adaptive road scene semantic segmentation network, and perform parameter training on the adaptive road scene semantic segmentation network based on a neutral cross entropy loss function to obtain an adaptive road scene semantic segmentation network guided by a trained multimodal information.

[0087] It is understandable that the semantic prediction module of the adaptive road scene semantic segmentation network structure can adopt the decoder in the conventional semantic segmentation network, and only adjust the encoder in the conventional semantic segmentation network.

[0088] Preferably, the calculation formula of the neutral cross entropy loss function is:

[0089]

[0090]

[0091] Among them, G i (x,y) and S i (x, y) represents the probability distribution of the true category of pixel (x, y) and the probability distribution of the output prediction respectively; W and H are the width and height of the input respectively, l i is the loss function of the i-th output prediction, and N is the number of classification labels of the output prediction.

[0092] For example, when the semantic segmentation categories of the road scene are roads, vehicles and pedestrians, and trees, that is, the classification label of the semantic segmentation is 3,

[0093] It can be seen that the neutral cross entropy loss function is defined as the average loss of output predictions, which can optimize the performance of the semantic segmentation model and guide the model to learn to distinguish features of different categories.

[0094] Specifically, the specific processing flow of the mask module, the positioning module, the cross-modal correction module, and the multimodal fusion module can be found in Example 1, which will not be repeated here.

[0095] Verification Example 1

[0096] This verification example uses an adaptive road scene semantic segmentation method guided by target area requirements to process the MFNet dataset. Figure 5 As shown, the following steps are included:

[0097] Step 1: Input the MFNet dataset to the target region demand-guided complete feature aggregation network. The MFNet dataset contains 1569 pairs of RGB images and TIR images, the training set consists of 784 pairs of RGB images and TIR images, the validation set consists of 392 pairs of RGB images and TIR images, and the remaining images are used for testing. The input size is adjusted to a fixed size of 480×640.

[0098] Step 2: Apply Gaussian distribution to the original multimodal data to generate a mask. Specifically, obtain the weight mask of the RGB image and the inverted mask of the TIR image.

[0099] Step 3: Based on the weighted mask, inverted mask and original multimodal data, accurate features of different modalities are obtained to locate the foreground area of ​​interest.

[0100] Step 4: Align the modal information of the foreground regions of interest in different modalities from both channel and spatial aspects.

[0101] Preferably, the specific steps of Step 4 are:

[0102] Step 4.1, guide the channel-by-channel calibration to extract texture features and color features from the two modal information;

[0103] Step 4.2, modal mixture space correction, establishes an interactive space along the spatial dimension to capture the spatial details and category-distinguishing semantic clues in the multimodal input images.

[0104] Step 4.3: Use the channel detail features and spatial features of one mode to correct the features of another mode to obtain complete features of different modes.

[0105] Step 5: Fuse the aligned multimodal features and use lightweight channel embedding To merge the updated RGB features and thermal infrared features, the merged features are fed to the depthwise separable convolutional layer DWconv 3×3 , element-by-element addition is performed through residual connection to enhance the expression of features.

[0106] Step 6: Adaptive scene semantic segmentation model, construct a loss function to supervise the model, and establish an adaptive road scene semantic segmentation model guided by multi-dimensional information.

[0107] Step 7, segmentation results, attached Figure 2 The first row is the input RGB image, the second row is the TIR image, and the third row is the segmentation result image of the adaptive road scene semantic segmentation method proposed in this embodiment.

[0108] Verification Example 2

[0109] This verification example uses an adaptive road scene semantic segmentation method guided by target area demand to process the PST900 dataset, including the following steps:

[0110] Step 1: Input the PST900 dataset to the target region demand-guided complete feature aggregation network. The PST900 dataset contains 894 pairs of RGB images and TIR images, and the training set consists of 597 pairs of RGB images and TIR images. The validation set consists of 288 pairs of RGB images and TIR images, and the remaining RGB images and TIR images are used for testing. The input size is adjusted to a fixed size of 720×1280.

[0111] Step 2: Apply Gaussian distribution to the original multimodal data to generate a mask. Specifically, obtain the weight mask of the RGB image and the inverted mask of the TIR image.

[0112] Step 3: Based on the weighted mask, inverted mask and original multimodal data, accurate features of different modalities are obtained to locate the foreground area of ​​interest.

[0113] Step 4: Align the feature differences of the foreground areas of interest in different modalities to prepare for the next step of multimodal feature fusion.

[0114] Preferably, the specific steps of Step 4 are:

[0115] Step 4.1, guided channel-by-channel calibration, eliminates the inevitable camera noise before fusion, and first removes this noise through channel-by-channel attention.

[0116] Step 4.2, modal mixture space correction, establish an interactive space along the spatial dimension to weight the spatial details and categories in the image.

[0117] Step 4.3: Use the channel detail features and spatial features of one mode to correct the features of another mode to obtain complete features of different modes.

[0118] Step 5: Fuse the aligned multimodal features and use lightweight channel embedding To merge the updated RGB features and thermal infrared features, the merged features are fed to the depthwise separable convolutional layer DWconv 3×3 , element-by-element addition is performed through residual connection to enhance the expression of features.

[0119] Step 6: Adaptive scene semantic segmentation model, construct a loss function to supervise the model, and establish an adaptive road scene semantic segmentation model guided by multi-dimensional information.

[0120] Step 7, segmentation results, attached Figure 3 The first row is the input RGB image, the second row is the TIR image, and the third row is the segmentation result image of the adaptive road scene semantic segmentation method proposed in this embodiment.

[0121] Example 3

[0122] This embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in Embodiment 1 or Embodiment 2 when executing the computer program.

[0123] Example 4

[0124] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 or Embodiment 2 are implemented.

[0125] Example 5

[0126] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 or Embodiment 2 are implemented.

Claims

1. A complete feature aggregation method based on target area demand guidance, characterized in that: The following steps are involved: A mask module is used to generate a mask by applying Gaussian distribution to the color image of the RGB modality and the thermal infrared image of the thermal infrared modality, and the mask is used to locate the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image; The cross-modal correction module is used to align the feature differences between the foreground region of interest in the color image and the foreground region of interest in the thermal infrared image. The multimodal fusion module is used to fuse the features of differential alignment to obtain fused complete features.

2. The complete feature aggregation method based on target area demand guidance according to claim 1 is characterized in that: The specific steps of applying Gaussian distribution to generate a mask for a color image in RGB mode and a thermal infrared image in thermal infrared mode are as follows: Calculate the average gray value of the three channels of the color image to obtain a gray image; Apply a Gaussian filter to the gray image to obtain the weight mask M of the color image ask , used to indicate the importance of RGB information; Subtract the weight mask M of the color image obtained in the previous step from the all-one matrix ask , generate the inverted mask IM ask , used to indicate the importance of each location of the thermal infrared image.

3. The complete feature aggregation method based on target area demand guidance according to claim 2 is characterized in that: The steps for locating the foreground region of interest in the RGB modality and the foreground region of interest in the depth modality based on the mask are as follows: Using the weight mask M ask and inverted mask IM ask The color image and thermal infrared image are weighted in a residual connection manner to obtain the foreground region of interest in the RGB modality and the foreground region of interest in the depth modality. The formula is as follows: M RGB =f DWconv3×3 [M ask ;RGB in ]×f conv1×1 (RGB in ) M T =f DWconv3×3 [IM ask ;TIR in ]×f conv1×1 (TIR in ) Among them, f DWconv3×3 represents a depth-separable convolution with a convolution kernel of 3×3, f conv1×1 Indicates a convolution with a convolution kernel of 1×1, RGB in Represents the input color image, TIR in represents the input thermal infrared image, M RGB Represents the feature map of the foreground region of interest in the color image, M TIR Represents the feature map of the foreground region of interest in a thermal infrared image.

4. The complete feature aggregation method based on target area demand guidance according to claim 3 is characterized in that: The specific steps of using the cross-modal correction module to perform feature difference alignment on the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image are as follows: Feature map M of the foreground region of interest in color images based on channel attention mechanism RGB and the feature map M of the foreground region of interest in the thermal infrared image T Denoising is performed to obtain the RGB and M T Extract channel detail features, the formula is as follows: in, is the channel detail feature of the color image, is the channel detail feature of the thermal infrared image, and Channel attention maps for color images and thermal infrared images respectively; Based on the spatial attention mechanism, the channel detail features are weighted and corrected to obtain fine features. The formula is as follows: in, and f t s Representing the spatial attention maps of color images and thermal infrared images respectively; is the fine feature of the color image, It is the fine features of thermal infrared images; The channel detail features and fine features of one image modality are used to correct the features of another image modality to obtain complete features of different image modalities: Among them, λ c and λ s is a pre-set hyperparameter value, ranging from 0 to 1; D RGB is the complete feature of the color image, D T It is a complete feature of thermal infrared image.

5. The complete feature aggregation method based on target area demand guidance according to claim 4 is characterized in that: The multimodal fusion module is used to fuse the features of the difference alignment. The specific steps to obtain the fused complete features are as follows: Concatenate the complete features of the color image and the complete features of the thermal infrared image, and convert the feature size back to the original feature size; Use a large kernel for depthwise separable convolution and re-weight the fused features through a simple Hadamard product, as follows: Among them, Linear( ) represents the linear layer function, f DWconv3×3 Indicates a convolution with a convolution kernel of 3×3, C fuse is the fusion of complete features; Q is the Hadamard product.

6. An adaptive road scene semantic segmentation method based on target area demand guidance, characterized in that: The following steps are involved: An adaptive road scene semantic segmentation network is constructed, wherein the adaptive road scene semantic segmentation network includes a mask module, a positioning module, a cross-modal correction module, a multimodal fusion module, and a semantic prediction module, wherein the mask module is used to apply Gaussian distribution to the color image and the thermal infrared image to generate a mask, and the mask is used to locate the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image; the cross-modal correction module is used to perform feature difference alignment on the foreground region of interest of the color image and the foreground region of interest of the thermal infrared image; The multimodal fusion module is used to fuse the features of differential alignment to obtain fused complete features; The semantic prediction module is used to make the final semantic prediction based on the fusion of complete features; Acquire color images and thermal infrared images of road scenes, send the color images and thermal infrared images to the adaptive road scene semantic segmentation network, and perform parameter training on the adaptive road scene semantic segmentation network based on a neutral cross entropy loss function to obtain an adaptive road scene semantic segmentation network guided by a trained multimodal information.

7. The method for adaptive road scene semantic segmentation based on target area demand guidance according to claim 6, characterized in that: The calculation formula of the neutral cross entropy loss function is: Among them, G i (x,y) and S i (x, y) represents the probability distribution of the true category of pixel (x, y) and the probability distribution of the output prediction respectively; W and H are the width and height of the input respectively, l i is the loss function of the i-th output prediction, and N is the number of classification labels of the output prediction.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the processor implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 7.

Images