A visible-thermal infrared scene understanding method based on explicit frequency decoupling in intelligent vehicles

Abstract

This invention discloses a visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling, belonging to the field of artificial intelligence technology. The method includes the following sub-steps: acquiring visible light images and thermal infrared images from visible light sensors and thermal infrared sensors respectively, and performing preprocessing; constructing an explicit frequency decoupling model, which sequentially includes a dual-branch encoder, an explicit frequency decoupling module, a component-specific fusion module, and a decoder; training the model end-to-end using a training set, calculating the composite loss function between the pixel-level scene semantic prediction map output by the model and the ground truth labels, and iteratively updating the network model parameters using a backpropagation algorithm and optimizer until the model converges; after training, applying the trained model to a test set, where the model predicts images in the test set, generates the final segmentation result, and compares it with the ground truth labels to evaluate the model's generalization ability and final performance.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for understanding visible light-thermal infrared scenes in intelligent vehicles based on explicit frequency decoupling. Background Technology

[0002] With the development of applications such as autonomous driving and intelligent security, accurate scene understanding technology has become the core of artificial intelligence vision systems. Traditional scene understanding methods mainly rely on visible light images, which can provide rich color and texture information under good lighting conditions, ensuring high segmentation accuracy. However, in practical application scenarios such as nighttime, inclement weather, or complex lighting (such as backlighting), the performance of a single visible light sensor has inherent limitations. The degradation of image quality can directly lead to the failure of scene understanding, posing a serious safety hazard. To address this issue, thermal infrared imaging, which is unaffected by lighting conditions, has been introduced as a key complementary sensing modality. Thermal infrared imaging is unaffected by lighting conditions, but its images typically have low resolution and lack fine texture. Therefore, fusing visible light and thermal infrared information has become a research hotspot.

[0003] In early research, researchers employed a direct fusion method based on simple operations. The core idea was to extract features from RGB and thermal infrared images separately, and then directly merge them through simple mathematical operations at one or more stages of the network. Common operations included element-wise operations and channel-level concatenation. For example, RTFNet, FuseSeg, and MFFENet use element-wise addition, directly adding the pixel values ​​at the same location in the two feature maps; while PST900 uses channel-level concatenation, stacking the two feature maps together along the channel dimension to form a "thicker" feature map.

[0004] To overcome the shortcomings of direct fusion methods, recent research has widely introduced attention mechanisms. The core idea is to allow the network model to learn autonomously which modality's information should be "focused on" in different scenes and spatial locations. This method no longer simply adds features together; instead, it first calculates an "importance weight" for each modality's features and then performs weighted fusion based on this weight. This weight is dynamically changed and learned autonomously by the network based on the input image content. Representative methods include: CAINet, which designs a context-aware interactive network that dynamically adjusts the level of attention to features of another modality based on the feature content of one modality; CCFFNet, which uses a cascaded attention module to perform complementary weighted fusion of features in the spatial and channel dimensions to achieve more refined information integration; and AGFNet, which designs an adaptive gated fusion module that uses a learnable "gating" mechanism to control the information flow and fusion ratio of RGB and thermal infrared features at the pixel level.

[0005] While attention-based methods represent a significant improvement over direct fusion, they still suffer from a deeper problem: they continue to weight feature maps of each modality as an indivisible whole. However, feature maps themselves contain information of different natures, including both "content" information representing their macroscopic contours and shapes, and "detail" information representing their fine textures and edges. Existing attention methods, while learning that "visible light features are more important in a certain region," indiscriminately apply this judgment to both the "content" and "detail" of the visible light features in that region. Ideally, the model should be able to simultaneously incorporate visible light "details" (such as sharp edges) and thermal infrared "content" (such as contours unaffected by illumination) within a region, as needed. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention proposes a method for understanding visible light-thermal infrared scenes in intelligent vehicles based on explicit frequency decoupling. The method is rationally designed, overcomes the shortcomings of existing technologies, and has good results.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for understanding visible light-thermal infrared scenes in intelligent vehicles based on explicit frequency decoupling includes the following sub-steps:

[0009] Step 1: Acquire visible light images and thermal infrared images from the visible light sensor and thermal infrared sensor respectively, and perform preprocessing;

[0010] Step 2: Construct a visible light-thermal infrared scene understanding model The model consists of a dual-branch encoder, an explicit frequency decoupling module, a component-specific fusion module, and a decoder.

[0011] Step 3: Use the training set to train the model Perform end-to-end training and compute the pixel-level scene semantic prediction map output by the model. The composite loss function between the truth label and the truth label Using the backpropagation algorithm and optimizer, based on The parameters of the network model are iteratively updated until the model converges;

[0012] Step 4: After training is complete, transfer the trained model... The model is applied to the test set, predicts images in the test set, generates the final segmentation result, and compares it with the real labels to evaluate the model's generalization ability and final performance.

[0013] Furthermore, in step 1, the visible light image and the thermal infrared image are spatially aligned using an image registration algorithm and uniformly scaled to a preset size to construct an original dataset of pixel-level annotations for the visible light-thermal infrared images. And divided into training sets and test set .

[0014] Furthermore, the dual-branch encoder employs two parallel Transformer-based backbone networks, serving as encoders for the visible light branch and the thermal imaging branch, respectively. The encoder comprises four layers. For the input visible light image and thermal imaging image, the two encoders extract features layer by layer and output feature maps respectively. and , Dimension is represented as ( ),in For batch size, For the number of channels, For height, For width.

[0015] Furthermore, the feature map output by the dual-branch encoder is fed into the explicit frequency decoupling module, which achieves decoupling through the following three sub-steps:

[0016] Step 2.1.1: By analyzing the feature map and Apply a low-pass filter operation to generate surrogate features that represent the low-frequency information of the feature map. and ;

[0017] Step 2.1.2: Transfer the feature map and After a convolutional transformation, the channels are uniformly divided, thereby pre-assigning independent channel groups to the content components and detail components, as shown in Equation (1):

[0018] (1)

[0019] in, This represents a convolutional transformation, which includes a standard 3×3 convolutional layer, layer normalization, and activation function. This indicates that the channels of the feature map are changed from... Evenly divided into , and They are respectively Feature maps of the segmented content components and detail components. and They are respectively Feature maps of the segmented content components and detail components;

[0020] Step 2.1.3: First, , , , , and Flatten the spatial dimensions, and then perform operations on the flattened spatial dimensions. Normalization yields the set of unit vectors. , , , , and Calculate the similarity between unit vectors As shown in equation (2):

[0021] (2)

[0022] in, Indicates the inner product. This refers to temperature hyperparameters.

[0023] By using the frequency decoupling contrast loss function, the content component is forced to be similar to the feature representation of the low-frequency proxy, while the detail component is forced to be dissimilar to the feature representation of the low-frequency proxy, as shown in Equation (3):

[0024] (3)

[0025] in, and The first Frequency decoupling comparison loss function for visible light and thermal infrared branches;

[0026] The final output of this module is four sets of decoupled feature maps: .

[0027] Furthermore, the decoupled feature map input component-specific fusion module includes the following three stages:

[0028] The general preprocessing stage includes the following sub-steps:

[0029] Step 2.2.1: A regularization strategy driven by cosine similarity loss is adopted. First, layer normalization and instance normalization are used to extract modality-specific styles. Then, style differences are removed through feature transformation. Finally, the cosine similarity loss function is used to guide the network to learn modality-invariant features, as shown in Equation (4):

[0030] (4)

[0031] in, Represents the cosine similarity function. This indicates element-wise subtraction. Representation layer normalization, Indicates instance normalization, and Let these represent the cosine similarity loss functions for the content component and the detail component, respectively; these are minimized during training. and The network is guided to learn content and detail representations that maximize intermodal similarity;

[0032] Step 2.2.2: Using the visible light component as a reference, calculate a transformation matrix from thermal imaging to the visible light semantic space, as shown in equation (5):

[0033] (5)

[0034] in, and These are the transformation matrices for the content components and the detail components, respectively; This is a flattening operation used to flatten a 3D feature map ( Flattened in space, it becomes a two-dimensional matrix. ); For min-max normalization;

[0035] The transformation matrix is ​​then used to guide the alignment of thermal imaging components, resulting in aligned content features. and As shown in equation (6):

[0036] (6)

[0037] in, This is an inverse flattening operation used to flatten a two-dimensional matrix ( Reverse flattening into a three-dimensional feature map ( ).

[0038] In the parallel component-specific enhancement and fusion stage, different strategies are used to enhance the content and detail components, specifically:

[0039] The content component fusion path is as follows: a spatial noise suppression method is adopted, which smooths features through downsampling and upsampling operations to generate spatial attention weights. Used to enhance the features of aligned content. To obtain the final content fusion features As shown in equation (7):

[0040] (7)

[0041] in, This indicates a global average pooling layer. This indicates a doubling of upsampling. This represents the Sigmoid activation function. This indicates element-wise multiplication;

[0042] The detail component fusion path is as follows: A hollow space pyramid pooling operation is used to obtain the detail fused features. As shown in equation (8):

[0043] (8)

[0044] in, This represents the pyramid pooling operation for empty spaces;

[0045] In the adaptive fusion stage, firstly and The features are concatenated along the channel dimension, and then channel attention weight vectors are learned from the concatenated features through global average pooling and 1×1 convolutional layers. As shown in equation (9):

[0046] (9)

[0047] in, This indicates that channel concatenation is performed on the feature. This represents a standard 1×1 convolutional layer.

[0048] Finally, using this weight The content features and detail features are weighted and summed to obtain the final fused features. As shown in equation (10):

[0049] (10).

[0050] Furthermore, the decoder uses cascaded decoding units ( The resolution of the feature map is gradually restored from the bottom up, for each decoding unit. First, the output of the previous layer decoding unit... Adaptive fusion of features with the current level Element-wise addition is performed, followed by upsampling using transposed convolution and feature refinement using standard convolution, ultimately generating a full-size pixel-level scene semantic prediction map. ;when At that time, the output of the decoding unit in the previous layer does not exist.

[0051] Furthermore, in step 4, the composite loss function Including scene semantic loss function Decoupling supervision loss function Cosine similarity loss function As shown in equations (11)-(13):

[0052] (11)

[0053] (12)

[0054] (13)

[0055] in, and Hyperparameters used to balance weights.

[0056] The beneficial technical effects of this invention are as follows:

[0057] This invention proposes a visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling. Compared with existing technologies, this invention decomposes features into low-frequency content components and high-frequency detail components through explicit frequency decoupling, thereby achieving independent modeling of different attribute information. This strategy fundamentally avoids category confusion and boundary blurring caused by information mixing, significantly improving the segmentation accuracy of small targets and object edges.

[0058] Furthermore, building upon successful decoupling, this invention employs a component-specific alignment and fusion strategy. This strategy effectively reduces the differences in distribution and semantics between different modalities before fusion through multimodal regularization difference reduction and pixel-level semantic alignment. This ensures that the subsequent fusion process can suppress cross-modal noise, improve fusion quality, and ultimately significantly enhance the model's segmentation performance and robustness in complex scenarios such as nighttime and severe weather. Attached Figure Description

[0059] Figure 1 This is a flowchart of the visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling in this invention.

[0060] Figure 2 This is a block diagram of the overall structure of the visible light-thermal infrared scene understanding model in this invention; Detailed Implementation

[0061] The specific embodiments of the present invention will be further described below with reference to specific examples:

[0062] A method for understanding visible light-thermal infrared scenes in intelligent vehicles based on explicit frequency decoupling includes the following sub-steps:

[0063] Step 1: Acquire visible light images and thermal infrared images from the visible light sensor and thermal infrared sensor respectively, and perform preprocessing;

[0064] Visible light and thermal infrared images are spatially aligned using image registration algorithms (such as feature point-based or optical flow methods), and then uniformly scaled to a preset size to construct a raw dataset of pixel-level annotations for visible light-thermal infrared images. And divided into training sets and test set .

[0065] Step 2: Construct a visible light-thermal infrared scene understanding model The model consists of a dual-branch encoder, an explicit frequency decoupling module, a component-specific fusion module, and a decoder.

[0066] The forward propagation process of the model is as follows:

[0067] The dual-branch encoder employs two parallel Transformer-based backbone networks (such as SegFormer-B3), serving as encoders for the visible light branch and the thermal imaging branch, respectively. The encoder consists of four layers. For the input visible light image and thermal imaging image, the two encoders extract features layer by layer and output feature maps respectively. and , Its dimension is represented as ( ),in For batch size, For the number of channels, For height, For width.

[0068] The feature map output from the dual-branch encoder is sent to the explicit frequency decoupling module, based on ( The intrinsic information attributes of an object are decomposed into two orthogonal components: the content component captures the macroscopic semantic information of the object, corresponding to the low-frequency components in the feature map; the detail component captures the microscopic texture and edges of the object, corresponding to the high-frequency components in the feature map. This module achieves decoupling through the following three sub-steps:

[0069] Step 2.1.1: Generate low-frequency agent information. This is done by analyzing the feature map. and Apply a low-pass filter operation (such as channel averaging) to each feature map to generate surrogate features that represent the low-frequency information of the feature map. and ;

[0070] Step 2.1.2: Structured separation of feature channels. The feature map... and After a convolutional transformation, the channels are uniformly divided, thereby pre-assigning independent channel groups to the content components and detail components, as shown in Equation (1):

[0071] (1)

[0072] in, Represents convolutional transformation, including a standard 3×3 convolutional layer, layer normalization (LayerNorm), and activation function (GELU). This indicates that the channels of the feature map are changed from... Evenly divided into , and They are respectively Feature maps of the segmented content components and detail components. and They are respectively Feature maps of the segmented content components and detail components;

[0073] Step 2.1.3: Compare loss-driven decoupling. First, […]. , , , , and Spatial dimension ( Flatten the surface, and then perform spatial dimension analysis on the flattened surface. Normalization yields the set of unit vectors. , , , , and Calculate the similarity between unit vectors As shown in equation (2):

[0074] (2)

[0075] in, Indicates the inner product. This refers to temperature hyperparameters.

[0076] By using the frequency decoupling contrast loss function, the content component is forced to be similar to the feature representation of the low-frequency proxy, while the detail component is forced to be dissimilar to the feature representation of the low-frequency proxy, as shown in Equation (3):

[0077] (3)

[0078] in, and The first Frequency decoupling comparison loss function for visible light and thermal infrared branches;

[0079] The final output of this module is four sets of decoupled feature maps: .

[0080] The decoupled feature map input component-specific fusion module includes the following three stages:

[0081] The general preprocessing stage includes the following sub-steps:

[0082] Step 2.2.1: Multimodal difference reduction. To reduce the difference in feature distribution between visible light and thermal infrared, a regularization strategy driven by cosine similarity loss is adopted. First, layer normalization (LN) and instance normalization (IN) are used to extract modality-specific styles. Then, feature transformation is used to remove style differences. Finally, the cosine similarity loss function is used to guide the network to learn modality-invariant features, as shown in Equation (4):

[0083] (4)

[0084] in, Represents the cosine similarity function. This indicates element-wise subtraction. Representation layer normalization, Indicates instance normalization, and Let these represent the cosine similarity loss functions for the content component and the detail component, respectively; these are minimized during training. and The network is guided to learn content and detail representations that maximize intermodal similarity;

[0085] Step 2.2.2: Semantic Alignment. To ensure feature fusion within the same or similar semantic context, a transformation matrix from thermal imaging to the visible light semantic space is calculated based on the visible light component, as shown in Equation (5):

[0086] (5)

[0087] in, and These are the transformation matrices for the content components and the detail components, respectively; This is a flattening operation used to flatten a 3D feature map ( Flattened in space, it becomes a two-dimensional matrix. ); For min-max normalization;

[0088] The transformation matrix is ​​then used to guide the alignment of thermal imaging components, resulting in aligned content features. and As shown in equation (6):

[0089] (6)

[0090] in, This is an inverse flattening operation used to flatten a two-dimensional matrix ( Reverse flattening into a three-dimensional feature map ( ).

[0091] In the parallel component-specific enhancement and fusion stage, different strategies are used to enhance the content and detail components, specifically:

[0092] The content component fusion path is as follows: To suppress background noise and highlight the main semantics, a spatial noise suppression method is adopted, which smooths features through downsampling and upsampling operations to generate spatial attention weights. Used to enhance the features of aligned content. To obtain the final content fusion features As shown in equation (7):

[0093] (7)

[0094] in, This indicates a global average pooling layer. This indicates a doubling of upsampling. This represents the Sigmoid activation function. This indicates element-wise multiplication;

[0095] The detail component fusion path is as follows: To fully exploit the texture and edge information between detail components, a dilated spatial pyramid pooling (ASPP) operation is employed. ASPP utilizes multiple parallel dilated convolutions with different dilation rates to capture and aggregate fine texture and contextual information from multiple receptive field scales, thereby enhancing the aligned detail features. The representational ability is used to obtain detailed fusion features. As shown in equation (8):

[0096] (8)

[0097] in, This represents the pyramid pooling operation for empty spaces;

[0098] In the adaptive fusion stage, firstly and The features are concatenated along the channel dimension, and then channel attention weight vectors are learned from the concatenated features through global average pooling and 1×1 convolutional layers. As shown in equation (9):

[0099] (9)

[0100] in, This indicates that channel splicing is performed on the feature. This represents a standard 1×1 convolutional layer.

[0101] Finally, using this weight The content features and detail features are weighted and summed to obtain the final fused features. As shown in equation (10):

[0102] (10).

[0103] The decoder uses cascaded decoding units ( The resolution of the feature map is gradually restored from the bottom up. Each decoding unit... First, the output of the previous layer decoding unit... Adaptive fusion of features with the current level Element-wise addition is performed, followed by upsampling using transposed convolution and feature refinement using standard convolution, ultimately generating a full-size pixel-level scene semantic prediction map. It is important to note that when At that time, the output of the decoding unit in the previous layer does not exist.

[0104] Step 3: Use the training set to train the model Perform end-to-end training and compute the pixel-level scene semantic prediction map output by the model. The composite loss function between the truth label and the truth label Using the backpropagation algorithm and optimizer, based on The parameters of the network model are iteratively updated until the model converges;

[0105] Step 4: After training is complete, transfer the trained model... The model is applied to the test set, predicts images in the test set, generates the final segmentation result, and compares it with the real labels to evaluate the model's generalization ability and final performance.

[0106] Composite loss function Including scene semantic loss function Decoupling supervision loss function Cosine similarity loss function As shown in equations (11)-(13):

[0107] (11)

[0108] (12)

[0109] (13)

[0110] in, and Hyperparameters used to balance weights.

[0111] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for understanding visible light-thermal infrared scenes in intelligent vehicles based on explicit frequency decoupling, comprising the following sub-steps: Step 1: Acquire visible light images and thermal infrared images from the visible light sensor and thermal infrared sensor respectively, and perform preprocessing; Step 2: Constructing a visible light-thermal infrared scene understanding model The model comprises, in sequence, a double-branch encoder, an explicit frequency decoupling module, a component-specific fusion module and a decoder. The feature map output from the dual-branch encoder is fed into the explicit frequency decoupling module, which achieves decoupling through the following three sub-steps: Step 2.1.1: Generating proxy feature maps representing low frequency information of the feature maps by applying a low pass filter operation to the feature maps and respectively and ; Step 2.1.2: Transfer the feature map and After a convolutional transformation, the channels are uniformly divided, thereby pre-assigning independent channel groups to the content components and detail components, as shown in Equation (1): ；(1) in, This represents a convolutional transformation, which includes a standard 3×3 convolutional layer, layer normalization, and activation function. This indicates that the channels of the feature map are changed from... Evenly divided into , and They are respectively Feature maps of the segmented content components and detail components. and They are respectively Feature maps of the segmented content components and detail components; Step 2.1.3: First, , , , , and Flatten the spatial dimensions, and then perform operations on the flattened spatial dimensions. Normalization yields the set of unit vectors. , , , , and Calculate the similarity between unit vectors As shown in equation (2): ；(2) in, Indicates the inner product. This refers to temperature hyperparameters. By using the frequency decoupling contrast loss function, the content component is forced to be similar to the feature representation of the low-frequency proxy, while the detail component is forced to be dissimilar to the feature representation of the low-frequency proxy, as shown in Equation (3): ；(3) in, and The first Frequency decoupling comparison loss function for visible light and thermal infrared branches; The final output of this module is four sets of decoupled feature maps: ; Step 3: Use the training set to train the model Perform end-to-end training and compute the pixel-level scene semantic prediction map output by the model. The composite loss function between the truth label and the truth label Using the backpropagation algorithm and optimizer, based on The parameters of the network model are iteratively updated until the model converges; Step 4: After training is complete, transfer the trained model... The model is applied to the test set, predicts images in the test set, generates the final segmentation result, and compares it with the real labels to evaluate the model's generalization ability and final performance.

2. The visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling according to claim 1, characterized in that, In step 1, the visible light image and the thermal infrared image are spatially aligned using an image registration algorithm and uniformly scaled to a preset size to construct an original dataset of pixel-level annotations for the visible light-thermal infrared images. And divided into training sets and test set .

3. The visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling according to claim 1, characterized in that, The dual-branch encoder employs two parallel Transformer-based backbone networks, serving as encoders for the visible light branch and the thermal imaging branch, respectively. The encoder comprises four layers; for the input visible light image and thermal imaging image, the two encoders extract features layer by layer and output feature maps respectively. and , Dimension is represented as ( ),in For batch size, For the number of channels, For height, For width.

4. The visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling according to claim 1, characterized in that, The decoupled feature map input component-specific fusion module includes the following three stages: The general preprocessing stage includes the following sub-steps: Step 2.2.1: A regularization strategy driven by cosine similarity loss is adopted. First, layer normalization and instance normalization are used to extract modality-specific styles. Then, style differences are removed through feature transformation. Finally, the cosine similarity loss function is used to guide the network to learn modality-invariant features, as shown in Equation (4): ； (4) in, Represents the cosine similarity function. This indicates element-wise subtraction. Representation layer normalization, Indicates instance normalization, and Let these represent the cosine similarity loss functions for the content component and the detail component, respectively; minimized during training. and The network is guided to learn content and detail representations that maximize intermodal similarity; Step 2.2.2: Using the visible light component as a reference, calculate a transformation matrix from thermal imaging to the visible light semantic space, as shown in equation (5): ；(5) in, and These are the transformation matrices for the content components and the detail components, respectively; This is a flattening operation used to flatten a 3D feature map ( Flattened in space, it becomes a two-dimensional matrix. ); For min-max normalization; The transformation matrix is ​​then used to guide the alignment of thermal imaging components, resulting in aligned content features. and As shown in equation (6): ；(6) in, This is an inverse flattening operation used to flatten a two-dimensional matrix ( Reverse flattening into a three-dimensional feature map ( ); In the parallel component-specific enhancement and fusion stage, different strategies are used to enhance the content and detail components, specifically: The content component fusion path is as follows: a spatial noise suppression method is adopted, which smooths features through downsampling and upsampling operations to generate spatial attention weights. Used to enhance the features of aligned content. To obtain the final content fusion features As shown in equation (7): ；(7) in, This indicates a global average pooling layer. This indicates a doubling of upsampling. This represents the Sigmoid activation function. This indicates element-wise multiplication; The detail component fusion path is as follows: A hollow space pyramid pooling operation is used to obtain the detail fused features. As shown in equation (8): ；(8) in, This represents the pyramid pooling operation for empty spaces; In the adaptive fusion stage, firstly and The features are concatenated along the channel dimension, and then channel attention weight vectors are learned from the concatenated features through global average pooling and 1×1 convolutional layers. As shown in equation (9): ；(9) in, This indicates that channel concatenation is performed on the feature. This represents a standard 1×1 convolutional layer. Finally, using this weight The content features and detail features are weighted and summed to obtain the final fused features. As shown in equation (10): (10)。 5. The visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling according to claim 1, characterized in that, The decoder uses cascaded decoding units ( The resolution of the feature map is gradually restored from the bottom up, for each decoding unit. First, the output of the previous layer decoding unit... Adaptive fusion of features with the current level Element-wise addition is performed, followed by upsampling using transposed convolution and feature refinement using standard convolution, ultimately generating a full-size pixel-level scene semantic prediction map. ;when At that time, the output of the decoding unit in the previous layer does not exist.

6. The visible light-thermal infrared scene understanding method for intelligent vehicles based on explicit frequency decoupling according to claim 1, characterized in that, In step 4, the composite loss function Including scene semantic loss function Decoupling supervision loss function Cosine similarity loss function As shown in equations (11)-(13): ；(11) ；(12) ； (13) in, and Hyperparameters used to balance weights.

Images