Visible light and infrared image multi-modal fusion method and system based on transition state

Abstract

The invention relates to a visible light and infrared image multi-modal fusion method and system based on a transition state, and the method comprises the steps: constructing a feature extraction network, inputting an original visible light image and an original infrared image after training, and outputting corresponding low-frequency features and high-frequency features through an encoder; embedding an improved multi-modal fusion model between an encoder and a decoder of the feature extraction network to obtain an improved feature extraction network, acquiring transition state features by using low-frequency features and high-frequency features output by the encoder of the trained feature extraction network, and fusing the transition state features by using the decoder; and inputting a visible light image and an infrared image which are paired into the trained improved feature extraction network, and outputting a fused image. According to the method, the detail features of the extra visible light image are reserved, the robustness of the infrared image is exerted, the distinguishing capability of the model for different frequency features is enhanced, and the fusion quality and downstream task performance are improved; the transition state generation quality is improved, and the end-to-end trainability is kept; and the feature decoupling capability is improved.

Description

Technical Field

[0001] This invention relates to the technical field of image or video recognition or understanding, and in particular to a method and system for multimodal fusion of visible light and infrared images based on transition states. Background Technology

[0002] With the rapid development of intelligent driving technology, existing application scenarios are placing higher demands on environmental perception capabilities in traffic situations. Environmental perception is a key prerequisite for realizing core functions such as autonomous driving, traffic monitoring, and pedestrian protection. Statistics show that 80% of traffic accidents worldwide occur in low-light environments, such as at night, in tunnels, and at dusk. Pedestrian collisions account for more than 65% of fatal accidents, especially in rainy or foggy weather, where traditional visible light cameras have a failure rate of up to 40% in vehicle recognition. This demonstrates that a single sensor cannot reliably acquire comprehensive environmental information, severely impacting the accuracy and safety of system decisions.

[0003] Visible light image sensors, such as those used in automotive cameras, can capture rich details in color and texture, making them suitable for identifying features like traffic lights, lane lines, and vehicle brands. However, in scenarios such as nighttime, low-light conditions in tunnels, fog, or heavy rain, their image quality deteriorates drastically, easily losing crucial information like pedestrian outlines and obstacle edges. In contrast, infrared image sensors, such as infrared thermal imagers, detect the thermal radiation of objects themselves, are not limited by lighting conditions, and can clearly distinguish between living targets (such as pedestrians and animals) and non-living targets (such as vehicles and buildings) in darkness and inclement weather. This represents a core approach to overcoming the bottleneck of environmental perception. However, infrared images lack color and texture details, making it difficult to accurately identify information such as traffic signs and license plates.

[0004] Therefore, multimodal fusion technology of visible light and infrared images has become an important means to solve the bottleneck of environmental perception in traffic scenarios. By fusing complementary information from the two modalities, it is possible to retain the detailed features of visible light images while leveraging the robustness of infrared images, thereby providing more comprehensive environmental perception for applications such as path planning for autonomous driving, abnormal event detection in traffic monitoring, and pedestrian collision warning.

[0005] However, existing multimodal fusion methods still have the following limitations in traffic scenarios:

[0006] (1) Insufficient feature alignment: The modal differences between visible light and infrared images are large, such as imaging principles and pixel distribution. Traditional fusion methods such as weighted averaging and feature stitching are difficult to achieve accurate alignment of low-level features, resulting in blurring and artifacts in the fusion results, affecting the edge integrity of targets such as vehicles and pedestrians.

[0007] (2) Frequency domain feature coupling distortion: In traffic scenarios, low-frequency structural features such as road topology and vehicle overall shape and high-frequency detail features such as turn signal flashing, pedestrians, and gestures need to be accurately preserved at the same time. However, existing deep models, such as Transformer, use a single encoder to process dual modes, which easily leads to frequency domain feature coupling distortion.

[0008] (3) Insufficient enhancement of key targets: In traffic scenarios, pedestrians and vehicles are the key targets that need to be focused on. However, existing fusion methods often process global features equally and fail to enhance the significant features of the target area, including the thermal radiation features of pedestrians and the contour features of vehicles. This reduces the accuracy of subsequent downstream tasks, such as target detection.

[0009] (4) Lack of state continuity: Dynamic processes such as high-speed vehicle driving and pedestrians suddenly crossing require the perception system to capture the continuous evolution of the target state. However, existing technologies use discrete fusion strategies, such as dual-channel attention, which cannot model the continuous transition law from infrared feature dominance to visible light feature dominance.

[0010] To address the aforementioned issues, there is an urgent need for a fusion method that can accurately align visible light and infrared features, dynamically capture modal evolution patterns, and specifically enhance key traffic targets (such as vehicles and pedestrians). Summary of the Invention

[0011] To address the problems existing in the prior art, this invention provides a method and system for multimodal fusion of visible light and infrared images based on transition states.

[0012] The technical solution adopted in this invention is a multimodal fusion method for visible light and infrared images based on transition states. The method constructs a feature extraction network, including an encoder and a decoder. After training, the network is used to input the original visible light and infrared images, and the encoder outputs the corresponding low-frequency and high-frequency features.

[0013] An improved multimodal fusion model is embedded between the encoder and decoder of the feature extraction network to obtain an improved feature extraction network. The low-frequency and high-frequency features output by the encoder of the trained feature extraction network are used to obtain transition state features and then fused with the decoder.

[0014] Input pairs of visible light images and infrared images into the trained and improved feature extraction network, and output a fused image.

[0015] Preferably, the feature extraction model includes a shared feature encoder, a dual-branch independent encoder, and a decoder arranged sequentially;

[0016] The shared feature encoder extracts common shallow features from the original visible light image and infrared image;

[0017] The dual-branch independent encoder includes a basic feature encoder and a detail feature encoder corresponding to the visible light image and the infrared image, respectively. The two basic feature encoders are used to extract the corresponding low-frequency features based on the common shallow features of the visible light image and the infrared image, respectively. The two detail feature encoders are used to extract the corresponding high-frequency features based on the common shallow features of the visible light image and the infrared image, respectively.

[0018] The low-frequency and high-frequency features of the corresponding visible light image are stitched together, and the low-frequency and high-frequency features of the corresponding infrared image are stitched together. Both are then input into the decoder, and the reconstructed visible light and infrared images are output.

[0019] Preferably, the feature extraction model is trained based on the original visible light image and infrared image, and the reconstructed visible light image and infrared image, and the corresponding loss is calculated for each.

[0020] Preferably, the improved multimodal fusion model includes a convolution-attention fusion module, a linear transformation-group convolution module, a transition state feature fusion module, and a gated forward propagation module arranged sequentially. The convolution-attention fusion module is connected to the output of the dual-branch independent encoder of the trained feature extraction network, and concatenates the low-frequency features corresponding to the visible light image and the infrared image, and concatenates the high-frequency features corresponding to the visible light image and the infrared image, and inputs them into the convolution-attention fusion module.

[0021] The output of the gated forward propagation module is connected to the input of the decoder of the trained feature extraction network.

[0022] Preferably, the convolution-attention fusion module includes a local branch module and a global branch module;

[0023] The local branch module includes sequentially arranged convolutional layers and deformable convolutional layers;

[0024] The global branch module includes a first convolutional layer, a self-attention unit, and a second convolutional layer arranged sequentially. The Q, K, and V matrices of the self-attention unit are respectively provided with deformable convolutional layers between them and the first convolutional layer. The input terminals of the first convolutional layer and the second convolutional layer are connected in a skip connection.

[0025] The output of the deformable convolutional layer is fused with the output of the second convolutional layer before being output.

[0026] Preferably, the linear transformation-group convolution module includes a convolutional layer and a group convolutional layer. The output of the convolutional layer is split by the group convolutional layer to obtain K transition states, thereby obtaining two transition state image feature sets corresponding to visible light and infrared images, respectively.

[0027] Preferably, the transition state feature fusion module includes a linear reshaping segmentation unit, a multi-head attention unit, a linear layer, and a residual connection layer arranged sequentially.

[0028] The linear reshaping segmentation unit includes a convolutional layer, a reshaping layer, and a segmentation output head, which correspond to the transition state image feature sets of the visible light and infrared images, respectively. After operating on the transition state image feature sets of the infrared and visible light images of the two branches, the output is sent to the Q matrix of the multi-head attention unit. After operating on the transition state image feature sets of the visible light and infrared images of the two branches, the output is sent to the K and V matrices of the multi-head attention unit.

[0029] The output of the multi-head attention unit is passed through a linear layer and a residual connection layer to obtain K state outputs corresponding to the upper branch visible light and the lower branch infrared images;

[0030] By aggregating and splicing along the state dimension, the transition state image fusion coding features are obtained.

[0031] Preferably, the gated forward propagation module includes a linear expansion layer, followed by a depthwise separable convolutional layer and an activation function layer. The output of the activation function layer is multiplied by the linear expansion layer and then input into a linear compression layer. The output of the linear compression layer is fused with the original feature map through residual connections to output the enhanced transition state features.

[0032] Preferably, for each enhanced transition state, after fusing the preliminary features with a convolutional layer, the result is concatenated with the upsampled output of the previous decoder module along the channel dimension. The concatenated result is then mapped to the pixel space through a convolutional layer, activation function, and linear transformation layer to generate the final fused image.

[0033] A visible light and infrared image multimodal fusion system based on transition states, comprising:

[0034] At least one processor; and

[0035] A memory that is communicatively connected to at least one of the processors;

[0036] The memory stores instructions that can be executed by the processor to implement the transition state-based visible light and infrared image multimodal fusion method.

[0037] This invention relates to a method and system for multimodal fusion of visible light and infrared images based on transition states. A feature extraction network, including an encoder and a decoder, is constructed and trained to take as input the original visible light and infrared images. The encoder outputs corresponding low-frequency and high-frequency features. An improved multimodal fusion model is embedded between the encoder and decoder of the feature extraction network to obtain an improved feature extraction network. The low-frequency and high-frequency features output by the encoder of the trained feature extraction network are used to obtain transition state features, which are then fused using the decoder. Pairs of visible light and infrared images are input to the trained improved feature extraction network, and a fused image is output. The system is implemented using this method.

[0038] The beneficial effects of this invention are as follows:

[0039] (1) Multimodal fusion is performed using visible light image information and infrared image information. By using the complementary information of the two modes, the detailed features of the visible light image are preserved while the robustness of the infrared image is brought into play, thus solving the problem of incomplete information caused by relying on a single sensor to obtain environmental information in traditional technology.

[0040] (2) A two-stage training method is adopted. By optimizing the feature decomposition and fusion process step by step, the difficulty of unsupervised training in multimodal image fusion is solved, the model's ability to distinguish features of different frequencies is enhanced, and the fusion quality and downstream task performance are ultimately improved.

[0041] (3) Before fusion, a dual-path convolution-attention fusion module is used to preprocess the visible light image and the infrared image respectively. The long-range dependency relationship is modeled by adding the local and global branch outputs and capturing spatial detail features. This pre-feature enhancement method improves the quality of transition state generation while maintaining end-to-end trainability.

[0042] (4) The decomposition of the transition state can more finely represent the key information of the two source images, effectively reduce information loss, and better explore the relationship between and within the modality. This continuous decomposition method is more in line with the needs of complex image fusion tasks, avoids the limitation of traditional methods that regard the relationship as a binary system. The internal multi-head cross attention mechanism can effectively distinguish the specific information between different modalities and reduce the information redundancy caused by the decomposition of the transition state, improve the feature decoupling ability. The feedforward network adopts a dual-path gating mechanism to suppress the noise introduced by cross-state attention, while enhancing the cross-scale representation ability of the transition state, and obtain the enhanced transition state features. Attached Figure Description

[0043] Figure 1 This is a flowchart of the method of the present invention;

[0044] Figure 2 This is a schematic diagram of the convolution-attention fusion module structure of the present invention;

[0045] Figure 3 This is a schematic diagram of the transition state feature fusion module structure of the present invention;

[0046] Figure 4 This is a schematic diagram of the gated forward propagation module structure of the present invention.

[0047] Figure 5 This is a schematic diagram of an embodiment of the present invention based on a feature extraction network. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] This invention relates to a multimodal fusion method for visible light and infrared images based on transition states. The method constructs a feature extraction network, including an encoder and a decoder. After training, the network is used to input the original visible light and infrared images, and the encoder outputs the corresponding low-frequency and high-frequency features.

[0050] An improved multimodal fusion model is embedded between the encoder and decoder of the feature extraction network to obtain an improved feature extraction network. The low-frequency and high-frequency features output by the encoder of the trained feature extraction network are used to obtain transition state features and then fused with the decoder.

[0051] Input pairs of visible light images and infrared images into the trained and improved feature extraction network, and output a fused image.

[0052] This invention is achieved through a two-step process. The first step involves training a basic feature extraction network to ensure that the output visible light and infrared light images, after feature extraction and reconstruction based on the original visible light and infrared light images, meet preset conditions. The second step involves embedding an improved multimodal fusion model between the encoder and decoder of the basic feature extraction network. This model uses a transitional state feature fusion module to simulate the continuous change trajectory of the two modal features and combines an attention mechanism to focus on key target areas. This effectively improves the information integrity and target recognition of the fused image in complex traffic scenarios, providing technical support for the reliable operation of intelligent transportation systems.

[0053] The following explanation is based on the "two-step method".

[0054] (1) Original feature extraction network

[0055] The feature extraction model includes a shared feature encoder, a dual-branch independent encoder, and a decoder arranged in sequence;

[0056] The shared feature encoder extracts common shallow features from the original visible light image and infrared image. and ;

[0057] The dual-branch independent encoder includes a basic feature encoder and a detail feature encoder corresponding to visible light images and infrared images, respectively. The two basic feature encoders are used to identify common shallow features of visible light images and infrared images. and Extract the corresponding low-frequency features and Two detail encoders are used to extract common shallow features from visible light and infrared images, respectively. and Extract the corresponding high-frequency features and ;

[0058] The low-frequency features corresponding to visible light images and high frequency characteristics The images are stitched together to capture the low-frequency features of the corresponding infrared images. and high frequency characteristics The images are stitched together and input into the decoder, which outputs the reconstructed visible light and infrared images.

[0059] The feature extraction model is trained by calculating the corresponding losses based on the original visible light image, infrared image, and reconstructed visible light image and infrared image.

[0060] In this invention, the shared feature encoder is composed of a Restormer module, the basic feature encoder is an encoder based on a Lite Transformer block, and the detail feature encoder is a CNN encoder based on an INN block. For the original feature extraction network technology, please refer to the setup and training of the multimodal shared feature encoder, the improved dual-branch independent encoder, and the decoder in Chinese Patent No. CN120032216A, "A Multimodal Fusion Method for Feature Enhancement of SAR and Visible Light Images".

[0061] (2) Improved multimodal fusion model

[0062] The improved multimodal fusion model includes a convolution-attention fusion module, a linear transformation-group convolution module, a transition state feature fusion module, and a gated forward propagation module arranged sequentially. The convolution-attention fusion module is connected to the output of the dual-branch independent encoder of the trained feature extraction network, and concatenates the low-frequency features corresponding to the visible light image and the infrared image, and concatenates the high-frequency features corresponding to the visible light image and the infrared image, and inputs them into the convolution-attention fusion module.

[0063] The output of the gated forward propagation module is connected to the input of the decoder of the trained feature extraction network.

[0064] The following details the technical content and implementation methods of the above modules.

[0065] (2-1) Convolution-Attention Fusion Module

[0066] In this invention, for , , and After being concatenated according to frequency features, they are input into the convolution-attention fusion module.

[0067] The convolution-attention fusion module includes a local branch module and a global branch module;

[0068] The local branch module includes sequentially arranged convolutional layers (1x1Conv) and deformable convolutional layers (3x3x3Conv); the former is used to adjust the channel dimensions, the channel shuffling layer reorganizes the channels and generates new features, and the latter is used to extract local features;

[0069] The global branch module includes a first convolutional layer (1x1Conv), a self-attention unit, and a second convolutional layer arranged sequentially. A deformable convolutional layer (3x3DConv) is provided between the Q, K, and V matrices of the self-attention unit and the first convolutional layer. The inputs of the first and second convolutional layers are connected in a skip connection. The outputs of the deformable convolutional layers and the second convolutional layer are fused together before outputting the final output. The output of the global branch module is...

[0070]

[0071] in,

[0072]

[0073]

[0074] Y represents the image feature map stitched together according to image category. For channel shuffling, a depthwise separable convolution operation (3x3DConv) is used within each group. For convolution operations, This involves a 1×1 convolution followed by a 3×3 depthwise convolution operation. This is a self-attention mechanism.

[0075] (2-2) Linear Transformation - Grouped Convolution Module

[0076] The linear transformation-group convolution module includes a convolutional layer and a group convolutional layer. The output of the convolutional layer is split by the group convolutional layer to obtain K transition states, thereby obtaining two transition state image feature sets corresponding to visible light and infrared images, respectively.

[0077] Here, the convolutional layer of the linear transformation-grouped convolution module uses 1x1Conv, which is essentially a linear transformation used to adjust the number of channels and generate an initial transition state. The channels are then grouped into two parts by the grouped convolutional layer for feature extraction and splitting, respectively. At this time, the number of groups is K, that is, K transition states.

[0078] For example, suppose the feature map has 40 channels after linear processing. It is then split into K=4 transition states from infrared to visible light. At this time, the grouped convolutional layer splits it into 4 groups of channels with 10 channels in each group, and then processes the 4 groups of channels separately.

[0079] In this invention, images corresponding to visible light and infrared light are ultimately obtained respectively. and Two transition state image feature sets and To maintain the alignment of the generated visible light image and infrared image on the continuous trajectory state, k is a positive integer between 1 and K, and T is an identifier;

[0080] The transition state here refers to the gradual transition from visible light to infrared light, which is reorganized into two groups of states: Group A consists of visible light characteristics and K transition states, and Group B consists of K transition states and infrared light characteristics.

[0081] (2-3) Transition State Feature Fusion Module

[0082] The transition state feature fusion module mainly includes operations such as linear transformation, multi-head attention and cross attention calculation, and dimension splicing.

[0083] The transition state feature fusion module includes a linear reshaping segmentation unit, a multi-head attention unit, a linear layer, and a residual connection layer arranged sequentially.

[0084] The linear reshaping segmentation unit includes a convolutional layer, a reshaping layer, and a segmentation output head, which correspond to the transition state image feature sets of the visible light and infrared images, respectively. After operating on the transition state image feature sets of the infrared and visible light images of the two branches, the output is sent to the Q matrix of the multi-head attention unit. After operating on the transition state image feature sets of the visible light and infrared images of the two branches, the output is sent to the K and V matrices of the multi-head attention unit.

[0085] The output of the multi-head attention unit is passed through a linear layer and a residual connection layer to obtain K state outputs corresponding to the upper branch visible light and the lower branch infrared images;

[0086] By aggregating and splicing along the state dimension, the transition state image fusion coding features are obtained.

[0087] Regarding the upper branch, in group A... After performing a linear transformation (1x1D convolution), reshaping, and multi-head segmentation, the output Q is obtained. Each transition feature in the process undergoes a linear transformation (1x1D convolution), reshaping, and multi-head segmentation to output K and V. Matrix multiplication and softmax are performed on Q and K to extract the attention map. The attention map is then multiplied by V to obtain the output of each head. The outputs of each head are concatenated and reshaped (compressing all spatial and channel information into a single long feature vector) before outputting the final output. Then, a projection layer (linear projection, 1x1D convolution) is added to obtain a feature map S with the same shape as the original input transition state feature map. This feature map is then incorporated into S through residual connections to obtain the final K output states. ;

[0088] For the lower branch, in group B... Perform linear transformation (1x1 Conv convolution), reshape, and multi-head segmentation to obtain Q. Each transition feature in the process undergoes a linear transformation (1x1 Conv convolution), reshaping, and multi-head segmentation to output K and V. Then, a matrix multiplication and softmax function are performed on Q and K to extract the attention map. The attention map is then multiplied by V to obtain the output of each head. Finally, the outputs of each head are concatenated and reshaped to obtain the final output. Another projection layer (1x1D convolution) is input to obtain a feature map S with the same shape as the original input transition state feature map. This feature map is then incorporated into S through residual connections, resulting in the final K-state output. ;

[0089] Sum the outputs from the upper and lower branches The transition state image fusion encoding features are obtained by aggregating and stitching along the state dimension. .

[0090] (2-4) Gated Forward Propagation Module

[0091] The gated forward propagation module includes a linear expansion layer, followed by a depthwise separable convolutional layer and an activation function layer. The output of the activation function layer is multiplied by the linear expansion layer and then input into a linear compression layer. The output of the linear compression layer is fused with the original feature map through residual connections to output the enhanced transition state features.

[0092] In this invention, for each transition state, its feature channel number is first doubled using a 1x1 Conv convolution operation.

[0093]

[0094] The linear expansion operation expands the number of channels by a factor of η using a 1x1Dconv operation;

[0095] The expanded features are processed through two parallel branches—one directly retains the original features, and the other generates gated weights using a depthwise separable convolutional layer (3x3 Dconv) and the GELU activation function to better capture contextual information.

[0096]

[0097] Then, the features output from the two parallel branches are multiplied element-wise, and each input channel is independently convolved using a 3x3 spatial convolution to achieve dynamic feature selection.

[0098]

[0099] Finally, a linear compression layer (1x1Conv) is used to compress the number of channels back to the original dimension.

[0100]

[0101] Then, the output is obtained by residual concatenation with the original feature map. .

[0102] (2-5) Fusion

[0103] For each enhanced transition state, the initial features are fused using a convolutional layer (3x3Conv), and then concatenated with the upsampled output of the previous decoder module along the channel dimension. The concatenated result is mapped to the pixel space through a convolutional layer (two 3x3Convs), an activation function ReLU, and a linear transformation layer (1x1Conv) to generate the final fused image.

[0104] In fact, this describes the concatenation and decoding of transition states. The concatenated result is mapped to the pixel space through a convolutional layer (two 3x3 Convs), an activation function ReLU, and a linear transformation layer (1x1 Conv).

[0105] This invention also relates to a visible light and infrared image multimodal fusion system based on transition states, comprising:

[0106] At least one processor; and

[0107] A memory that is communicatively connected to at least one of the processors;

[0108] The memory stores instructions that can be executed by the processor to implement the transition state-based visible light and infrared image multimodal fusion method.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0114] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for multimodal fusion of visible light and infrared images based on transition states, characterized in that: The method constructs a feature extraction network, including an encoder and a decoder. After training, it is used as input to the original visible light image and infrared image, and the encoder outputs the corresponding low-frequency features and high-frequency features. An improved multimodal fusion model is embedded between the encoder and decoder of the feature extraction network to obtain an improved feature extraction network. The low-frequency and high-frequency features output by the encoder of the trained feature extraction network are used to obtain transition state features and then fused with the decoder. Input pairs of visible light images and infrared images into the trained and improved feature extraction network, and output a fused image.

2. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 1, characterized in that: The feature extraction model includes a shared feature encoder, a dual-branch independent encoder, and a decoder arranged in sequence; The shared feature encoder extracts common shallow features from the original visible light image and infrared image; The dual-branch independent encoder includes a basic feature encoder and a detail feature encoder corresponding to the visible light image and the infrared image, respectively. The two basic feature encoders are used to extract the corresponding low-frequency features based on the common shallow features of the visible light image and the infrared image, respectively. The two detail feature encoders are used to extract the corresponding high-frequency features based on the common shallow features of the visible light image and the infrared image, respectively. The low-frequency and high-frequency features of the corresponding visible light image are stitched together, and the low-frequency and high-frequency features of the corresponding infrared image are stitched together. Both are then input into the decoder, and the reconstructed visible light and infrared images are output.

3. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 2, characterized in that: The feature extraction model is trained by calculating the corresponding losses based on the original visible light image, infrared image, and reconstructed visible light image and infrared image.

4. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 2, characterized in that: The improved multimodal fusion model includes a convolution-attention fusion module, a linear transformation-group convolution module, a transition state feature fusion module, and a gated forward propagation module arranged sequentially. The convolution-attention fusion module is connected to the output of the dual-branch independent encoder of the trained feature extraction network, and concatenates the low-frequency features corresponding to the visible light image and the infrared image, and concatenates the high-frequency features corresponding to the visible light image and the infrared image, and inputs them into the convolution-attention fusion module. The output of the gated forward propagation module is connected to the input of the decoder of the trained feature extraction network.

5. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 4, characterized in that: The convolution-attention fusion module includes a local branch module and a global branch module; The local branch module includes sequentially arranged convolutional layers and deformable convolutional layers; The global branch module includes a first convolutional layer, a self-attention unit, and a second convolutional layer arranged sequentially. The Q, K, and V matrices of the self-attention unit are respectively provided with deformable convolutional layers between them and the first convolutional layer. The input terminals of the first convolutional layer and the second convolutional layer are connected in a skip connection. The output of the deformable convolutional layer is fused with the output of the second convolutional layer before being output.

6. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 4, characterized in that: The linear transformation-group convolution module includes a convolutional layer and a group convolutional layer. The output of the convolutional layer is split by the group convolutional layer to obtain K transition states, thereby obtaining two transition state image feature sets corresponding to visible light and infrared images, respectively.

7. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 6, characterized in that: The transition state feature fusion module includes a linear reshaping segmentation unit, a multi-head attention unit, a linear layer, and a residual connection layer arranged sequentially. The linear reshaping segmentation unit includes a convolutional layer, a reshaping layer, and a segmentation output head, which correspond to the transition state image feature sets of the visible light and infrared images, respectively. After operating on the transition state image feature sets of the infrared and visible light images of the two branches, the output is sent to the Q matrix of the multi-head attention unit. After operating on the transition state image feature sets of the visible light and infrared images of the two branches, the output is sent to the K and V matrices of the multi-head attention unit. The output of the multi-head attention unit is passed through a linear layer and a residual connection layer to obtain K state outputs corresponding to the upper branch visible light and the lower branch infrared images; By aggregating and splicing along the state dimension, the transition state image fusion coding features are obtained.

8. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 7, characterized in that: The gated forward propagation module includes a linear expansion layer, followed by a depthwise separable convolutional layer and an activation function layer. The output of the activation function layer is multiplied by the linear expansion layer and then input into a linear compression layer. The output of the linear compression layer is fused with the original feature map through residual connections to output the enhanced transition state features.

9. The method for multimodal fusion of visible light and infrared images based on transition states according to claim 8, characterized in that: For each enhanced transition state, the initial features are fused by a convolutional layer and then concatenated with the upsampled output of the previous decoder module along the channel dimension. The concatenated result is mapped to the pixel space through a convolutional layer, activation function, and linear transformation layer to generate the final fused image.

10. A multimodal fusion system for visible light and infrared images based on transition states, characterized in that: include: At least one processor; and A memory that is communicatively connected to at least one of the processors; The memory stores instructions executable by the processor to implement the visible light and infrared image multimodal fusion method based on transition states as described in any one of claims 1 to 9.

Images