Infrared and visible light video fusion method and system for low-illumination scene

Abstract

The application provides an infrared and visible light video fusion method and system for a low-illumination scene, and the method comprises the following steps: pre-processing infrared and visible light video sequences; inputting data into an encoding network for multi-scale coding to generate modal features and perception priors, performing channel enhancement, prior-guided interaction, feedforward network, residual connection and long-distance dependence modeling to obtain enhanced modal features of multi-scale coding; performing discrete wavelet transform on the enhanced modal features of multi-scale coding, fusing low-frequency and high-frequency subbands to obtain frequency domain fusion features, and then performing spatial modulation to obtain spatial domain-frequency domain collaborative fusion features; and in the decoding process, adjacent frame timing modeling is performed by using timing consistency, multi-scale features are fused by step-by-step upsampling, and a fusion video is generated. The application realizes regional-level adaptive cross-modal fusion through spatial domain-frequency domain collaborative fusion, and improves the utilization ability of complementary information of infrared and visible light.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for fusing infrared and visible light videos in low-light scenes. Background Technology

[0002] Infrared and low-illumination visible light fusion aims to leverage the complementary information between the two imaging modalities, fusing the stable target thermal response in the infrared modality with the rich texture details in the visible light modality to obtain a fusion result with strong scene perception capabilities. Under low-illumination conditions, visible light images are susceptible to illumination degradation and noise interference, leading to the loss of structural and texture information. Infrared images provide relatively stable thermal response information, but due to imaging mechanisms, they typically lack rich texture details and fine-grained structural representations. Therefore, effectively fusing the infrared thermal response with reliable structural information from low-illumination visible light images is a crucial challenge in infrared and visible light fusion.

[0003] With the development of deep learning technology, infrared and visible light fusion methods based on convolutional neural networks, Transformers, generative adversarial networks, state-space models, and diffusion models have made some progress. However, some existing methods typically use a uniform feature modeling approach to extract infrared and visible light features, failing to fully consider the differences between the two modalities in imaging mechanisms and information representation. The infrared modality mainly characterizes the thermal radiation information of the target, while the low-illumination visible light modality mainly contains structural and texture information affected by illumination degradation. Using the same feature extraction strategy makes it difficult to separately highlight the thermal response of the infrared target and the reliable structures in the low-illumination visible light image, easily leading to insufficient utilization of modality-specific information.

[0004] To improve the representational power of fused features, existing technologies have begun to utilize prior information to guide the extraction of infrared and visible light features. Depending on the type of prior used, related methods mainly include those based on low-level visual priors and those based on high-level semantic priors. Low-level visual priors typically utilize texture, content, gradient, or saliency information from the source image to guide the network to preserve structure and detail; high-level semantic priors utilize textual descriptions, scene semantics, or semantic information extracted from pre-trained models to guide the fusion network to focus on regions with task value. These methods demonstrate that prior information can improve fusion results to some extent, but existing priors usually focus on general visual information or scene semantic information, lacking modality-specific modeling for different imaging mechanisms. This makes it difficult to simultaneously describe brightness degradation, effective structure, and local texture in low-light visible light images, as well as thermal response, target boundaries, and salient regions in infrared images.

[0005] Besides modality-specific feature extraction, the complementarity of infrared and visible light information in the spatial and frequency domains is also a crucial factor affecting fusion quality. Some existing methods incorporate frequency domain information into the fusion process, enhancing salient targets and texture details through discrete wavelet transform, Fourier transform, or other frequency analysis techniques. Other methods further establish the interaction between spatial and frequency domain features, enabling frequency information to participate in spatial feature fusion. However, existing methods typically treat frequency domain features as an independent branch, limiting the interaction between the spatial and frequency domains. Some cross-domain fusion methods rely on pre-defined fixed fusion strategies, making it difficult to adapt to different scene content and imaging conditions. Since the information reliability of infrared and visible light modes may differ within the same spatial region, fixed stitching, addition, or weighting methods struggle to dynamically select more effective modal features based on the regional content.

[0006] Furthermore, most existing infrared and visible light fusion methods are geared towards static images. Real-world multimodal sensors typically acquire dynamic video continuously. Video fusion not only requires utilizing complementary information between different modalities but also maintaining spatiotemporal consistency between consecutive frames. Applying single-frame image fusion methods directly to video sequences easily overlooks the motion correspondence and temporal dependencies between adjacent frames, leading to inter-frame flickering and structural instability in the fused video. While existing video fusion methods improve inter-frame stability through multi-frame feature learning or temporal context aggregation, they still struggle to simultaneously consider the target saliency, texture details, and continuous changes between adjacent frames in the current frame.

[0007] Therefore, existing infrared and low-light visible light video fusion methods still struggle to extract reliable thermal response information and low-light structural information based on the imaging differences between the two modalities. The collaborative interaction between the spatial and frequency domains, as well as the dynamic modality selection capability in different regions, also need improvement. Furthermore, it is necessary to further suppress inter-frame flicker and structural inconsistencies in the continuous fusion results. How to achieve modality-specific feature enhancement, spatial-frequency domain collaborative fusion, and effective utilization of temporal information from adjacent frames while preserving salient infrared targets and visible light texture details are the technical problems that need to be solved in the field of infrared and low-light visible light video fusion. Summary of the Invention

[0008] In view of the above, the main objective of this invention is to propose a method and system for fusing infrared and visible light video in low-light scenes, so as to solve the above-mentioned technical problems.

[0009] This invention proposes a method for fusing infrared and visible light videos in low-light scenes, the method comprising the following steps: Step 1: Preprocess the acquired infrared video sequence and visible light video sequence to obtain infrared video input data and visible light video input data; Step 2: Input the infrared video input data and the visible light video input data into the coding network for multi-scale coding to generate enhanced modal features with multi-scale coding. Step 3: Generate low-frequency and high-frequency sub-bands by performing discrete wavelet transform on the enhanced modal features of multi-scale coding; obtain frequency domain fusion features by fusing the low-frequency and high-frequency sub-bands with each mode; spatially modulate the frequency domain fusion features and process them through a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features. Step 4: Based on the spatial domain-frequency domain collaborative fusion features, the temporal consistency of adjacent frames is used for temporal modeling during the decoding process, and upsampling is performed step by step to fuse multi-scale features and generate fused video.

[0010] This invention also proposes an infrared and visible light video fusion system for low-light scenes, the system comprising: The video sequence preprocessing module is used for: The acquired infrared video sequence and visible light video sequence are preprocessed to obtain infrared video input data and visible light video input data; The multi-scale feature encoding module is used for: Infrared video input data and visible light video input data are input into the coding network for multi-scale coding to generate enhanced modal features with multi-scale coding. The cross-modal feature fusion module is used for: Low-frequency and high-frequency subbands are generated by performing discrete wavelet transform on the enhanced modal features of multi-scale coding; based on the low-frequency and high-frequency subbands, frequency domain fusion features are obtained by mutual fusion between modes; spatial modulation is applied to the frequency domain fusion features and processed by a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features. The temporal modeling and video reconstruction module is used for: Based on the spatial domain-frequency domain collaborative fusion features, the temporal consistency of adjacent frames is used to model the temporal sequence during the decoding process, while upsampling is performed step by step to fuse multi-scale features and generate fused video.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention addresses the different imaging characteristics of infrared mode and low-light visible light mode by constructing thermal sensing priors and low-light sensing priors respectively. The sensing priors guide the Transformer to achieve dynamic interaction between the sensing priors and the corresponding modal features, which can enhance the characterization of infrared thermal targets and improve the ability to retain visible light texture and structural information under low-light conditions.

[0012] 2. This invention performs frequency decomposition of infrared and visible light features through spatial-frequency domain collaborative fusion, and fuses low-frequency structural information and high-frequency texture and target boundary information of the two modes respectively; further, it uses the frequency fusion result to guide spatial feature modulation, and combines a hybrid expert mechanism to dynamically select and aggregate spatial features, thereby realizing regional adaptive cross-modal fusion and improving the utilization capability of complementary information of infrared and visible light.

[0013] 3. This invention introduces the timing information of adjacent frames during the video decoding process to improve the timing consistency of the fusion result and help reduce inter-frame flicker and structural inconsistency in the fused video. Attached Figure Description

[0014] Figure 1 This is a flowchart of an infrared and visible light video fusion method for low-light scenes proposed in this invention; Figure 2 This is a network diagram illustrating an infrared and visible light video fusion method for low-light scenes proposed in this invention. Figure 3 This is a schematic diagram of the low-light perception prior extraction mechanism of the infrared and visible light video fusion method for low-light scenes proposed in this invention. Figure 4 This is a schematic diagram of the thermal sensing prior extraction mechanism of an infrared and visible light video fusion method for low-light scenes proposed in this invention. Figure 5 This is a schematic diagram of the spatial domain-frequency domain collaborative fusion mechanism of an infrared and visible light video fusion method for low-light scenes proposed in this invention. Figure 6 This is a schematic diagram of the framework of the infrared and visible light video fusion system for low-light scenes proposed in this invention. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0017] Please see Figure 1This invention proposes a method for fusing infrared and visible light videos in low-light scenes, which includes the following steps: Step 1: Preprocess the acquired infrared video sequence and visible light video sequence to obtain infrared video input data and visible light video input data.

[0018] Specifically, in step 1, the acquired infrared video sequence and visible light video sequence are preprocessed to obtain infrared video input data and visible light video input data. The specific steps are as follows: Acquire infrared video sequences and visible light video sequences; Select consecutive video frames that correspond in time from the infrared video sequence and the visible light video sequence respectively in chronological order to generate consecutive infrared video frames and consecutive visible light video frames. The continuous infrared video frames and continuous visible light video frames are subjected to size unification and numerical normalization processing, and then stacked in chronological order along the time dimension to obtain the infrared video input data and visible light video input data.

[0019] Furthermore, the visible light input data is three-channel data, the infrared video input data is single-channel data, and the model output is a three-channel color fused video.

[0020] Step 2: Input the infrared video input data and the visible light video input data into the coding network for multi-scale coding to generate enhanced modal features with multi-scale coding.

[0021] Please see Figure 2 Specifically, in step 2, the infrared video input data and the visible light video input data are input into the coding network for multi-scale coding to obtain the enhanced modal features of multi-scale coding. The specific steps are as follows: In the shallow coding scale of multi-scale coding, infrared video input data and visible light video input data are input into the coding network, and multi-scale brightness modeling is performed on the visible light features in the visible light video input data to obtain local brightness features and extended brightness features. Please see Figure 3 The importance weights are generated through a gating network; the local brightness features and extended brightness features are weighted and fused using the importance weights to obtain the brightness degradation prior; the gating network can be composed of convolution operations, global average pooling, multilayer perceptron and softmax function. The Sobel operator is used to extract gradient features from visible light features to obtain visible light gradient features; average pooling is performed on the visible light features to obtain low-frequency background features; and then the low-frequency background is subtracted from the visible light features to obtain local high-frequency texture features. Spatial adaptive weights are generated through a first attention gate; structural reconstruction representations are generated based on the spatial adaptive weights, visible light gradient features, and local high-frequency texture features; the first attention gate may consist of a convolutional layer, a SiLU activation function, and a Sigmoid function, used to generate spatial adaptive weights and dynamically weight and aggregate visible light gradient features and local high-frequency texture features. Low-light enhancement weights are generated based on brightness degradation priors. The low-light enhancement weight is used to modulate the structural recovery characterization to obtain the low-light enhancement prior; The Sobel operator is used to extract gradient features from the low-light enhancement prior to obtain the enhancement gradient features; Structure selection weights are generated through a second attention gate; structural consistency priors are generated based on the structure selection weights, enhanced gradient features, and visible light gradient features; the second attention gate has the same structure as the first attention gate. Adaptive weighting is applied to visible light features, low-light enhancement priors, and structural consistency priors, and then processed by the Sigmoid function to obtain local saliency priors; The low-light enhancement prior, structural consistency prior, and local saliency prior are concatenated along the channel dimension, and the concatenation result is sequentially subjected to channel rearrangement and convolution operations to obtain the low-light perception prior. Please see Figure 4 The Sobel operator is used to extract the infrared gradient features from the infrared video input data to obtain the infrared gradient features. Multi-scale structural modeling of infrared gradient features was performed using 3×3 depthwise separable convolution, 5×5 dilated convolution, and asymmetric strip convolution consisting of 1×7 and 7×1 convolutions, respectively, to obtain thermal structural features. A gating network is used to generate dynamic weights corresponding to each thermal structure feature; based on each thermal structure feature and its corresponding dynamic weights, a thermal structure prior is generated by weighted fusion. The mean and standard deviation of the infrared features are calculated. Based on the mean, standard deviation, and infrared features, the features are standardized and then processed by the Sigmoid function to obtain the initial thermal response characteristics of the infrared features. The initial thermal response features of infrared features are structurally modulated using thermal structure priors to obtain structurally refined thermal response features. Extract the local neighborhood of the center of the spatial location of the structurally refined thermal response features and calculate the similarity weight; based on the similarity weight and the structurally refined thermal response features, calculate the thermal response prior. Adaptive weighted aggregation of infrared features, thermal structure priors, and thermal response priors is performed, followed by processing with a Sigmoid function to obtain thermally significant priors. The thermal structure prior, thermal response prior, and thermal saliency prior are concatenated along the channel dimension, and the concatenation result is sequentially subjected to channel rearrangement and convolution operations to obtain the thermal perception prior. Visible light features or infrared features are sequentially processed by layer normalization, channel self-attention, and residual connection to obtain channel-enhanced features; Channel enhancement features are mapped to query features, and low-light perception priors or thermal perception priors are mapped to key features and value features. Based on query features, key features, and value features, feature interaction is performed through cross attention, and then prior guided interaction features are obtained through residual connections. The prior guided interaction features are processed sequentially through a normalization layer, a feedforward network, and a residual connection to obtain enhanced modal features at the shallow coding scale. In multi-scale coding, at the deep coding scale, the Transformer feature extraction mechanism is used to perform long-distance dependency modeling on the enhanced modal features of the shallow coding scale in order to obtain the enhanced modal features of the deep coding scale. The enhanced modal features at the shallow coding scale and the enhanced modal features at the deep coding scale are used together as the enhanced modal features for multi-scale coding.

[0022] Furthermore, multi-scale brightness modeling is performed on the visible light features in the visible light video input data to obtain local brightness features and extended brightness features. The corresponding process has the following relationship: ; in, Indicates the first Visible light features at the coding scale Indicates local brightness characteristics, Indicates extended brightness characteristics; This represents a multi-scale brightness modeling operation, which includes a 3×3 depthwise separable convolution branch and a 5×5 dilated convolution branch. The depthwise separable convolution branch in the multi-scale brightness modeling operation uses a 3×3 convolution kernel to extract local brightness change information; the dilated convolution branch uses a 5×5 convolution kernel to extract extended lighting context information. By utilizing importance weights, local brightness features and extended brightness features are weighted and fused to obtain the brightness degradation prior. The corresponding process has the following relationship: ; in, This indicates a priori luminance degradation. and All represent importance weights. This represents element-wise multiplication; The Sobel operator is used to extract gradient features from visible light features to obtain visible light gradient features; average pooling is then performed on the visible light features to obtain low-frequency background features; finally, the low-frequency background is subtracted from the visible light features to obtain local high-frequency texture features. The corresponding process has the following relationship: ; in, Represents the characteristics of visible light gradient. This indicates the Sobel gradient extraction operation. Represents local high-frequency texture features. This indicates an average pooling operation; Based on spatial adaptive weights, visible light gradient features, and local high-frequency texture features, a structural reconstruction representation is generated. The corresponding process has the following relationship: ; in, Indicates structural recovery characterization, Indicates spatial adaptive weights; Based on the prior knowledge of brightness degradation, low-light enhancement weights are generated, and the corresponding process has the following relationship: ; in, Indicates low-light enhancement weight, Indicates learnable parameters, Represents a constant used to maintain numerical stability; The structural recovery characterization is modulated using low-light enhancement weights to obtain low-light enhancement priors. The corresponding process has the following relationship: ; in, This indicates a priori enhancement of low light intensity; through the above modulation method, regions with insufficient illumination but clear structural responses can achieve strong enhancement, while dark regions lacking effective structure are enhanced due to... The response is weak, and it will not be amplified indiscriminately due to low brightness, thus avoiding the misinterpretation of dark area noise as texture details; The Sobel operator is used to extract gradient features from the low-light enhancement prior to obtain the enhancement gradient features. The corresponding process has the following relationship: ; in, This indicates enhanced gradient features; Structure selection weights are generated through a second attention gating process; structural consistency priors are generated based on the structure selection weights, enhanced gradient features, and visible light gradient features. The corresponding process has the following relationship: ; in, This indicates a priori structural consistency. Indicates the structural selection weights; Adaptively weighting visible light features, low-light enhancement priors, and structural consistency priors is performed, followed by processing with the Sigmoid function to obtain locally significant priors. The corresponding process has the following relationship: ; in, This indicates a priori significance of local significance. , and All represent learnable parameters, initialized to 1. Since the low-light enhancement prior, structural consistency prior, and local saliency prior have different feature distributions, directly concatenating them may limit the effective interaction between different prior representations. Therefore, channel rearrangement is used to promote the interaction of different prior information in the channel dimension, and subsequent convolution operations are used to unify the number of channels and feature distribution. The Sobel operator is used to extract the infrared gradient features from the infrared video input data to obtain the infrared gradient features. The corresponding process has the following relationship: ; in, Indicates infrared gradient characteristics, Indicates the first Infrared features at the coding scale; Based on each thermal structure feature and its corresponding dynamic weight, a priori thermal structure is generated through weighted fusion. The corresponding process has the following relationship: ; in, Indicates thermal structure prior, Indicates passage Thermal structure features obtained by multi-scale structure modeling using depthwise separable convolution. Indicates passage Thermal structure features obtained by multi-scale structure modeling using dilated convolution. Indicates passage Convolution and Thermal structure features obtained by multi-scale structural modeling using asymmetric strip convolutions. , and Both represent dynamic weights; Based on the mean and standard deviation of the infrared features, and after standardization, the infrared features are processed using the Sigmoid function to obtain the initial thermal response characteristics. The corresponding process has the following relationship: ; in, The initial thermal response characteristics representing infrared features. The standard deviation of infrared features The mean value of the infrared signature; By using thermal structure priors to modulate the initial thermal response features of infrared characteristics, refined thermal response features are obtained. The corresponding process has the following relationship: ; in, This indicates the thermal response characteristics of the refined structure; Based on similarity weights and structurally refined thermal response features, the thermal response prior is calculated, and the corresponding process has the following relationship: ; in, Indicates thermal response prior, Indicates the center position. This represents a local neighborhood of size k×k centered at a given location. Indicates the neighborhood location, The similarity weight is calculated based on the response differences between the center location and its neighboring locations. The input features represent local neighborhood aggregation; when the responses of the center position and the neighborhood position are similar, a larger similarity weight is assigned, indicating that they are more likely to belong to the same hot region; when the two are significantly different, a smaller similarity weight is assigned, thereby reducing the influence of pixels with inconsistent thermal responses during the enhancement process. An adaptive weighted aggregation of infrared features, thermal structure priors, and thermal response priors is performed, followed by processing with a Sigmoid function to obtain thermally significant priors. The corresponding process has the following relationship: ; in, This indicates a priori significance of thermal properties; , and All of these represent learnable parameters, initialized to 1. Visible light features or infrared features are sequentially processed by layer normalization, channel self-attention, and residual connection to obtain channel-enhanced features. The corresponding process has the following relationship: ; in, Indicates the first Channel enhancement features at the coding scale This indicates a channel self-attention operation. Presentation layer normalization operation, This represents the channel enhancement features of the infrared mode at the l-th coding scale. Indicates the first Channel enhancement features of visible light modes at the coding scale. Indicates the first Visible light or infrared features at the coding scale Represents a mode; Channel enhancement features are mapped to query features, and low-light perception priors or thermal perception priors are mapped to key features and value features. Based on query features, key features, and value features, feature interaction is performed through cross-attention, and then prior-guided interaction features are obtained through residual connections. The corresponding process has the following relationship: ; in, Indicates the first Prior-guided interaction features at the coding scale Indicates the first Prior guided interaction features of infrared modes at the coding scale. Indicates the first Prior-guided interaction features of visible light modalities at the coding scale. Represents perceptual priors. This indicates a priori knowledge of low-light perception. This indicates prior knowledge of heat perception. This indicates a cross-attention operation; The prior guided interaction features are processed sequentially through a normalization layer, a feedforward network, and a residual connection to obtain the enhanced modal features. The corresponding process has the following relationship: ; in, This represents the enhanced modal features at the shallow coding scale under the first coding scale. This represents the enhanced modal features at the shallow coding scale under the second coding scale. This represents the enhanced modal features at the shallow coding scale of the infrared modality at the first coding scale. This represents the enhanced modal features at the shallow coding scale of the infrared modality under the second coding scale. This represents the enhanced modal features at the shallow coding scale of the visible light modality at the first coding scale. This represents the enhanced modal features at the shallow coding scale of the visible light modality at the second coding scale. This indicates feedforward network processing.

[0023] Step 3: Generate low-frequency and high-frequency sub-bands by performing discrete wavelet transform on the enhanced modal features of multi-scale coding; obtain frequency domain fusion features by fusing the low-frequency and high-frequency sub-bands with each mode; spatially modulate the frequency domain fusion features and process them through a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features.

[0024] Please see Figure 5 Specifically, in step 3, low-frequency and high-frequency sub-bands are generated by performing discrete wavelet transform on the enhanced modal features of multi-scale coding; based on the low-frequency and high-frequency sub-bands, frequency domain fusion features are obtained through inter-modal fusion; the frequency domain fusion features are spatially modulated and processed through a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features. The specific steps are as follows: Discrete wavelet transform is performed on the enhanced modal features to obtain low-frequency subbands, horizontal high-frequency subbands, vertical high-frequency subbands, and diagonal high-frequency subbands; The horizontal high-frequency subband, vertical high-frequency subband, and diagonal high-frequency subband are spliced ​​along the channel dimension to obtain a unified high-frequency feature. The low-frequency subbands of the infrared mode and the visible light mode are fused to obtain low-frequency fusion features. Specifically, the low-frequency subbands of the infrared mode and the visible light mode are added element-wise, and the addition results are sequentially convolved and SiLU activated to obtain initial low-frequency joint features. Max pooling and average pooling are then performed on the initial low-frequency joint features, and the resulting max pooling and average pooling features are concatenated along the channel dimension. The concatenated results are sequentially convolved and passed through the Sigmoid function to obtain low-frequency fusion weights. The low-frequency subbands of the infrared mode and the visible light mode are adaptively fused according to the low-frequency fusion weights, and the fusion results are sequentially convolved and SiLU activated to obtain low-frequency fusion features. The unified high-frequency features of the infrared mode and the unified high-frequency features of the visible light mode are fused to obtain high-frequency fused features. Specifically, the unified high-frequency features of the infrared mode and the unified high-frequency features of the visible light mode are added element-wise, and the addition results are sequentially convolved and SiLU activated to obtain initial high-frequency joint features. The initial high-frequency joint features are then averaged to obtain high-frequency smooth features. The difference between the initial high-frequency joint features and the high-frequency smooth features is calculated, and the difference results are sequentially convolved and passed through the Sigmoid function to obtain a high-frequency spatial attention map. The high-frequency spatial attention map is then multiplied element-wise with the initial high-frequency joint features, and the multiplication results are sequentially convolved and SiLU activated to obtain high-frequency fused features. Based on low-frequency and high-frequency fusion features, frequency domain fusion features are obtained through inverse discrete wavelet transform. Enhanced modal features include enhanced infrared features and enhanced visible light features. At each coding scale, low-frequency fusion features and high-frequency fusion features are concatenated along the channel dimension, and then frequency-guided vectors are obtained through fully connected mapping and global average pooling. The frequency-guided vectors are used to modulate the enhanced modal features encoded at multiple scales to obtain frequency-guided infrared spatial features and frequency-guided visible light spatial features. Frequency-guided infrared spatial features and frequency-guided visible light spatial features are spliced ​​along the channel dimension to obtain joint spatial features; The response weights of each expert branch are generated through a hybrid expert mechanism; the joint spatial features are adaptively aggregated based on the response weights of the expert branches to obtain spatial domain fusion features; the hybrid expert mechanism includes a gating network and multiple expert branches; the gating network is used to generate the response weights of each expert branch according to the joint spatial features, and the K expert branches with the largest response weights are activated using a Top-K sparse selection strategy. After splicing the frequency domain fusion features and the spatial domain fusion features along the channel, a convolution operation is performed to obtain the spatial domain-frequency domain collaborative fusion features.

[0025] Furthermore, the enhanced modal features encoded by the multi-scale code are modulated using frequency-guided vectors to obtain frequency-guided infrared spatial features and frequency-guided visible light spatial features. The corresponding process has the following relationship: ; in, This indicates frequency-guided infrared spatial characteristics. This indicates the spatial characteristics of visible light guided by frequency. Represents the frequency-guided vector. This represents the enhanced infrared features at the deep coding scale under the third coding scale. This represents the enhanced infrared features at the deep coding scale under the fourth coding scale. This represents the enhanced visible light features at the deep coding scale under the third coding scale. This represents the enhanced visible light features at the deep coding scale under the fourth coding scale; Adaptive aggregation of joint spatial features is performed based on the response weights corresponding to expert branches to obtain spatial domain fusion features. The corresponding process has the following relationship: ; in, Indicates spatial domain fusion characteristics, This represents the set of indexes that indicate the activated expert branches. This represents the response weight corresponding to the i-th expert branch. This indicates that the process has been completed by the i-th expert branch. Indicates joint spatial characteristics; After concatenating the frequency domain fusion features and the spatial domain fusion features along the channels, and then performing a convolution operation, the spatial domain-frequency domain co-fusion features are obtained. The corresponding process has the following relationship: ; in, Indicates the first Spatial-frequency domain co-fusion features at the coding scale This represents the convolution operation. This indicates a splicing operation along the channel dimension.

[0026] Step 4: Based on the spatial domain-frequency domain collaborative fusion features, the temporal consistency of adjacent frames is used for temporal modeling during the decoding process, and upsampling is performed step by step to fuse multi-scale features and generate fused video.

[0027] Specifically, in step 4, based on the spatial-frequency domain collaborative fusion features, temporal consistency is used to perform temporal modeling of adjacent frames during the decoding process, while upsampling is performed step by step to fuse multi-scale features and generate a fused video. The specific steps are as follows: The correlation between the spatial domain-frequency domain co-fusion features of the current video frame and the spatial domain-frequency domain co-fusion features of the previous video frame is calculated using the first cross-attention branch to obtain the temporal information output by the first cross-attention branch. The correlation between the spatial-frequency domain co-fusion features of the current video frame and the spatial-frequency domain co-fusion features of the next video frame is calculated using the second cross-attention branch to obtain the temporal information output by the second cross-attention branch. The temporal information output by the first cross-attention branch is fused with the temporal information output by the second cross-attention branch to generate bidirectional collaborative attention information; Based on bidirectional collaborative attention information, the complementary features in the spatial-frequency domain collaborative fusion features of the previous video frame and the spatial-frequency domain collaborative fusion features of the next video frame are weighted and aggregated to obtain the temporal aggregation features of adjacent frames. The feedforward network is used to refine the temporal aggregation features of adjacent frames. The refined feature results are then residually fused with the spatial-frequency domain collaborative fusion features of the current video frame to obtain the temporal enhancement fusion features corresponding to the current video frame. By using skip connections, the temporal enhancement fusion features at each coding scale are upsampled stepwise and multi-scale feature reconstruction is performed to generate fused videos.

[0028] Furthermore, this invention trains the model by constructing a total loss function, wherein the total loss expression is: ; in, This represents the total loss value. Represents the structural similarity loss term; This represents the weight coefficients of the structural similarity loss term, and ; Represents the gradient loss term; Let represent the weight coefficients of the gradient loss term, and ; Indicates the color loss term; This represents the weighting coefficient of the color loss term, and ; Represents the time-series consistency loss term. This represents the weighting coefficient of the time-series consistency loss term, and ; The structural similarity loss is used to constrain the fused video to preserve the main structural information in both the infrared and visible light videos, and can be expressed as: ; in, Indicates the total number of frames in the video. Indicates the time index of a video frame. This represents a function for calculating structural similarity. This represents the visible light image of the x-th frame. This represents the fused image of frame x. This represents the infrared image of frame x. The gradient loss is used to constrain the fused video to retain gradient information from both source videos, and can be expressed as: ; in, This represents the Sobel gradient of the fused image in frame x. Represents the Sobel operator. This represents the Sobel gradient after adding the visible light image and the infrared image of the x-th frame. Represents the L1 norm; Color loss is used to constrain the chromaticity consistency between the fused video and the visible light video, and can be expressed as: ; in, This represents the extraction of Cb and Cr chromaticity components from the YCbCr color space. This represents the Cb and Cr chromaticity components of the fused image in frame x. Represents the Cb and Cr chromaticity components of the x-th visible light image; The temporal consistency loss simultaneously constrains the consistency of inter-frame variations between the fused video and the infrared video, as well as the consistency of gradient variations between adjacent frames of the fused video, and can be expressed as: ; in, This represents the temporal difference between the (x+1)th frame and the xth frame of the merged video. This represents the time difference between the (x+1)th frame and the xth frame of the infrared video. This represents the Sobel gradient of the fused image in frame x+1. This represents the Sobel gradient of the fused image in frame x.

[0029] It should be noted that the infrared and visible light video fusion model for low-light scenes proposed in this invention adopts a U-shaped encoder-decoder structure and sets four encoding scales, involving two modalities (infrared mode and visible light mode). As the scale increases, the spatial resolution of the features decreases stepwise while the channel dimension increases stepwise. At the shallow encoding scale (layers 1 and 2), the features mainly contain modality-specific edge, texture, and local structural information. Therefore, a perceptual prior-guided feature extraction mechanism is used to construct low-light perceptual priors and thermal perceptual priors respectively, and the perceptual priors are used to guide the Transformer to realize the dynamic interaction between each perceptual prior and the corresponding modal features, thereby obtaining the visible light and infrared features at the shallow encoding scale. At the deep encoding scale (layers 3 and 4), the features gradually transform into high-level semantics and global context information. Therefore, a Transformer feature extraction unit is used to perform long-distance dependency modeling to obtain the visible light and infrared features at the deep encoding scale, avoiding the continuous application of modal priors to deep features and thus limiting cross-modal semantic interaction. After obtaining infrared and visible light features at each coding scale, a spatial-frequency domain collaborative fusion mechanism is set at each coding scale to obtain multi-scale fused features. The spatial-frequency domain collaborative fusion mechanism first performs discrete wavelet transform on the two modal features and fuses low-frequency and high-frequency features respectively to obtain low-frequency fused features and high-frequency fused features. The low-frequency fused features and high-frequency fused features are then transformed by inverse discrete wavelet transform to obtain frequency domain fused features. Frequency guidance information is generated based on the low-frequency fused features and high-frequency fused features to modulate spatial features, and spatial domain fused features are obtained through a hybrid expert mechanism. Finally, the frequency domain fused features and spatial domain fused features are combined to obtain spatial-frequency domain collaborative fused features. During the decoding process, the spatial-frequency domain collaborative fusion features of each coding scale are used as input. The temporal consistency modeling mechanism is used to establish the temporal interaction between the current video frame and the adjacent video frames before and after it, and to aggregate the complementary temporal information in the adjacent video frames. Combined with skip connections, and through step-by-step upsampling and multi-scale feature reconstruction, the fused video is finally generated.

[0030] Please see Figure 6 This invention also provides an infrared and visible light video fusion system for low-light scenes, the system comprising: The video sequence preprocessing module is used for: The acquired infrared video sequence and visible light video sequence are preprocessed to obtain infrared video input data and visible light video input data; The multi-scale feature encoding module is used for: Infrared video input data and visible light video input data are input into the coding network for multi-scale coding to generate enhanced modal features with multi-scale coding. The cross-modal feature fusion module is used for: Low-frequency and high-frequency subbands are generated by performing discrete wavelet transform on the enhanced modal features of multi-scale coding; based on the low-frequency and high-frequency subbands, frequency domain fusion features are obtained by mutual fusion between modes; spatial modulation is applied to the frequency domain fusion features and processed by a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features. The temporal modeling and video reconstruction module is used for: Based on the spatial domain-frequency domain collaborative fusion features, the temporal consistency of adjacent frames is used to model the temporal sequence during the decoding process, while upsampling is performed step by step to fuse multi-scale features and generate fused video.

[0031] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0032] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0033] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for fusing infrared and visible light video in low-light scenes, characterized in that, The method includes the following steps: Step 1: Preprocess the acquired infrared video sequence and visible light video sequence to obtain infrared video input data and visible light video input data; Step 2: Input the infrared video input data and the visible light video input data into the coding network for multi-scale coding to generate enhanced modal features with multi-scale coding. Step 3: Generate low-frequency and high-frequency sub-bands by performing discrete wavelet transform on the enhanced modal features of multi-scale coding; obtain frequency domain fusion features by fusing the low-frequency and high-frequency sub-bands with each mode; spatially modulate the frequency domain fusion features and process them through a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features. Step 4: Based on the spatial domain-frequency domain collaborative fusion features, the temporal consistency of adjacent frames is used for temporal modeling during the decoding process, and upsampling is performed step by step to fuse multi-scale features and generate fused video.

2. The infrared and visible light video fusion method for low-light scenes according to claim 1, characterized in that, In step 1, the acquired infrared video sequence and visible light video sequence are preprocessed to obtain infrared video input data and visible light video input data. The specific steps are as follows: Acquire infrared video sequences and visible light video sequences; Select consecutive video frames that correspond in time from the infrared video sequence and the visible light video sequence respectively in chronological order to generate consecutive infrared video frames and consecutive visible light video frames. The continuous infrared video frames and continuous visible light video frames are subjected to size unification and numerical normalization processing, and then stacked in chronological order along the time dimension to obtain the infrared video input data and visible light video input data.

3. The infrared and visible light video fusion method for low-light scenes according to claim 2, characterized in that, In step 2, the infrared video input data and the visible light video input data are input into the coding network for multi-scale coding to obtain enhanced modal features of multi-scale coding. The specific steps are as follows: In the shallow coding scale of multi-scale coding, infrared video input data and visible light video input data are input into the coding network, and multi-scale brightness modeling is performed on the visible light features in the visible light video input data to obtain local brightness features and extended brightness features. The importance weights are generated by a gating network; the local brightness features and the extended brightness features are weighted and fused using the importance weights to obtain the brightness degradation prior. The Sobel operator is used to extract gradient features from visible light features to obtain visible light gradient features; average pooling is performed on the visible light features to obtain low-frequency background features; and then the low-frequency background is subtracted from the visible light features to obtain local high-frequency texture features. Spatial adaptive weights are generated through first attention gating; Based on spatial adaptive weights, visible light gradient features, and local high-frequency texture features, a structural reconstruction representation is generated. Low-light enhancement weights are generated based on brightness degradation priors. The low-light enhancement weight is used to modulate the structural recovery characterization to obtain the low-light enhancement prior; The Sobel operator is used to extract gradient features from the low-light enhancement prior to obtain the enhancement gradient features; Structure selection weights are generated through a second attention gating method; a structure consistency prior is generated based on the structure selection weights, enhanced gradient features, and visible light gradient features. Adaptive weighting is applied to visible light features, low light enhancement priors, and structural consistency priors, and then processed by the Sigmoid function to obtain local saliency priors; The low-light enhancement prior, structural consistency prior, and local saliency prior are concatenated along the channel dimension, and the concatenation result is sequentially subjected to channel rearrangement and convolution operations to obtain the low-light perception prior. The Sobel operator is used to extract the infrared gradient features from the infrared video input data to obtain the infrared gradient features. Multi-scale structural modeling of infrared gradient features was performed using 3×3 depthwise separable convolution, 5×5 dilated convolution, and asymmetric strip convolution consisting of 1×7 and 7×1 convolutions, respectively, to obtain thermal structural features. A gating network is used to generate dynamic weights corresponding to each thermal structure feature; based on each thermal structure feature and its corresponding dynamic weights, a thermal structure prior is generated through weighted fusion. The mean and standard deviation of the infrared features are calculated. Based on the mean, standard deviation, and infrared features, the features are standardized and then processed by the Sigmoid function to obtain the initial thermal response characteristics of the infrared features. The initial thermal response features of infrared features are structurally modulated using thermal structure priors to obtain structurally refined thermal response features. Extract the local neighborhood of the center of the spatial location of the refined thermal response features of the structure, and calculate the similarity weight; Based on similarity weights and structurally refined thermal response features, thermal response priors are obtained through calculation. Adaptive weighted aggregation of infrared features, thermal structure priors, and thermal response priors is performed, followed by processing with the Sigmoid function to obtain thermally significant priors; The thermal structure prior, thermal response prior, and thermal saliency prior are concatenated along the channel dimension, and the concatenation result is sequentially subjected to channel rearrangement and convolution operations to obtain the thermal perception prior. Visible light features or infrared features are sequentially processed by layer normalization, channel self-attention, and residual connection to obtain channel-enhanced features; Map channel enhancement features to query features, and map low-light perception priors or heat perception priors to key features and value features; Based on query features, key features, and value features, feature interaction is achieved through cross attention, and then prior guided interaction features are obtained through residual connections. The prior guided interaction features are processed sequentially through a normalization layer, a feedforward network, and a residual connection to obtain enhanced modal features at the shallow coding scale. In multi-scale coding, at the deep coding scale, the Transformer feature extraction mechanism is used to perform long-distance dependency modeling on the enhanced modal features of the shallow coding scale in order to obtain the enhanced modal features of the deep coding scale. The enhanced modal features at the shallow coding scale and the enhanced modal features at the deep coding scale are used together as the enhanced modal features for multi-scale coding.

4. The infrared and visible light video fusion method for low-light scenes according to claim 3, characterized in that, Multi-scale brightness modeling is performed on the visible light features in the visible light video input data to obtain local brightness features and extended brightness features. The corresponding process has the following relationship: ； in, Indicates the first Visible light features at the coding scale Indicates local brightness characteristics, Indicates extended brightness characteristics, This represents a multi-scale brightness modeling operation; By utilizing importance weights, local brightness features and extended brightness features are weighted and fused to obtain the brightness degradation prior. The corresponding process has the following relationship: ； in, This indicates a priori luminance degradation. and All represent importance weights. This represents element-wise multiplication; The Sobel operator is used to extract gradient features from visible light features to obtain visible light gradient features; average pooling is then performed on the visible light features to obtain low-frequency background features; finally, the low-frequency background is subtracted from the visible light features to obtain local high-frequency texture features. The corresponding process has the following relationship: ； in, Represents the visible light gradient characteristics. This indicates the Sobel gradient extraction operation. Represents local high-frequency texture features. This indicates an average pooling operation; Based on spatial adaptive weights, visible light gradient features, and local high-frequency texture features, a structural reconstruction representation is generated. The corresponding process has the following relationship: ； in, Indicates structural recovery characterization. Indicates spatial adaptive weights; Based on the prior knowledge of brightness degradation, low-light enhancement weights are generated, and the corresponding process has the following relationship: ； in, Indicates low-light enhancement weight, Indicates learnable parameters, Represents a constant used to maintain numerical stability; The structural recovery characterization is modulated using low-light enhancement weights to obtain low-light enhancement priors. The corresponding process has the following relationship: ； in, This indicates a priori enhancement in low light; The Sobel operator is used to extract gradient features from the low-light enhancement prior to obtain the enhancement gradient features. The corresponding process has the following relationship: ； in, This indicates enhanced gradient features; Structure selection weights are generated through a second attention gating process; structural consistency priors are generated based on the structure selection weights, enhanced gradient features, and visible light gradient features. The corresponding process has the following relationship: ； in, This indicates a priori structural consistency. Indicates the structural selection weights; Adaptively weighting visible light features, low-light enhancement priors, and structural consistency priors is performed, followed by processing with the Sigmoid function to obtain locally significant priors. The corresponding process has the following relationship: ； in, This indicates a locally significant prior. , and All of these represent learnable parameters.

5. The infrared and visible light video fusion method for low-light scenes according to claim 4, characterized in that, The Sobel operator is used to extract the infrared features from the infrared video input data to obtain the infrared gradient features. The corresponding process has the following relationship: ； in, Indicates infrared gradient characteristics, Indicates the first Infrared features at the coding scale; Based on each thermal structure feature and its corresponding dynamic weight, a priori thermal structure is generated through weighted fusion. The corresponding process has the following relationship: ； in, Indicates thermal structure prior, Indicates passage Thermal structure features obtained by multi-scale structure modeling using depthwise separable convolution. Indicates passage Thermal structure features obtained by multi-scale structure modeling using dilated convolution. Indicates passage Convolution and Thermal structure features obtained by multi-scale structural modeling using asymmetric strip convolutions. , and Both represent dynamic weights; Based on the mean and standard deviation of the infrared features, and after standardization, the infrared features are processed using the Sigmoid function to obtain the initial thermal response characteristics. The corresponding process has the following relationship: ； in, The initial thermal response characteristics representing infrared features. The standard deviation of infrared features The mean value of the infrared signature; By using thermal structure priors to modulate the initial thermal response features of infrared characteristics, refined thermal response features are obtained. The corresponding process has the following relationship: ； in, This indicates the thermal response characteristics of the refined structure; Based on similarity weights and structurally refined thermal response features, the thermal response prior is calculated, and the corresponding process has the following relationship: ； in, Indicates thermal response prior, Indicates the center position. This represents a local neighborhood of size k×k centered at a given location. Indicates the neighborhood location, The similarity weight is calculated to represent the response differences between the center location and its neighboring locations. Input features representing local neighborhood aggregation; An adaptive weighted aggregation of infrared features, thermal structure priors, and thermal response priors is performed, followed by processing with a Sigmoid function to obtain thermally significant priors. The corresponding process has the following relationship: ； in, Indicates the significance of thermal priors. , and All of these represent learnable parameters.

6. The infrared and visible light video fusion method for low-light scenes according to claim 5, characterized in that, Visible light features or infrared features are sequentially processed by layer normalization, channel self-attention, and residual connection to obtain channel-enhanced features. The corresponding process has the following relationship: ； in, Indicates the first Channel enhancement features at the coding scale This indicates a channel self-attention operation. Presentation layer normalization operation, This represents the channel enhancement features of the infrared mode at the l-th coding scale. Indicates the first Channel enhancement features of visible light modes at the coding scale. Indicates the first Visible light or infrared features at the coding scale Represents a mode; Channel enhancement features are mapped to query features, and low-light perception priors or thermal perception priors are mapped to key features and value features. Based on query features, key features, and value features, feature interaction is performed through cross-attention, and then prior-guided interaction features are obtained through residual connections. The corresponding process has the following relationship: ； in, Indicates the first Prior-guided interaction features at the coding scale Indicates the first Prior guided interaction features of infrared modes at the coding scale. Indicates the first Prior-guided interaction features of visible light modalities at the coding scale. Represents perceptual priors. This indicates a priori knowledge of low-light perception. This indicates prior knowledge of heat perception. This indicates a cross-attention operation; The prior-guided interaction features are processed sequentially through a normalization layer, a feedforward network, and a residual connection to obtain enhanced modal features at the shallow coding scale. The corresponding process has the following relationship: ； in, This represents the enhanced modal features at the shallow coding scale under the first coding scale. This represents the enhanced modal features at the shallow coding scale under the second coding scale. This represents the enhanced modal features at the shallow coding scale of the infrared modality at the first coding scale. This represents the enhanced modal features at the shallow coding scale of the infrared modality under the second coding scale. This represents the enhanced modal features at the shallow coding scale of the visible light modality at the first coding scale. This represents the enhanced modal features at the shallow coding scale of the visible light modality at the second coding scale. This indicates feedforward network processing.

7. The infrared and visible light video fusion method for low-light scenes according to claim 6, characterized in that, In step 3, low-frequency and high-frequency sub-bands are generated by performing discrete wavelet transform on the enhanced modal features encoded by multi-scale; based on the low-frequency and high-frequency sub-bands, frequency domain fusion features are obtained through inter-modal fusion; the frequency domain fusion features are spatially modulated and processed through a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features. The specific steps are as follows: Discrete wavelet transform is performed on the enhanced modal features of multi-scale coding to obtain low-frequency subbands, horizontal high-frequency subbands, vertical high-frequency subbands, and diagonal high-frequency subbands; The horizontal high-frequency subband, vertical high-frequency subband, and diagonal high-frequency subband are spliced ​​along the channel dimension to obtain a unified high-frequency feature. The low-frequency subband of the infrared mode is fused with the low-frequency subband of the visible light mode to obtain low-frequency fusion features; The unified high-frequency characteristics of the infrared mode and the unified high-frequency characteristics of the visible light mode are fused to obtain high-frequency fused characteristics; Based on low-frequency and high-frequency fusion features, frequency domain fusion features are obtained through inverse discrete wavelet transform. Enhanced modal characteristics include enhanced infrared characteristics and enhanced visible light characteristics; At each coding scale, the low-frequency fusion features and high-frequency fusion features are concatenated along the channel dimension, and then the frequency-guided vector is obtained through fully connected mapping and global average pooling. The frequency-guided vector is used to modulate the enhanced modal features of multi-scale coding to obtain frequency-guided infrared spatial features and frequency-guided visible light spatial features. Frequency-guided infrared spatial features and frequency-guided visible light spatial features are spliced ​​along the channel dimension to obtain joint spatial features; The response weights of each expert branch are generated through a hybrid expert mechanism; the joint spatial features are adaptively aggregated based on the response weights of the expert branches to obtain spatial domain fusion features. After splicing the frequency domain fusion features and the spatial domain fusion features along the channel, a convolution operation is performed to obtain the spatial domain-frequency domain collaborative fusion features.

8. The infrared and visible light video fusion method for low-light scenes according to claim 7, characterized in that, The enhanced modal features encoded by multi-scale codes are modulated using frequency-guided vectors to obtain frequency-guided infrared spatial features and frequency-guided visible light spatial features. The corresponding process has the following relationship: ； in, This indicates frequency-guided infrared spatial characteristics. This indicates the spatial characteristics of visible light guided by frequency. Represents the frequency-guided vector. This represents the enhanced infrared features at the deep coding scale under the third coding scale. This represents the enhanced infrared features at the deep coding scale under the fourth coding scale. This represents the enhanced visible light features at the deep coding scale under the third coding scale. This represents the enhanced visible light features at the deep coding scale under the fourth coding scale; Adaptive aggregation of joint spatial features is performed based on the response weights corresponding to expert branches to obtain spatial domain fusion features. The corresponding process has the following relationship: ； in, Indicates spatial domain fusion characteristics, This represents the set of indexes that indicate the activated expert branches. This represents the response weight corresponding to the i-th expert branch. This indicates that the process has been completed by the i-th expert branch. Indicates joint spatial characteristics; After concatenating the frequency domain fusion features and the spatial domain fusion features along the channels, and then performing a convolution operation, the spatial domain-frequency domain co-fusion features are obtained. The corresponding process has the following relationship: ； in, Indicates the first Spatial-frequency domain co-fusion features at the coding scale This represents the convolution operation. This indicates a splicing operation along the channel dimension.

9. The infrared and visible light video fusion method for low-light scenes according to claim 8, characterized in that, In step 4, based on the spatial-frequency domain collaborative fusion features, temporal consistency is used to model the temporal sequence of adjacent frames during the decoding process, and upsampling is performed step by step to fuse multi-scale features and generate a fused video. The specific steps are as follows: The correlation between the spatial domain-frequency domain co-fusion features of the current video frame and the spatial domain-frequency domain co-fusion features of the previous video frame is calculated using the first cross-attention branch to obtain the temporal information output by the first cross-attention branch. The correlation between the spatial-frequency domain co-fusion features of the current video frame and the spatial-frequency domain co-fusion features of the next video frame is calculated using the second cross-attention branch to obtain the temporal information output by the second cross-attention branch. The temporal information output by the first cross-attention branch is fused with the temporal information output by the second cross-attention branch to generate bidirectional collaborative attention information; Based on bidirectional collaborative attention information, the complementary features in the spatial-frequency domain collaborative fusion features of the previous video frame and the spatial-frequency domain collaborative fusion features of the next video frame are weighted and aggregated to obtain the temporal aggregation features of adjacent frames. The feedforward network is used to refine the temporal aggregation features of adjacent frames. The refined feature results are then residually fused with the spatial-frequency domain collaborative fusion features of the current video frame to obtain the temporal enhancement fusion features corresponding to the current video frame. By using skip connections, the temporal enhancement fusion features at each coding scale are upsampled stepwise and multi-scale feature reconstruction is performed to generate fused videos.

10. An infrared and visible light video fusion system for low-light scenes, characterized in that, The system employs the infrared and visible light video fusion method for low-light scenes according to any one of claims 1 to 9, and the system comprises: The video sequence preprocessing module is used for: The acquired infrared video sequence and visible light video sequence are preprocessed to obtain infrared video input data and visible light video input data; The multi-scale feature encoding module is used for: Infrared video input data and visible light video input data are input into the coding network for multi-scale coding to generate enhanced modal features with multi-scale coding. The cross-modal feature fusion module is used for: Low-frequency and high-frequency subbands are generated by performing discrete wavelet transform on the enhanced modal features of multi-scale coding; based on the low-frequency and high-frequency subbands, frequency domain fusion features are obtained by mutual fusion between modes; spatial modulation is applied to the frequency domain fusion features and processed by a hybrid expert mechanism to obtain spatial-frequency domain collaborative fusion features. The temporal modeling and video reconstruction module is used for: Based on the spatial domain-frequency domain collaborative fusion features, the temporal consistency of adjacent frames is used to model the temporal sequence during the decoding process, while upsampling is performed step by step to fuse multi-scale features and generate fused video.

Images