Multimodal video fusion method and device based on spatio-temporal hierarchical attention network

Abstract

The application discloses a multi-modal video fusion method and device based on a space-time hierarchical attention network, and relates to the technical field of computer vision. The method comprises the following steps: based on an inter-frame consistency loss function and a bidirectional consistency loss function, a two-stage training is performed on a video fusion model according to a first training data set and a second training data set; an infrared video to be fused and a visible light video to be fused are acquired; based on a frame-by-frame sliding mode, three frames of the infrared video to be fused and the visible light video to be fused are continuously and synchronously sliced to obtain a first infrared segment set and a first visible light segment set; three-dimensional position coding is added to the first infrared segment set and the first visible light segment set; and multi-modal video fusion is performed on the second infrared segment set and the second visible light segment set by using an optimized video fusion model to obtain a fused video. The application is an efficient and coherent multi-modal video fusion method based on space-time hierarchical attention.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multimodal video fusion method and apparatus based on a spatiotemporal hierarchical attention network. Background Technology

[0002] Multimodal video fusion technology, especially the fusion of infrared and visible light video, has significant application value in key areas such as autonomous driving, security monitoring, and military reconnaissance. Its core objective is to combine the advantages of different sensors: visible light sensors provide rich texture details, while infrared sensors can penetrate smoke and identify people and objects at night, ensuring the system's perception reliability and stability in all weather conditions and scenarios.

[0003] The development of video fusion technology has undergone a paradigm shift from traditional methods to deep learning. Early methods were mainly based on classical image processing theories, such as wavelet transform, Laplacian pyramid, and sparse representation, relying on manually designed feature extraction and fusion rules. While these methods are effective under specific conditions, they generally suffer from limitations such as poor adaptability, susceptibility to artifacts, and high computational complexity, making it difficult to cope with complex and ever-changing real-world scenarios.

[0004] Current advanced still image fusion techniques, especially those based on visual transformer architectures, possess a core advantage in their powerful spatial feature representation capabilities. These architectures typically employ an encoder-decoder design, where the encoder progressively extracts multi-scale features from coarse to fine through a hierarchical approach. To capture global information while maintaining computational efficiency, a self-attention mechanism based on local windows is designed. This mechanism first divides the feature map into multiple non-overlapping local regions, computes self-attention in parallel within each window, and achieves cross-window information interaction through specific mechanisms (such as window shifting or feature propagation), approximating the global receptive field with linear computational complexity. At the feature fusion level, these methods often employ sophisticated fusion strategies, such as evaluating the activity level or saliency of features by calculating a certain norm, and adaptively assigning fusion weights based on this level to ensure that key information in the source image is preserved to the greatest extent possible.

[0005] However, the aforementioned techniques, from feature extraction to attention calculation, are strictly limited to the two-dimensional space, namely the height and width of the image, lacking consideration for time. This "temporal blind spot" directly leads to serious problems. On the one hand, because the feature extraction process is performed independently frame by frame, the model cannot utilize the valuable motion and change information contained in adjacent frames, resulting in the loss of temporal information and rendering the model's ability to perceive dynamic changes in the scene almost zero. On the other hand, due to the lack of cross-frame consistency constraints, the fusion result of each frame is optimized independently, leading to random and inconsistent differences between adjacent frames, causing visual quality degradation. During video playback, these differences manifest as unstable jumps in overall brightness or local details, i.e., "flickering," while the edges of moving objects may also exhibit jitter or artifacts, severely damaging the smoothness and realism of the video.

[0006] Besides the limitations of the mainstream image fusion technologies mentioned above, other approaches exploring video fusion also have significant shortcomings. For example, while methods based on three-dimensional convolutional neural networks (3D CNNs) can handle spatiotemporal data, their enormous number of parameters and computational complexity make them difficult to deploy in practice. On the other hand, structures relying on optical flow or recurrent neural networks (RNNs) are limited by problems such as motion estimation errors and difficulties in learning long-range dependencies, respectively.

[0007] In the existing technology, there is a lack of an efficient and coherent multimodal video fusion method based on spatiotemporal hierarchical attention. Summary of the Invention

[0008] To address the problem that existing technologies simply treat video as a series of independent images and process it frame by frame, completely ignoring the inherent temporal continuity and dynamic correlation of video data, this invention provides a multimodal video fusion method and apparatus based on a spatiotemporal hierarchical attention network. The technical solution is as follows:

[0009] On the one hand, a multimodal video fusion method based on a spatiotemporal hierarchical attention network is provided. This method is implemented by a multimodal video fusion device and includes:

[0010] The first training dataset was constructed based on the publicly available REDS and DAVIS datasets; the second training dataset was constructed based on the publicly available TNO, RoadScene, and MSRS datasets.

[0011] A video fusion model is constructed based on the network structure of the spatiotemporal hierarchical attention network and the model structure of the Transformer model;

[0012] Based on the inter-frame consistency loss function and the bi-directional consistency loss function, the video fusion model is trained in two stages according to the first training dataset and the second training dataset to obtain an optimized video fusion model.

[0013] Acquire the infrared video and the visible light video to be fused; based on the frame-by-frame sliding method, perform three consecutive frames of synchronous slicing on the infrared video and the visible light video to be fused to obtain the first infrared segment set and the first visible light segment set.

[0014] Add three-dimensional position encoding to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set;

[0015] Based on the second infrared and second visible light fragment sets, an optimized video fusion model is used to perform multimodal video fusion to obtain a fused video.

[0016] On the other hand, a multimodal video fusion device based on a spatiotemporal hierarchical attention network is provided. This device is applied to a multimodal video fusion method based on a spatiotemporal hierarchical attention network. The device includes:

[0017] The training data construction module is used to construct the first training dataset based on the publicly available REDS and DAVIS datasets; and to construct the second training dataset based on the publicly available TNO, RoadScene, and MSRS datasets.

[0018] The video fusion model building module is used to build a video fusion model based on the network structure of the spatiotemporal hierarchical attention network and the model structure of the Transformer model.

[0019] The video fusion model training module is used to perform two-stage training on the video fusion model based on the inter-frame consistency loss function and the bidirectional consistency loss function, according to the first training dataset and the second training dataset, to obtain an optimized video fusion model.

[0020] The data processing module to be fused is used to acquire the infrared video to be fused and the visible light video to be fused; based on the frame-by-frame sliding method, the infrared video to be fused and the visible light video to be fused are synchronously sliced ​​for three consecutive frames to obtain the first infrared segment set and the first visible light segment set.

[0021] A three-dimensional position encoding module is used to add three-dimensional position encoding to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set.

[0022] The video fusion module is used to perform multimodal video fusion using an optimized video fusion model based on the second infrared segment set and the second visible light segment set to obtain a fused video.

[0023] On the other hand, a multimodal video fusion device is provided, the multimodal video fusion device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the methods described above for multimodal video fusion based on a spatiotemporal hierarchical attention network is implemented.

[0024] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described multimodal video fusion methods based on spatiotemporal hierarchical attention networks.

[0025] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0026] This invention proposes a multimodal video fusion method based on a spatiotemporal hierarchical attention network. The method takes the target frame to be fused and its two adjacent frames as a processing unit input. It utilizes the temporal context information of adjacent frames to enhance the feature representation of the target frame, realizes cross-modal feature fusion, and reconstructs and outputs a high-quality fused image.

[0027] By calculating the relative displacement of any two image patches in the three dimensions of time, height, and width, a three-dimensional relative position vector is constructed, and this vector is injected element by element into the "key" and "value" vectors of the self-attention mechanism, enabling the model to obtain accurate spatiotemporal perception capabilities.

[0028] The training process is decomposed into two stages: pre-training and joint training. In the pre-training stage, the feature fusion module is trained independently. The goal of this stage is to enable the module to learn to generate temporally smooth, flicker-free outputs directly from the raw video data. Its core optimization objective is an inter-frame consistency loss function. In the joint training stage, the model is jointly trained end-to-end to optimize the final fusion performance. This invention is an efficient and coherent multimodal video fusion method based on spatiotemporal hierarchical attention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1This is a flowchart of a multimodal video fusion method based on a spatiotemporal hierarchical attention network provided by an embodiment of the present invention;

[0031] Figure 2 This is a block diagram of a multimodal video fusion device based on a spatiotemporal hierarchical attention network provided in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the structure of a multimodal video fusion device provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0034] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0035] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0036] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0037] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0038] This invention provides a multimodal video fusion method based on a spatiotemporal hierarchical attention network. This method can be implemented by a multimodal video fusion device, which can be a terminal or a server. Figure 1 The flowchart shown is for a multimodal video fusion method based on a spatiotemporal hierarchical attention network. The processing flow of this method may include the following steps:

[0039] S1. Construct the first training dataset based on the publicly available REDS and DAVIS datasets; construct the second training dataset based on the publicly available TNO, RoadScene, and MSRS datasets.

[0040] In one feasible implementation, the first training dataset in this invention includes a video restoration dataset (REalistic and Dynamic Scenes dataset, REDS) for tasks such as video super-resolution / deblurring and a dataset (Densely Annotated VIdeo Segmentation dataset, DAVIS) for video object segmentation.

[0041] The second training dataset includes the TNO Multiband ImageData Collection (TNO), the RoadScene: A New Dataset of Aligned Infrared and Visible Images (RoadScene), and the Multi-Spectral Road Scenarios for Practical Infrared and Visible ImageFusion (MSRS).

[0042] S2. Construct a video fusion model based on the network structure of the spatiotemporal hierarchical attention network and the model structure of the Transformer model;

[0043] The video fusion model includes a feature extraction module, a feature fusion module, and a video reconstruction module.

[0044] The feature extraction module is a global feature extraction network composed of multiple cascaded spatiotemporal hierarchical attention networks.

[0045] In one feasible implementation, the feature extraction module is a global feature extraction network composed of multiple cascaded spatiotemporal hierarchical attention modules, which extracts high-level feature representations containing long-range spatiotemporal dependencies through parallel processing.

[0046] In the feature fusion module, temporal and cross-modal feature fusion is performed. The model first aggregates the feature information of the auxiliary frame into the main frame through temporal fusion to generate an enhanced single-frame feature sum, and then performs cross-modal fusion to generate a single information-rich fused feature.

[0047] In the video reconstruction module, the fused feature is smoothly projected back from the feature space to the pixel space through inverse reshaping and a series of upsampling operations, ultimately reconstructing a single frame of high-quality fused video image.

[0048] S3. Based on the inter-frame consistency loss function and the bi-directional consistency loss function, the video fusion model is trained in two stages according to the first training dataset and the second training dataset to obtain the optimized video fusion model.

[0049] Optionally, based on the inter-frame consistency loss function and the bi-directional consistency loss function, the video fusion model is trained in two stages using the first training dataset and the second training dataset to obtain an optimized video fusion model, including:

[0050] Based on the first training dataset, the feature fusion module is used to perform multimodal feature fusion to obtain a fused feature dataset;

[0051] The inter-frame consistency loss is calculated based on the first training dataset and the fused feature dataset, using the inter-frame consistency loss function.

[0052] Based on the inter-frame consistency loss, the parameters of the feature fusion module are optimized to obtain the first optimized feature fusion module;

[0053] Based on the second training dataset, video reconstruction is performed using the feature extraction module, the first optimized feature fusion module, and the video reconstruction module to obtain the reconstructed video dataset.

[0054] Based on the bidirectional consistency loss function, the bidirectional consistency loss is calculated using the second training dataset and the reconstructed video dataset.

[0055] Based on the bidirectional consistency loss, the parameters of the feature extraction module, the first optimized feature fusion module, and the video reconstruction module are optimized to obtain the optimized feature extraction module, the second optimized feature fusion module, and the optimized video reconstruction module.

[0056] An optimized video fusion model is obtained based on the optimized feature extraction module, the second optimized feature fusion module, and the optimized video reconstruction module.

[0057] In one feasible implementation, to address the flickering and jitter problems commonly found in existing fusion methods when processing video, this invention introduces an inter-frame consistency loss function, denoted as... .

[0058] Traditional image fusion loss functions typically optimize only the quality of a single frame, completely ignoring the temporal continuity of the video. This frame-by-frame optimization approach can easily lead to small but inconsistent differences between adjacent output frames, which visually manifest as flickering artifacts that disrupt the viewing experience.

[0059] This invention addresses this deficiency through a dedicated pre-training phase. The training objective of this phase is to enable the module to learn to generate temporally coherent results directly from the raw video data. The loss function is shown in equation (1) below.

[0060] (1);

[0061] Where N is the total number of frames in the video sequence; t is the index of the current time frame; This is the timing balance constraint function; The fusion features are defined at time frame index t; the optimization objective at this stage is the inter-frame consistency loss function. It explicitly constrains temporal smoothness by calculating and minimizing the feature differences between two adjacent frames generated by the fusion network. This difference is calculated by passing both frames through a temporal smoothness constraint function. Quantization is performed using the mapped Euclidean distance.

[0062] After pre-training, the first optimized feature fusion module, which possesses prior knowledge of temporal consistency, will be integrated into the complete network architecture, which includes spatiotemporal feature extraction, for the second stage of end-to-end joint training. This strategy separates the learning of temporal stability from the complex spatiotemporal feature fusion, enabling the network to fundamentally generate flicker-free, smooth videos.

[0063] During the joint training process, optimization is performed on both the image and feature aspects, using a bidirectional consistency loss function, which is shown in equations (2) and (3) below:

[0064] (2);

[0065] (3);

[0066] Among them, F i To reconstruct the frame images of the video; S 1i Infrared source image; S 2i Image of a visible light source; F feat,j To reconstruct the frame image features of the video; S 1feat,j Features of infrared source images; S 2feat,j Features of visible light source images.

[0067] During joint training, for low-level features, attention-weighted fusion is used to preserve detailed information and avoid losing high-frequency information such as texture and edges. For high-level features, semantic consistency is emphasized, and cross-attention fusion is used. Mutual attention of deep image features is calculated through an additional Transformer layer to capture global dependencies. For the fused features at each level, skip connections are used to ensure that the fusion result contains both detailed and semantic information.

[0068] S4. Obtain the infrared video to be fused and the visible light video to be fused; Based on the frame-by-frame sliding method, perform three consecutive frames of synchronous slicing on the infrared video to be fused and the visible light video to be fused to obtain the first infrared segment set and the first visible light segment set.

[0069] In one feasible implementation, the present invention employs a "three-in-one-out" video processing paradigm, using time frames from the registered infrared and visible light video streams. Three consecutive frames centered on As a two-stream parallel input, where the first... The first frame is the main frame to be fused, while the two frames before and after it serve as auxiliary information for the temporal context.

[0070] S5. Add three-dimensional position encoding to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set;

[0071] Optionally, three-dimensional position encoding is added to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set, including:

[0072] The three-dimensional relative displacement vectors between image blocks are calculated based on the first infrared fragment set and the first visible light fragment set to obtain the infrared relative displacement set and the visible light relative displacement set.

[0073] Based on a preset learnable scalar, the infrared relative displacement set and the visible light relative displacement set are scaled in time dimension to obtain the scaled infrared relative displacement set and the scaled visible light relative displacement set.

[0074] Based on the preset first trainable projection matrix, the first infrared segment set is injected with three-dimensional position encoding according to the scaled infrared relative displacement set to obtain the second infrared segment set;

[0075] Based on the preset second trainable projection matrix, the first visible light fragment set is injected with three-dimensional position encoding according to the scaled visible light relative displacement set to obtain the second visible light fragment set.

[0076] In one feasible implementation, for three-dimensional (time, height, width) data such as video sequences, the model must be able to understand the relative spatiotemporal relationships between image blocks in order to effectively model the motion of objects and the dynamic changes of the scene. This invention injects relative positional information as a learnable bias into the self-attention module, enabling the model to acquire spatiotemporal awareness.

[0077] To quantify the spatiotemporal relationship between any two image patches, their spatiotemporal coordinates are defined. For any image patch... And another image block Their positions can be determined by a three-dimensional coordinate system. To uniquely determine, among which This represents the time frame index where the block is located. and These represent the vertical (height) and horizontal (width) coordinates within the frame, respectively. The relative displacements of these two image patches in the three dimensions of time, height, and width are calculated to obtain a three-dimensional relative displacement vector. .in, .

[0078] Considering the significant differences in physical scale and intrinsic meaning between the time and spatial dimensions, in order to enable the model to better adapt to information at these two different scales, the time displacement component... Nonlinear scaling is performed, as shown in equation (1).

[0079] (4);

[0080] in, As a learnable temporal scaling factor, it allows the network to adaptively adjust the perceptual scale of temporal distance based on data characteristics during training. Equation (4) uses a logarithmic function to compress larger temporal distances, making the model more sensitive to temporally adjacent frames while preserving the positional relationships of distant frames.

[0081] The processed three-dimensional relative displacement vector A self-attention module is injected to achieve spatiotemporal awareness. As shown in Equation (5), a trainable projection matrix is ​​used. and Will This is mapped to a positional bias vector. This bias vector is then added element-wise to the original key vector. Sum value vector In this process, a new location-aware key / value vector containing spatiotemporal relationships is generated. and .

[0082] (5);

[0083] S6. Based on the second infrared segment set and the second visible light segment set, use the optimized video fusion model to perform multimodal video fusion to obtain the fused video.

[0084] Optionally, based on the second infrared segment set and the second visible light segment set, multimodal video fusion is performed using an optimized video fusion model to obtain a fused video, including:

[0085] Based on the spatiotemporal hierarchical attention mechanism, feature extraction is performed on the second infrared fragment set and the second visible light fragment set to obtain the infrared frame image feature set and the visible light frame image feature set.

[0086] Contextual information is aggregated based on the infrared frame image feature set and the visible light frame image feature set to obtain the enhanced infrared frame image feature set and the enhanced visible light frame image feature set.

[0087] Based on the L1 norm strategy, cross-modal feature fusion is performed on the feature sets of enhanced infrared frame images and enhanced visible light frame images to obtain the fused frame image feature set.

[0088] Feature decoding is performed on the feature set of the fused frame images to obtain the fused frame image set; the fused frame image set is then sorted in time sequence to obtain the fused video.

[0089] In one feasible implementation, the input data format of the second infrared fragment set and the second visible light fragment set is as follows: That is, each segment contains 3 frames with a height of Width ,have Images from 1 input channel. These input segments are injected with 3D positional encoding to introduce crucial prior positional information. Among them, the 1st... The first frame is the main frame to be fused, while the two frames before and after it serve as temporal context to provide auxiliary information. After model processing, a high-quality fused result is finally output. .

[0090] Optionally, based on a spatiotemporal hierarchical attention mechanism, feature extraction is performed according to the second infrared fragment set and the second visible light fragment set to obtain infrared frame image feature sets and visible light frame image feature sets, including:

[0091] Attention calculations are performed within a three-dimensional spatiotemporal window based on the second infrared fragment set and the second visible light fragment set to obtain local spatiotemporal domain attention weights.

[0092] Based on the second infrared fragment set and the second visible light fragment set, attention calculation is performed between three-dimensional spatiotemporal windows to obtain cross-domain spatiotemporal attention weights;

[0093] Based on local spatiotemporal attention weights and cross-domain spatiotemporal attention weights, weighted feature aggregation is performed according to the second infrared fragment set and the second visible light fragment set to obtain the infrared frame image feature set and the visible light frame image feature set.

[0094] In one feasible implementation, to efficiently model long-range spatiotemporal dependencies in video data, the core feature extraction module of this scheme adopts and integrates a spatiotemporal hierarchical attention mechanism. This mechanism constructs the global spatiotemporal receptive field in a computationally efficient manner, and its core lies in the alternating use of two attention calculation modes in consecutive modules.

[0095] By employing local spatiotemporal attention, the feature map is divided into non-overlapping three-dimensional spatiotemporal windows, and self-attention is independently computed within each window, thereby efficiently capturing local spatiotemporal features. The next module utilizes cross-domain spatiotemporal attention, cleverly achieving information interaction between adjacent spatiotemporal domains by iteratively reorganizing the feature map before dividing it into windows. This allows information to flow globally to establish long-range spatiotemporal dependencies.

[0096] As shown in equation (6), the attention calculation within the window follows the standard scaled dot product attention model. This equation calculates the query matrix ( ) and the key matrix ( The similarity is evaluated by the dot product of () and by a scaling factor. Perform normalization (where (where the dimension is the key vector), then incorporate the three-dimensional relative position offset. By injecting spatiotemporal information, ultimately through The attention weights calculated by the function are applied to the value matrix ( To obtain the weighted features.

[0097] (6);

[0098] in, This is a timing mask matrix used to constrain timing relationships.

[0099] In one feasible implementation, the robust multimodal video fusion method for complex scenarios proposed in this invention, by introducing three-dimensional relative position coding, spatiotemporal hierarchical attention mechanism and inter-frame consistency loss function, has achieved significant beneficial effects in terms of fusion quality, temporal stability and operating efficiency compared with the prior art.

[0100] This invention generates fusion results with richer information and clearer details. Quantitative experimental results show that, compared with traditional wavelet transform fusion and mainstream fusion networks based on convolutional neural networks (CNN), this invention achieves the best results in several key objective evaluation indicators. As shown in Table 1 (Quantitative Evaluation Results of the Fusion Algorithm), it achieves the highest scores in indicators such as information entropy (7.45±0.15), spatial frequency (17.92±0.76), and mutual information (13.88±0.31). This proves that the fused video generated by this method contains richer total information, clearer texture details, and can transfer and retain more effective information from the source video.

[0101] Table 1

[0102]

[0103] From a qualitative visual perspective, this invention can clearly present background textures and details (such as vehicle outlines and ground) in visible light images while fully preserving bright thermal targets (such as pedestrians) in infrared images, resulting in excellent overall visual perception.

[0104] This invention is specifically optimized for video processing, significantly improving the inter-frame smoothness and stability of the output video. In the key metric of temporal consistency, this method scores 0.98±0.01, significantly outperforming methods based on CNN networks (0.94±0.03) and wavelet transforms (0.91±0.04). This fully demonstrates that by introducing inter-frame consistency loss and spatiotemporal joint modeling, this invention effectively suppresses visual artifacts such as flickering and jitter caused by traditional frame-by-frame processing methods, ensuring the smoothness and continuity of the fused video during playback, which is more in line with human visual habits, and also achieving the optimal visual information fidelity metric of 0.73±0.02.

[0105] This invention improves fusion performance while also maintaining a lightweight and efficient model. Compared to traditional CNN fusion architectures, the framework proposed in this invention demonstrates improved model parameters by 23% and inference speed by 39% on publicly available datasets, under the same input resolution and RTX 4060 GPU conditions. This advantage means that this invention can achieve faster processing speeds with less computational resources, making it more feasible and valuable for deployment in real-world applications such as autonomous driving and real-time monitoring, where high processing efficiency is required.

[0106] This invention proposes a multimodal video fusion method based on a spatiotemporal hierarchical attention network. The method takes the target frame to be fused and its two adjacent frames as a processing unit input. It utilizes the temporal context information of adjacent frames to enhance the feature representation of the target frame, realizes cross-modal feature fusion, and reconstructs and outputs a high-quality fused image.

[0107] By calculating the relative displacement of any two image patches in the three dimensions of time, height, and width, a three-dimensional relative position vector is constructed, and this vector is injected element by element into the "key" and "value" vectors of the self-attention mechanism, enabling the model to obtain accurate spatiotemporal perception capabilities.

[0108] The training process is decomposed into two stages: pre-training and joint training. In the pre-training stage, the feature fusion module is trained independently. The goal of this stage is to enable the module to learn to generate temporally smooth, flicker-free outputs directly from the raw video data. Its core optimization objective is an inter-frame consistency loss function. In the joint training stage, the model is jointly trained end-to-end to optimize the final fusion performance. This invention is an efficient and coherent multimodal video fusion method based on spatiotemporal hierarchical attention.

[0109] Figure 2 This is a block diagram of a multimodal video fusion device based on a spatiotemporal hierarchical attention network provided in an embodiment of the present invention. This device is used in a multimodal video fusion method based on a spatiotemporal hierarchical attention network. (Refer to...) Figure 2 The device includes a training data construction module 210, a video fusion model construction module 220, a video fusion model training module 230, a data to be fused processing module 240, a three-dimensional position encoding module 250, and a video fusion module 260. Among them:

[0110] Training data construction module 210 is used to construct the first training dataset based on the publicly available REDS dataset and DAVIS dataset; and to construct the second training dataset based on the publicly available TNO dataset, RoadScene dataset and MSRS dataset.

[0111] The video fusion model building module 220 is used to build a video fusion model based on the network structure of the spatiotemporal hierarchical attention network and the model structure of the Transformer model.

[0112] The video fusion model training module 230 is used to perform two-stage training on the video fusion model based on the inter-frame consistency loss function and the bidirectional consistency loss function, according to the first training dataset and the second training dataset, to obtain an optimized video fusion model.

[0113] The data processing module 240 is used to acquire the infrared video to be fused and the visible light video to be fused; based on the frame-by-frame sliding method, the infrared video to be fused and the visible light video to be fused are synchronously sliced ​​for three consecutive frames to obtain the first infrared segment set and the first visible light segment set.

[0114] The three-dimensional position encoding module 250 is used to add three-dimensional position encoding to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set.

[0115] The video fusion module 260 is used to perform multimodal video fusion using an optimized video fusion model based on the second infrared segment set and the second visible light segment set to obtain a fused video.

[0116] The video fusion model includes a feature extraction module, a feature fusion module, and a video reconstruction module.

[0117] The feature extraction module is a global feature extraction network composed of multiple cascaded spatiotemporal hierarchical attention networks.

[0118] Optionally, the video fusion model training module 230 is further used for:

[0119] Based on the first training dataset, the feature fusion module is used to perform multimodal feature fusion to obtain a fused feature dataset;

[0120] The inter-frame consistency loss is calculated based on the first training dataset and the fused feature dataset, using the inter-frame consistency loss function.

[0121] Based on the inter-frame consistency loss, the parameters of the feature fusion module are optimized to obtain the first optimized feature fusion module;

[0122] Based on the second training dataset, video reconstruction is performed using the feature extraction module, the first optimized feature fusion module, and the video reconstruction module to obtain the reconstructed video dataset.

[0123] Based on the bidirectional consistency loss function, the bidirectional consistency loss is calculated using the second training dataset and the reconstructed video dataset.

[0124] Based on the bidirectional consistency loss, the parameters of the feature extraction module, the first optimized feature fusion module, and the video reconstruction module are optimized to obtain the optimized feature extraction module, the second optimized feature fusion module, and the optimized video reconstruction module.

[0125] An optimized video fusion model is obtained based on the optimized feature extraction module, the second optimized feature fusion module, and the optimized video reconstruction module.

[0126] Optionally, the three-dimensional position encoding module 250 is further used for:

[0127] The three-dimensional relative displacement vectors between image blocks are calculated based on the first infrared fragment set and the first visible light fragment set to obtain the infrared relative displacement set and the visible light relative displacement set.

[0128] Based on a preset learnable scalar, the infrared relative displacement set and the visible light relative displacement set are scaled in time dimension to obtain the scaled infrared relative displacement set and the scaled visible light relative displacement set.

[0129] Based on the preset first trainable projection matrix, the first infrared segment set is injected with three-dimensional position encoding according to the scaled infrared relative displacement set to obtain the second infrared segment set;

[0130] Based on the preset second trainable projection matrix, the first visible light fragment set is injected with three-dimensional position encoding according to the scaled visible light relative displacement set to obtain the second visible light fragment set.

[0131] Optionally, the video fusion module 260 is further used for:

[0132] Based on the spatiotemporal hierarchical attention mechanism, feature extraction is performed on the second infrared fragment set and the second visible light fragment set to obtain the infrared frame image feature set and the visible light frame image feature set.

[0133] Contextual information is aggregated based on the infrared frame image feature set and the visible light frame image feature set to obtain the enhanced infrared frame image feature set and the enhanced visible light frame image feature set.

[0134] Based on the L1 norm strategy, cross-modal feature fusion is performed on the feature sets of enhanced infrared frame images and enhanced visible light frame images to obtain the fused frame image feature set.

[0135] Feature decoding is performed on the feature set of the fused frame images to obtain the fused frame image set; the fused frame image set is then sorted in time sequence to obtain the fused video.

[0136] Optionally, the video fusion module 260 is further used for:

[0137] Attention calculations are performed within a three-dimensional spatiotemporal window based on the second infrared fragment set and the second visible light fragment set to obtain local spatiotemporal domain attention weights.

[0138] Based on the second infrared fragment set and the second visible light fragment set, attention calculation is performed between three-dimensional spatiotemporal windows to obtain cross-domain spatiotemporal attention weights;

[0139] Based on local spatiotemporal attention weights and cross-domain spatiotemporal attention weights, weighted feature aggregation is performed according to the second infrared fragment set and the second visible light fragment set to obtain the infrared frame image feature set and the visible light frame image feature set.

[0140] This invention proposes a multimodal video fusion method based on a spatiotemporal hierarchical attention network. The method takes the target frame to be fused and its two adjacent frames as a processing unit input. It utilizes the temporal context information of adjacent frames to enhance the feature representation of the target frame, realizes cross-modal feature fusion, and reconstructs and outputs a high-quality fused image.

[0141] By calculating the relative displacement of any two image patches in the three dimensions of time, height, and width, a three-dimensional relative position vector is constructed, and this vector is injected element by element into the "key" and "value" vectors of the self-attention mechanism, enabling the model to obtain accurate spatiotemporal perception capabilities.

[0142] The training process is decomposed into two stages: pre-training and joint training. In the pre-training stage, the feature fusion module is trained independently. The goal of this stage is to enable the module to learn to generate temporally smooth, flicker-free outputs directly from the raw video data. Its core optimization objective is an inter-frame consistency loss function. In the joint training stage, the model is jointly trained end-to-end to optimize the final fusion performance. This invention is an efficient and coherent multimodal video fusion method based on spatiotemporal hierarchical attention.

[0143] Figure 3 This is a schematic diagram of the structure of a multimodal video fusion device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the multimodal video fusion device may include the above-mentioned Figure 2 The illustrated multimodal video fusion apparatus is based on a spatiotemporal hierarchical attention network. Optionally, the multimodal video fusion apparatus 310 may include a first processor 2001.

[0144] Optionally, the multimodal video fusion device 310 may also include a memory 2002 and a transceiver 2003.

[0145] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0146] The following is combined with Figure 3 A detailed introduction to each component of the multimodal video fusion device 310 is provided below:

[0147] The first processor 2001 is the control center of the multimodal video fusion device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0148] Optionally, the first processor 2001 can perform various functions of the multimodal video fusion device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0149] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0150] In a specific implementation, as one example, the multimodal video fusion device 310 may also include multiple processors, for example... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0151] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0152] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the multimodal video fusion device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0153] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0154] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0155] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the multimodal video fusion device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0156] It should be noted that, Figure 3 The structure of the multimodal video fusion device 310 shown in the diagram does not constitute a limitation on the router. Actual multimodal video fusion devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0157] Furthermore, the technical effects of the multimodal video fusion device 310 can be referenced from the technical effects of the multimodal video fusion method based on spatiotemporal hierarchical attention network described in the above method embodiments, and will not be repeated here.

[0158] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0159] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0160] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0161] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0162] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0163] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0164] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0166] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0169] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal video fusion method based on a spatiotemporal hierarchical attention network, characterized in that, The method includes: The first training dataset was constructed based on the publicly available REDS and DAVIS datasets; the second training dataset was constructed based on the publicly available TNO, RoadScene, and MSRS datasets. A video fusion model is constructed based on the network structure of the spatiotemporal hierarchical attention network and the model structure of the Transformer model; Based on the inter-frame consistency loss function and the bi-directional consistency loss function, the video fusion model is trained in two stages according to the first training dataset and the second training dataset to obtain an optimized video fusion model. The video fusion model includes a feature extraction module, a feature fusion module, and a video reconstruction module. The feature extraction module is a global feature extraction network composed of multiple cascaded spatiotemporal hierarchical attention networks; The step of training the video fusion model in two stages based on the inter-frame consistency loss function and the bidirectional consistency loss function, using the first training dataset and the second training dataset, to obtain an optimized video fusion model includes: Based on the first training dataset, the feature fusion module is used to perform multimodal feature fusion to obtain a fused feature dataset; The inter-frame consistency loss is calculated based on the first training dataset and the fused feature dataset, using the inter-frame consistency loss function. Based on the inter-frame consistency loss, the parameters of the feature fusion module are optimized to obtain the first optimized feature fusion module; Based on the second training dataset, video reconstruction is performed using the feature extraction module, the first optimized feature fusion module, and the video reconstruction module to obtain the reconstructed video dataset. Based on the bidirectional consistency loss function, the bidirectional consistency loss is calculated using the second training dataset and the reconstructed video dataset. Based on the bidirectional consistency loss, the parameters of the feature extraction module, the first optimized feature fusion module, and the video reconstruction module are optimized to obtain the optimized feature extraction module, the second optimized feature fusion module, and the optimized video reconstruction module. Based on the optimized feature extraction module, the second optimized feature fusion module, and the optimized video reconstruction module, an optimized video fusion model is obtained; Acquire the infrared video and the visible light video to be fused; based on the frame-by-frame sliding method, perform three consecutive frames of synchronous slicing on the infrared video and the visible light video to be fused to obtain the first infrared segment set and the first visible light segment set. Add three-dimensional position encoding to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set; Based on the second infrared and second visible light fragment sets, an optimized video fusion model is used to perform multimodal video fusion to obtain a fused video.

2. The multimodal video fusion method based on spatiotemporal hierarchical attention network according to claim 1, characterized in that, The step of adding three-dimensional position encoding to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set includes: The three-dimensional relative displacement vectors between image blocks are calculated based on the first infrared fragment set and the first visible light fragment set to obtain the infrared relative displacement set and the visible light relative displacement set. Based on a preset learnable scalar, the infrared relative displacement set and the visible light relative displacement set are scaled in time dimension to obtain the scaled infrared relative displacement set and the scaled visible light relative displacement set. Based on the preset first trainable projection matrix, the first infrared segment set is injected with three-dimensional position encoding according to the scaled infrared relative displacement set to obtain the second infrared segment set; Based on the preset second trainable projection matrix, the first visible light fragment set is injected with three-dimensional position encoding according to the scaled visible light relative displacement set to obtain the second visible light fragment set.

3. The multimodal video fusion method based on spatiotemporal hierarchical attention network according to claim 1, characterized in that, The step of performing multimodal video fusion using an optimized video fusion model based on a second infrared fragment set and a second visible light fragment set to obtain a fused video includes: Based on the spatiotemporal hierarchical attention mechanism, feature extraction is performed on the second infrared fragment set and the second visible light fragment set to obtain the infrared frame image feature set and the visible light frame image feature set. Contextual information is aggregated based on the infrared frame image feature set and the visible light frame image feature set to obtain the enhanced infrared frame image feature set and the enhanced visible light frame image feature set. Based on the L1 norm strategy, cross-modal feature fusion is performed on the feature sets of enhanced infrared frame images and enhanced visible light frame images to obtain the fused frame image feature set. Feature decoding is performed on the feature set of the fused frame images to obtain the fused frame image set; the fused frame image set is then sorted in time sequence to obtain the fused video.

4. The multimodal video fusion method based on spatiotemporal hierarchical attention network according to claim 1, characterized in that, The spatiotemporal hierarchical attention mechanism extracts features from the second infrared fragment set and the second visible light fragment set to obtain infrared frame image feature sets and visible light frame image feature sets, including: Attention calculations are performed within a three-dimensional spatiotemporal window based on the second infrared fragment set and the second visible light fragment set to obtain local spatiotemporal domain attention weights. Based on the second infrared fragment set and the second visible light fragment set, attention calculation is performed between three-dimensional spatiotemporal windows to obtain cross-domain spatiotemporal attention weights; Based on local spatiotemporal attention weights and cross-domain spatiotemporal attention weights, weighted feature aggregation is performed according to the second infrared fragment set and the second visible light fragment set to obtain the infrared frame image feature set and the visible light frame image feature set.

5. A multimodal video fusion device based on a spatiotemporal hierarchical attention network, wherein the multimodal video fusion device based on a spatiotemporal hierarchical attention network is used to implement the multimodal video fusion method based on a spatiotemporal hierarchical attention network as described in any one of claims 1-4, characterized in that, The device includes: The training data construction module is used to construct the first training dataset based on the publicly available REDS and DAVIS datasets; and to construct the second training dataset based on the publicly available TNO, RoadScene, and MSRS datasets. The video fusion model building module is used to build a video fusion model based on the network structure of the spatiotemporal hierarchical attention network and the model structure of the Transformer model. The video fusion model training module is used to perform two-stage training on the video fusion model based on the inter-frame consistency loss function and the bidirectional consistency loss function, according to the first training dataset and the second training dataset, to obtain an optimized video fusion model. The data processing module to be fused is used to acquire the infrared video to be fused and the visible light video to be fused; based on the frame-by-frame sliding method, the infrared video to be fused and the visible light video to be fused are synchronously sliced ​​for three consecutive frames to obtain the first infrared segment set and the first visible light segment set. A three-dimensional position encoding module is used to add three-dimensional position encoding to the first infrared fragment set and the first visible light fragment set to obtain the second infrared fragment set and the second visible light fragment set. The video fusion module is used to perform multimodal video fusion using an optimized video fusion model based on the second infrared segment set and the second visible light segment set to obtain a fused video.

6. A multimodal video fusion device, characterized in that, The multimodal video fusion device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 4.

Images