Multimodal video fusion method and device based on semantic time sequence collaborative modeling

Abstract

The application discloses a robust multi-modal video fusion method and device based on semantic time sequence collaborative modeling. The method comprises the following steps: extracting multi-scale hierarchical spatial features from a visible light video sequence and an infrared video sequence respectively, obtaining degenerative embedding based on the multi-scale hierarchical spatial features, and performing multi-scale degenerative modulation on the multi-scale hierarchical spatial features based on the degenerative embedding after constraint alignment; searching for the most relevant quantization prototype of the degenerative enhanced multi-scale hierarchical spatial features in the trained learnable dictionary, performing cross attention based on the quantization prototype and the degenerative enhanced multi-scale hierarchical spatial features, and performing fusion to obtain fusion features; performing time sequence modeling and bidirectional propagation to obtain bidirectional enhanced fusion features, performing space-time aggregation on the bidirectional enhanced fusion features to obtain space-time fusion features; and performing reconstruction based on the fusion features and the space-time fusion features to obtain a fusion video. The application can generate a fusion video with clear details, complete structure and smooth time sequence.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a robust multimodal video fusion method, apparatus, storage medium, and electronic device based on semantic temporal collaborative modeling. Background Technology

[0002] Due to limitations in imaging principles, single-modal sensors struggle to provide a comprehensive representation of a scene. For example, infrared sensors can highlight thermal targets in low light, at night, or in complex weather conditions, while visible light sensors can provide rich texture, color, and structural details under good lighting conditions. By leveraging the complementary properties of different modalities, image fusion can achieve more accurate scene representation and generate fusion results more suitable for human observation and subsequent visual tasks, making it valuable for applications in autonomous driving, security monitoring, scene perception, and target recognition.

[0003] In recent years, multimodal image fusion methods have been extensively studied, effectively aggregating salient thermal target information from infrared images and texture structure information from visible light images in static image fusion. However, most of these methods, such as CDDFuse, revolve around single-frame images, primarily learning the spatial complementarity between different modalities at the same time. When directly applied to video data, they typically employ a frame-by-frame independent fusion strategy, lacking modeling of motion correlations and temporal redundancy information between adjacent frames. This easily leads to problems such as brightness flickering, edge jumps, and unstable target structures between consecutive frames. To address the lack of temporal consistency in frame-by-frame fusion, existing video fusion methods such as TemCoCo and VideoFusion have begun to introduce spatiotemporal representations, adjacent frame interactions, or temporal attention mechanisms to enhance information correlation between consecutive frames. However, these methods often employ holistic aggregation or coarse-grained interactions of features from adjacent frames, making it difficult to determine which cross-frame changes can provide effective compensation for the current frame. Therefore, the model is prone to introducing invalid or even interfering temporal information, resulting in insufficient detail recovery, unstable motion region fusion, and low temporal compensation efficiency. Furthermore, infrared and visible light videos in real-world scenarios are often affected by complex environmental and sensor factors, such as rain and snow occlusion, as well as complex degradation caused by multiple types of degradation occurring simultaneously. Existing video fusion methods either assume high-quality input and easily inherit degradation from the source video, or rely on data augmentation and pre-augmentation modules to improve robustness, but they typically lack explicit modeling of degradation categories and intensities, making it difficult to adapt to the random occurrence of multiple types and complex degradations in real-world scenarios. Simultaneously, the quality of the fused video depends not only on whether low-level textures and thermal targets are preserved, but also closely related to the integrity of the high-level semantic structure. Existing methods often treat the fusion task as a low-level visual reconstruction problem, neglecting semantic knowledge. Therefore, the fusion results are prone to problems such as blurred semantic boundaries, damage to target structures, and decreased performance in downstream semantic understanding. Summary of the Invention

[0004] This invention provides a robust multimodal video fusion method, apparatus, storage medium, and electronic device based on semantic-temporal collaborative modeling, which can solve the problems of insufficient robustness of existing multimodal video fusion methods in complex degradation scenarios, insufficient utilization of high-quality semantic structures, and poor temporal consistency.

[0005] This invention provides a robust multimodal video fusion method based on semantic-temporal collaborative modeling, comprising: Multi-scale hierarchical spatial features are extracted from visible light video sequences and infrared video sequences respectively. Degenerate embeddings are obtained based on the multi-scale hierarchical spatial features. Multi-scale degradation modulation is performed on the multi-scale hierarchical spatial features based on the constraint-aligned degradation embeddings to obtain degenerate-enhanced multi-scale hierarchical spatial features. A learnable dictionary is pre-built and trained. The quantization prototype most relevant to the degenerate-enhanced multi-scale hierarchical spatial features is retrieved from the trained learnable dictionary. Cross attention is performed based on the quantization prototype and the degenerate-enhanced multi-scale hierarchical spatial features to obtain the visible light features and infrared features after dictionary interaction enhancement. These features are then fused to obtain the fused features. Temporal attention is calculated based on fusion features. The temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction are added to obtain forward-enhanced features or backward-enhanced features. The fusion features, forward-enhanced features and backward-enhanced features are concatenated to obtain bidirectional enhanced fusion features. The bidirectional enhanced fusion features are spatiotemporally aggregated to obtain spatiotemporal fusion features. The fused video is obtained by reconstructing based on fusion features and spatiotemporal fusion features.

[0006] Furthermore, according to the robust multimodal video fusion method based on semantic-temporal collaborative modeling described above, multi-scale hierarchical spatial features are extracted from visible light video sequences and infrared video sequences respectively. Degradational embeddings are obtained based on these multi-scale hierarchical spatial features. Multi-scale degradation modulation is then applied to the multi-scale hierarchical spatial features based on the constrained-aligned degradational embeddings to obtain degraded and enhanced multi-scale hierarchical spatial features, including: Visible light video sequences are input into a visible light encoder to obtain visible light multi-scale hierarchical spatial features, and infrared video sequences are input into an infrared encoder to obtain infrared multi-scale hierarchical spatial features. Each scale feature in the multi-scale hierarchical spatial features is divided into multiple spatial regions. The average response, significant response, and structural change response of each region are calculated. The responses of different scales and regions are concatenated in the channel dimension and input into a multilayer perceptron for nonlinear mapping to generate degenerate embeddings. Text prompts are constructed in advance based on samples during the training phase. The text prompts are then fed into a frozen text encoder to obtain degenerate text features. Based on the degenerate text, the normalized degenerate embeddings are constrained and aligned to obtain constrained and aligned degenerate embeddings. The constrained and aligned degenerate embeddings are input into linear mapping layers of the same scale as the multi-scale hierarchical spatial features to generate channel scaling parameters and offset parameters for each scale. The channel scaling parameters and offset parameters are then applied to the visible light multi-scale hierarchical spatial features and infrared multi-scale hierarchical spatial features of the corresponding scales to obtain degenerate and enhanced multi-scale hierarchical spatial features.

[0007] Furthermore, according to the robust multimodal video fusion method based on semantic-temporal collaborative modeling described above, a learnable dictionary is pre-constructed and trained, including: Acquire high-quality infrared and high-quality visible light samples to construct learnable dictionaries for each type of sample; Any high-quality infrared sample or high-quality visible light sample is input into a pre-trained visual geometric convolutional neural network to extract visual features. Then, the sample is input into a frozen self-supervised visual base model to extract semantic features. The visual features and the semantic features are fused to obtain high-quality visual semantic features. Initialize a set of learnable vectors as a learnable dictionary, and select the dictionary prototype that is closest to the high-quality visual semantic features as the quantization representation in the learnable dictionary; The quantized representation is input into the image reconstruction encoder to recover a high-quality image; The high-quality image is input into the semantic segmentation decoder to obtain the predicted segmentation result; The reconstruction loss is calculated based on the restored high-quality image and real high-quality samples. The cross-entropy loss is calculated based on the predicted segmentation result and the corresponding real label. The vector quantization constraint loss is calculated based on the high-quality visual semantic features and the quantization representation. The learnable dictionary is trained based on the reconstruction loss, the cross-entropy loss and the vector quantization constraint loss.

[0008] Furthermore, according to the robust multimodal video fusion method based on semantic-temporal collaborative modeling described above, temporal attention is calculated based on the fusion features. The temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction are added to obtain forward-enhanced features or backward-enhanced features. The fusion features, forward-enhanced features, and backward-enhanced features are concatenated to obtain bidirectionally enhanced fusion features. The bidirectionally enhanced fusion features are spatiotemporally aggregated to obtain spatiotemporal fusion features, including: The difference features are calculated based on the fusion features of the current frame and the fusion features of the adjacent frames. The temporal attention is then calculated based on the difference features and the fusion features of the adjacent frames to obtain the temporal attention output. The forward-enhanced feature is obtained by adding the temporal attention output, the memory compensation term for forward propagation, the fused feature, the visible light feature enhanced by dictionary interaction, and the infrared feature. The temporal attention output, the backward propagation memory compensation term, the fused features, the visible light features enhanced by dictionary interaction, and the infrared features are added together to obtain the backward enhancement features; The fusion feature, the forward enhancement feature, and the backward enhancement feature are concatenated to obtain the bidirectional enhanced fusion feature. The bidirectional enhanced fusion feature is then spatiotemporally aggregated to obtain the spatiotemporal fusion feature.

[0009] Furthermore, according to the robust multimodal video fusion method based on semantic temporal collaborative modeling described above, when performing forward propagation, differential features are calculated based on the fusion features of the current frame and the fusion features of the previous frame; the memory compensation term for forward propagation includes the fusion features of the previous frame and the fusion features after bidirectional enhancement. When backpropagation is performed, differential features are calculated based on the fusion features of the current frame and the fusion features of the next frame; the memory compensation term for backpropagation includes the fusion features of the next frame and the fusion features after bidirectional enhancement.

[0010] Furthermore, according to the robust multimodal video fusion method based on semantic-temporal collaborative modeling described above, the spatiotemporal fusion features are obtained by spatiotemporal aggregation of the bidirectional enhanced fusion features, including: The fused features after bidirectional enhancement are divided into multiple groups along the channel dimension:

[0011] in, The fusion features are after time-shifting. This represents the fusion feature shifted forward along the time dimension. This represents the fusion feature shifted along the time dimension. Indicates fusion features that remain unchanged. This represents the total number of channels in the fusion feature after bidirectional enhancement. The current number of channels for the fused features; The temporally shifted fusion features are input into a 3D convolutional layer to obtain spatiotemporal fusion features.

[0012] Furthermore, according to the robust multimodal video fusion method based on semantic-temporal collaborative modeling described above, the fused video is reconstructed based on fusion features and spatiotemporal fusion features, including: The fusion feature is upsampled and concatenated with the spatiotemporal fusion feature in the channel dimension to obtain the spliced ​​feature; The stitching features are sequentially passed through a convolutional layer, an activation layer, and a reconstruction head to obtain a fused video.

[0013] The present invention also provides a robust multimodal video fusion device based on semantic temporal collaborative modeling, comprising: The multi-scale feature extraction and degradation perception module is used to extract multi-scale hierarchical spatial features from visible light video sequences and infrared video sequences respectively, obtain degradation embeddings based on multi-scale hierarchical spatial features, and perform multi-scale degradation modulation on multi-scale hierarchical spatial features based on the constraint-aligned degradation embeddings to obtain degradation-enhanced multi-scale hierarchical spatial features. The visual-semantic interaction module is used to pre-build and train a learnable dictionary. It retrieves the quantization prototype most relevant to the degenerate-enhanced multi-scale hierarchical spatial features from the trained learnable dictionary. Based on the quantization prototype and the degenerate-enhanced multi-scale hierarchical spatial features, it performs cross-attention to obtain the visible light features and infrared features enhanced by dictionary interaction, and then fuses them to obtain the fused features. The differential temporal modeling and spatiotemporal aggregation module is used to calculate temporal attention based on fusion features. It adds the temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction to obtain forward-enhanced features or backward-enhanced features. It concatenates the fusion features, forward-enhanced features, and backward-enhanced features to obtain bidirectional enhanced fusion features. It then performs spatiotemporal aggregation on the bidirectional enhanced fusion features to obtain spatiotemporal fusion features. The reconstruction module is used to reconstruct the fused video based on fusion features and spatiotemporal fusion features.

[0014] The present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described robust multimodal video fusion methods based on semantic-temporal collaborative modeling.

[0015] The present invention also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the robust multimodal video fusion method based on semantic temporal collaborative modeling described in any of the preceding claims.

[0016] This invention provides a robust multimodal video fusion method, apparatus, storage medium, and electronic device based on semantic-temporal collaborative modeling. By extracting multi-scale hierarchical spatial features from visible light and infrared videos and performing multi-scale degradation modulation on these features based on degradation embedding, this invention can dynamically perceive and adapt to the degradation type and intensity in the scene, achieving active blocking of invalid noise while accurately decoupling and preserving effective structural information, significantly improving the degradation robustness of fused features under harsh imaging conditions. By pre-constructing a learnable dictionary and retrieving the quantized prototype most relevant to the degradation enhancement features from the dictionary, combined with a cross-attention mechanism for feature interaction enhancement, this invention utilizes high-order structural semantic priors to explicitly constrain low-order physical features, effectively compensating for local information loss and semantic gaps in low-light scenes, significantly enhancing the fusion performance. This invention demonstrates the ability of fused features to preserve topological structure and edge details. By calculating temporal attention and introducing memory compensation terms to generate forward / backward enhanced features, and then concatenating the original fused features with bidirectional enhanced features for spatiotemporal aggregation, this invention can efficiently suppress redundant background interference while adaptively capturing cross-frame dynamically changing regions with information gain. It establishes a strong consistent representation of moving targets on the time axis, effectively eliminating temporal flicker and abrupt changes between video frames. By constructing a collaborative fusion mechanism of "degradation modulation - semantic dictionary constraint - bidirectional temporal enhancement", this invention not only significantly improves the visual perception quality and temporal smoothness of fused videos in all-weather and harsh environments, but also provides reliable multimodal perception input for downstream intelligent tasks such as autonomous driving and robot navigation, possessing broad application prospects and practical value. Attached Figure Description

[0017] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0018] Figure 1 The flowchart illustrates a robust multimodal video fusion method based on semantic-temporal collaborative modeling, as provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the text guidance feature modulation process provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the visual semantic prior interaction process provided in an embodiment of the present invention.

[0021] Figure 4 The flowchart for time-series dynamic modeling and bidirectional memory propagation provided in the embodiments of the present invention.

[0022] Figure 5 This is a schematic diagram comparing the method of the present invention with existing methods, as provided in the embodiments of the present invention.

[0023] Figure 6 This diagram illustrates a comparison between the method of the present invention and other advanced fusion methods in certain scenarios provided by embodiments of the present invention.

[0024] Figure 7 This is a schematic diagram of the structure of a robust multimodal video fusion device based on semantic temporal collaborative modeling provided in an embodiment of the present invention.

[0025] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] To address the aforementioned problems, embodiments of the present invention provide a robust multimodal video fusion method, apparatus, storage medium, and electronic device based on semantic-temporal collaborative modeling. The robust multimodal video fusion apparatus based on semantic-temporal collaborative modeling provided by embodiments of the present invention can be integrated into an electronic device, which can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0028] Please see Figure 1 , Figure 1 The flowchart illustrates a robust multimodal video fusion method based on semantic-temporal collaborative modeling, provided in an embodiment of the present invention. This method, applied in electronic devices, includes the following steps: S1. Extract multi-scale hierarchical spatial features from visible light video sequences and infrared video sequences respectively. Obtain degenerate embeddings based on multi-scale hierarchical spatial features. Perform multi-scale degenerate modulation on the multi-scale hierarchical spatial features based on the constrained aligned degenerate embeddings to obtain degenerate-enhanced multi-scale hierarchical spatial features.

[0029] In one embodiment, step S1 includes the following steps: S11, input the visible light video sequence into the visible light encoder to obtain the visible light multi-scale hierarchical spatial features, and input the infrared video sequence into the infrared encoder to obtain the infrared multi-scale hierarchical spatial features.

[0030] Acquire time-synchronized visible light video sequences and infrared video sequences Input to the visible light encoder respectively and infrared encoder The hierarchical spatial features at high, medium, and low scales (i.e., multi-scale hierarchical spatial features) are extracted and expressed by the following formula:

[0031]

[0032] in, and They represent high-, medium-, and low-scale features, respectively. This represents the multi-scale hierarchical spatial characteristics of visible light. It represents the multi-scale hierarchical spatial characteristics of infrared radiation.

[0033] The multi-scale features specifically include high-scale, medium-scale, and low-scale features. High-scale features are used to preserve edge, texture, and thermal object details, medium-scale features are used to describe local structures, and low-scale features are used to carry contextual and semantic information.

[0034] S12 divides each scale feature in the multi-scale hierarchical spatial features into multiple spatial regions, calculates the average response, significant response and structural change response of each region, cascades the responses of different scales and regions in the channel dimension, and inputs them into a multilayer perceptron for nonlinear mapping to generate degenerate embeddings.

[0035] Since real-world video degradation is typically regional and non-uniform, this embodiment of the invention does not employ a single global average pooling method to directly compress the entire frame's features. Instead, it performs multi-region spatial pooling on high-scale, mid-scale, and low-scale features separately. Figure 2 This is a schematic diagram of the text guidance feature modulation process provided in an embodiment of the present invention, such as... Figure 2 As shown.

[0036] Each scale feature is divided into multiple spatial regions, and the average response, significant response, and structural change response in each region are calculated to obtain degradation sensitivity statistics that can reflect the distribution of local rain streaks, fog scattering, low-light dark areas, noise disturbances, and complex degradation.

[0037] Specifically, the average response is calculated using regional average pooling, the significant response is calculated using max pooling, and the structural change response is calculated using spatial gradient variance response.

[0038] Subsequently, the responses from different scales and regions are concatenated and spliced ​​along the channel dimension to form a degradation-sensitive statistical vector. This degradation-sensitive statistical vector is then input into a multilayer perceptron for nonlinear mapping to generate a compact degradation embedding. :

[0039] in, This can be represented by region-average pooling, max pooling, or spatial gradient variance. For splicing operations, It is a multilayer perceptron.

[0040] This degradation embedding is used to describe the degradation type, degradation intensity, and spatial distribution trend of the current video segment.

[0041] S13. Text prompts are constructed in advance based on samples during the training phase. The text prompts are then fed into the frozen text encoder to obtain degenerate text features. Based on the degenerate text, the normalized degenerate embedding is constrained and aligned to obtain the constrained and aligned degenerate embedding.

[0042] To ensure that the degenerate embedding z has a clear degeneracy discriminative meaning, natural language text prompts are constructed during the training phase based on sample degeneracy labels or degeneracy synthesis strategies. For example, ... Figure 2 As shown, a degraded description containing the keywords "noise" and "low-light" is constructed for samples containing both noise and low light. This text prompt is input into a frozen text encoder to obtain degraded text features. Simultaneously, the degraded embedding z obtained in step S12 is normalized, the cosine similarity between the visual degraded embedding and the degraded text features is calculated, and alignment is performed using bidirectional InfoNCE loss (image-to-text and text-to-image).

[0043] in, Indicates the number of image samples. This represents the total number of candidate text features. Indicates the first -th image features and the first - Similarity scores between text features It is a temperature parameter that controls the sharpness of the similarity distribution. This represents the set of positive samples corresponding to the same degradation category.

[0044] The loss for symmetric text-to-image conversion is calculated analogously, resulting in the final degenerate alignment loss. Defined as:

[0045] This constraint allows samples with the same degradation type to cluster in the embedding space, while samples with different degradation types or different combinations of degradation are separated from each other, enabling the network to obtain a degradation modulation signal with semantic interpretability.

[0046] S14. The constrained and aligned degenerate embeddings are input into linear mapping layers of the same scale as the multi-scale hierarchical spatial features to generate channel scaling parameters and offset parameters for each scale. The channel scaling parameters and offset parameters are applied to the visible light multi-scale hierarchical spatial features and infrared multi-scale hierarchical spatial features of the corresponding scale to obtain degenerate and enhanced multi-scale hierarchical spatial features.

[0047] like Figure 2 As shown, the degradation embedding Linear mapping layers corresponding to high-scale, mid-scale, and low-scale hierarchical spatial features are input separately to generate channel scaling parameters for each scale. and offset parameters Subsequently, and By applying feature-level linear modulation to the visible and infrared multi-scale hierarchical spatial features at corresponding scales, the channel response is adjusted to obtain degraded and enhanced multi-scale hierarchical spatial features. Taking the low scale as an example, this process is expressed by the following equation:

[0048] For channels heavily affected by noise, rain, fog, or low light, the modulation parameters reduce their response intensity; for channels containing target contours, texture edges, thermally significant regions, and stable structures, the modulation parameters enhance their response. Thus, the network can initially suppress degradation-related interference and enhance effective information before proceeding to subsequent fusion and temporal modeling.

[0049] S2: A learnable dictionary is pre-built and trained. The quantization prototype most relevant to the degenerate-enhanced multi-scale hierarchical spatial features is retrieved from the trained learnable dictionary. Cross-attention is performed based on the quantization prototype and the degenerate-enhanced multi-scale hierarchical spatial features to obtain the visible light features and infrared features after dictionary interaction enhancement, and then fused to obtain the fused features.

[0050] In one embodiment, the pre-construction and training of the learnable dictionary in step S2 includes the following steps: Acquire high-quality infrared and high-quality visible light samples to construct learnable dictionaries for each type of sample; S21. Input any high-quality infrared sample or high-quality visible light sample into a pre-trained visual geometric convolutional neural network to extract visual features. Then, input the sample into a frozen self-supervised visual base model to extract semantic features. Finally, fuse the visual features and semantic features to obtain high-quality visual semantic features.

[0051] It should be noted that high-quality samples refer to samples that are free from significant degradation due to rain, fog, noise, low light, motion blur, etc., and whose target outlines, texture edges, and thermal target responses are clear.

[0052] In one specific embodiment, the visual ensemble convolutional neural network is a Vgg19 network, and the self-supervised visual base model is a Dinov2 backbone network.

[0053] Specifically, for any high-quality sample First, a pre-trained Vgg19 network is used to extract visual features containing edges, textures, and local structures. Then, the frozen Dinov2 backbone network is used to extract semantic features containing target category, scene layout, and regional semantic relationships. Because the two have different channel dimensions and expression levels, they are first passed through... Convolution or linear mapping maps structural visual features to a dimension consistent with semantic features, and then multi-head attention is used to inject semantic context into the structural visual features, resulting in high-quality visual semantic features. :

[0054] in, As a multi-head attention mechanism, Weight matrix.

[0055] S22, initialize a set of learnable vectors as a learnable dictionary, and select the dictionary prototype that is closest to the high-quality visual semantic features as the quantization representation.

[0056] In a learnable dictionary Choose the nearest dictionary prototype as the quantization representation :

[0057] in, This is a vector quantization operation.

[0058] Therefore, each dictionary prototype learns reliable structural and semantic patterns in common scenarios step by step.

[0059] S23, input the quantized representation into the image reconstruction encoder to recover a high-quality image.

[0060] S24, input the high-quality image into the semantic segmentation decoder to obtain the predicted segmentation result.

[0061] S25: Calculate the reconstruction loss based on the restored high-quality image and real high-quality samples; calculate the cross-entropy loss based on the predicted segmentation results and corresponding real labels; calculate the vector quantization constraint loss based on high-quality visual semantic features and quantization representation; and train the learnable dictionary based on the reconstruction loss, cross-entropy loss and vector quantization constraint loss.

[0062] Steps S23-S25 are as follows: Configure the image reconstruction decoder to quantize features. Restore to a high-quality input image, and with reconstruction loss The constraint dictionary preserves texture, edge, and hot object structure. Secondly, a semantic segmentation decoder is set up to quantize the features. Predicted as pixel-level semantic labels, and using cross-entropy loss. The constraint dictionary maintains the semantic separability of categories such as vehicles, pedestrians, roads, buildings, sky, and background; finally, vector quantization constraints are introduced. :

[0063] in, This represents the gradient stopping symbol.

[0064] Vector quantization constraint loss brings high-quality visual semantic features closer to the selected dictionary prototypes in the feature space, while stabilizing dictionary updates by stopping gradient operations. After these optimizations, the prototypes in the dictionary can serve as both visual structural priors and high-level semantic priors.

[0065] After the learnable dictionary is trained, its parameters are fixed and used as a priori library for the main fusion network. Figure 3 This is a schematic diagram of the visual semantic prior interaction process provided in an embodiment of the present invention, such as... Figure 3 As shown, for the current degraded and enhanced visible light multi-scale hierarchical spatial features and the degraded and enhanced infrared multi-scale hierarchical spatial features, the most relevant quantization prototype is first dynamically retrieved based on the distance or similarity between the feature and the dictionary prototype. Subsequently, the degraded modulated features (i.e., degraded enhanced visible light multi-scale hierarchical spatial features and degraded enhanced infrared multi-scale hierarchical spatial features) are used as queries, and the retrieved dictionary prototypes are used as keys and values ​​to perform cross-attention. Cross-attention adaptively selects the high-quality structures and semantic patterns to be injected based on the current content, thereby compensating for local information loss caused by rain, fog, low illumination, or noise, resulting in visible light features enhanced through dictionary interaction. and infrared features Finally, the visible light features (visible light texture and color information) enhanced by dictionary interaction and the infrared features (infrared thermal target information) enhanced by dictionary interaction are integrated through modality-aware attention, and the consistency of semantic structure in adjacent frames is further constrained through temporal cross-attention. This process can be represented as:

[0066] in, For time-crossing attention mechanism, These are the initial fusion features.

[0067] S3 calculates temporal attention based on fusion features, adds the temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction to obtain forward-enhanced features or backward-enhanced features, concatenates the fusion features, forward-enhanced features and backward-enhanced features to obtain bidirectional enhanced fusion features, and performs spatiotemporal aggregation on the bidirectional enhanced fusion features to obtain spatiotemporal fusion features.

[0068] In one embodiment, step S3 includes the following steps: S31. Calculate the difference features based on the fusion features of the current frame and the fusion features of adjacent frames, and calculate the temporal attention based on the difference features and the fusion features of adjacent frames to obtain the temporal attention output.

[0069] Specifically, regarding the video, the first... Frame fusion features The features are fused with those of neighboring frames (when propagating forward, the features are fused with those of the previous frame). When propagating backwards, it becomes the fused feature of the next frame. First, calculate the difference features between the two. Differential features can highlight regions where changes occur between adjacent frames, including moving targets, target boundaries, occlusion changes, and areas with missing information due to transient degradation within a frame. Subsequently, the differential features are mapped to attention queries, and the features of neighboring frames are mapped to keys and values, thus calculating the temporal attention output of the differential-aware algorithm. :

[0070] Unlike directly aggregating neighboring features of an entire frame, this process selects cross-frame information based on whether the change is useful to the current frame, enabling the network to prioritize extracting effective temporal cues from adjacent frames that can compensate for the current frame, while reducing redundant interference from static backgrounds and repetitive textures.

[0071] S32 adds the temporal attention output, the memory compensation term for forward propagation, the fused features, and the visible light and infrared features enhanced by dictionary interaction to obtain the forward-enhanced features.

[0072] S33 adds the temporal attention output, the memory compensation term for backpropagation, the fused features, and the visible light and infrared features enhanced by dictionary interaction to obtain the back-enhanced features.

[0073] S34. The fusion feature, forward enhancement feature and backward enhancement feature are concatenated to obtain the bidirectional enhanced fusion feature. The bidirectional enhanced fusion feature is then spatiotemporally aggregated to obtain the spatiotemporal fusion feature.

[0074] In order to utilize contextual information over a long period of time, embodiments of the present invention set up a lightweight memory state. It serves as a memory compensation item to preserve historical characteristics that have already been disseminated.

[0075] It should be noted that when performing forward propagation, the difference features are calculated based on the fusion features of the current frame and the fusion features of the previous frame; the memory compensation term for forward propagation includes the fusion features of the previous frame and the fusion features after bidirectional enhancement; when performing backward propagation, the difference features are calculated based on the fusion features of the current frame and the fusion features of the next frame; the memory compensation term for backward propagation includes the fusion features of the next frame and the fusion features after bidirectional enhancement.

[0076] Figure 4 The flowchart of time-series dynamic modeling and bidirectional memory propagation provided in the embodiments of the present invention is as follows: Figure 4 As shown, the processing of the first At frame rate, first process the lightweight memory state. Perform convolution or linear transformation to obtain a memory compensation term with the same feature dimension as the current frame; then add the current frame features (fused features, visible light features enhanced by dictionary interaction, and infrared features), the differential attention output obtained in step S31, and the memory compensation term to obtain the forward-enhanced features. The memory state is then updated jointly by the current memory and the features of neighboring frames:

[0077] in To update the network for memory, For the updated memory compensation term, the historical context is recursively accumulated according to the above formula.

[0078] Similarly, a backpropagation is performed from the end of the video to the beginning frame to obtain future contextual information. Finally, the fused features, forward enhancement features, and backward enhancement features are concatenated along the channel dimension and input into a 3D convolution for integration, so that each frame utilizes effective information from both the past and the future.

[0079] In one embodiment, spatiotemporal aggregation of the bidirectional enhanced fusion features is performed to obtain spatiotemporal fusion features, including the following steps: S341, the bidirectional enhanced fused features are divided into multiple groups along the channel dimension, of which one-quarter of the channels are shifted forward along the time dimension to receive the information of the previous frame; one-quarter of the channels are shifted backward along the time dimension to receive the information of the next frame; the remaining channels retain the information of the current frame unchanged, as expressed by the following formula:

[0080] in, The fusion features are after time-shifting. This represents the fusion feature shifted forward along the time dimension. This represents the fusion feature shifted along the time dimension. Indicates fusion features that remain unchanged. This represents the total number of channels in the fusion feature after bidirectional enhancement. The current number of channels for the fusion features.

[0081] This timing shift operation can complete the feature exchange between adjacent frames without introducing additional learnable parameters.

[0082] S342 inputs the temporally shifted fusion features into a 3D convolutional layer to obtain spatiotemporal fusion features.

[0083] The shifted features are input into a 3D convolutional layer, which simultaneously learns local motion patterns, cross-frame structural continuity, and degradation compensation relationships in both spatial and temporal dimensions, resulting in more stable spatiotemporal fusion features.

[0084] S4, based on fusion features and spatiotemporal fusion features, reconstructs the fused video to obtain the fused video.

[0085] In one embodiment, step S4 includes the following steps: S41, after upsampling the fused features, concatenate them with the spatiotemporal fused features in the channel dimension to obtain the spliced ​​features.

[0086] S42 sequentially passes the stitched features through a convolutional layer, an activation layer, and a reconstruction head to obtain the fused video.

[0087] Specifically, the low-scale fusion features are first upsampled to match their spatial dimensions with the high-scale spatiotemporal fusion features. Then, the two are concatenated along the channel dimension and gradually mapped back to the image domain through convolutional layers, activation layers, and a reconstruction head to obtain the final fused video. The reconstruction process utilizes high-scale features to restore sharp edges, texture details, and thermal target outlines, while using low-scale visual semantic features to maintain scene layout, target categories, and regional semantic structure, thereby generating a fused video that combines visual fidelity and semantic reliability.

[0088] The above method can be applied to video fusion models, which perform end-to-end optimization using the original high-quality video or non-degraded video as constraint anchor points. The video fusion model is jointly trained based on a total loss, which includes: degraded text alignment loss. Dictionary learning loss is used to constrain the consistency between degenerate embeddings and text degenerate cues. Used to constrain the reconstruction capability, semantic discrimination capability, and quantization stability of the visual semantic dictionary, with content preservation loss. Temporal consistency loss is used to constrain the fusion of videos to preserve significant intensity information and gradient details from infrared and visible light source videos. This is used to constrain the direction of change between adjacent frames of the fused video to remain consistent with the source video, reducing brightness flicker, structural abruptness, and color consistency loss. This constraint is used to preserve the chromaticity information of the visible light video in the fusion result. The total loss is expressed as:

[0089] in, , , and Hyperparameters are used to balance various losses.

[0090] In practical implementation, a robust multimodal video fusion method based on semantic temporal collaborative modeling can be implemented using software. Through degradation alignment, structural semantic priors, and differential temporal memory propagation mechanisms, high-quality fusion of multimodal videos can be achieved in all-weather and harsh environments.

[0091] Figure 5 This is a schematic diagram comparing the method of the present invention with existing methods, as shown in the embodiments of the present invention. Figure 5 As shown, compared with existing methods, this invention not only mines the complementary features of multiple modes, but also performs adaptive feature adjustment around the degradation state of the current video, scene structure and cross-frame changes, which significantly improves the robustness of the model in dynamic and complex environments. Figure 6 This diagram illustrates a comparison between the method of this invention and other advanced fusion methods in certain scenarios provided by embodiments of the invention. Existing methods are generally affected by degradation interference during the fusion process (such as low light and blurring in infrared-visible light fusion), leading to problems such as information loss, temporal flicker, and semantic deficiencies in the fusion results. In contrast, this invention, through text-guided degradation alignment, explicit constraints of the visual semantic dictionary, and joint optimization of differential temporal modeling, can actively block invalid degradation noise and repair spatial structure deficiencies and temporal artifacts. In the highly challenging all-weather infrared and visible light video fusion task, this invention can stably generate fused videos with clear details, complete structure, and smooth temporal sequence, significantly improving the fused visual perception quality in complex real-world scenarios and the reliability of downstream intelligent perception tasks.

[0092] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a robust multimodal video fusion device based on semantic temporal collaborative modeling. Specifically, the robust multimodal video fusion device based on semantic temporal collaborative modeling can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0093] Please see Figure 7 , Figure 7This invention specifically describes a robust multimodal video fusion device based on semantic-temporal collaborative modeling, applied in electronic devices. The robust multimodal video fusion device based on semantic-temporal collaborative modeling may include: The multi-scale feature extraction and degradation perception module is used to extract multi-scale hierarchical spatial features from visible light video sequences and infrared video sequences respectively, obtain degradation embeddings based on multi-scale hierarchical spatial features, and perform multi-scale degradation modulation on multi-scale hierarchical spatial features based on the constraint-aligned degradation embeddings to obtain degradation-enhanced multi-scale hierarchical spatial features. The visual-semantic interaction module is used to pre-build and train a learnable dictionary. It retrieves the quantization prototype most relevant to the degenerate-enhanced multi-scale hierarchical spatial features from the trained learnable dictionary. Based on the quantization prototype and the degenerate-enhanced multi-scale hierarchical spatial features, it performs cross-attention to obtain the visible light features and infrared features enhanced by dictionary interaction, and then fuses them to obtain the fused features. The differential temporal modeling and spatiotemporal aggregation module is used to calculate temporal attention based on fusion features. It adds the temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction to obtain forward-enhanced features or backward-enhanced features. It concatenates the fusion features, forward-enhanced features, and backward-enhanced features to obtain bidirectional enhanced fusion features. It then performs spatiotemporal aggregation on the bidirectional enhanced fusion features to obtain spatiotemporal fusion features. The reconstruction module is used to reconstruct the fused video based on fusion features and spatiotemporal fusion features.

[0094] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0095] In addition, this embodiment of the invention also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the robust multimodal video fusion method based on semantic-temporal collaborative modeling provided in this embodiment of the invention. Therefore, it can achieve the beneficial effects that any robust multimodal video fusion method based on semantic-temporal collaborative modeling provided in this embodiment of the invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0096] Figure 8The diagram illustrates a specific structural block diagram of an electronic device provided in an embodiment of the present invention. This electronic device can be used to implement the robust multimodal video fusion method based on semantic-temporal collaborative modeling provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0097] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0098] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control.

[0099] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0100] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0101] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0102] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0103] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Multi-scale hierarchical spatial features are extracted from visible light video sequences and infrared video sequences respectively. Degenerate embeddings are obtained based on the multi-scale hierarchical spatial features. Multi-scale degradation modulation is performed on the multi-scale hierarchical spatial features based on the constraint-aligned degradation embeddings to obtain degenerate-enhanced multi-scale hierarchical spatial features. A learnable dictionary is pre-built and trained. The quantization prototype most relevant to the degenerate-enhanced multi-scale hierarchical spatial features is retrieved from the trained learnable dictionary. Cross attention is performed based on the quantization prototype and the degenerate-enhanced multi-scale hierarchical spatial features to obtain the visible light features and infrared features after dictionary interaction enhancement. These features are then fused to obtain the fused features. Temporal attention is calculated based on fusion features. The temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction are added to obtain forward-enhanced features or backward-enhanced features. The fusion features, forward-enhanced features and backward-enhanced features are concatenated to obtain bidirectional enhanced fusion features. The bidirectional enhanced fusion features are spatiotemporally aggregated to obtain spatiotemporal fusion features. The fused video is obtained by reconstructing based on fusion features and spatiotemporal fusion features.

[0104] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0105] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the robust multimodal video fusion method based on semantic-temporal collaborative modeling provided by the present invention.

[0106] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0107] Since the instructions stored in the storage medium can execute the steps in any embodiment of the robust multimodal video fusion method based on semantic-temporal collaborative modeling provided in the embodiments of the present invention, the beneficial effects that any robust multimodal video fusion method based on semantic-temporal collaborative modeling provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0108] The foregoing has provided a detailed description of a robust multimodal video fusion method, apparatus, storage medium, and electronic device based on semantic temporal collaborative modeling provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A robust multimodal video fusion method based on semantic-temporal collaborative modeling, characterized in that, include: Multi-scale hierarchical spatial features are extracted from visible light video sequences and infrared video sequences respectively. Degenerate embeddings are obtained based on the multi-scale hierarchical spatial features. Multi-scale degradation modulation is performed on the multi-scale hierarchical spatial features based on the constraint-aligned degradation embeddings to obtain degenerate-enhanced multi-scale hierarchical spatial features. A learnable dictionary is pre-built and trained. The quantization prototype most relevant to the degenerate-enhanced multi-scale hierarchical spatial features is retrieved from the trained learnable dictionary. Cross attention is performed based on the quantization prototype and the degenerate-enhanced multi-scale hierarchical spatial features to obtain the visible light features and infrared features after dictionary interaction enhancement. These features are then fused to obtain the fused features. Temporal attention is calculated based on fusion features. The temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction are added to obtain forward-enhanced features or backward-enhanced features. The fusion features, forward-enhanced features and backward-enhanced features are concatenated to obtain bidirectional enhanced fusion features. The bidirectional enhanced fusion features are spatiotemporally aggregated to obtain spatiotemporal fusion features. The fused video is obtained by reconstructing based on fusion features and spatiotemporal fusion features.

2. The robust multimodal video fusion method based on semantic-temporal collaborative modeling according to claim 1, characterized in that, Multi-scale hierarchical spatial features are extracted from visible light video sequences and infrared video sequences respectively. Degraded embeddings are obtained based on these multi-scale hierarchical spatial features. Multi-scale degradation modulation is then applied to the multi-scale hierarchical spatial features based on the constrained aligned degradation embeddings to obtain degraded and enhanced multi-scale hierarchical spatial features, including: Visible light video sequences are input into a visible light encoder to obtain visible light multi-scale hierarchical spatial features, and infrared video sequences are input into an infrared encoder to obtain infrared multi-scale hierarchical spatial features. Each scale feature in the multi-scale hierarchical spatial features is divided into multiple spatial regions. The average response, significant response, and structural change response of each region are calculated. The responses of different scales and regions are concatenated in the channel dimension and input into a multilayer perceptron for nonlinear mapping to generate degenerate embeddings. Text prompts are constructed in advance based on samples during the training phase. The text prompts are then fed into a frozen text encoder to obtain degenerate text features. Based on the degenerate text, the normalized degenerate embeddings are constrained and aligned to obtain constrained and aligned degenerate embeddings. The constrained and aligned degenerate embeddings are input into linear mapping layers of the same scale as the multi-scale hierarchical spatial features to generate channel scaling parameters and offset parameters for each scale. The channel scaling parameters and offset parameters are then applied to the visible light multi-scale hierarchical spatial features and infrared multi-scale hierarchical spatial features of the corresponding scales to obtain degenerate and enhanced multi-scale hierarchical spatial features.

3. The robust multimodal video fusion method based on semantic temporal collaborative modeling according to claim 1, characterized in that, A pre-built and trained learnable dictionary includes: Acquire high-quality infrared and high-quality visible light samples to construct learnable dictionaries for each type of sample; Any high-quality infrared sample or high-quality visible light sample is input into a pre-trained visual geometric convolutional neural network to extract visual features. Then, the sample is input into a frozen self-supervised visual base model to extract semantic features. The visual features and the semantic features are fused to obtain high-quality visual semantic features. Initialize a set of learnable vectors as a learnable dictionary, and select the dictionary prototype that is closest to the high-quality visual semantic features as the quantization representation in the learnable dictionary; The quantized representation is input into the image reconstruction encoder to recover a high-quality image; The high-quality image is input into the semantic segmentation decoder to obtain the predicted segmentation result; The reconstruction loss is calculated based on the restored high-quality image and real high-quality samples. The cross-entropy loss is calculated based on the predicted segmentation result and the corresponding real label. The vector quantization constraint loss is calculated based on the high-quality visual semantic features and the quantization representation. The learnable dictionary is trained based on the reconstruction loss, the cross-entropy loss and the vector quantization constraint loss.

4. The robust multimodal video fusion method based on semantic-temporal collaborative modeling according to claim 1, characterized in that, Temporal attention is calculated based on fusion features. The temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction are added to obtain forward-enhanced features or backward-enhanced features. The fusion features, forward-enhanced features, and backward-enhanced features are concatenated to obtain bidirectionally enhanced fusion features. Spatiotemporal aggregation is performed on the bidirectionally enhanced fusion features to obtain spatiotemporal fusion features, including: The difference features are calculated based on the fusion features of the current frame and the fusion features of the adjacent frames. The temporal attention is then calculated based on the difference features and the fusion features of the adjacent frames to obtain the temporal attention output. The forward-enhanced feature is obtained by adding the temporal attention output, the memory compensation term for forward propagation, the fused feature, the visible light feature enhanced by dictionary interaction, and the infrared feature. The temporal attention output, the backward propagation memory compensation term, the fused features, the visible light features enhanced by dictionary interaction, and the infrared features are added together to obtain the backward enhancement features; The fusion feature, the forward enhancement feature, and the backward enhancement feature are concatenated to obtain the bidirectional enhanced fusion feature. The bidirectional enhanced fusion feature is then spatiotemporally aggregated to obtain the spatiotemporal fusion feature.

5. The robust multimodal video fusion method based on semantic-temporal collaborative modeling according to claim 4, characterized in that, When propagating forward, differential features are calculated based on the fusion features of the current frame and the fusion features of the previous frame; The forward propagation memory compensation term includes the fusion features of the previous frame and the fusion features after bidirectional enhancement. When backpropagation is performed, differential features are calculated based on the fusion features of the current frame and the fusion features of the next frame; the memory compensation term for backpropagation includes the fusion features of the next frame and the fusion features after bidirectional enhancement.

6. The robust multimodal video fusion method based on semantic-temporal collaborative modeling according to claim 4, characterized in that, Spatiotemporal aggregation of the bidirectional enhanced fusion features yields spatiotemporal fusion features, including: The fused features after bidirectional enhancement are divided into multiple groups along the channel dimension: in, The fusion features are after time-shifting. This represents the fusion feature shifted forward along the time dimension. This represents the fusion feature shifted along the time dimension. Indicates fusion features that remain unchanged. This represents the total number of channels in the fusion feature after bidirectional enhancement. The current number of channels for the fused features; The temporally shifted fusion features are input into a 3D convolutional layer to obtain spatiotemporal fusion features.

7. The robust multimodal video fusion method based on semantic-temporal collaborative modeling according to claim 1, characterized in that, The fused video is reconstructed based on fusion features and spatiotemporal fusion features, including: The fusion feature is upsampled and concatenated with the spatiotemporal fusion feature in the channel dimension to obtain the spliced ​​feature; The stitching features are sequentially passed through a convolutional layer, an activation layer, and a reconstruction head to obtain a fused video.

8. A robust multimodal video fusion device based on semantic-temporal collaborative modeling, characterized in that, include: The multi-scale feature extraction and degradation perception module is used to extract multi-scale hierarchical spatial features from visible light video sequences and infrared video sequences respectively, obtain degradation embeddings based on multi-scale hierarchical spatial features, and perform multi-scale degradation modulation on multi-scale hierarchical spatial features based on the constraint-aligned degradation embeddings to obtain degradation-enhanced multi-scale hierarchical spatial features. The visual-semantic interaction module is used to pre-build and train a learnable dictionary. It retrieves the quantization prototype most relevant to the degenerate-enhanced multi-scale hierarchical spatial features from the trained learnable dictionary. Based on the quantization prototype and the degenerate-enhanced multi-scale hierarchical spatial features, it performs cross-attention to obtain the visible light features and infrared features enhanced by dictionary interaction, and then fuses them to obtain the fused features. The differential temporal modeling and spatiotemporal aggregation module is used to calculate temporal attention based on fusion features. It adds the temporal attention output, memory compensation term, fusion features, and visible light and infrared features enhanced by dictionary interaction to obtain forward-enhanced features or backward-enhanced features. It concatenates the fusion features, forward-enhanced features, and backward-enhanced features to obtain bidirectional enhanced fusion features. It then performs spatiotemporal aggregation on the bidirectional enhanced fusion features to obtain spatiotemporal fusion features. The reconstruction module is used to reconstruct the fused video based on fusion features and spatiotemporal fusion features.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the robust multimodal video fusion method based on semantic temporal collaborative modeling as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The device includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps of the robust multimodal video fusion method based on semantic temporal collaborative modeling as described in any one of claims 1 to 7.

Images