Multi-modal visual fusion method and device based on double teacher distillation and self-supervision

Abstract

The application relates to a kind of multi-modal visual fusion methods and devices based on double teacher distillation and self-supervision, to solve the problems that existing methods lack unified framework, time sequence stability is poor, structure information degeneration.The method comprises the following steps: collecting multi-modal visual fusion dataset;Two-stage training process is constructed: the first stage trains auto-encoder, realizes knowledge transfer through double teacher distillation and time sequence matcher, combines reconstruction loss self-supervised optimization student model;The second stage freezes the encoder, embeds cross attention fusion module, optimizes the fusion network through adaptive gradient loss, second-order time smoothing loss, perception loss, consistency loss;When testing, output fusion result and performance index.The application realizes dynamic and static scene unified fusion, improves fusion quality and time sequence stability, and is suitable for target tracking, unmanned driving and the like.

Description

Technical Field

[0001] This application relates to the field of multimodal visual fusion technology, and in particular to a multimodal visual fusion method and apparatus based on dual-teacher distillation and self-supervision. Background Technology

[0002] In real-world scenarios, different sensors are needed to detect and perceive target scenes. For example, visible light sensors can provide high-definition, high-resolution image information in good environments, but their anti-interference capabilities are weak. Infrared sensors, relying on their thermal radiation recognition characteristics, can effectively detect thermal targets in complex environments (such as fog, rain, and nighttime), but they suffer from low spatial resolution and limited scene details. Multimodal visual fusion technology can extract effective information and eliminate redundant information from source images from different sensors, generating a fused image with richer information and stronger perception capabilities, thereby enhancing the understanding of real-world scenes.

[0003] However, existing multimodal visual fusion methods have the following key problems: First, there is a lack of a unified fusion framework. Existing methods employ different modeling paradigms for video sequences and single-frame scenes, failing to establish a universal and unified framework. In dynamic scenes, methods sensitive to subtle changes between frames are susceptible to temporal instability and flicker artifacts. Although there are video fusion methods with explicit temporal modeling that can improve temporal consistency, they rely on the assumption of "smooth and continuous changes between adjacent frames," resulting in weak generalization ability on complex or unseen datasets. Furthermore, because they require explicit modeling of inter-frame relationships, they cannot be directly applied to single-frame scenes.

[0004] Second, the teacher model and the fusion task objectives are inconsistent. Existing methods, when introducing pre-trained teacher models, often directly inherit their feature representation capabilities, neglecting the inherent differences between the teacher model's optimization objectives and the requirements of the fusion task. Mainstream pre-trained teacher models (such as those used for classification and recognition) emphasize high-level semantic discriminative information, lacking sufficient modeling ability for low-level structural and detailed information; while the core requirement of multimodal visual fusion is "preserving the structural consistency and fine-grained details of different modal inputs." This difference in objectives can easily lead to the student model being excessively influenced by the semantic bias of the teacher model during the distillation process, causing structural information degradation.

[0005] Therefore, there is an urgent need for a unified multimodal visual fusion method that can ensure the temporal stability of video fusion in dynamic scenes and generate high-quality fusion results in both dynamic and static environments, thereby promoting the implementation of multimodal visual fusion in real-world scenarios. Summary of the Invention

[0006] Therefore, it is necessary to provide a multimodal visual fusion method and apparatus based on dual-teacher distillation and self-supervision to address the aforementioned technical problems.

[0007] Firstly, this application provides a multimodal visual fusion method based on dual-teacher distillation and self-supervision. The method includes: We collected multiple multimodal visual fusion datasets, including video fusion datasets and image fusion datasets. The video fusion datasets were divided into training, validation, and test sets, while the image fusion datasets were all used as test sets. The training and test data were independent of each other. A training process for constructing a multimodal visual fusion model is described, which includes two stages. The first stage trains an autoencoder model, which includes an encoder and a decoder. The second stage freezes the encoder in the autoencoder architecture. The autoencoder and the intermediate embedded fusion module constitute a multimodal visual fusion network. In the first stage, a dual-teacher knowledge distillation learning strategy is adopted to obtain representations from the teacher network consisting of an image-based pre-trained large model and a video-based pre-trained large model. At the same time, a temporal matcher is constructed to align and match the temporal features output by the video-based pre-trained large model with the temporal features output by the student model. The mean absolute error loss is introduced to measure the difference between the reconstructed image and the input image and to serve as the self-supervised training objective. Under the joint constraints of the self-supervised signal and the distillation supervision signal, the weights of the student model are obtained and saved based on the video fusion dataset. In the second stage, adaptive gradient loss and second-order temporal smoothing constraint loss are designed for the multimodal visual fusion network. Combined with consistency loss and perceptual loss, the weights of the multimodal visual fusion model are obtained and saved based on the video fusion dataset. During testing, the video fusion dataset test set and the image fusion dataset are input into the multimodal visual fusion model to obtain the fusion results, and the performance metrics of the fusion results are calculated.

[0008] Optionally, in one embodiment of this application, after collecting multiple multimodal visual fusion datasets, the method further includes: Perform frame interpolation on low frame rate multimodal video datasets; Data augmentation techniques are employed during the model training phase, including random flipping, random rotation, and random cropping.

[0009] Optionally, in one embodiment of this application, the autoencoder model uses ViT as its skeleton, and each layer of the encoder and decoder is composed of ViT blocks. The working process of the autoencoder model includes: The input data is linearly mapped to obtain an initial feature representation; The initial feature representation is obtained by hierarchical feature extraction through the encoder to obtain multi-scale features, and each encoder layer is composed of stacked ViT blocks; Multi-scale features are reconstructed hierarchically through a decoder, with each decoder layer consisting of stacked ViT blocks. Finally, the reconstructed image is obtained through inverse linear mapping and reconstruction function.

[0010] Optionally, in one embodiment of this application, the intermediate embedded fusion module is a cross-attention module, and the working process of the cross-attention module includes: Multimodal features generate corresponding queries, keys, and values; The single attention head is calculated by normalizing the product of the query and the transpose of the key using the Softmax function and then multiplying it by the value. Multiple attention heads are concatenated and mapped back to the original dimension to obtain a multi-head cross-attention output; Multi-head self-attention output is obtained through residuals and layer normalization. The multi-head self-attention output is then processed by a feedforward neural network and reinforced with residuals and layer normalization to obtain enhanced features. The enhanced features of the two modalities are input into a fully connected layer for fusion to obtain the fused features.

[0011] Optionally, in one embodiment of this application, the dual-teacher knowledge distillation learning strategy includes: A projection head is attached after each encoder layer, which is implemented using a lightweight multilayer perceptron and activated only during training; The outputs of all projection heads are aggregated to form an enhanced distillation representation; Distillation loss is a combination of cosine constraints and smoothing L1 constraints imposed on student and teacher characteristics; A teacher discard regularization strategy is introduced, assigning binary coefficients to each teacher model to control whether it participates in the distillation process.

[0012] Optionally, in one embodiment of this application, the operation of the timing matcher includes: A model capable of predicting bidirectional optical flow is used to estimate the reverse optical flow between two adjacent frames; Pixel-level optical flow is mapped and merged onto a block mesh to obtain a block-level optical flow representation; The block-level optical flow representation is used to perform temporal alignment of features in adjacent frames, and the aligned features are then fused with the features of the current frame.

[0013] Optionally, in one embodiment of this application, the total loss function of the first stage is the sum of the mean absolute error loss and the distillation loss, wherein the mean absolute error loss is calculated using the L1 norm to determine the difference between the reconstructed image and the input image.

[0014] Optionally, in one embodiment of this application, the calculation process of the adaptive gradient loss includes: The gradient magnitude is defined as the sum of the absolute responses along the horizontal and vertical directions; Based on the difference in gradient magnitude between the two modal inputs, the Sigmoid function is used to obtain pixel-wise adaptive weights; The target gradient is a weighted combination of the gradients of the two modal inputs; The gradient loss is the L1 norm of the fused image gradient and target gradient.

[0015] Optionally, in one embodiment of this application, the second-order temporal smoothing constraint loss is the sum of the L1 norms of the inter-frame second-order difference of the fusion result and the inter-frame second-order difference of the two modal inputs. The perceptual loss is a weighted sum of the fusion result and the L1 norm of the feature representations of the two modal inputs at each layer; The consistency loss is the sum of the fusion result and the L1 norms of the two modal inputs, with the L1 norm of one of the modal inputs multiplied by the weighting coefficient. The total loss function in the second stage is the sum of adaptive gradient loss, second-order time smoothing constraint loss, perceptual loss, and consistency loss.

[0016] Secondly, this application also provides a multimodal visual fusion device based on dual-teacher distillation and self-supervision. The device includes: The dataset collection and processing module is used to collect multiple multimodal visual fusion datasets, including video fusion datasets and image fusion datasets. The video fusion datasets are divided into training sets, validation sets, and test sets, and the image fusion datasets are all used as test sets. The training and test data are independent of each other. A multimodal visual fusion model training process construction module is used to construct a multimodal visual fusion model training process. The process includes two stages: the first stage trains an autoencoder model, which includes an encoder and a decoder; the second stage freezes the encoder in the autoencoder architecture, and the autoencoder and the intermediate embedded fusion module constitute a multimodal visual fusion network. The first-stage training module of the model employs a dual-teacher knowledge distillation learning strategy in the first stage. It acquires representations from the teacher network consisting of an image-based pre-trained large model and a video-based pre-trained large model. Simultaneously, it constructs a temporal matcher to align and match the temporal features output by the video-based pre-trained large model with the temporal features output by the student model. It introduces the mean absolute error loss to measure the difference between the reconstructed image and the input image and uses it as the self-supervised training objective. Under the joint constraints of the self-supervised signal and the distillation supervision signal, the student model weights are obtained and saved based on the video fusion dataset. The second-stage training module of the model is used in the second stage to design adaptive gradient loss and second-order temporal smoothing constraint loss for the multimodal visual fusion network. It combines consistency loss and perceptual loss to train and save the weights of the multimodal visual fusion model based on the video fusion dataset. The multimodal visual fusion module is used to input the video fusion dataset test set and the image fusion dataset into the multimodal visual fusion model during testing to obtain the fusion results and calculate the performance metrics of the fusion results.

[0017] Compared with the prior art, the beneficial effects of the present invention are: First, dual-teacher distillation enables complementary knowledge transfer: through the heterogeneous combination of "image-based pre-trained large model" and "video-based pre-trained large model", the student model learns complementary knowledge of structural details and spatiotemporal representation; in particular, the video-based teacher model can provide spatiotemporal structural priors, improving the fusion stability of dynamic scenes.

[0018] Second, joint distillation and self-supervised optimization of feature representation: In the first stage, through the joint effect of self-supervised constraints (reconstruction loss) and distillation supervision (dual teacher loss), features with consistent structure and good alignment are generated in the shared feature space, taking into account both fine-grained details and semantic information, and avoiding structural degradation caused by semantic bias of the teacher model.

[0019] Third, second-order temporal smoothing constraints improve temporal consistency: by modeling the changing trends of consecutive frames through second-order inter-frame differences, temporal jitter caused by local motion or perceptual inconsistencies is suppressed, ensuring the temporal stability of video fusion.

[0020] Fourth, a unified framework adapts to dynamic and static scenarios: the two-stage training process does not require adjustments to the modeling paradigm for video / single-frame scenarios, and can be directly adapted to video fusion and image fusion tasks, improving the versatility and generalization of the method. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multimodal visual fusion method based on dual-teacher distillation and self-supervision in one embodiment. Figure 2 This is a schematic diagram of the model framework used in the first stage of training in one embodiment; Figure 3 This is a schematic diagram of the model framework used in the second stage of training in one embodiment; Figure 4 This is a schematic diagram of the temporal matcher used in the first stage of training in one embodiment. Figure 5 This is a structural block diagram of a multimodal visual fusion device based on dual-teacher distillation and self-supervision in one embodiment; Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] In one embodiment, such as Figure 1 As shown, a multimodal visual fusion method based on dual-teacher distillation and self-supervision is provided, including the following steps: S101: Collect multiple multimodal visual fusion datasets. Multimodal video datasets need to be divided into training, validation, and test sets. Multimodal image datasets are all used as test sets. These data are independent and invisible to each other during training and testing.

[0024] S102: Construct a multimodal visual fusion model training process, which includes two stages: In the first stage, train an autoencoder model, which includes an encoder and a decoder. In the second stage, freeze the encoder in the above autoencoder architecture, and the above autoencoder and the intermediate embedded fusion module constitute a multimodal visual fusion network.

[0025] S103: In the first stage, a dual-teacher knowledge distillation learning strategy is designed to obtain stronger representations from teacher networks with complementary characteristics. The teacher models include an image-based pre-trained large model and a video-based pre-trained large model. For example... Figure 2 As shown, in the first stage, a temporal matcher is constructed to align and match the temporal features output by the pre-trained teacher model based on video with those output by the student model during knowledge distillation, thereby achieving knowledge mapping between the teacher and student models in the temporal dimension. In the first stage, mean absolute error loss is introduced to measure the difference between the reconstructed image and the input image, and this mean absolute error loss is used as the self-supervised training objective. Under the joint constraints of the self-supervised signal and the distillation supervision signal, the first-stage model is trained and optimized for a preset number of rounds, and training is completed based on the video dataset. The weights of the student model with the best performance are obtained and saved.

[0026] S104: In the second stage, adaptive gradient loss and second-order temporal smoothing constraint loss are designed for the multimodal visual fusion network. Combined with consistency loss and perceptual loss, the second-stage model is trained and optimized for a preset number of rounds. The training is completed based on the video dataset, and the weights of the fusion model with the best performance are obtained and saved.

[0027] S105: The input data for testing includes a multimodal video fusion dataset and a multimodal image fusion dataset as the test set. The input data is fed into the multimodal visual fusion model to be tested to obtain the fusion result. Performance metrics of the fusion result are calculated, including but not limited to peak signal-to-noise ratio, structural similarity coefficient, and correlation coefficient. The fusion result and performance metrics of the fusion model on the test input data are reported to evaluate the fusion quality and generalization ability of the present invention.

[0028] In one embodiment of this application, a publicly available multimodal visual fusion dataset is used in a closed scene. For video fusion datasets, this invention includes infrared and visible light video fusion datasets, and medical video fusion datasets, denoted as... , These video datasets will be partially split into training and testing sets according to an appropriate ratio. For some low frame rate infrared and visible light video fusion datasets, this invention employs a frame interpolation strategy to make the transitions between frames smoother. Specifically, this invention performs frame interpolation between two adjacent original video frames, the process of which includes: first, acquiring adjacent first and second frame images, denoted as... , Second, based on the first frame image and the second frame image, the temporal correspondence between them is estimated. Third, according to the temporal correspondence, at least one intermediate transition frame is generated between the first frame image and the second frame image. Fourth, the intermediate transition frames are inserted sequentially between the first and second frame images to form a continuous video sequence, thereby improving the smoothness of the video. Additionally, during the model training phase, this invention utilizes data augmentation techniques, including random flipping, random rotation, and random cropping, to enhance the model's recovery and generalization capabilities. For the image fusion dataset, this invention includes a near-infrared and visible light image fusion dataset and a polarization image fusion dataset, denoted as... , All of these image datasets will be used as the test set.

[0029] In one embodiment of this application, an autoencoder framework is used in the distillation stage, primarily employing ViT as the skeleton. Each layer of the encoder and decoder consists of one ViT block. The encoder progressively establishes global dependencies of the image through multiple layers of ViT modules, forming a set of multimodal feature representations at different semantic levels. The decoder, in reverse order, combines the features output from the last layer of the encoder to progressively reconstruct the structural information of the image.

[0030] In the first stage, let the input data be... First, it goes through a linear mapping Obtain the initial feature representation :

[0031] Hierarchical feature extraction via encoder:

[0032] Each of them It consists of a stack of 1 ViT blocks.

[0033] During the decoding stage, the multi-scale features are reconstructed hierarchically using the decoder:

[0034] Each of them It consists of a stack of 1 ViT blocks.

[0035] Finally, it undergoes an inverse linear mapping and reconstruction function. The reconstructed image is obtained. .

[0036]

[0037] In the second stage, let the input data for the two modalities be... The data is input into the frozen encoder to obtain multimodal features. The final layer output of the encoder can be represented as... Therefore, multimodal features can be represented as Then, the fusion module can obtain the fused features. The formula is as follows:

[0038] The obtained fusion features are then processed by a decoder to obtain the final fusion result. .

[0039] In one embodiment of this application, such as Figure 3 The diagram shows the model framework used in the second stage of training. The fusion module is a cross-attention module. This is to obtain the enhanced features. First, use the obtained multimodal features Generate the corresponding query ,key Sum :

[0040] in, To generate The weights are calculated for each attention head.

[0041] in, This represents the matrix transpose operation. for The dimension of the vector, where Softmax is the normalized exponential function. The dimensions are concatenated and mapped back to the original dimensions:

[0042] MHCA stands for Multi-head Cross-Attention. This is the weight matrix used for output projection, which can project the concatenated vector back to the original dimensions. Concat is the channel concatenation operation.

[0043] Finally, the final multi-head self-attention output is obtained using residuals and layer normalization:

[0044] Here, LayerNorm is the layer normalization operation. After the multi-head self-attention output passes through the feedforward neural network, residuals and layer normalization are also applied to obtain the final output of the cross-attention module. :

[0045] FFN stands for Feedforward Neural Network.

[0046] Similarly, after the above steps, an enhanced version can be obtained. Finally, the enhanced features... , The input is fed into a fully connected (FC) layer for fusion to obtain the final fused features. The formula is as follows:

[0047] In one embodiment of this application, a projection ladder and teacher dropout regularization are introduced. Specifically, after each encoder layer, the invention attaches a scalable projection head P, implemented using a lightweight multilayer perceptron. These projection heads are activated only during training to stabilize gradients and facilitate the extraction of structural information from intermediate layers. Furthermore, the outputs of all projection heads are aggregated to form an enhanced distillation representation. The formula is as follows:

[0048] Where L represents the number of encoder layers, and D and V represent the image-based pre-trained large model and the video-based pre-trained large model, respectively. This represents the projection head of the l-th encoder layer specific to the teacher model M, while This represents the output of the l-th encoder layer specific to the teacher model M.

[0049] Based on the above cross-layer representation, the distillation loss in this process is calculated by applying cosine constraints and smoothing L1 constraints to both student and teacher features. The calculation method is as follows:

[0050] Here, k represents the token index. This represents the feature token output by the teacher model M.

[0051] Furthermore, for a single sample (i.e., a video sequence), the single-teacher distillation loss can be expressed as:

[0052] Here, K represents the number of tags, and T represents the number of frames.

[0053] Furthermore, a teacher dropout regularization strategy is introduced based on the aforementioned loss formula. For each period and each sample, this strategy assigns a binary coefficient to teacher M. Accordingly, the distillation loss of a sample can be expressed as:

[0054] in, The teacher model is set to zero with probability p to control whether it participates in the distillation process of a given sample. To prevent both teacher models from being discarded simultaneously, thus causing the distillation signal to disappear, this invention retains the teacher model with the larger loss (i.e., sets the teacher model with the larger loss). Another teacher model with less loss would be randomly discarded with probability 1-δ, where the random variable δ follows a Bernoulli distribution with parameter p.

[0055] In one embodiment of this application, given a video sequence ,in B express, T Indicates the number of video frames. C Indicates the number of channels. H and W Let these represent the spatial height and width, respectively. The encoder features generated by the student model can be represented as... The encoder features generated by the image-based pre-trained teacher large model can be matched with the student network, and can also be represented as... In the encoding process of video-based pre-trained teacher large models, multiple consecutive video frames are aggregated into temporal block representations along the temporal dimension. Therefore, their encoder output feature representation is... , where τ represents the length of the timing block.

[0056] To alleviate the above problems, a temporal matcher is proposed. By estimating dense optical flow, it aligns features from adjacent frames to the current time step, thereby providing a consistent spatiotemporal reference for feature-level distillation. For example... Figure 4 As shown, specifically, given two adjacent frames of data and This invention employs a model capable of predicting bidirectional optical flow. Estimated from point to The reverse optical flow is expressed by the following formula:

[0057] Because ViT models at the block level, it processes feature representations by discretizing them into a series of block tokens. Optical flow, on the other hand, is defined at the pixel level, enabling the depiction of much finer-grained motion information. This difference in modeling granularity leads to an inherent incompatibility between the two in their representation spaces.

[0058] Therefore, this invention maps and merges dense pixel-level optical flow onto a block mesh, thereby obtaining a block-level optical flow representation aligned with the ViT feature space. Specifically, this invention spatially averages the optical flow vectors within each block region to obtain the corresponding block-level displacement, thereby achieving an effective conversion of the optical flow field from pixel space to block space. Subsequently, this invention utilizes the estimated optical flow field to analyze the features of adjacent frames. Perform time alignment, mapping it to the current time step. This operation is formally represented as follows:

[0059] here This indicates a feature alignment operation guided by an optical flow field.

[0060] Finally, feature fusion is performed, using the following formula:

[0061] In one embodiment of this application, when there is a difference between the optimization objective of the pre-trained teacher model and the requirements of the multimodal visual fusion task, a self-supervised learning mechanism is introduced to impose structural consistency constraints on the encoding process of the student model. The teacher model is primarily optimized for high-level semantic tasks, and its feature representations are more biased towards semantic discriminative information, while the multimodal visual fusion task emphasizes the accurate preservation of low-level structural and detailed information.

[0062] To address this, a self-supervised signal based on the difference between input and reconstruction is constructed to constrain the feature representation of the student encoder. This allows the student encoder to learn the semantic knowledge provided by the teacher model while further preserving structural consistency and fine-grained information from the multimodal input. This guides the student encoder to form a low-level structural representation that better meets the needs of multimodal visual fusion tasks and reduces the inherent semantic bias introduced by the semantically driven teacher model. Therefore, this invention employs [a specific method / mechanism] in the self-supervised stage. 1. Reconstruction loss. Its formula is as follows:

[0063] here express 1-norm.

[0064] Therefore, the total loss function of the entire self-supervised training process can be expressed as the following formula:

[0065] In one embodiment of this application, the present invention labels the fusion result and the inputs of the two modalities as follows: , and .

[0066] For gradient loss, this invention employs an adaptive selection strategy guided by gradient magnitude to enhance edge details. First, the gradient magnitude is defined as the sum of the absolute responses along the horizontal x-direction and the vertical y-direction:

[0067] Based on two inputs and The difference in gradient magnitude between them is measured using the Sigmoid function. The pixel-wise adaptive weight γ is obtained from the following formula:

[0068] here It's a hyperparameter.

[0069] With weight Target gradient This can be represented as a weighted combination of the source gradients:

[0070] Finally, the gradient maximum consistency loss minimizes the difference between the gradient of the fused image and the gradient of the target image. It is defined by the 1-norm. Its formula is as follows:

[0071] In one embodiment of this application, to enhance the temporal stability of video sequence tasks, this invention designs a second-order temporal smoothness constraint across consecutive frames, which operates along the time dimension. This constraint ensures that the dynamic change pattern of the fusion result remains consistent with the source sequence, thereby making the evolution of the fusion result smoother in the time domain. The specific formula is as follows:

[0072] in The second-order difference between frames is expressed by the following formula:

[0073] in Represents the t-th frame. .

[0074] In addition, this invention introduces perceptual loss. and consistency loss The formulas for these two losses are as follows:

[0075]

[0076] in This represents the feature representation of the l-th layer.

[0077] Therefore, the final loss function for the second stage can be expressed as:

[0078] In one embodiment of this application, actual operational analysis was performed under the following data conditions to verify the feasibility and performance of the technical solution described in this invention, as follows: 1. Data Preparation: This invention selects certain scenes from the HDO video dataset and BraTS2020 video dataset sequences for training. For testing, this invention selects the remaining HDO video dataset, BraTS2020 video dataset sequences, and M3FD video dataset for multimodal video fusion testing. For multimodal image fusion testing, this invention selects the Polarization image dataset and near-infrared / visible light image dataset.

[0079] 2. The hyperparameters and loss function hyperparameters of each module proposed in this invention are set as follows in this embodiment: Autoencoder: In this embodiment, ViT has a patch size of 16, an encoder depth of 6, and an output hidden dimension of 768. Its decoder has a depth of 4 and an output hidden dimension of 512.

[0080] Loss function: In this embodiment, for the two hyperparameters in the loss function and They were set to 0.4 and 0.5 respectively. And for... It will use a decreasing configuration {1, 1 / 4, 1 / 8, 1 / 16, 1 / 32}.

[0081] 3. The hyperparameter settings for the multi-stage training setup proposed in this invention are as follows in this embodiment: Phases 1 and 2: In this embodiment, the input video sequence is randomly cropped, and the spatial resolution is standardized to 384×384 with a duration of 6 frames. The entire framework is optimized using the Adam optimizer, with an initial learning rate of... Each stage involves 2000 independent training sessions.

[0082] 4. Baseline Model: To verify the effectiveness of the proposed solution in this embodiment, other existing multimodal visual fusion methods with better performance were used to conduct comparative experiments, including CoCoNet, EMMA, MAEFuse, Mask-Dif, TemCoCo, and VF-Bench.

[0083] 5. Evaluation indicators for fusion effect: This embodiment uses objective image quality evaluation metrics to measure the actual performance of the model, as detailed below: SSIM (Structural Similarity Index): The structural similarity index is used to measure the similarity between the fused image and the reference image in terms of brightness, contrast, and structural information. The higher the value, the stronger the ability of the fused result to preserve the structure.

[0084] PSNR (Peak Signal-to-Noise Ratio): The peak signal-to-noise ratio is used to measure the degree of distortion of the fused image relative to the reference image. The higher the value, the more noise and distortion there is in the fused result.

[0085] CC (Correlation Coefficient): The correlation coefficient is used to measure the degree of linear correlation between the fused image and the reference image. The larger the value, the stronger the correlation between the fused result and the source image.

[0086] BiSWE (Bi-Directional Self-Warping Error): BiSWE quantifies the temporal alignment error between frames in a fused video sequence. The smaller the value, the better the temporal alignment of the fused result.

[0087] MS2R (Motion Smoothness with Dual Reference Videos): MS2R is used to evaluate the coherence of motion changes in the fused video and two reference sequences. The smaller the value, the smoother the fusion result.

[0088] 6. Experimental Results:

[0089] Table 1. Objective metrics obtained from the fusion results of each method on the HDO dataset. It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0090] Based on the same inventive concept, this application also provides a device for implementing the aforementioned multimodal visual fusion method based on dual-teacher distillation and self-supervised methods. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the device based on dual-teacher distillation and self-supervised methods provided below can be found in the limitations of the multimodal visual fusion method based on dual-teacher distillation and self-supervised methods described above, and will not be repeated here.

[0091] In one embodiment, such as Figure 5 As shown, a multimodal visual fusion device 500 based on dual-teacher distillation and self-supervision is provided, including: a dataset collection and processing module 501, a multimodal visual fusion model training process construction module 502, a first-stage model training module 503, a second-stage model training module 504, and a multimodal visual fusion module 505, wherein: The dataset collection and processing module is used to collect multiple multimodal visual fusion datasets, including video fusion datasets and image fusion datasets. The video fusion datasets are divided into training sets, validation sets, and test sets, and the image fusion datasets are all used as test sets. The training and test data are independent of each other. A multimodal visual fusion model training process construction module is used to construct a multimodal visual fusion model training process. The process includes two stages: the first stage trains an autoencoder model, which includes an encoder and a decoder; the second stage freezes the encoder in the autoencoder architecture, and the autoencoder and the intermediate embedded fusion module constitute a multimodal visual fusion network. The first-stage training module of the model employs a dual-teacher knowledge distillation learning strategy in the first stage. It acquires representations from the teacher network consisting of an image-based pre-trained large model and a video-based pre-trained large model. Simultaneously, it constructs a temporal matcher to align and match the temporal features output by the video-based pre-trained large model with the temporal features output by the student model. It introduces the mean absolute error loss to measure the difference between the reconstructed image and the input image and uses it as the self-supervised training objective. Under the joint constraints of the self-supervised signal and the distillation supervision signal, the student model weights are obtained and saved based on the video fusion dataset. The second-stage training module of the model is used in the second stage to design adaptive gradient loss and second-order temporal smoothing constraint loss for the multimodal visual fusion network. It combines consistency loss and perceptual loss to train and save the weights of the multimodal visual fusion model based on the video fusion dataset. The multimodal visual fusion module is used to input the video fusion dataset test set and the image fusion dataset into the multimodal visual fusion model during testing to obtain the fusion results and calculate the performance metrics of the fusion results.

[0092] In one embodiment of this application, after collecting multiple multimodal visual fusion datasets, the method further includes: Perform frame interpolation on low frame rate multimodal video datasets; Data augmentation techniques are employed during the model training phase, including random flipping, random rotation, and random cropping.

[0093] In one embodiment of this application, the autoencoder model uses ViT as its skeleton, and each layer of the encoder and decoder is composed of ViT blocks. The working process of the autoencoder model includes: The input data is linearly mapped to obtain an initial feature representation; The initial feature representation is obtained by hierarchical feature extraction through the encoder to obtain multi-scale features, and each encoder layer is composed of stacked ViT blocks; Multi-scale features are reconstructed hierarchically through a decoder, with each decoder layer consisting of stacked ViT blocks. Finally, the reconstructed image is obtained through inverse linear mapping and reconstruction function.

[0094] In one embodiment of this application, the intermediate embedded fusion module is a cross-attention module, and the operation of the cross-attention module includes: Multimodal features generate corresponding queries, keys, and values; The single attention head is calculated by normalizing the product of the query and the transpose of the key using the Softmax function and then multiplying it by the value. Multiple attention heads are concatenated and mapped back to the original dimension to obtain a multi-head cross-attention output; Multi-head self-attention output is obtained through residuals and layer normalization. The multi-head self-attention output is then processed by a feedforward neural network and reinforced with residuals and layer normalization to obtain enhanced features. The enhanced features of the two modalities are input into a fully connected layer for fusion to obtain the fused features.

[0095] In one embodiment of this application, the dual-teacher knowledge distillation learning strategy includes: A projection head is attached after each encoder layer, which is implemented using a lightweight multilayer perceptron and activated only during training; The outputs of all projection heads are aggregated to form an enhanced distillation representation; Distillation loss is a combination of cosine constraints and smoothing L1 constraints imposed on student and teacher characteristics; A teacher discard regularization strategy is introduced, assigning binary coefficients to each teacher model to control whether it participates in the distillation process.

[0096] In one embodiment of this application, the operation of the timing matcher includes: A model capable of predicting bidirectional optical flow is used to estimate the reverse optical flow between two adjacent frames; Pixel-level optical flow is mapped and merged onto a block mesh to obtain a block-level optical flow representation; The block-level optical flow representation is used to perform temporal alignment of features in adjacent frames, and the aligned features are then fused with the features of the current frame.

[0097] In one embodiment of this application, the total loss function of the first stage is the sum of the mean absolute error loss and the distillation loss, wherein the mean absolute error loss is calculated using the L1 norm to determine the difference between the reconstructed image and the input image.

[0098] In one embodiment of this application, the calculation process of the adaptive gradient loss includes: The gradient magnitude is defined as the sum of the absolute responses along the horizontal and vertical directions; Based on the difference in gradient magnitude between the two modal inputs, the Sigmoid function is used to obtain pixel-wise adaptive weights; The target gradient is a weighted combination of the gradients of the two modal inputs; The gradient loss is the L1 norm of the fused image gradient and target gradient.

[0099] In one embodiment of this application, the second-order temporal smoothing constraint loss is the sum of the L1 norms of the inter-frame second-order difference of the fusion result and the inter-frame second-order difference of the two modal inputs; The perceptual loss is a weighted sum of the fusion result and the L1 norm of the feature representations of the two modal inputs at each layer; The consistency loss is the sum of the fusion result and the L1 norms of the two modal inputs, with the L1 norm of one of the modal inputs multiplied by the weighting coefficient. The total loss function in the second stage is the sum of adaptive gradient loss, second-order time smoothing constraint loss, perceptual loss, and consistency loss.

[0100] The modules in the aforementioned multimodal visual fusion device based on dual-teacher distillation and self-supervision can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0101] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a multimodal visual fusion method based on dual-teacher distillation and self-supervision. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0102] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0103] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0104] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0105] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0106] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0107] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimodal visual fusion method based on dual-teacher distillation and self-supervised learning, characterized in that, The method includes: We collected multiple multimodal visual fusion datasets, including video fusion datasets and image fusion datasets. The video fusion datasets were divided into training, validation, and test sets, while the image fusion datasets were all used as test sets. The training and test data were independent of each other. A training process for a multimodal visual fusion model is constructed, which includes two stages. The first stage trains an autoencoder model, which includes an encoder and a decoder. The second stage freezes the encoder in the autoencoder architecture. The autoencoder and the intermediate embedded fusion module constitute a multimodal visual fusion network. In the first stage, a dual-teacher knowledge distillation learning strategy is adopted to obtain representations from the teacher network consisting of an image-based pre-trained large model and a video-based pre-trained large model. At the same time, a temporal matcher is constructed to align and match the temporal features output by the video-based pre-trained large model with the temporal features output by the student model. The mean absolute error loss is introduced to measure the difference between the reconstructed image and the input image and to serve as the self-supervised training objective. Under the joint constraints of the self-supervised signal and the distillation supervision signal, the weights of the student model are obtained and saved based on the video fusion dataset. In the second stage, an adaptive gradient loss and a second-order temporal smoothing constraint loss are designed for the multimodal visual fusion network. Combined with consistency loss and perceptual loss, the weights of the multimodal visual fusion model are obtained and saved based on the video fusion dataset. During testing, the video fusion dataset test set and the image fusion dataset are input into the multimodal visual fusion model to obtain the fusion results, and the performance metrics of the fusion results are calculated.

2. The multimodal visual fusion method based on dual-teacher distillation and self-supervised learning according to claim 1, characterized in that, The process of collecting multiple multimodal visual fusion datasets also includes: Perform frame interpolation on low frame rate multimodal video datasets; Data augmentation techniques are employed during the model training phase, including random flipping, random rotation, and random cropping.

3. The multimodal visual fusion method based on dual-teacher distillation and self-supervised learning according to claim 1, characterized in that, The autoencoder model uses ViT as its skeleton, and each layer of the encoder and decoder is composed of ViT blocks. The working process of the autoencoder model includes: The input data is linearly mapped to obtain an initial feature representation; The initial feature representation is obtained by hierarchical feature extraction through the encoder to obtain multi-scale features, and each encoder layer is composed of stacked ViT blocks; Multi-scale features are reconstructed hierarchically through a decoder, with each decoder layer consisting of stacked ViT blocks. Finally, the reconstructed image is obtained through inverse linear mapping and reconstruction function.

4. The multimodal visual fusion method based on dual-teacher distillation and self-supervision as described in claim 1, characterized in that, The intermediate embedded fusion module is a cross-attention module, and the working process of the cross-attention module includes: Multimodal features generate corresponding queries, keys, and values; The single attention head is calculated by normalizing the product of the query and the transpose of the key using the Softmax function and then multiplying it by the value. Multiple attention heads are concatenated and mapped back to the original dimension to obtain a multi-head cross-attention output; Multi-head self-attention output is obtained through residuals and layer normalization. The multi-head self-attention output is then processed by a feedforward neural network and reinforced with residuals and layer normalization to obtain enhanced features. The enhanced features of the two modalities are input into a fully connected layer for fusion to obtain the fused features.

5. The multimodal visual fusion method based on dual-teacher distillation and self-supervision as described in claim 1, characterized in that, The dual-teacher knowledge distillation learning strategy includes: A projection head is attached after each encoder layer, which is implemented using a lightweight multilayer perceptron and activated only during training; The outputs of all projection heads are aggregated to form an enhanced distillation representation; Distillation loss is a combination of cosine constraints and smoothing L1 constraints imposed on student and teacher characteristics; A teacher discard regularization strategy is introduced, assigning binary coefficients to each teacher model to control whether it participates in the distillation process.

6. The multimodal visual fusion method based on dual-teacher distillation and self-supervision as described in claim 1, characterized in that, The operation of the timing matcher includes: A model capable of predicting bidirectional optical flow is used to estimate the reverse optical flow between two adjacent frames; Pixel-level optical flow is mapped and merged onto a block mesh to obtain a block-level optical flow representation; The block-level optical flow representation is used to perform temporal alignment of features in adjacent frames, and the aligned features are then fused with the features of the current frame.

7. The multimodal visual fusion method based on dual-teacher distillation and self-supervision as described in claim 1, characterized in that, The total loss function in the first stage is the sum of the mean absolute error loss and the distillation loss. The mean absolute error loss is calculated using the L1 norm to determine the difference between the reconstructed image and the input image.

8. The multimodal visual fusion method based on dual-teacher distillation and self-supervision as described in claim 1, characterized in that, The calculation process of the adaptive gradient loss includes: The gradient magnitude is defined as the sum of the absolute responses along the horizontal and vertical directions; Based on the difference in gradient magnitude between the two modal inputs, the Sigmoid function is used to obtain pixel-wise adaptive weights; The target gradient is a weighted combination of the gradients of the two modal inputs; The gradient loss is the L1 norm of the fused image gradient and target gradient.

9. The multimodal visual fusion method based on dual-teacher distillation and self-supervision as described in claim 1, characterized in that, The second-order temporal smoothing constraint loss is the sum of the L1 norms of the inter-frame second-order difference of the fusion result and the inter-frame second-order difference of the two modal inputs. The perceptual loss is a weighted sum of the fusion result and the L1 norm of the feature representations of the two modal inputs at each layer; The consistency loss is the sum of the fusion result and the L1 norms of the two modal inputs, with the L1 norm of one of the modal inputs multiplied by the weighting coefficient. The total loss function in the second stage is the sum of adaptive gradient loss, second-order time smoothing constraint loss, perceptual loss, and consistency loss.

10. A multimodal visual fusion device based on dual-teacher distillation and self-supervision, characterized in that, The device includes: The dataset collection and processing module is used to collect multiple multimodal visual fusion datasets, including video fusion datasets and image fusion datasets. The video fusion datasets are divided into training sets, validation sets, and test sets, and the image fusion datasets are all used as test sets. The training and test data are independent of each other. A multimodal visual fusion model training process construction module is used to construct a multimodal visual fusion model training process. The process includes two stages: the first stage trains an autoencoder model, which includes an encoder and a decoder; the second stage freezes the encoder in the autoencoder architecture, and the autoencoder and the intermediate embedded fusion module constitute a multimodal visual fusion network. The first-stage training module of the model employs a dual-teacher knowledge distillation learning strategy in the first stage. It acquires representations from the teacher network consisting of an image-based pre-trained large model and a video-based pre-trained large model. Simultaneously, it constructs a temporal matcher to align and match the temporal features output by the video-based pre-trained large model with the temporal features output by the student model. It introduces the mean absolute error loss to measure the difference between the reconstructed image and the input image and uses it as the self-supervised training objective. Under the joint constraints of the self-supervised signal and the distillation supervision signal, the student model weights are obtained and saved based on the video fusion dataset. The second-stage training module of the model is used in the second stage to design adaptive gradient loss and second-order temporal smoothing constraint loss for the multimodal visual fusion network. It combines consistency loss and perceptual loss to train and save the weights of the multimodal visual fusion model based on the video fusion dataset. The multimodal visual fusion module is used to input the video fusion dataset test set and the image fusion dataset into the multimodal visual fusion model during testing to obtain the fusion results and calculate the performance metrics of the fusion results.

Images