Multi-modal fusion progressive pre-training and self-adaptive fine tuning method based on visual large model
By employing a progressive pre-training and adaptive fine-tuning method based on a large visual model for multimodal fusion, the problem of detail loss in multimodal image fusion is solved, achieving better fusion results and model robustness.
Patent Information
- Application Number
- CN202511423544.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-01-13
AI Technical Summary
The problem of loss of multimodal image details and poor fusion effect during multimodal image fusion.
We employ a progressive pre-training and adaptive fine-tuning method based on a large visual model with multimodal fusion. By combining self-supervised learning and a self-supervised loss function with multimodal images, we train the large visual model and then fine-tune it by constructing a total task loss function in downstream tasks.
It improves the effect of multimodal image fusion, enhances the robustness and adaptability of the model, and improves its practicality in real-world scenarios.
Smart Images
Figure CN121329784A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of deep learning and image fusion, and more particularly to a multi-modal fusion progressive pre-training and adaptive fine-tuning method based on a visual large model. BACKGROUND
[0002] Single RGB images include limited image information, and it is difficult to perform target detection, image segmentation and other tasks based on single RGB images. In contrast, fusing infrared images, depth maps and other modal data can provide complementary information and significantly improve the robustness and adaptability of the system. Therefore, multi-modal image fusion technology has emerged.
[0003] Multi-modal image fusion integrates infrared images, depth information, thermal imaging and other multi-source data to form complementary feature expressions, and RGB images retain texture and color details. Combining multi-modal features can build more robust environmental perception capabilities. However, during multi-modal image fusion, the loss function design in the training process of the fusion model is unreasonable, and adaptive fine-tuning for downstream tasks is not considered, resulting in loss of multi-modal image details and poor fusion effect. SUMMARY
[0004] To solve the problem of loss of multi-modal image details and poor fusion effect during multi-modal image fusion, the present application proposes a multi-modal fusion progressive pre-training and adaptive fine-tuning method based on a visual large model. Through self-supervised learning, multi-modal optimization is achieved, and the fusion effect of the model on multi-modal images is improved.
[0005] To achieve the above technical effects, the technical solutions of the present application are as follows: In a first aspect, the present application proposes a multi-modal fusion progressive pre-training and adaptive fine-tuning method based on a visual large model, comprising the following steps: S1: Obtain images of different modalities and pre-process the images to obtain pre-processed multi-modal images; S2: Input the multi-modal images into the visual large model constructed, construct a self-supervised loss function, and train the visual large model based on the self-supervised loss function using the multi-modal images; S3: Based on the multi-modal image fusion downstream task, construct a total task loss function, and fine-tune the visual large model considering the multi-modal image fusion downstream task.
[0006] Preferably, in the S1, the images of other modalities are converted to RGB format through statistical normalization processing, forming an image pair with the original RGB image, and the image data is mapped to a standard normal distribution through linear transformation, and the process satisfies the expression:
[0007]
[0008]
[0009] wherein, represents the mean value of pixels of a single image, represents the standard deviation of pixels; The statistical normalized gradient calculation expression is:
[0010] is the gradient of the original image, as a scaling factor.
[0011] Preferably, the multi-modal image after statistical normalization processing is preprocessed, and the process is: using a beam splitter to collect multi-modal images, ensuring spatial alignment, introducing data enhancement technology, designing a specific preprocessing pipeline according to the characteristics of different modal images; using batch normalization to accelerate image data convergence, the expression is:
[0012] wherein, and are the batch mean and standard deviation respectively, γ and β are learnable parameters, and ϵ is a small constant; Finally, the alignment accuracy is verified by mean square error, and the expression is:
[0013] wherein, H and W represent the number of pixels in the vertical direction and the vertical direction of the image respectively.
[0014] Preferably, the visual large model comprises: an image division module based on a Transformer model, a feature extraction module, and a feature fusion module, the feature extraction module extracts high-level semantic features of the original RGB image in the image pair using an RGB-Transformer model extracts high-level features of other modal images using an X-Transformer model uses a multi-layer perception MLP to realize feature interaction of and ; the feature fusion module is based on a double-layer fully connected network, and adds a ReLU activation function and a residual connection.
[0015] Preferably, in the S2, the image division module is used to divide the images in the image pair into image blocks, each block is flattened into a vector to form an image block sequence P; the position encoding E is generated by a sine-cosine function, and the expression is:
[0016]
[0017] wherein pos is the absolute position of the image patch, is the model dimension, 2i and 2i+1 correspond to even and odd dimensions respectively; The combination of the image patch sequence P and the position encoding E is iteratively processed using the feature extraction module, and the expression is:
[0018] extracting high-level semantic features of the original RGB image in the image pair and high-level features of other modal images , splicing and The feature representation after row splicing is: The spliced features are nonlinearly fused using a multi-layer perceptron to generate preliminary fusion features , and the expression is:
[0019]
[0020] wherein, represents feature splicing, and ReLU activation function , is the first layer weight, is the second layer weight, , is the bias; The basic information of the RGB mode is retained, and a residual connection is constructed, and the expression is:
[0021] is the weight of the adaptive adjustment of multiple features, and an attention mechanism is introduced, and the expression is:
[0022] wherein, is the weighting coefficient; The input of all neurons in each layer is normalized using layer normalization to stabilize the output distribution of each layer.
[0023] Preferably, based on the feature relationship similarity , the image patch matching loss , the image patch prediction loss and the contrast loss are weighted and summed to construct a self-supervised loss function, and the expression is:
[0024] wherein the image block matching loss is expressed as:
[0025]
[0026] wherein sim is the cosine similarity, i, j, k are index variables, control distribution softening; the image block prediction loss is expressed as:
[0027]
[0028] wherein, is the true value, is the predicted value, is the prediction layer weight matrix, is the context feature weight matrix, is the context feature, is the context feature bias term, is the prediction layer bias term; the contrastive loss is expressed as:
[0029] wherein, is the positive sample, is the negative sample, control distribution temperature; finally, a multi-modal image fusion model is obtained.
[0030] Preferably, the optimization strategy based on the self-supervised loss function is: setting the initial value of , , in the training process, dynamically adjusting the proportion of , , , sampling negative samples in real time within each training batch, and using momentum update to optimize the selection of positive samples and negative samples.
[0031] Preferably, the visual large model is adapted to the downstream task, and a total task loss function is constructed, which is weighted fusion of classification loss and regression loss , and the expression is:
[0032]
[0033]
[0034] wherein, and is the task weight.
[0035] Preferably, the optimization strategy based on multi-modal image fusion downstream task fine-tuning is: Cosine annealing learning rate is used by using the learning rate scheduling , the expression is:
[0036] wherein, is the initial learning rate, t is the current epoch, and T is the total epoch; The pre-training features are retained, 90% of the self-supervised pre-training layers are frozen, and only the subsequent downstream task head is trained.
[0037] In a second aspect, the present application further provides a multi-modal image fusion method based on a visual large model, comprising the following steps: S10: training a visual large model based on a multi-modal fusion progressive pre-training and adaptive fine-tuning method of the visual large model to obtain a trained visual large model; S11: acquiring images of different modalities and pre-processing the images to obtain multi-modal images; S12: fusing the multi-modal images based on the trained visual large model.
[0038] Compared with the prior art, the technical scheme of the present application has the following beneficial effects: The present application provides a multi-modal image fusion model training method and a multi-modal image fusion method. In the multi-modal image fusion model training method, images of different modalities are acquired, and the images are pre-processed to obtain pre-processed multi-modal images. The pre-processing process enhances the generalization ability of the model and further enriches the diversity of the input data. The multi-modal images are input into the constructed multi-modal image fusion model, a self-supervised loss function is constructed, and based on the self-supervised loss function, the multi-modal image fusion model is trained using the multi-modal images. Through self-supervised learning of unlabeled data, the model can learn the internal correlation between multi-modal images. Based on the multi-modal image fusion downstream task, a total task loss function is constructed, and the multi-modal image fusion model considering the multi-modal image fusion downstream task is fine-tuned, which improves the practicability of the model in actual scenarios. In the multi-modal image fusion method, the multi-modal images are fused based on the trained multi-modal image fusion model. The present application gradually constructs and optimizes the multi-modal image fusion model by using the strategy of progressive pre-training and adaptive fine-tuning. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 a flow chart of a multi-modal fusion progressive pre-training and adaptive fine-tuning method based on a visual large model proposed in an embodiment of the present application; Figure 2 a schematic diagram of a visual large model proposed in an embodiment of the present application; Figure 3 a flow chart of a multi-modal image fusion method based on a visual large model proposed in an embodiment of the present application. DETAILED DESCRIPTION
[0040] The accompanying drawings are only for illustrative purposes and should not be construed as limiting the present patent; In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size; It is understandable to some skilled persons in the art that some known content in the drawings may be omitted.
[0041] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.
[0042] The positional relationship described in the drawings is only for illustrative purposes and should not be construed as limiting the present patent; Embodiment 1 The present embodiment proposes a multi-modal fusion progressive pre-training and adaptive fine-tuning method based on a visual large model, as shown in Figure 1 , comprising the following steps: S1: obtaining images of different modalities and pre-processing the images to obtain pre-processed multi-modal images; S2: inputting the multi-modal images into the visual large model constructed, constructing a self-supervised loss function, and training the visual large model based on the self-supervised loss function using the multi-modal images; S3: constructing a total task loss function based on the multi-modal image fusion downstream task, and fine-tuning the visual large model considering the multi-modal image fusion downstream task.
[0043] In the present embodiment, by obtaining images of different modalities and pre-processing the images to obtain pre-processed multi-modal images, the pre-processing process enhances the generalization ability of the model and further enriches the diversity of the input data. The multi-modal images are inputted into the visual large model constructed, a self-supervised loss function is constructed, and the visual large model is trained based on the self-supervised loss function using the multi-modal images. Through self-supervised learning of unlabeled data, the model can learn the internal correlation between multi-modalities. Based on the multi-modal image fusion downstream task, a total task loss function is constructed, and the visual large model considering the multi-modal image fusion downstream task is fine-tuned, which improves the practicability of the model in actual scenarios.
[0044] In some preferred embodiments, in S1, the images of other modalities are converted into RGB format by statistical normalization processing, form image pairs with the original RGB images, and the image data is mapped to a standard normal distribution by linear transformation, and the process satisfies the expression:
[0045]
[0046]
[0047] wherein, represents the pixel mean of a single image, represents the pixel standard deviation; The gradient calculation expression after statistical normalization is:
[0048] is the gradient of the original image, as a scaling factor.
[0049] In this embodiment, in order to enhance the generalization ability of the model, data augmentation techniques such as random flipping, cropping and color jittering are introduced to further enrich the diversity of input data. Statistical normalization maps data to a standard normal distribution through linear transformation, reduces the numerical difference between modalities, eliminates the influence of light and sensor differences, and provides a unified input for subsequent feature extraction.
[0050] In some preferred embodiments, the multi-modal images after statistical normalization processing are preprocessed, and the process is: using a beam splitter to collect multi-modal images to ensure spatial alignment, introducing data augmentation techniques, designing specific preprocessing pipelines according to the characteristics of different modal images; using batch normalization to accelerate image data convergence, and the expression is:
[0051] wherein, and are the batch mean and standard deviation respectively, γ and β are learnable parameters, and ϵ is a small constant; Finally, the alignment accuracy is verified by mean square error, and the expression is:
[0052] wherein, H and W represent the number of pixels in the vertical and vertical directions of the image respectively.
[0053] In this embodiment, batch normalization standardizes each batch of input x to a distribution with mean 0 and variance 1, adjusts the distribution through learnable parameters, preserves the feature expression ability of the network, and ϵ is usually 1e-5 to avoid division by zero.
[0054] In some preferred embodiments, referring to Figure 2 , the visual large model comprises an image division module based on a Transformer model, a feature extraction module, and a feature fusion module. The feature extraction module extracts high-level semantic features of the original RGB image in the image pair using an RGB-Transformer model , extracts high-level features of other modal images using an X-Transformer model , and realizes feature interaction of and using a multi-layer perception (MLP); the feature fusion module is based on a double-layer fully connected network, with a ReLU activation function and a residual connection.
[0055] In this embodiment, in the image division module, the sequence processing of the Transformer model can better preserve the global context information, providing a richer semantic basis for multi-modal fusion. The feature extraction process iterates through multiple Transformer encoders, gradually refining the high-level semantic representation, and improving the representation ability of the features of each modality. The fusion module designs a double-layer fully connected network, with a ReLU activation and a residual connection, to enhance the non-linear representation ability of the features and the stability of gradient propagation. The fusion process also includes an attention mechanism to dynamically weight the contributions of different modalities.
[0056] In some preferred embodiments, in S2, the images in the image pair are divided into image blocks using the image division module, each block is flattened into a vector, forming an image block sequence P; a position encoding E is generated using a sine-cosine function, with the expression:
[0057]
[0058] where pos is the absolute position of the image block, is the model dimension, and 2i and 2i+1 correspond to even and odd dimensions, respectively; The combination of the image block sequence P and the position encoding E is iteratively processed using the feature extraction module, with the expression:
[0059] extracting high-level semantic features of the original RGB image in the image pair and high-level features of other modal images , concatenating and The feature representation after row concatenation is: A multi-layer perception is used to perform non-linear fusion on the concatenated features, generating preliminary fusion features , with the expression:
[0060]
[0061] where, denotes feature concatenation, ReLU activation function , is the first layer weight, is the second layer weight, , is the bias; reserve the basic information of the RGB mode, construct the residual connection, and the expression is:
[0062] is the weight of adaptive adjustment of multiple features, and the attention mechanism is introduced, and the expression is:
[0063] where, is the weighting coefficient; The layer normalization is used to normalize the input of all neurons of each layer, and the output distribution of each layer is stabilized.
[0064] In the embodiment, based on the ImageNet-21K unlabeled data, a Vision Transforme (ViT) model is pre-trained, the image is divided into 16x16 non-overlapping blocks by using the ViT, and a image block sequence is formed The Swin-Transformer further introduces a window attention mechanism, decomposes the global self-attention into local window calculation, significantly reduces the calculation complexity, and retains the spatial locality information. The position encoding E is generated by the sine-cosine function, Generally, 768, the encoding value decays with the position and dimension index, ensuring that the model captures local and global spatial relationships.
[0065] In the feature extraction module, the Transformer model includes a multi-head self-attention layer and a feedforward network, and the attention mechanism expression is:
[0066] Q, K, and V are query, key, and value matrices, (h is the number of heads) is the dimension of each head, and is a scaling factor to prevent gradient explosion. The window attention of the Swin-Transformer further optimizes the expression:
[0067] where, , , B is the relative position bias matrix for queries, keys and values in the window.
[0068] The optimization strategy of Swin-Transformer is to set the window size to 7x7, set the Dropout rate to 0.1 to control overfitting, use the Shifted Window mechanism to enhance cross-window interaction, and verify the feature quality through principal component analysis (PCA) dimensionality reduction visualization.
[0069] In the feature fusion module, , C is the number of feature channels, N is the number of features, is the first layer weight, is the second layer weight, and the ReLU activation function introduces nonlinearity, and the gradient is
[0070] If , otherwise 0.
[0071] Residual connection alleviates the gradient vanishing problem of deep network through skip connection, and the gradient is:
[0072] The optimization strategy is to use LayerNorm to stabilize the output, adjust to enhance the representation ability, and the attention weight is optimized through gradient descent.
[0073] In some preferred embodiments, based on the feature relationship similarity , the image block matching loss , the image block prediction loss and the contrast loss are weighted and summed to construct a self-supervised loss function, and the expression is:
[0074] Among them, the expression of image block matching loss is:
[0075]
[0076] Among them, sim is the cosine similarity, i, j, k are index variables, to control the softening of the distribution; The expression of image block prediction loss is:
[0077]
[0078] in, For the true value, For predicted values, This is the prediction layer weight matrix. For context feature weight matrix, For contextual features, For context feature bias terms, For the prediction layer bias term; Comparative loss The expression is:
[0079] in, As a positive sample, For negative samples, The temperature distribution is controlled; finally, a multimodal image fusion model is obtained.
[0080] In this embodiment, Minimize the negative log-likelihood to approximate the high similarity of positive sample pairs. Image patch prediction loss. gradient Comparative losses Based on InfoNCE loss, modality discriminability is enhanced by learning the semantic consistency between modalities through positive and negative sample pairs, maximizing positive sample similarity and suppressing negative sample interference.
[0081] In some preferred embodiments, the optimization strategy based on the self-supervised loss function is as follows: set , , The initial value is dynamically adjusted during training. , , The ratio of positive to negative samples is used to sample negative samples in real time within each training batch, and momentum updates are used to optimize the selection of positive and negative samples.
[0082] In this embodiment, during the optimization process, the weighting coefficients of the loss function were first reasonably allocated, wherein... , , 3. This proportion setting reflects a balance consideration for different objectives: As the primary weighting factor, emphasis is placed on optimizing core tasks, while , This maintains a balance on secondary objectives, ensuring that the overall training process highlights the main task without neglecting auxiliary signals, thus achieving a good trade-off between generalization performance and convergence speed.
[0083] During training, parameters are dynamically adjusted through validation set monitoring. At each stage or after several iterations, the performance on the validation set is evaluated, and adjustments to the learning rate, loss weights, or positive / negative sample selection strategy are determined accordingly. This dynamic adjustment mechanism based on validation set feedback helps prevent the model from overfitting or underfitting, and allows for gradual refinement of optimization directions in the later stages of training, improving model stability and final performance.
[0084] In terms of sample processing, large-scale intra-batch negative sampling (K=4096) was introduced. Compared with traditional small-scale negative sampling, this approach significantly improves training efficiency and representation learning performance. By introducing more potential negative samples in the same batch, the model can learn discriminative features in a richer contrastive environment, thereby improving the discriminative power of the embedding vector space. In addition, large-scale negative sampling can also mitigate the impact of sampling bias to some extent, making the optimization results more robust.
[0085] To further enhance the effectiveness of positive and negative sample selection, this embodiment also introduces a momentum update mechanism. Under this mechanism, negative samples are not only randomly selected from the current batch but also updated in conjunction with historical states. This approach maintains the diversity and continuity of the sample distribution, thereby avoiding excessive limitation to short-term sample distributions. For positive sample selection, momentum update can also help the model gradually accumulate high-quality feature representations, improving the model's convergence and robustness during long-term training.
[0086] Example 2 This embodiment proposes a progressive pre-training and adaptive fine-tuning method based on a large visual model for multimodal fusion. The method adapts the large visual model to downstream tasks and constructs a total task loss function, which consists of a classification loss. and regression loss The weighted fusion is expressed as:
[0087]
[0088]
[0089] in, and This represents the task weight.
[0090] In this embodiment, the self-supervised pre-trained model is fine-tuned for multimodal downstream tasks, covering three major tasks: detection, tracking, and segmentation. 5 million pairs of samples are labeled using the KAIST dataset, covering complex scenarios. Data augmentation and online hard negative mining are introduced during training. Data augmentation uses MixUp (mixing sample features and labels) to enhance the model's generalization ability. Online hard negative mining dynamically selects difficult-to-classify or difficult-to-regress samples during training, focusing on optimizing these samples to improve the model's adaptability to difficult samples.
[0091] The total task loss function consists of classification loss. and regression loss The weighted composition aims to balance the optimization objectives of multiple tasks, and its function expression is:
[0092] in, To balance task weights, classification is the core task of detection, tracking, and segmentation, so the weight of classification loss is higher than that of regression loss.
[0093]
[0094]
[0095] in, Used to control the balance between easy and difficult samples. For cross-entropy loss, Introducing a focus mechanism to reduce the weight of easily classified samples and amplify the weight of difficult-to-classify samples; gradient. .
[0096] Regression loss The expression is:
[0097] in SmoothL1 combines L1 and L2 losses to smooth the transition and reduce the impact of outliers, and the gradient... .
[0098] For single-modal downstream tasks, multimodal data is used to enhance the training of the single-modal model. To balance sub-task performance and single-modal robustness, an enhanced loss function is adopted. During fine-tuning, cross-modal regularization is introduced to reduce the risk of overfitting. The process expression is as follows:
[0099]
[0100] in, , For mission losses, This is a cross-modal regularization term that constrains the consistency between single-modal features and multimodal features, where N is the number of samples. and The RGB features of the i-th sample and other modal features are respectively; gradient .
[0101] The optimization strategy is to freeze the pre-trained layers to retain the pre-trained knowledge of a single-modality model; fine-tune the task head to train only the task-specific layers of the model; adjust the output layer parameters for sub-tasks; and fine-tune for 10 epochs.
[0102] In some preferred embodiments, the optimization strategy for downstream task fine-tuning based on multimodal image fusion is as follows: Using learning rate scheduling with cosine annealing learning rate The expression is:
[0103] in, The initial learning rate is t, the current epoch is t, and the total epochs are T. Pre-trained features are retained, 90% of the self-supervised pre-trained layers are frozen, and only the subsequent downstream task heads are trained.
[0104] In this embodiment, the learning rate is set. This method allows for fine-tuning of model parameters to adapt to specific multimodal tasks. With a training epoch count of 50 epochs, the pre-trained model's parameters are already close to their optimal solutions, resulting in fast convergence during fine-tuning and requiring fewer epochs. In the early stages of training, a larger learning rate allows the model to quickly converge to the approximate parameter space of the target task; in the later stages, a smaller learning rate allows for fine-tuning of parameters to find better local optima. Compared to fixed learning rates or step descent, cosine annealing dynamically adjusts the learning rate, more effectively optimizing model performance. Pre-trained layers have already learned general features; freezing these layers avoids destroying these effective features, and training 10% of the layers reduces computation by 90% compared to training all layers, significantly improving training efficiency and preventing overfitting.
[0105] Example 3 This embodiment proposes a multimodal image fusion method based on a large visual model. See [link to relevant documentation]. Figure 3 This includes the following steps: S10: The visual large model is trained using a multimodal fusion progressive pre-training and adaptive fine-tuning method based on a large visual model to obtain a well-trained visual large model. S11: Acquire images of different modalities and preprocess the images to obtain multimodal images; S12: Fuse multimodal images based on the trained large visual model.
[0106] In this embodiment, the process of multimodal image fusion is as follows: First, based on the progressive pre-training and adaptive fine-tuning method of multimodal fusion based on visual large model in Embodiments 1 and 2, the training of visual large model is completed to obtain the trained visual large model.
[0107] High-performance dedicated sensors, including infrared cameras, pulse cameras, and depth cameras, are used to acquire images of other modalities. The selection of these devices considers resolution, frame rate, and environmental adaptability; for example, infrared cameras need to support night vision, and depth cameras need high-precision ranging capabilities. To improve spatial alignment accuracy, a beam splitter is introduced as an auxiliary tool, enabling simultaneous acquisition of multimodal data through optical path segmentation. In the initial stage, 5 million image pairs are acquired for each modality type. The data volume is dynamically adjusted based on subsequent training results, with the final goal of reaching 100-200 million pairs. Data preprocessing includes resizing images to 224x224 pixels to standardize the input size, achieving modal alignment through timestamp correction (target error less than 10 milliseconds), and applying Gaussian filtering. Remove noise to ensure data cleanliness and consistency.
[0108] The trained model fuses multimodal images. It utilizes the RGB-Transformer model in the feature extraction module to extract high-level semantic features (such as texture and shape) from the original RGB image, and the X-Transformer model (e.g., infrared Transformer) to extract complementary features from other modalities (such as temperature and distance). The features from the two modalities are then concatenated, and feature interaction (non-linear fusion) is achieved through a multilayer perceptron (MLP). A two-layer fully connected network is employed, incorporating ReLU activation and residual connections to preserve the basic information of the RGB modalities. An attention mechanism dynamically adjusts the weights of the multimodal features, ultimately yielding the fused multimodal image. The large-scale visual model first captures the intrinsic relationships between multimodalities through self-supervised learning, and then fine-tunes downstream to adapt to specific tasks, avoiding overfitting or convergence difficulties caused by directly training complex models.
[0109] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model, characterized in that, Includes the following steps: S1: Acquire images of different modalities and preprocess the images to obtain preprocessed multimodal images; S2: Input the multimodal images into the constructed large visual model, construct a self-supervised loss function, and train the large visual model using the multimodal images based on the self-supervised loss function; S3: Based on the downstream task of multimodal image fusion, construct the total task loss function and fine-tune the large visual model that considers the downstream task of multimodal image fusion.
2. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 1, characterized in that, In step S1, images of other modalities are converted to RGB format through statistical normalization, forming an image pair with the original RGB image. The image data is mapped to a standard normal distribution through a linear transformation, and the process satisfies the expression: in, This represents the average pixel value of a single image. Indicates the standard deviation of pixels; The statistically normalized gradient calculation expression is as follows: The gradient of the original image. As a scaling factor.
3. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 2, characterized in that, The multimodal images after statistical normalization are preprocessed as follows: Multimodal images are acquired using a beam splitter to ensure spatial alignment; data augmentation techniques are introduced; and a specific preprocessing pipeline is designed based on the characteristics of different modalities. Batch normalization is used to accelerate image data convergence, as expressed by: in, and denoted as batch mean and standard deviation, respectively; γ and β are learnable parameters; and ϵ is a small constant. Finally, the alignment accuracy is verified using the mean square error, expressed as: Where H and W represent the number of pixels in the vertical and center directions of the image, respectively.
4. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 1, characterized in that, The large-scale visual model includes: an image segmentation module based on the Transformer model, a feature extraction module, and a feature fusion module. The feature extraction module uses the RGB-Transformer model to extract high-level semantic features of the original RGB images in the image pair. High-level features of other modal images are extracted using the X-Transformer model. Implemented using a multilayer perceptron (MLP) and The feature interaction; the feature fusion module is based on a two-layer fully connected network, with the addition of ReLU activation function and residual connection.
5. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 4, characterized in that, In step S2, the image segmentation module divides the image in the image pair into image blocks, and each block is flattened into a vector to form an image block sequence P; the position code E is generated using a sine-cosine function, with the expression: Where pos is the absolute position of the image patch. For model dimensions, 2i and 2i+1 correspond to even and odd dimensions, respectively; The feature extraction module iteratively processes the combination of image patch sequence P and position code E, as expressed by: Extracting high-level semantic features from the original RGB images in the image pair High-level features of other modal images splicing and The features after row concatenation are represented as follows: The spliced features are nonlinearly fused using a multilayer perceptron to generate preliminary fused features. The expression is: in, Represents feature concatenation, ReLU activation function , For the first layer of weights, For the second layer of weights, , For bias; Preserving the basic information of the RGB modes, a residual connection is constructed, expressed as: To adaptively adjust the weights of multiple features, an attention mechanism is introduced, expressed as: in, These are weighting coefficients; Layer normalization is used to normalize the input of all neurons in each layer, thereby stabilizing the output distribution of each layer.
6. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 5, characterized in that: Based on feature relationship similarity Using image patch matching loss Image patch prediction loss And comparative loss We use weighted summation to construct a self-supervised loss function, expressed as: Among them, image patch matching loss The expression is: Where sim is the cosine similarity, and i, j, and k are index variables. Controlled distribution softening; Image patch prediction loss The expression is: in, For the true value, For predicted values, This is the prediction layer weight matrix. For context feature weight matrix, For contextual features, For context feature bias terms, For the prediction layer bias term; Comparative loss The expression is: in, As a positive sample, For negative samples, The temperature distribution is controlled; finally, a multimodal image fusion model is obtained.
7. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 6, characterized in that, The optimization strategy based on the self-supervised loss function is as follows: Set , , The initial value is dynamically adjusted during training. , , The ratio of positive to negative samples is used to sample negative samples in real time within each training batch, and momentum updates are used to optimize the selection of positive and negative samples.
8. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 6, characterized in that, The large visual model is adapted to downstream tasks, and a total task loss function is constructed, which consists of classification loss. and regression loss The weighted fusion is expressed as: in, and This represents the task weight.
9. The method for progressive pre-training and adaptive fine-tuning of multimodal fusion based on a large visual model according to claim 8, characterized in that, The optimization strategy for fine-tuning downstream tasks based on multimodal image fusion is as follows: Using learning rate scheduling with cosine annealing learning rate The expression is: in, The initial learning rate is t, the current epoch is t, and the total epochs are T. Pre-trained features are retained, 90% of the self-supervised pre-trained layers are frozen, and only the subsequent downstream task heads are trained.
10. A multimodal image fusion method based on a large visual model, characterized in that, Includes the following steps: S10: The visual large model is trained based on the progressive pre-training and adaptive fine-tuning method of multimodal fusion based on the visual large model as described in any one of claims 1 to 9, to obtain a trained visual large model. S11: Acquire images of different modalities and preprocess the images to obtain multimodal images; S12: Fuse multimodal images based on the trained large visual model.