Optimization method and system for adaptive multi-level fine-tuning of multimodal large models

Abstract

The present invention relates to the field of multimodal large models, and discloses an optimization method and system for adaptive multi-level fine-tuning of multimodal large models. The method constructs a multimodal data set; encodes the modal data in a single group of samples based on a pre-trained multimodal basic large model to generate a feature representation; calculates the task relevance score, feature information score and gradient scale score of each modal data feature to generate a single modality score; calculates the modal alignment score, information complementarity score and synergy gain score between different modalities to generate a cross-modal interaction score; constructs a depth adjustment demand index and an interaction intensity demand index based on the two scores, selects a combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction based on the index value, and adaptively optimizes the multimodal basic large model; verifies whether the model performance meets the requirements to iteratively optimize until the requirements are met. The present invention saves computing resources and improves the representation and generalization capabilities of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to an optimization method and system for adaptive multi-level fine-tuning multimodal large models. Background Art

[0002] With the rapid development of the Transformer architecture, it is now possible to convert inputs from different modalities (including text, images, speech, and video) into semantic representations using their own dedicated encoders. This can then be aligned across modalities within a unified, multi-layered Transformer model, achieving true multimodal fusion. This technological breakthrough supports the construction of powerful large multimodal models. By integrating information from diverse data sources (such as text, images, speech, and video), large multimodal models achieve a comprehensive perception and understanding of the real world, providing more comprehensive and in-depth insights. This lays the foundation for building more intelligent and natural human-computer interaction systems and is crucial for a variety of practical applications. Large multimodal models can fully exploit the inherent connections between data from different modalities, enabling cross-modal information fusion and reasoning, helping to improve the decision-making capabilities and generalization performance of AI systems. Furthermore, the application of large multimodal models in various vertical fields has also brought significant value. For example, in the medical field, multimodal big models can comprehensively analyze multi-source data such as patients' images, medical records, genes, etc., to assist doctors in making more comprehensive and accurate diagnoses; in the field of autonomous driving, multimodal big models can achieve real-time perception and prediction of the vehicle's surrounding environment, improving driving safety and decision-making intelligence.

[0003] However, fine-tuning large multimodal models to suit specific tasks presents several challenges. First, large multimodal models typically contain a massive number of parameters, placing heavy demands on computing resources and training time. Traditional full-model fine-tuning methods are inefficient and struggle to meet real-time and cost constraints. Second, the data distribution and characteristics of different tasks vary widely. The heterogeneity of multimodal data requires models to maintain a high degree of flexibility and adaptability when processing different types of data. A unified fine-tuning strategy is difficult to address across all tasks.

[0004] Existing fine-tuning technologies for large multimodal models have the following shortcomings: First, most fine-tuning methods make adjustments to the parameter space of the entire model, while ignoring the differences between different task levels and modalities. This one-size-fits-all strategy is not only computationally inefficient, but also makes it difficult to fully explore the characteristics of task data and the correlation between modalities. Secondly, existing methods usually rely on manually designed hyperparameters and fine-tuning strategies, and lack the ability to adapt to task characteristics. This leads to a large amount of manual debugging and trial and error in practical applications, affecting the practicality and generalization performance of the model. Furthermore, most fine-tuning algorithms focus on the optimization of a single modality and do not consider the interaction and fusion between different modalities. This limits the representation ability of large multimodal models and cannot fully utilize their advantages in cross-modal understanding and generation. Summary of the Invention

[0005] To solve the technical problems existing in the prior art, the present invention provides an optimization method and system for adaptive multi-level fine-tuning of a multi-modal large model. The present invention designs a multi-granularity fine-tuning strategy that can adaptively select the optimal fine-tuning method and range according to the task hierarchy and modal characteristics, thereby avoiding waste of computing resources. In addition, during the fine-tuning process, the interaction between different modalities is modeled to achieve effective fusion of cross-modal information and enhance the representation and generalization capabilities of the model.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention discloses an optimization method for an adaptive multi-stage fine-tuning multi-modal large model, including S1 to S6.

[0008] S1. Construct a multimodal dataset, which includes multiple groups of samples, each group of samples contains different types of modal data.

[0009] S2. Encode the modal data in a single set of samples based on a pre-trained multimodal basic model to generate standardized feature representations.

[0010] S3. Calculate the task relevance score, feature information score, and gradient scale score for each modal data feature to generate a single modal score; among them, the task relevance score is used to measure the semantic association strength between the sample feature and the task goal, the feature information score is used to measure the information richness of the sample feature; and the gradient scale score is used to measure the importance of the sample feature.

[0011] S4. Calculate the modal alignment score, information complementarity score, and synergistic gain score between different modalities to generate a cross-modal interaction score; among them, the modal alignment score is used to measure the degree of matching between sample features of different modalities in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between sample features of different modalities, and the synergistic gain score is used to measure the synergistic effect generated by the fusion of sample features of different modalities.

[0012] S5. Based on the single-modal comprehensive score and cross-modal interaction score, construct the depth adjustment demand index and interaction intensity demand index. According to the index values, dynamically select the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction to adaptively optimize the multimodal basic large model.

[0013] S6. Verify whether the performance of the multimodal basic large model meets the requirements. If so, complete fine-tuning. Otherwise, return to S2 to iteratively optimize the model using the next set of samples until the requirements are met.

[0014] The present invention discloses an optimization system for an adaptive multi-stage fine-tuning multimodal large model, which applies the optimization method for an adaptive multi-stage fine-tuning multimodal large model as described above; the system includes: a data acquisition module, an encoding module, a single-modal multi-granularity adaptive fine-tuning module, a cross-modal interaction modeling module, a fine-tuning strategy implementation module, and a verification feedback module.

[0015] The data acquisition module is used to construct a multimodal dataset, which includes multiple groups of samples, each group of samples contains different types of modal data;

[0016] The encoding module is used to encode the modal data in a single set of samples based on the pre-trained multimodal basic model to generate standardized feature representations;

[0017] The unimodal multi-granularity adaptive fine-tuning module is used to calculate the task relevance score, feature information score, and gradient scale score of each modal data feature to generate a unimodal score. The task relevance score is used to measure the semantic association strength between the feature and the specific task goal, the feature information score is used to measure the information richness of the feature, and the gradient scale score is used to measure the importance of the feature.

[0018] The cross-modal interaction modeling module is used to calculate the modal alignment score, information complementarity score, and synergy gain score between different modalities to generate a cross-modal interaction score. The modal alignment score is used to measure the degree of matching between different modal features in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between different modalities, and the synergy gain score is used to measure the synergistic effect generated by the fusion of different modal features.

[0019] The fine-tuning strategy implementation module is used to construct deep adjustment demand indicators and interaction intensity demand indicators based on the single-modal comprehensive score and cross-modal interaction score. It dynamically selects the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction according to the indicator values ​​to adaptively optimize the multimodal basic large model.

[0020] The verification feedback module is used to verify whether the performance of the multimodal basic large model meets the requirements. If so, fine-tuning is completed; otherwise, it returns to the encoding module to iteratively optimize the model using the next set of samples until the requirements are met.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention innovatively proposes a dual-path parallel evaluation system. Through the two technical routes of single-modality multi-granularity adaptive fine-tuning and cross-modal interactive modeling, it breaks through the limitations of traditional methods that only focus on single-modality optimization and realizes fine-grained adjustment of model parameters. In the single-modality path, fine-grained feature evaluation is achieved through analysis of three dimensions: task characteristics, feature quality, and semantic structure, avoiding the "one-size-fits-all" global fine-tuning in the existing technology and significantly improving computational efficiency. In the cross-modal path, deep interactive modeling between modalities is achieved through modal alignment evaluation, information supplementation, and collaborative gain analysis, solving the problem of insufficient modal fusion in the existing technology.

[0023] In addition, based on the score fusion results, the present invention can adaptively select the optimal combination strategy of deep-shallow fine-tuning and strong-weak interaction, avoiding the limitations of traditional methods that rely on manual parameter adjustment and improving the practicality and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flowchart of the optimization method for adaptive multi-stage fine-tuning multimodal large model in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] See also Figure 1 The present invention provides an optimization method for an adaptive multi-level fine-tuning multi-modal large model, including S1~S6.

[0027] S1. Construct a multimodal dataset, which includes multiple groups of samples, each group of samples contains different types of modal data.

[0028] S2. Encode the modal data in a single set of samples based on a pre-trained multimodal basic model to generate standardized feature representations.

[0029] S3. Calculate the task relevance score, feature information score, and gradient scale score for each modal data feature to generate a single modal score; among them, the task relevance score is used to measure the semantic association strength between the sample feature and the task goal, the feature information score is used to measure the information richness of the sample feature; and the gradient scale score is used to measure the importance of the sample feature.

[0030] S4. Calculate the modal alignment score, information complementarity score, and synergistic gain score between different modalities to generate a cross-modal interaction score; among them, the modal alignment score is used to measure the degree of matching between sample features of different modalities in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between sample features of different modalities, and the synergistic gain score is used to measure the synergistic effect generated by the fusion of sample features of different modalities.

[0031] S5. Based on the single-modal comprehensive score and cross-modal interaction score, construct the depth adjustment demand index and interaction intensity demand index. According to the index values, dynamically select the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction to adaptively optimize the multimodal basic large model.

[0032] S6. Verify whether the performance of the multimodal basic large model meets the requirements. If so, complete fine-tuning. Otherwise, return to S2 to iteratively optimize the model using the next set of samples until the requirements are met.

[0033] This embodiment also provides an optimization system for an adaptive multi-level fine-tuning multimodal large model, which applies the optimization method for the adaptive multi-level fine-tuning multimodal large model as described above; the system includes: a data acquisition module, an encoding module, a single-modal multi-granularity adaptive fine-tuning module, a cross-modal interaction modeling module, a fine-tuning strategy implementation module and a verification feedback module, which correspond to the above 6 steps respectively. Each module can also be constructed by multiple sub-modules to achieve multiple functions.

[0034] To address the numerous challenges that current large multimodal models face in adapting to downstream tasks, such as the difficulty in quantifying the importance of different modal parameters, underutilization of intermodal interaction information, and the waste of computational resources, this paper proposes an adaptive fine-tuning framework based on multi-granularity evaluation. By designing an adaptive evaluation mechanism and fine-tuning strategy selection algorithm, this framework achieves an optimal balance between model performance and computational resources.

[0035] The core innovations of this invention are mainly reflected in two aspects: first, a multi-granular parameter importance assessment method is proposed, which can accurately quantify the importance of parameters at different levels from three dimensions: task relevance, feature information content, and gradient sensitivity; second, a novel cross-modal interaction modeling mechanism is designed to achieve efficient fusion of modal information by evaluating the alignment, complementarity, and synergistic gain between modalities. These innovations enable this invention to significantly reduce computing resource consumption while ensuring model performance.

[0036] The adaptive multi-stage fine-tuning optimization solution for large multimodal models provided by this invention can automatically select the optimal fine-tuning strategy based on the characteristics of different modal data and task requirements, achieving efficient optimization of model performance. The following details the overall architecture of this embodiment and the specific implementation of each core module.

[0037] 1. System input and preprocessing

[0038] At the entry point of the entire system, the input data from different modalities must first be processed and normalized. The multimodal dataset received by the system is formally represented as:

[0039]

[0040] Here, m represents the number of modes supported by the system. Indicates the samples, is the corresponding label set, where (C is the number of label categories), R is the set of real numbers, n This unified data representation lays the foundation for subsequent multimodal processing.

[0041] 1.1 Text Mode Processing

[0042] As one of the most fundamental modalities in a multimodal system, the quality of its preprocessing directly affects the model's understanding ability. This paper adopts an advanced word segmentation strategy to convert the input text into a standardized sequence representation:

[0043] ;

[0044] Where, Represents text modality features; Represents a word segmenter; Represents input text, is the secondary table size; is the length of the text sequence, where The default setting is 768, which is consistent with the mainstream pre-training model.

[0045] The word segmentation process here uses the BPE (Byte-Pair Encoding) algorithm implemented by SentencePiece, which can effectively balance the size of the vocabulary and the granularity of word segmentation, and is particularly suitable for processing multilingual text. Specifically, the configuration and processing flow of the word segmenter are as follows:

[0046] ① Vocabulary setting: The vocabulary size is set to 32K, which can well balance the vocabulary coverage and model complexity.

[0047] ② Contains special tags: [CLS] for sequence representation, [SEP] for separating different text segments, [MASK] for masked language modeling, and [PAD] for sequence alignment.

[0048] ③ Sequence processing: The maximum sequence length L can be configured to 512 / 1024 / 2048, depending on the specific task requirements. A sliding window strategy is used to segment very long sequences to ensure that important information is not lost. Short sequences are padded with [PAD] markers to ensure batch consistency.

[0049] ④ Position encoding strategy: Combining sinusoidal position encoding and learnable position encoding, sinusoidal encoding provides prior position information: , the learnable encoding can be adapted to specific task requirements through training.

[0050] After these processing steps, the text data is converted into a Tensor of .

[0051] 1.2 Image Modality Processing

[0052] As one of the most information-rich modalities in a multimodal system, the quality of image preprocessing directly affects the model's ability to understand visual information. To ensure effective extraction and representation of image features, this paper uses image preprocessing. After processing, the image data is mapped into a standardized tensor representation:

[0053] ;

[0054] Where, is the image modality feature; For standardization, Standardize the image; is the resolution normalization process. H and W are the height and width of the image respectively.

[0055] This process consists of three core steps, each addressing specific technical challenges. The following details the implementation of each step:

[0056] Resolution Normalization (Resize Module): The base resolution is 224×224, suitable for common vision tasks; the HD resolution is 384×384, used for scenarios requiring greater detail; and the UHD resolution is 512×512, suitable for fine-grained recognition tasks. This resolution selection uses an adaptive strategy, dynamically adjusting based on the input image's native resolution and task requirements. The resizing process uses a bicubic interpolation algorithm to maintain image quality while ensuring geometric feature continuity.

[0057] ① Resolution Normalization (Resize Module): The base resolution is 224×224, suitable for common vision tasks; the HD resolution is 384×384, used for scenarios requiring greater detail; and the UHD resolution is 512×512, suitable for fine-grained recognition tasks. This resolution selection uses an adaptive strategy, dynamically adjusting based on the input image's native resolution and task requirements. The resizing process utilizes a bicubic interpolation algorithm to maintain image quality while ensuring geometric feature continuity.

[0058] ② Image Enhancement (ImageAugment module): To improve the robustness and generalization capabilities of the model, a multi-layered data augmentation strategy was designed. First, color enhancement was applied, with brightness adjusted within a ±40% range to simulate varying lighting conditions; contrast adjusted within a ±40% range to enhance feature contrast; saturation adjusted within a ±40% range to enrich color representation; and hue shift adjusted within a ±10% range to provide color diversity. Geometric transformations were also applied, with the image randomly flipped horizontally with a probability of 0.5 to enhance directional invariance; and the original region was randomly cropped at a ratio of 0.8-1.0, while maintaining the aspect ratio to avoid image distortion.

[0059] ③ Normalization (Normalize module): To stabilize model training, images are normalized at the pixel level: normalize = Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]). These normalization parameters are statistically derived from the ImageNet dataset and are universally applicable. Subtracting the mean eliminates overall brightness differences in the image, dividing by the standard deviation unifies the feature scale, and channel-wise processing accounts for differences in the distribution of the three RGB channels.

[0060] After these carefully designed preprocessing steps, the input image is converted into a standardized feature representation, providing high-quality visual features for subsequent multimodal fusion and task learning. This processing method not only ensures the integrity of the information, but also significantly improves the learning efficiency and generalization ability of the model.

[0061] 1.3 Video Modality Processing

[0062] As a composite modality of temporal and spatial features, video data processing requires simultaneous consideration of two key aspects: spatial feature extraction and temporal modeling. This paper designs an end-to-end video processing flow to convert input video into a standardized spatiotemporal feature representation:

[0063] ;

[0064] Where, is the video modality feature; It is time series sampling processing; For spatial feature processing; T 1×3× H × W is the four-dimensional structure of the video feature tensor, T 1 represents the number of sampling time frames, 3 represents the RGB three-color channels, H and W Respectively represent the height and width of each frame image; For input video.

[0065] ① Temporal Sampling (TemporalProcess module): An adaptive sampling strategy is designed to process video sequences of varying lengths. Sampling configuration includes: frame count settings, supporting dynamic configurations of 8 / 16 / 32 frames; sampling mode settings, including uniform sampling (suitable for overall motion understanding), dense sampling (focusing on motion details), and random sampling (increasing training sample diversity); and temporal enhancement settings, including random segment sampling, speed perturbation, and temporal masking.

[0066] ② Spatial Feature Processing (SpatialProcess module): This module inherits and expands the functionality of image enhancement processing, and also adds temporal consistency enhancement processing. Specifically, it prevents changes between multiple frames of the video stream, otherwise temporal consistency is destroyed.

[0067] ③ Motion feature extraction: In order to better capture dynamic information, multiple motion feature extraction methods are integrated, including motion optical flow feature extraction and motion boundary detection.

[0068] 1.4 Audio Modal Processing

[0069] As a time-frequency domain signal, audio data processing requires special attention to the extraction of spectral features and the preservation of timing characteristics. The present invention designs a complete audio processing flow:

[0070] ;

[0071] Where, is the audio modal feature; For audio feature extraction processing; As basic pretreatment; For input audio.

[0072] The audio processing flow mainly includes the following key steps:

[0073] ① Basic preprocessing: First, standardize the raw audio to lay the foundation for subsequent feature extraction. This includes audio resampling, which converts the audio sampling rate to 16kHz (speech) or 44.1kHz (music) to ensure consistent feature extraction. Next, volume normalization normalizes the audio signal to the range [-1, 1] to eliminate the impact of volume differences. Finally, silence detection and processing uses an energy-based detection algorithm to identify and remove silent segments below -60dB. The audio is also segmented. For longer audio, segments are processed in 10-second increments, with a 25% overlap between segments to avoid information loss.

[0074] ② Spectral feature extraction: After completing basic preprocessing, the system enters the feature extraction phase. Considering the needs of different application scenarios, the system supports multiple spectral feature extraction methods. The most basic is the Log-Mel Spectrogram (Log-Mel Spectrogram). This process first converts the time domain signal to the frequency domain using a short-time Fourier transform (SFT). A 2048-point FFT and a 512-point frame shift are used to ensure a balance between time and frequency resolution. A Mel filter bank is then used to map the linear frequency axis to a Mel scale that better aligns with human auditory perception. Depending on the application scenario, 80 (for speech processing) or 128 (for music processing) Mel filters can be selected. Finally, the spectrogram is compressed using a logarithmic transform to better highlight features in low-energy regions, making the resulting feature map more consistent with human hearing.

[0075] ③ Audio feature enhancement and data augmentation: In order to further improve the robustness and generalization ability of the model, the present invention also designs a comprehensive audio enhancement strategy. These enhancement methods are carried out in both the frequency domain and the time domain. In the frequency domain, the system adopts SpecAugment technology to enhance the anti-interference ability of the model by randomly blocking parts of the frequency and time dimensions. Specifically, the system will randomly block up to 30 Mel filter channels and 40 time frames, and can flexibly combine multiple masked areas. In the time domain, the system implements a series of signal-level enhancement methods, including adding background noise with different signal-to-noise ratios, simulating different degrees of room reverberation effects, adjusting pitch and playback speed within a specific range, etc. The combined use of these enhancement methods significantly improves the model's adaptability to various acoustic environments.

[0076] After these carefully designed processing steps, the original audio signal is finally converted into a standardized feature representation with the dimension of ,inF represents the frequency dimension (usually 80 or 128), T 2 represents the time dimension (dynamically adjusted based on the audio length). This unified feature representation not only preserves the key information of the audio signal but also provides a standardized input format for subsequent modal fusion and model training. Through this complete processing pipeline, the system can effectively capture and utilize the time-frequency features in audio, providing high-quality audio feature representation for multimodal learning tasks.

[0077] 2. Multimodal Basic Large Model Encoding

[0078] After completing the standardized preprocessing of each modality data, the present invention uses a large-scale pre-trained multimodal basic model M (such as the Unified-IO-2 multimodal large model) as the core framework for feature encoding. This model uses a unified encoding-decoding architecture and can simultaneously process multiple modal inputs such as text, images, videos, and audio. Specifically, the preprocessed text sequence , image features , video frame sequence and audio features are simultaneously fed into model M. The model first performs preliminary encoding of the data from each modality using specially designed modality-specific encoders, mapping the inputs from different modalities into a unified feature space. This unified feature representation lays the foundation for subsequent modal interaction and semantic understanding. After encoding by model M, the system obtains a set of standardized feature representations, which are then fed into the subsequent basic semantic feature modeling and abstract semantic modeling modules for in-depth analysis.

[0079] After encoding the multimodal basic model M, the system obtains a preliminary feature representation. However, in order to better explore the deep semantic information within the modality and the correlation between modalities, the present invention designs two key optimization modules: a single-modal multi-granularity adaptive fine-tuning module and a cross-modal interaction modeling module. These two modules use different optimization strategies and evaluation mechanisms to improve the system's understanding ability from two dimensions: within the modality and between modalities. Among them, the single-modal multi-granularity adaptive fine-tuning module is mainly responsible for fine-grained optimization and adjustment of the features of each modality, while the cross-modal interaction modeling module is committed to enhancing the semantic association and information complementarity between different modalities. This dual optimization mechanism ensures that the system can fully and deeply understand the semantic connotation of multimodal data.

[0080] 3. Single-modality multi-granularity adaptive fine-tuning

[0081] The single-modal multi-granularity adaptive fine-tuning module in the present invention is designed to optimize the feature representation after being encoded by the basic model M. For the input multimodal data, the basic model M first generates the corresponding feature matrix representation, which is recorded as , where the subscript m represents the modality type, which can be any of text, image, video, or audio, and d represents the dimension of the feature space. In order to achieve accurate feature evaluation and optimization, the present invention designs three scoring submodules with different optimization objectives.

[0082] 3.1 Task Relevance Scoring Module

[0083] The task relevance scoring module uses a deep semantic matching mechanism to evaluate the semantic association strength between features and specific task objectives. First, the input task description text is processed by the task encoder. The encoder uses a bidirectional LSTM structure and inputs the task description sequence. ,in Indicates the The encoder’s hidden state update formula is:

[0084] ;

[0085] ;

[0086] in and denote the hidden states of the forward and backward LSTMs respectively, and Represent forward and backward LSTM processing respectively; Represents the word unit input at the tth time step in the task description sequence.

[0087] By concatenating the bidirectional hidden states and performing a linear transformation, we get the vector representation of the task:

[0088] ;

[0089] in, Vectorized representation of tasks, capturing the semantic information of task description; is the learnable weight matrix, h is the LSTM hidden layer dimension, is the bias term, Represents the concatenated last hidden state of the forward LSTM and the first hidden state of the backward LSTM, which contains the complete semantic information of the sequence.

[0090] After obtaining the task vector representation, the present invention calculates the relevance between features and tasks by constructing a multi-level attention network. The network first performs local context encoding on the feature sequence, using a sliding window mechanism with a window size of k. For each position in the feature sequence, , extract its local context features:

[0091] ;

[0092] in, Represents a one-dimensional convolution operation, which is used to capture local feature dependencies; For the i local context features; is the i-th feature vector in the feature sequence, representing the original feature information processed by the model.

[0093] Then, the semantic matching degree between the features and the task is calculated through the multi-head attention mechanism:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] Where, Q 、 K 、 V They are query, key, and value respectively; , , are the transformation matrices for query, key, and value respectively, is the dimension of the attention head; For the j The output of an attention head; is the activation function; It is the matrix product of the query vector and the key vector, which is used to calculate the attention weight distribution.

[0099] The outputs of multiple attention heads are concatenated and linearly transformed to obtain the attention score :

[0100] ;

[0101] in is the number of attention heads. is the linear transformation matrix of the multi-head attention output, which is used to map the concatenated attention head output to the final representation space; is the corresponding bias vector.

[0102] Finally, the attention score is adjusted through the position weighting mechanism, and the adjusted attention score :

[0103] ;

[0104] in is the position encoding function, which is used to introduce the influence of position information. In this way, a scoring sequence reflecting the relevance of the task is obtained :

[0105] .

[0106] 3.2 Feature Information Scoring Module

[0107] The feature information scoring module quantitatively evaluates the value of features from the perspective of information theory. This module first constructs a probabilistic model of feature distribution and then measures the importance of features using statistics such as information entropy and mutual information.

[0108] The specific implementation process is as follows:

[0109] In order to accurately estimate the probability distribution of features, the present invention adopts an adaptive kernel density estimation method. , its probability density estimation formula is:

[0110] ;

[0111] Where, is the number of observed samples, indicating the total number of feature vectors contained in the feature sequence; is the bandwidth parameter, which controls the smoothness of the kernel function and determines the smoothness of the probability density estimation; is the dimension of the feature space, indicating the dimension of each feature vector; is the eigenvector of the jth sample, which is used to estimate the reference point of the probability density.

[0112] is a multivariate Gaussian kernel function:

[0113] ;

[0114] H is the bandwidth matrix, which is determined adaptively by the Silverman criterion:

[0115] ;

[0116] in, is the empirical covariance matrix of the features.

[0117] This adaptive bandwidth selection strategy can dynamically adjust the shape of the kernel function according to the local density characteristics of the data, thereby improving the accuracy of probability estimation.

[0118] After obtaining the probability distribution estimate of the feature, the system calculates the conditional entropy of the feature. In order to reduce the computational complexity, the feature sequence is divided into several overlapping subsequences, each of which has a length of l and an overlapping length of l / 2. , and its conditional entropy calculation formula is:

[0119] ;

[0120] in express The context window of eigenvectors. Conditional probability Estimate by Bayes' formula:

[0121] ;

[0122] In order to capture the interdependence between features, the system calculates the mutual information between subsequences. and , and the mutual information calculation formula is:

[0123] ;

[0124] Considering the computational efficiency, the mutual information estimation method based on k-nearest neighbors is adopted:

[0125] ;

[0126] in, K 1 for k The number of nearest neighbors in the nearest neighbor algorithm; is the digamma function, and Respectively and In their respective marginal spaces Number of neighbors, is the total number of samples.

[0127] In order to comprehensively consider the local and global importance of features, the system introduces a multi-scale feature importance scoring mechanism. At the local scale, the normalized conditional entropy of the feature is calculated:

[0128] ;

[0129] Where, Represents a characteristic subsequence The normalized conditional entropy of ; Represents a characteristic subsequence In the conditions The conditional entropy under reflects the characteristic subsequence local importance in a specific conditional context; Represents a characteristic subsequence In the conditions The conditional entropy under reflects the characteristic subsequence local importance in a specific context; Indicates that the conditional entropy in the brackets takes the maximum value;

[0130] On a global scale, calculate the average mutual information between a feature and other features:

[0131] ;

[0132] Where, Represents a characteristic subsequence The weighted average mutual information with all other features; Represents a characteristic subsequence and characteristic subsequences The mutual information measure between is the decay weight based on sequence distance, and the expression formula is:

[0133] ;

[0134] is the temperature parameter, which is used to control the rate of distance attenuation; It is an exponential function with the natural constant e as its base.

[0135] The final feature information score is obtained through weighted combination:

[0136] ;

[0137] Where, Represents a characteristic subsequence Comprehensive information score of is a learnable weight parameter, is the feature diversity measurement function:

[0138] ;

[0139] Where, Represents a characteristic subsequence and characteristic subsequences This multi-scale feature information evaluation mechanism can fully capture the information value of the feature and provide reliable importance guidance for subsequent feature optimization.

[0140] 3.3 Gradient Scale Scoring Module

[0141] The gradient scale scoring module evaluates the importance of features by analyzing the gradient behavior during the optimization process. This module not only considers first-order gradient information but also introduces second-order curvature information to build a complete gradient tracking and analysis system. The specific implementation process is as follows:

[0142] During model training, for each feature vector , the system first calculates its first-order gradient with respect to the loss function L:

[0143] ;

[0144] Where, is the partial derivative; in order to reduce the noise impact of stochastic gradient, this paper designs a dynamic gradient accumulation mechanism. This mechanism maintains an exponential moving average (EMA) gradient statistic:

[0145] ;

[0146] in is the momentum coefficient, Indicates the current time step; and Respectively l Parameters in time step t and time steps t The exponential moving average of the gradient is -1; For the i The actual gradient value of the parameter at the current time step. In order to eliminate the deviation in the early stage of training, the EMA is corrected for the deviation:

[0147] ;

[0148] Where, is the exponential moving average of the gradient after bias correction; is the momentum coefficient of t At the same time, the system also tracks the second moment estimate of the gradient:

[0149] ;

[0150] ;

[0151] in is the decay rate of the second-order moment estimate; for t to the power of and Respectively i Parameters in time step t and t The exponential moving average of the squared gradient of -1; For the i Parameters at the current time step t The actual gradient value of is the bias-corrected estimate of the second-order moment of the gradient. Based on these statistics, the system calculates the adaptive learning rate:

[0152] ;

[0153] in, For the Adaptive learning rate for parameters; is the basic learning rate, is a numerical stability constant.

[0154] In order to more accurately evaluate the optimization difficulty of the feature, the present invention uses the finite difference method to approximate the diagonal elements of the Hessian matrix:

[0155] ;

[0156] In order to reduce computational overhead, the present invention adopts a random sampling strategy, where only the second-order derivatives of some features are calculated in each small batch.

[0157] Based on the first-order gradient and second-order curvature information, the present invention designs an adaptive feature importance scoring function:

[0158] ;

[0159] Where, The first character sequence i Gradient sensitivity of each feature; 、 、 Weight coefficients to balance the contributions of different items; is the Sigmoid function; is the first-order gradient The exponential moving average gradient statistic of The deviation correction amount; is the diagonal element of the Hessian matrix approximated by the finite difference method, which is used to evaluate the optimization difficulty of the feature; is the second-order moment estimate.

[0160] In order to capture the dynamic change characteristics of features during the optimization process, this paper maintains a feature update history:

[0161] ;

[0162] in represents the update amount of the feature, For thei The update history set of features at time step t, and Represent the feature update amount at time step tk and time step t-1 respectively. Based on the update history, calculate the stability index of the feature:

[0163] ;

[0164] Where, For the i Stability index of each feature; Indicates the i The set of historical updates of features; To find the standard deviation function, is the averaging function.

[0165] The final gradient scale score is obtained by combining multiple indicators:

[0166] ;

[0167] in is the stability weight, is the direction consistency weight, is the consistency measure of the gradient direction:

[0168] ;

[0169] Where, For the i Gradient direction consistency measure of features; For the i Features at time step t The average gradient vector of K is the size of the historical time window, indicating the number of historical gradients considered; It is the cosine similarity function, which is used to measure the similarity between the directions of two gradient vectors. The closer the index value is to 1, the more consistent the gradient directions are. For the i Features at time step t - k The average gradient vector of .

[0170] After evaluation by the above three scoring modules, the comprehensive scoring vector of the feature is obtained:

[0171] ;

[0172] in is a learnable combination weight matrix, softmax is the activation function, 、 、 They are the feature stability scoring vector, feature information scoring vector and feature gradient scoring vector. Based on the comprehensive scoring, the present invention makes adaptive adjustments to the features:

[0173] ;

[0174] in represents element-wise multiplication, is the global adjustment step size, It is a layer normalization operation used to stabilize the scale of features. is the feature representation after adaptive adjustment; is the original feature representation;

[0175] In addition, the present invention also implements a feedback adjustment mechanism to dynamically adjust the weight of the scoring module by monitoring the performance indicators of downstream tasks:

[0176] ;

[0177] in is the regularization coefficient, which is used to prevent the weight from being overly concentrated in a certain scoring dimension. is the total loss function for feedback adjustment; It is the main loss function for downstream tasks; It is an L2 norm operation; through this adaptive scoring and optimization mechanism, it can accurately identify and enhance the most valuable feature representations for the task, significantly improving the performance of the model.

[0178] This multi-dimensional feature evaluation and optimization framework can not only accurately capture the importance of features, but also dynamically adjust the optimization strategy according to task requirements, providing a high-quality feature foundation for subsequent cross-modal interaction modeling.

[0179] 4. Cross-modal interaction modeling module

[0180] The cross-modal interaction modeling module mainly includes three key evaluation dimensions: modal alignment, information complementarity, and synergy gain, which are used to comprehensively evaluate and optimize the interaction quality between different modalities.

[0181] 4.1 Modal Alignment Scoring Module

[0182] Modality alignment is used to measure the matching degree of different modal features in the semantic space. Given the feature representation of two modalities and , and its alignment is calculated as follows:

[0183] (1) Feature normalization preprocessing to obtain preprocessing results and :

[0184] ;

[0185] .

[0186] (2) Calculate the semantic similarity matrix between modalities:

[0187] ;

[0188] in, for and The semantic similarity matrix between .

[0189] (3) Introducing local alignment metrics :

[0190] ;

[0191] Where, N is the characteristic sequence length; is the similarity matrix All elements of the row; Indicates the maximum value operation, which is used to find the The similarity between a feature and the best matching feature in the other modality.

[0192] (4) Global alignment calculate:

[0193] ;

[0194] Where, To find the average function; Used to measure the Euclidean distance between the average representations of two modal features.

[0195] (5) Final modal alignment score :

[0196] ;

[0197] Where, and is the balance coefficient; It is an exponential function with the natural constant e as its base.

[0198] 4.2 Information complementarity scoring module

[0199] Information complementarity measures the amount of complementary information carried by different modalities, and is specifically implemented as follows:

[0200] (1) Calculate the modality-specific information entropy:

[0201] ;

[0202] ;

[0203] Where, For modal i Information entropy of For modal j Information entropy of is the probability distribution function; For modal i The eigenvalues ​​in ; For modal j The eigenvalues ​​in .

[0204] It should be noted that the eigenvalues ​​in the mode ( and ) refers to the specific values ​​or states that may appear in a specific mode i or j.

[0205] The term "modal characteristics" is a broader concept, referring to the characteristics that can describe or characterize the overall properties of the modality, such as information entropy. It is a modal feature.

[0206] (2) Calculate conditional entropy:

[0207] ;

[0208] ;

[0209] Where, For known modes j Case modal i The conditional entropy of is the joint probability distribution; is the conditional probability distribution.

[0210] (3) Calculating mutual information :

[0211] ;

[0212] (4) Therefore, the information complementarity score For:

[0213] .

[0214] 4.3 Synergy Gain Scoring Module

[0215] The synergy gain is used to evaluate the synergy effect produced by the fusion of different modal features and is quantified by measuring the performance gain after fusion. The specific implementation is as follows:

[0216] (1) Basic performance evaluation: For any two modal features and , first evaluate its independent performance and :

[0217] ;

[0218] ;

[0219] in It is a performance evaluation function, which can be an evaluation indicator of a specific task (such as classification accuracy, retrieval accuracy, etc.).

[0220] (2) Collaborative feature generation: building a multi-level collaborative feature fusion network :

[0221] ;

[0222] ;

[0223] ;

[0224] in represents feature splicing, W 1~ W 3 and b 1~ b 3 is a learnable parameter. and They are the first and second layer networks respectively; is the Sigmoid function.

[0225] (3) Fusion performance calculation:

[0226] .

[0227] (4) Relative gain evaluation: Calculate the synergistic gain based on basic performance and fusion performance.

[0228] ;

[0229] ;

[0230] Where, is the absolute synergy gain, which represents the absolute improvement of the fusion feature performance relative to the best performance of a single modality; is the relative synergy gain, which represents the relative improvement rate of the fusion feature performance relative to the best performance of a single modality; Indicates the maximum value operation.

[0231] (5) Timing Stability Assessment: Introducing Timing Window , calculate the stability of the gain:

[0232] ;

[0233] Where, Indicates that at time step t The relative synergy gain value, ; To find the standard deviation function, is the averaging function.

[0234] (6) Final synergistic gain: After comprehensively considering relative gain and stability, we get:

[0235] ;

[0236] in To balance the parameters, they are dynamically adjusted according to the task characteristics.

[0237] 5. Adaptive fine-tuning strategy based on dual scoring

[0238] After completing unimodal multi-granularity adaptive fine-tuning and cross-modal interaction modeling, the system obtains two key scoring results. To fully leverage this scoring information to guide feature optimization, this paper designs a dual-scoring-based adaptive fine-tuning strategy framework. This framework comprehensively considers unimodal and cross-modal interaction scores to select the optimal fine-tuning strategy for different feature states.

[0239] In terms of unimodal scoring, its score Scored by task relevance , Feature Quality Score , and gradient scale scores composition:

[0240] ;

[0241] The weight coefficient Determined by grid search on the validation set, the weight coefficients are set to 0.4, 0.3, and 0.3 respectively; It can also be determined by grid search on the validation set.

[0242] In terms of cross-modal interaction scoring, the system utilizes modality alignment ( ), information complementarity ( ) and synergy gain ( ) three indicators. These indicators measure the semantic consistency, information complementarity, and synergy between different modal features. The comprehensive score of cross-modal interaction is calculated as follows:

[0243] ;

[0244] Where, and are the weight coefficients of the single modality score and modality alignment score in the deep adjustment demand index respectively; and are the weight coefficients of the information complementarity score and the synergy gain score in the interaction intensity demand index respectively; scoring the modal alignment component in cross-modal interactions; The information complementarity component in scoring cross-modal interactions; Scoring the synergy gain component in cross-modal interactions.

[0245] Based on these two types of scores, the present invention constructs a single-modal depth adjustment demand index and multimodal interaction intensity demand indicators :

[0246] ;

[0247] ;

[0248] Where, and are the weight coefficients of the single modality score and modality alignment score in the deep adjustment demand index respectively; and are the weight coefficients of the information complementarity score and the synergy gain score in the interaction intensity demand index respectively.

[0249] The above two indicators determine the depth and interaction strength of feature optimization respectively. The optimized threshold and The optimization threshold between samples across modalities. At this time, for each sample, the optimization process of the present invention will adaptively select one of the following four strategies:

[0250] (1) Deep fine-tuning + strong interaction strategy: and This strategy is used when . This strategy indicates that the quality of the single-modal features of the current sample needs to be significantly improved, and there is a strong complementary relationship between the modalities.

[0251] (2) Deep fine-tuning + weak interaction strategy: and This strategy indicates that the single-modal features of the current sample need to be deeply optimized, but the feature information between the modalities has been well aligned.

[0252] (3) Shallow fine-tuning + strong interaction strategy: and This strategy indicates that the quality of the single-modal features of the current sample is high, but the cross-modal feature complementarity information needs to be further strengthened.

[0253] (4) Shallow fine-tuning + weak interaction strategy: and This strategy indicates that the quality of the single-modal features of this sample is already good and the demand for inter-modal interaction is low.

[0254] In order to ensure the stability and adaptability of strategy selection, the system uses a sliding window mechanism to dynamically update the strategy selection threshold and Specifically, after each training cycle, the change in feature optimization effect is calculated based on the , the system will adjust the threshold accordingly. This adaptive mechanism ensures that the strategy selection can be continuously optimized as the training progresses, thereby achieving better feature optimization results.

[0255] 6. Optimization mechanism

[0256] Weight optimization update: Use gradient ascent method to optimize weight parameters:

[0257] ;

[0258] Where, is the objective function J For weight parameters w gradient; Represents partial derivatives; concatenation vectors ; is a regularization term used to prevent overfitting; is the regularization coefficient, which is used to control the regularization strength; For the The weight parameter at the iteration; is the learning rate, which is used to control the step size of each parameter update.

[0259] Termination condition judgment: Design a dual termination criterion. When any of the following conditions is met, the performance of the multimodal basic large model is judged to be qualified:

[0260] ;

[0261] ;

[0262] Where, and Respectively and The score value of the iteration; The score improvement threshold is defined as follows: when the score difference between two consecutive iterations is less than the threshold, the model performance is considered to be stable; is the gradient norm threshold. When the L2 norm of the gradient is less than the threshold, the model is considered to have reached the local optimum. is the L2 norm, which is used to measure the size of the gradient vector.

[0263] The dual termination criteria are designed for the following reasons: ① Condition 1 focuses on the convergence of performance metrics. If the performance improvement is already very small (condition 1 is satisfied), even if the gradient is not sufficiently small, further optimization may only bring about a small improvement. ② Condition 2 focuses on the stability of the parameter space. If the gradient is already very small (condition 2 is satisfied), even if the difference between the previous and next scores is not sufficiently small, the model is already close to the local minimum, and further optimization may lead to overfitting.

[0264] Feedback regulation mechanism: introducing a performance feedback loop

[0265] ;

[0266] ;

[0267] Where, is the performance change; and Respectively and Model performance indicators for iterations; and Respectively and The learning rate used in the iteration is the learning rate adjustment coefficient; is a symbolic function.

[0268] Through this multi-level scoring and optimization mechanism, the present invention can adaptively adjust the modal interaction strategy to maximize the synergistic gain effect while ensuring modal alignment. At the same time, the feedback adjustment mechanism can dynamically adjust the optimization strategy according to actual performance to ensure the stability and convergence of the system. The entire process ensures high-quality semantic alignment, information complementarity, and synergistic enhancement between different modal features. This multi-dimensional evaluation framework can not only comprehensively measure the quality of modal interaction, but also guide the system to perform targeted optimization and improve the effect of multimodal fusion.

[0269] Experimental / comparative verification:

[0270] In order to comprehensively evaluate the performance of this method on different modal combinations and task types, this embodiment selected multiple public multimodal datasets for experiments. Among them, MSCOCO is an image-text multimodal dataset containing more than 120,000 images and corresponding 5 description texts, which is widely used in image description and visual question answering tasks. CMU-MOSEI is a large-scale text-speech-video trimodal sentiment analysis dataset, which contains 23,453 annotated video clips from online video websites, covering 250 different topics and 1,000 different speakers. HowTo100M is a video-text dataset containing more than 1.36 million instructional videos and their corresponding subtitle texts. MIMIC-CXR is a multimodal dataset in the medical field, containing more than 227,000 chest X-rays and their corresponding radiology reports. This embodiment is divided into training set, validation set and test set in a ratio of 8:1:1.

[0271] This example selects the current mainstream multimodal basic large model as the optimization object, including:

[0272] 1. LLaVA-1.5: A vision-language model developed based on Vicuna-13B and CLIP ViT-L / 14, supporting image understanding and visual dialogue.

[0273] 2. BLIP-2: A visual-language model using the Q-Former architecture, based on FlanT5-XXL, capable of connecting visual encoders and large language models.

[0274] ImageBind: A multimodal model that supports six modalities (image, text, audio, depth, thermal, and IMU data), developed by the Meta AI research team.

[0275] 4. KOSMOS-2.5: A large-scale vision-language model developed by Microsoft that performs well in processing interleaved image and text inputs.

[0276] The experimental environment was configured with 8×NVIDIA A800 (80GB) GPUs and implemented in PyTorch 2.5.1. The batch size was 32, the initial learning rate was set to 1e-5, and a cosine annealing strategy was used. All experiments were repeated 10 times and the average was taken to eliminate randomness. This example compares the adaptive multi-level fine-tuning method of the present invention with mainstream fine-tuning methods, including full-parameter fine-tuning, LoRA (Low-Rank Adaptation), P-Tuning v2, Adapter-based fine-tuning, and QLoRA.

[0277] This example first compares the overall performance of the proposed method with mainstream fine-tuning methods on various datasets. All experiments were repeated 10 times, and the mean and standard deviation are reported. Performance comparisons were performed using the LLaVA-1.5 model on the MSCOCO dataset, and the results are shown in Table 1.

[0278] Table 1: Results of fine-tuning the LLaVA-1.5 model on MSCOCO using different methods

[0279] ;

[0280] On the MSCOCO dataset, our method achieves an accuracy improvement of 1.47 percentage points over the best baseline method (full parameter fine-tuning), while also achieving a lower standard deviation, indicating more stable performance. Of particular note, our method improves the generation quality metrics BLEU-4 and CIDEr by 0.92 and 1.75 points, respectively, demonstrating enhanced expressiveness for the model in vision-language generation tasks.

[0281] The performance of the ImageBind model is compared on the CMU-MOSEI multimodal sentiment analysis dataset. The results are shown in Table 2.

[0282] Table 2: Results of fine-tuning the ImageBind model on CMU-MOSEI using different methods

[0283] ;

[0284] On the CMU-MOSEI dataset, our method performs well in the multimodal sentiment analysis task, improving the F1 score by 1.82 percentage points and reducing the mean absolute error (MAE) by 0.022 over the best baseline. This demonstrates that our method can effectively integrate sentiment information from text, speech, and video modalities, improving the accuracy and stability of sentiment recognition.

[0285] The performance of the BLIP-2 model was compared on the MIMIC-CXR medical imaging dataset. The results are shown in Table 3.

[0286] Table 3: Results of fine-tuning the BLIP-2 model on MIMIC-CXR using different methods

[0287] ;

[0288] In the field of medical imaging, this method demonstrates significant advantages, improving diagnostic accuracy by 1.71 percentage points and the AUC by 0.013. Notably, the standard deviation of this method is 26.4% lower than the average of other methods, demonstrating its superior reliability and stability in demanding medical scenarios.

[0289] The performance comparison of the KOSMOS-2.5 model is performed on the HowTo100M video-text dataset. The results are shown in Table 4.

[0290] Table 4: Results of fine-tuning the KOSMOS-2.5 model on HowTo100M using different methods

[0291] ;

[0292] In the video understanding task, this method improved the action recognition accuracy by 1.62 percentage points and the retrieval R@5 by 1.67 percentage points. This shows that this method has certain advantages in processing time series data and long sequence features.

[0293] Computational resource consumption comparison is performed on the LLaVA-1.5 (13B) model. The results are shown in Table 5.

[0294] Table 5: Comparison of computational resource consumption on the LLaVA-1.5 (13B) model

[0295] ;

[0296] The proposed method not only offers advantages in performance but also demonstrates excellent resource efficiency. Compared to full parameter fine-tuning, this method reduces training time by 70.5% and GPU memory usage by 73.0%. Compared to other efficient parameter fine-tuning methods, this method reduces training time by an average of 9.7% and GPU memory usage by an average of 8.7%.

[0297] It is particularly noteworthy that the proportion of trainable parameters of this method is dynamically adjusted, ranging from 0.41% to 5.25%, which reflects the adaptability of the method - dynamically selecting the optimal parameter adjustment strategy according to different samples and task stages.

[0298] To verify the contribution of each component of the present invention, this embodiment conducted a detailed ablation experiment. Table 6 below shows the performance changes after removing different components from the LLaVA-1.5 model.

[0299] Table 6: Performance changes after removing different components from the LLaVA-1.5 model

[0300] ;

[0301] The ablation experiment results show that:

[0302] 1. The cross-modal interaction score is one of the most critical components. After removing it, the accuracy dropped by 2.36 percentage points, indicating that accurately capturing the interaction between modalities is important for multimodal models.

[0303] 2. Although the performance degradation of the adaptive strategy selection mechanism is relatively small (1.65 percentage points), it has a significant impact on computational efficiency. After its removal, the training time increases by 12.53%.

[0304] 3. The gradient scale score and synergy gain score each contributed approximately 0.95 and 1.30 percentage points of performance improvement.

[0305] 4. The sliding window dynamic threshold mechanism improves performance by 0.77 percentage points compared to the fixed threshold, while reducing training time by 8.34%.

[0306] This example further analyzes the contributions of the three subcomponents of the cross-modal interaction score, and the results are shown in Table 7.

[0307] Table 7: Contributions of the three subcomponents of the cross-modal interaction score

[0308] .

[0309] The results show that the importance of the three scoring mechanisms varies in different task types: the modal alignment score contributes more to the image-text description task; the information complementarity score is more critical in multimodal sentiment analysis; and the synergy gain score performs better in medical diagnosis and video understanding tasks that require subtle feature fusion.

[0310] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An optimization method for adaptive multi-stage fine-tuning of a multimodal large model, characterized in that: include: S1. Construct a multimodal dataset comprising multiple groups of samples, each group of samples containing different types of modal data; wherein the types of modalities include text modality, image modality, video modality, and audio modality; S2. Encode the modal data in a single set of samples based on a pre-trained multimodal base model to generate standardized feature representations; S3. Calculate the task relevance score, feature information score, and gradient scale score for each modal data feature to generate a single modality score; the task relevance score is used to measure the strength of the semantic association between the sample feature and the task goal, the feature information score is used to measure the information richness of the sample feature, and the gradient scale score is used to measure the importance of the sample feature; S4. Calculate the modal alignment score, information complementarity score, and synergy gain score between different modalities to generate a cross-modal interaction score; the modal alignment score is used to measure the degree of matching between sample features of different modalities in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between sample features of different modalities, and the synergy gain score is used to measure the synergistic effect generated by the fusion of sample features of different modalities; S5. Based on the single modality score and cross-modal interaction score, the depth adjustment demand index and the interaction intensity demand index are constructed. According to the index value, the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction is dynamically selected to adaptively optimize the multimodal basic large model. In step S5, the depth adjustment demand index D need and interaction intensity demand index I need The calculation formulas are as follows: Where κ1 and κ2 are the weight coefficients of the single modality score and modality alignment score in the depth adjustment demand index; μ1 and μ2 are the weight coefficients of the information complementarity score and synergy gain score in the interaction intensity demand index; scoring the modal alignment component in cross-modal interactions; The information complementarity component in scoring cross-modal interactions; is the synergy gain component in the cross-modal interaction score; S single Scoring for unimodality; Among them, by comparing D need , I need and the preset strategy selection threshold θ D ,θ I The size between them is selected as one of the following four strategies as the adaptive optimization direction: When D need >θ D And I need >θ I When using deep fine-tuning + strong interaction strategy; When D need >θ D And I need ≤θ I When , we adopt the deep fine-tuning + weak interaction strategy; When D need ≤θ D And I need >θ I When using shallow fine-tuning + strong interaction strategy; When D need ≤θ D And I need ≤θ I When , shallow fine-tuning + weak interaction strategy is adopted; Among them, the sliding window mechanism is used to dynamically update the strategy selection threshold θ D and θ I .

2. The optimization method of the adaptive multi-stage fine-tuning multi-modal large model according to claim 1 is characterized in that: Also includes: S6. Verify whether the performance of the multimodal basic large model meets the requirements. If so, complete fine-tuning. Otherwise, return to S2 to iteratively optimize the model using the next set of samples until the requirements are met. In step S3, the calculation method of the gradient scale score includes: Calculate the first-order gradient of each feature of the feature sequence relative to the loss function, and calculate the second-order moment estimate of the first-order gradient to calculate the gradient sensitivity index: Where, γ i represents the gradient sensitivity of the i-th feature of the feature sequence; w g 、w h 、w s is the weight coefficient for balancing the contributions of different items; σ(·) is the Sigmoid function; is the first-order gradient g i The exponential moving average gradient statistic of Deviation correction amount; H ii is the diagonal element of the Hessian matrix approximated by the finite difference method, which is used to evaluate the optimization difficulty of the feature; is the second-order moment estimate; Based on the update history of the feature, calculate the stability index of the feature: Where S i is the stability index of the i-th feature; U i represents the set of historical updates of the i-th feature; std(·) is the standard deviation function, and mean(·) is the mean value function; Calculate the consistency metric of the gradient direction: Where D i is the gradient direction consistency measure of the i-th feature; is the average gradient vector of the i-th feature at time step t; K is the size of the historical time window, indicating the number of historical gradients considered; cos(·) is the cosine similarity function, which is used to measure the similarity of the directions of two gradient vectors; is the average gradient vector of the i-th feature at time step tk; Based on the gradient sensitivity, the stability index and the gradient direction consistency index, the gradient scale score of the feature is calculated by weighting: Where, is the gradient scale score of the i-th feature; s is the stability weight; d is the direction consistency weight.

3. The optimization method of the adaptive multi-stage fine-tuning multi-modal large model according to claim 1 is characterized in that: In step S4, the calculation method of the synergy gain score includes: Evaluate any two modal features F i and F j Basic performance P on specific task evaluation indicators i and P j : P i =φ(F i ) P j =φ(F j ) Where φ(·) is the performance evaluation function; Construct a multi-level collaborative feature fusion network F fused : Where, and are the first and second layers of the network respectively; σ(·) is the Sigmoid function; [;] represents feature concatenation; W1~W3 and b1~b3 are learnable parameters; Evaluate the fusion performance P fused , P fused =φ(F fused ); Based on the basic performance and the fusion performance, the preliminary synergy gain is calculated: G abs =P fused -max(P i ,P j ) Where G abs is the absolute synergy gain, which indicates the absolute improvement of the fusion feature performance relative to the best performance of a single modality; G rel is the relative synergy gain, which indicates the relative improvement rate of the fusion feature performance relative to the best performance of a single modality; max(·) indicates the maximum value operation; Introduce the timing window T' and calculate the stability of the gain S gain : Where, represents the relative synergy gain value at time step t, t∈[1,T']; std(·) is the standard deviation function, and mean(·) is the mean value function; Calculate the synergy gain score Gain score : Gain score =αG rel +(1-α)S gain Where α is a balance parameter, which is dynamically adjusted according to the task characteristics.

4. The optimization method of the adaptive multi-stage fine-tuning multi-modal large model according to claim 1 is characterized in that: In step S3, the method for calculating the task relevance score includes: A task encoder with a bidirectional LSTM structure is used to convert the task description text encoding into a task vector representation; Construct a multi-level attention network and adopt a sliding window mechanism to encode the local context of the feature sequence; The multi-head attention mechanism calculates the semantic match between the local context features of each position and the task. The outputs of multiple attention heads are then concatenated and linearly transformed to obtain an attention score. The attention score is then adjusted using a position-weighted mechanism to obtain a score sequence that reflects the relevance of the task. The calculation method of the feature information score includes: Divide the feature sequence into several overlapping subsequences; On a local scale, the normalized conditional entropy of the subsequence is calculated; on a global scale, the average mutual information between subsequences is calculated; According to the normalized conditional entropy and the average mutual information, and by introducing a feature diversity measurement function, a feature information quantity score is calculated through weighted calculation.

5. The optimization method of the adaptive multi-stage fine-tuning multi-modal large model according to claim 1 is characterized in that: In step S4, the method for calculating the modality alignment score includes: For any two modal features F i and F j Perform normalization preprocessing to obtain preprocessing results and Calculate the semantic similarity matrix S between the two modalities ij : Where d is the dimension of the feature space; the superscript T represents transposition; Calculate the local alignment A local : Where N is the length of the feature sequence; [k * ,:] is the kth similarity matrix * All elements in the row; max(·) represents the maximum value operation, which is used to find the kth * The similarity between a feature and the best matching feature in the other modality; Calculate the global alignment A global : Where, mean(·) is the averaging function; ||·||2 is the L2 norm, which is used to measure the Euclidean distance between the average representations of two modal features; Calculate the modal alignment score Align score : Align score =λ1A local +λ2exp(-A global ) Where λ1 and λ2 are equilibrium coefficients; exp(·) is an exponential function with the natural constant e as the base; The calculation method of the information complementarity score includes: Calculate the mode-specific information entropy: H(F i )=-∑p(f i )logp(f i ) H(F j )=-∑p(f j )logp(f j ) Where, H(F i ) is the information entropy of mode i; H(F j ) is the information entropy of mode j; p(·) is the probability distribution function; f i is the eigenvalue in mode i; f j is the eigenvalue in mode j; Calculate conditional entropy: H(F i |F j )=-∑∑p(f i ,f j )logp(f i |f j ) H(F j |F i )=-∑∑p(f i ,f j )logp(f j |f i ) Where, H(F i |F j ) is the conditional entropy of mode i when mode j is known; p(·,·) is the joint probability distribution; p(·|·) is the conditional probability distribution; Calculate mutual information: MI(F i ,F j )=H(F i )-H(F i |F j ) Calculate the information complementarity score Comp score :

6. The optimization method of the adaptive multi-stage fine-tuning multi-modal large model according to claim 1, characterized in that: The calculation formulas for the unimodal score and the cross-modal interaction score are as follows: S single =α1S task +α2S feature +α3S gradient S cross =β1Align score +β2Comp score +β3Gain score Where S cross Scoring cross-modal interactions; S task 、S feature and S gradient They are task relevance score, feature information score and gradient scale score respectively; weight coefficients α1~α3 and β1~β3 are determined by grid search on the validation set; Align score 、Comp score and Gain score They are modal alignment score, information complementarity score and synergy gain score respectively.

7. The optimization method of the adaptive multi-stage fine-tuning multi-modal large model according to claim 6 is characterized in that: In the process of adaptive optimization of the multimodal basic model, the gradient ascent method is used to optimize the weight parameters, and the expression formula is: Where, is the gradient of the objective function J with respect to the weight parameter w; Represents partial derivatives; splicing vector Q score =[S single ;S Cross ]; R(w) is the regularization term, used to prevent overfitting; γ is the regularization coefficient, used to control the regularization strength; For the tth * The weight parameter at the iteration; η is the learning rate, which is used to control the step size of each parameter update; In step S6, the performance of the multimodal basic large model is determined to be qualified when any of the following conditions is met: Condition 1: Condition 2: Where, and are tth * +1 and t * The score value of the iteration; ∈1 is the score improvement threshold; ∈2 is the gradient norm threshold; ‖·‖2 is the L2 norm; The following feedback regulation mechanisms are also employed: Where, Δ perf is the performance change; and are tth * +1 and t * Model performance indicators for iterations; and are tth * +1 and t * The learning rate used in the iteration; ξ is the learning rate adjustment coefficient; sign(·) is the sign function.

8. The optimization method of the adaptive multi-stage fine-tuning multi-modal large model according to claim 1 is characterized in that: The multimodal dataset is represented as: D={(X1,y 1 ),…,(X i ,y i ),…,(X n ,y n )} Where, X i represents the i-th sample, m represents the number of supported modes; Y={y 1 ,y 2 ,…,y n } is a label set, where y∈R C , R is a set of real numbers, C is the number of label categories, and n is the total number of samples; In step S1, the different modal data are also preprocessed, specifically including: Text modality processing: The SentencePiece tokenizer is combined with the BPE algorithm to convert the input text into a standardized text sequence with a vocabulary size of 32KB. Special tags are configured, including the CLS tag for sequence representation, the SEP tag for separating different text segments, the MASK tag for masked language modeling, and the PAD tag for sequence alignment. A hybrid strategy of sinusoidal positional encoding and learnable positional encoding is adopted. Sliding window segmentation is performed on very long text sequences, and short sequences are padded with PAD tags. Image modality processing: Adopting an adaptive strategy to select resolution, dynamically adjusting it based on the original resolution of the input image and task requirements, using a bicubic interpolation algorithm during the adjustment process; performing multi-level image enhancement processing; and performing pixel-level normalization on the image. Video modality processing: The input video is sequentially sampled, spatially processed, and then subjected to motion feature extraction, thereby converting the input video into a standardized spatiotemporal feature representation. Audio modal processing: Basic preprocessing of raw audio, including unifying the audio sampling rate, volume normalization of the audio signal, removal of silent segments with energy below a preset threshold, segmentation of long audio into specific duration units, and retaining a set percentage of overlap between adjacent segments; converting the audio time domain signal into a spectrogram through short-time Fourier transform, mapping the linear frequency to the Mel scale through a Mel filter bank, logarithmically compressing the spectrogram, and enhancing audio features and data augmentation.

9. Adaptive multi-level fine-tuning multi-modal large model optimization system, characterized by: The method for optimizing a large adaptive multi-stage fine-tuning multi-modal model according to any one of claims 1 to 8 is applied; the system comprising: A data acquisition module is used to construct a multimodal dataset, which includes multiple groups of samples, each group of samples contains different types of modal data; The encoding module is used to encode the modal data in a single set of samples based on a pre-trained multimodal basic model to generate standardized feature representations; A single-modality multi-granularity adaptive fine-tuning module is used to calculate the task relevance score, feature information score, and gradient scale score of each modal data feature to generate a single-modality score. The task relevance score is used to measure the strength of the semantic association between the feature and the specific task goal, the feature information score is used to measure the information richness of the feature, and the gradient scale score is used to measure the importance of the feature. A cross-modal interaction modeling module is used to calculate the modal alignment score, information complementarity score, and synergy gain score between different modalities to generate a cross-modal interaction score. The modal alignment score is used to measure the degree of matching between different modal features in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between different modalities, and the synergy gain score is used to measure the synergistic effect generated by the fusion of different modal features. The fine-tuning strategy implementation module is used to construct the depth adjustment requirement index and the interaction intensity requirement index based on the single-modal comprehensive score and the cross-modal interaction score. It dynamically selects the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction according to the index values ​​to adaptively optimize the multimodal basic large model. The verification feedback module is used to verify whether the performance of the multimodal basic large model meets the requirements. If so, fine-tuning is completed; otherwise, the module returns to the encoding module to iteratively optimize the model using the next set of samples until the requirements are met.

Images