Optimization method and system for adaptive multi-stage fine-tuning multi-modal large model

Abstract

The invention relates to the field of multi-modal large models, and discloses an optimization method and system for a self-adaptive multi-stage fine-tuning multi-modal large model. The method comprises the following steps: constructing a multi-modal data set; coding modal data in a single group of samples based on a pre-trained multi-modal basic large model to generate feature representation; calculating a task correlation score, a feature information amount score and a gradient scale score of each modal data feature to generate a single-modal score; calculating a modal alignment degree score, an information complementation degree score and a collaborative gain degree score among different modals to generate a cross-modal interaction score; constructing a depth adjustment demand index and an interaction strength demand index according to the two scores, selecting a combination strategy of depth fine adjustment / shallow fine adjustment and strong interaction / weak interaction according to index values, and performing adaptive optimization on the multi-modal basic large model; and verifying whether the model performance meets the requirements or not so as to iteratively optimize to meet the requirements. According to the method, computing resources are saved, and the representation and generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to an optimization method and system for an adaptive multi-level fine-tuning multimodal large model. Background Art

[0002] With the rapid development of the Transformer architecture, it is now possible to convert inputs of different modalities (including text, images, voice, video and other different modal data) into semantic representations through their own dedicated encoders, and then perform cross-modal semantic alignment in a unified Transformer multi-layer structure model, thereby achieving true multimodal fusion. This technological breakthrough provides support for building a powerful multimodal large model. By integrating information from different data sources (such as text, images, voice, and video), the multimodal large model achieves a comprehensive perception and understanding of the real world, provides more comprehensive and in-depth insights, and lays the foundation for building a more intelligent and natural human-computer interaction system, which is crucial for various practical application scenarios. The multimodal large model can fully explore the intrinsic connections between different modal data, realize cross-modal information fusion and reasoning, and help improve the decision-making ability and generalization performance of artificial intelligence systems. In addition, the application of multimodal large models in vertical fields has also brought significant value. For example, in the medical field, multimodal big models can comprehensively analyze multi-source data such as patients' images, medical records, genes, etc., to assist doctors in making more comprehensive and accurate diagnoses; in the field of autonomous driving, multimodal big models can achieve real-time perception and prediction of the vehicle's surrounding environment, improving driving safety and decision-making intelligence.

[0003] However, there are also several challenges in fine-tuning large multimodal models to adapt to specific tasks. On the one hand, large multimodal models usually contain a large number of parameters, which require high computing resources and training time. Traditional full-model fine-tuning methods are inefficient and difficult to meet real-time and cost constraints. On the other hand, the data distribution and characteristics of different tasks vary greatly. The heterogeneity of multimodal data requires the model to maintain a high degree of flexibility and adaptability when processing different types of data. It is difficult to take into account the needs of all tasks using a unified fine-tuning strategy.

[0004] The existing fine-tuning technology for large multimodal models has the following shortcomings: First, most fine-tuning methods make adjustments in the parameter space of the entire model, while ignoring the differences between different task levels and modalities. This one-size-fits-all strategy is not only computationally inefficient, but also difficult to fully explore the characteristics of task data and the correlation between modalities. Secondly, existing methods usually rely on manually designed hyperparameters and fine-tuning strategies, and lack the ability to adapt to task characteristics, which leads to a lot of manual debugging and trial and error in practical applications, affecting the practicality and generalization performance of the model. Furthermore, most fine-tuning algorithms focus on the optimization of a single modality, without considering the interaction and fusion between different modalities, which limits the representation ability of large multimodal models and cannot give full play to their advantages in cross-modal understanding and generation. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention provides an optimization method and system for adaptive multi-level fine-tuning of a multi-modal large model. The present invention designs a multi-granularity fine-tuning strategy, which can adaptively select the optimal fine-tuning method and range according to the task level and modal characteristics to avoid waste of computing resources; in addition, the interaction between different modalities is modeled during the fine-tuning process to achieve effective fusion of cross-modal information and enhance the representation and generalization capabilities of the model.

[0006] To achieve the above object, the present invention provides the following technical solutions: The present invention discloses an optimization method for an adaptive multi-stage fine-tuning multi-modal large model, including S1-S6.

[0007] S1. Construct a multimodal dataset, which includes multiple groups of samples, each group of samples contains different types of modal data.

[0008] S2. Encode the modal data in a single group of samples based on the pre-trained multimodal basic large model to generate standardized feature representations.

[0009] S3. Calculate the task relevance score, feature information score and gradient scale score of each modal data feature to generate a unimodal score; among them, the task relevance score is used to measure the semantic association strength between the sample feature and the task goal, the feature information score is used to measure the information richness of the sample feature; the gradient scale score is used to measure the importance of the sample feature.

[0010] S4. Calculate the modal alignment score, information complementarity score and synergy gain score between different modalities to generate a cross-modal interaction score; among them, the modal alignment score is used to measure the matching degree of sample features of different modalities in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between sample features of different modalities, and the synergy gain score is used to measure the synergy effect produced by the fusion of sample features of different modalities.

[0011] S5. Based on the single-modal comprehensive score and the cross-modal interaction score, we construct the depth adjustment demand index and the interaction intensity demand index. According to the index values, we dynamically select the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction to adaptively optimize the multimodal basic large model.

[0012] S6. Verify whether the performance of the multimodal basic large model meets the requirements. If so, complete fine-tuning. Otherwise, return to S2 to iteratively optimize the model using the next set of samples until the requirements are met.

[0013] The present invention discloses an optimization system for an adaptive multi-level fine-tuning multimodal large model, which applies the optimization method for the adaptive multi-level fine-tuning multimodal large model as described above; the system comprises: a data acquisition module, an encoding module, a single-modal multi-granularity adaptive fine-tuning module, a cross-modal interaction modeling module, a fine-tuning strategy implementation module and a verification feedback module.

[0014] The data acquisition module is used to construct a multimodal data set, which includes multiple groups of samples, each group of samples contains different types of modal data; The encoding module is used to encode the modal data in a single group of samples based on the pre-trained multimodal basic large model to generate standardized feature representations; The single-modal multi-granularity adaptive fine-tuning module is used to calculate the task relevance score, feature information score and gradient scale score of each modal data feature to generate a single-modal score; among them, the task relevance score is used to measure the semantic association strength between the feature and the specific task goal, the feature information score is used to measure the information richness of the feature; the gradient scale score is used to measure the importance of the feature; The cross-modal interaction modeling module is used to calculate the modal alignment score, information complementarity score and synergy gain score between different modalities to generate a cross-modal interaction score; among them, the modal alignment score is used to measure the matching degree of different modal features in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between different modalities, and the synergy gain score is used to measure the synergy effect generated after the fusion of different modal features; The fine-tuning strategy implementation module is used to construct the depth adjustment demand index and the interaction intensity demand index based on the single-modal comprehensive score and the cross-modal interaction score, and dynamically select the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction according to the index value, so as to adaptively optimize the multi-modal basic large model; The verification feedback module is used to verify whether the performance of the multimodal basic large model meets the requirements. If so, fine-tuning is completed; otherwise, it returns to the encoding module to iteratively optimize the model using the next set of samples until the requirements are met.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention innovatively proposes a dual-path parallel evaluation system. Through the two technical routes of single-modality multi-granularity adaptive fine-tuning and cross-modal interactive modeling, it breaks through the limitations of traditional methods that only focus on single-modality optimization and realizes the refined adjustment of model parameters. In the single-modality path, fine-grained feature evaluation is achieved through the analysis of three dimensions: task characteristics, feature quality, and semantic structure, avoiding the "one-size-fits-all" global fine-tuning in the prior art and significantly improving computational efficiency. In the cross-modal path, deep interactive modeling between modalities is achieved through modal alignment evaluation, information supplementation, and synergistic gain analysis, solving the problem of insufficient modal fusion in the prior art.

[0016] In addition, based on the score fusion results, the present invention can adaptively select the optimal combination strategy of deep-shallow fine-tuning and strong-weak interaction, avoiding the limitations of traditional methods that rely on manual parameter adjustment and improving the practicality and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a flowchart of an optimization method for an adaptive multi-stage fine-tuning multi-modal large model in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1 The present invention provides an optimization method for an adaptive multi-level fine-tuning multi-modal large model, including S1~S6.

[0020] S1. Construct a multimodal dataset, which includes multiple groups of samples, each group of samples contains different types of modal data.

[0021] S2. Encode the modal data in a single group of samples based on the pre-trained multimodal basic large model to generate standardized feature representations.

[0022] S3. Calculate the task relevance score, feature information score and gradient scale score of each modal data feature to generate a unimodal score; among them, the task relevance score is used to measure the semantic association strength between the sample feature and the task goal, the feature information score is used to measure the information richness of the sample feature; the gradient scale score is used to measure the importance of the sample feature.

[0023] S4. Calculate the modal alignment score, information complementarity score and synergy gain score between different modalities to generate a cross-modal interaction score; among them, the modal alignment score is used to measure the matching degree of sample features of different modalities in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between sample features of different modalities, and the synergy gain score is used to measure the synergy effect produced by the fusion of sample features of different modalities.

[0024] S5. Based on the single-modal comprehensive score and the cross-modal interaction score, we construct the depth adjustment demand index and the interaction intensity demand index. According to the index values, we dynamically select the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction to adaptively optimize the multimodal basic large model.

[0025] S6. Verify whether the performance of the multimodal basic large model meets the requirements. If so, complete fine-tuning. Otherwise, return to S2 to iteratively optimize the model using the next set of samples until the requirements are met.

[0026] The present embodiment also provides an optimization system for an adaptive multi-level fine-tuning multimodal large model, which applies the optimization method for an adaptive multi-level fine-tuning multimodal large model as described above; the system includes: a data acquisition module, an encoding module, a single-modal multi-granularity adaptive fine-tuning module, a cross-modal interaction modeling module, a fine-tuning strategy implementation module and a verification feedback module, which correspond to the above 6 steps respectively, and each module can also be constructed of multiple sub-modules to achieve multiple functions.

[0027] In view of the many challenges that exist in the current multimodal large model in downstream task adaptation, such as the difficulty in quantifying the importance of different modal parameters, the underutilization of inter-modal interaction information, and the easy waste of computing resources, this paper proposes an adaptive fine-tuning framework based on multi-granularity evaluation. This framework achieves the optimal balance between model performance and computing resources by designing an adaptive evaluation mechanism and a fine-tuning strategy selection algorithm.

[0028] The core innovations of this invention are mainly reflected in two aspects: first, a multi-granular parameter importance evaluation method is proposed, which can accurately quantify the importance of parameters at different levels from three dimensions: task relevance, feature information, and gradient sensitivity; second, a novel cross-modal interaction modeling mechanism is designed, which achieves efficient fusion of modal information by evaluating the alignment, complementarity, and synergy gain between modalities. These innovations enable this invention to significantly reduce computing resource consumption while ensuring model performance.

[0029] The optimization scheme of the adaptive multi-level fine-tuning multi-modal large model provided by the present invention can automatically select the optimal fine-tuning strategy according to the characteristics of different modal data and task requirements, and realize efficient optimization of model performance. The overall architecture of this embodiment and the specific implementation of each core module are described in detail below.

[0030] 1. System input and preprocessing At the entry point of the entire system, the input data from different modalities needs to be processed and normalized first. The multimodal data set received by the system is formally represented as:

[0031] Here, m represents the number of modes supported by the system. Indicates samples, is the corresponding label set, where (C is the number of label categories), R is the set of real numbers, n This unified data representation lays the foundation for subsequent multimodal processing.

[0032] 1.1 Text mode processing As one of the most basic modes in a multimodal system, the quality of text data preprocessing directly affects the model's understanding ability. This paper adopts an advanced word segmentation strategy to convert the input text into a standardized sequence representation: ;

[0033] In the formula, Represents text modality features; Represents a tokenizer; Represents input text, is the secondary table size; is the length of the text sequence, where The default setting is 768, which is consistent with the mainstream pre-training model. The word segmentation process here uses the BPE (Byte-Pair Encoding) algorithm implemented by SentencePiece, which can effectively balance the size of the vocabulary and the granularity of word segmentation, and is particularly suitable for processing multilingual texts. Specifically, the configuration and processing flow of the word segmenter are as follows: ① Vocabulary setting: The vocabulary size is set to 32K. This size can well balance the vocabulary coverage and model complexity.

[0034] ② Contains special tags: [CLS] for sequence representation, [SEP] for separating different text segments, [MASK] for masked language modeling, and [PAD] for sequence alignment.

[0035] ③Sequence processing: The maximum sequence length L can be configured to 512 / 1024 / 2048, which can be selected according to specific task requirements. The sliding window strategy is used to split the super-long sequence to ensure that important information is not lost; the short sequence is padded with [PAD] markers to ensure the consistency of batch processing.

[0036] ④ Position coding strategy: Combining sinusoidal position coding and learnable position coding, sinusoidal coding provides prior position information: , the learnable encoding can be adapted to specific task requirements through training.

[0037] After these processing steps, the text data is converted into Tensor of .

[0038] 1.2 Image modality processing As one of the most information-rich modalities in a multimodal system, the quality of image preprocessing directly affects the model's ability to understand visual information. In order to ensure the effective extraction and representation of image features, this paper uses image preprocessing, and the image data is mapped into a standardized tensor representation after processing: ; In the formula, is the image modality feature; For standardized processing, Standardize the image; is the resolution normalization process. H and W are the height and width of the image respectively.

[0039] This process consists of three core steps, each of which addresses specific technical challenges. The following details the implementation of each step: Resolution normalization (Resize module): The base resolution is 224×224, which is suitable for conventional visual tasks; the HD resolution is 384×384, which is used for scenes that require more detailed information; the ultra-HD resolution is 512×512, which is suitable for fine recognition tasks. The resolution selection adopts an adaptive strategy and is dynamically adjusted according to the original resolution of the input image and the task requirements. The adjustment process uses a bicubic interpolation algorithm to ensure the continuity of geometric features while maintaining image quality.

[0040] ① Resolution standardization (Resize module): The base resolution is 224×224, which is suitable for conventional visual tasks; HD resolution: 384×384, which is used for scenes that require more detailed information; UHD resolution: 512×512, which is suitable for fine recognition tasks. The resolution selection adopts an adaptive strategy and is dynamically adjusted according to the original resolution of the input image and task requirements. The adjustment process uses a bicubic interpolation algorithm to ensure the continuity of geometric features while maintaining image quality.

[0041] ② Image enhancement processing (ImageAugment module): In order to improve the robustness and generalization ability of the model, a multi-level data enhancement strategy is designed: first, color enhancement is used, with a brightness adjustment range of ±40% to simulate different lighting conditions; the contrast adjustment range is ±40% to enhance feature contrast; the saturation adjustment range is ±40% to enrich color performance; the hue shift adjustment range is ±10% to provide color diversity. At the same time, geometric transformation is used, with a random horizontal flip probability of 0.5 to increase directional invariance; the original area is randomly cropped at a ratio of 0.8-1.0, while maintaining the aspect ratio to avoid image distortion.

[0042] ③ Normalization processing (Normalize module): In order to make the model training more stable, the image is normalized at the pixel level: normalize=Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]). These normalization parameters are obtained from the statistics of the ImageNet dataset and are universal. Subtracting the mean eliminates the overall brightness difference of the image, dividing by the standard deviation unifies the feature scale, and channel processing considers the distribution difference of the three RGB channels.

[0043] After these carefully designed preprocessing steps, the input image is converted into a standardized feature representation, providing high-quality visual features for subsequent multimodal fusion and task learning. This processing method not only ensures the integrity of the information, but also significantly improves the learning efficiency and generalization ability of the model.

[0044] 1.3 Video Mode Processing As a composite modality of temporal and spatial characteristics, video data processing needs to consider two key aspects: spatial feature extraction and temporal modeling. This paper designs an end-to-end video processing flow to convert the input video into a standardized spatiotemporal feature representation: ; In the formula, is the video modality feature; It is time series sampling processing; For spatial feature processing; T 1 ×3× H × W is the four-dimensional structure of the video feature tensor, T 1 Indicates the number of sampling time frames, 3 represents the RGB three-color channels, H and W Respectively represent the height and width of each frame image; For input video.

[0045] ① Temporal sampling (TemporalProcess module): In order to process video sequences of different lengths, an adaptive sampling strategy is designed. Sampling configuration includes: frame number setting, supporting 8 / 16 / 32 frame dynamic configuration; sampling mode setting, including uniform sampling (suitable for overall action understanding), dense sampling (focusing on action details), and random sampling (increasing the diversity of training samples); and temporal enhancement settings, including random segment sampling, speed disturbance, temporal masking, etc.

[0046] ② Spatial feature processing (SpatialProcess module): It inherits and expands the image enhancement processing function, and adds additional temporal consistency enhancement processing. Specifically, it means that multiple frames of the video stream are not allowed to be changed, so the temporal consistency is destroyed.

[0047] ③ Motion feature extraction: In order to better capture dynamic information, multiple motion feature extraction methods are integrated at the same time, including motion optical flow feature extraction and motion boundary detection.

[0048] 1.4 Audio Mode Processing As a time-frequency domain signal, audio data processing requires special attention to the extraction of spectral features and the preservation of timing characteristics. The present invention designs a complete audio processing flow: ; In the formula, is the audio modal feature; For audio feature extraction processing; As the basic pre-treatment; For input audio.

[0049] The audio processing flow mainly includes the following key steps: ① Basic preprocessing: First, standardize the original audio to lay a good foundation for subsequent feature extraction. This includes audio resampling, which uniformly converts the audio sampling rate to 16kHz (speech) or 44.1kHz (music) to ensure the consistency of feature extraction. Secondly, volume normalization is performed to normalize the audio signal to the range of [-1,1] to eliminate the impact of volume differences. Finally, silence detection and processing uses an energy-based detection algorithm to identify and remove silent segments below -60dB. At the same time, the audio is segmented. For longer audio, it is segmented in units of 10 seconds, and a 25% overlap is ensured between segments to avoid information loss.

[0050] ② Spectral feature extraction: After completing the basic preprocessing, the system will enter the feature extraction stage. Considering the needs of different application scenarios, the system supports a variety of spectral feature extraction methods. The most basic of these is the extraction of the Mel Spectrogram (Log-Mel Spectrogram). This process first converts the time domain signal to the frequency domain through short-time Fourier transform, and uses 2048-point FFT and 512-point frame shift to ensure the balance of time-frequency resolution. Then, the linear frequency axis is mapped to the Mel scale that is more in line with the human ear's perception characteristics through the Mel filter group. Depending on the application scenario, 80 (speech processing) or 128 (music processing) Mel filters can be selected. Finally, the spectrum graph is compressed through logarithmic transformation, which can better highlight the characteristics of the low-energy area, so that the final feature map is more in line with the auditory characteristics of the human ear.

[0051] ③ Audio feature enhancement and data augmentation: In order to further improve the robustness and generalization ability of the model, the present invention also designs a comprehensive audio enhancement strategy. These enhancement methods are carried out in both the frequency domain and the time domain. In the frequency domain, the system uses SpecAugment technology to enhance the anti-interference ability of the model by randomly blocking parts of the frequency and time dimensions. Specifically, the system will randomly block up to 30 Mel filter channels and 40 time frames, and can flexibly combine multiple masked areas. In the time domain, the system implements a series of signal-level enhancement methods, including adding background noise with different signal-to-noise ratios, simulating different degrees of room reverberation effects, adjusting pitch and playback speed within a specific range, etc. The combined use of these enhancement methods significantly improves the model's adaptability to various acoustic environments.

[0052] After these carefully designed processing steps, the original audio signal is finally converted into a standardized feature representation with the dimension of ,in F represents the frequency dimension (usually 80 or 128), T 2 Represents the time dimension (dynamically adjusted according to the length of the audio). This unified feature representation not only retains the key information of the audio signal, but also provides a standardized input format for subsequent modal fusion and model training. Through this complete processing flow, the system can effectively capture and utilize the time-frequency features in the audio and provide high-quality audio feature representation for multimodal learning tasks.

[0053] 2. Multimodal basic large model encoding After completing the standardized preprocessing of each modality data, the present invention uses a large-scale pre-trained multimodal basic model M (such as the Unified-IO-2 multimodal large model) as the core framework of feature encoding. This model uses a unified encoding-decoding architecture and can simultaneously process multiple modal inputs such as text, images, videos, and audio. Specifically, the preprocessed text sequence , Image features , video frame sequence And audio features are simultaneously fed into the model M. The model first preliminarily encodes the data of each modality through a specially designed modality-specific encoder, mapping the inputs of different modalities into a unified feature space. This unified feature representation lays the foundation for subsequent modal interaction and semantic understanding. After encoding by the model M, the system obtains a set of standardized feature representations, which will be fed into the subsequent basic semantic feature modeling and abstract semantic modeling modules for in-depth analysis.

[0054] After encoding the multimodal basic model M, the system obtains a preliminary feature representation. However, in order to better explore the deep semantic information within the modality and the correlation between modalities, the present invention designs two key optimization modules: a single-modal multi-granularity adaptive fine-tuning module and a cross-modal interaction modeling module. These two modules use different optimization strategies and evaluation mechanisms to improve the system's understanding ability from two dimensions: within the modality and between modalities. Among them, the single-modal multi-granularity adaptive fine-tuning module is mainly responsible for fine-grained optimization and adjustment of the features of each modality, while the cross-modal interaction modeling module is committed to enhancing the semantic association and information complementarity between different modalities. This dual optimization mechanism ensures that the system can fully and deeply understand the semantic connotation of multimodal data.

[0055] 3. Single-mode multi-granularity adaptive fine-tuning The single-modal multi-granularity adaptive fine-tuning module in the present invention is designed to optimize the feature representation after being encoded by the basic model M. For the input multimodal data, the basic model M first generates the corresponding feature matrix representation, which is recorded as , where the subscript m represents the modality type, which can be any of text, image, video or audio, and d represents the dimension of the feature space. In order to achieve accurate feature evaluation and optimization, the present invention designs three scoring submodules with different optimization objectives.

[0056] 3.1 Task Relevance Scoring Module The task relevance scoring module uses a deep semantic matching mechanism to evaluate the semantic association strength between features and specific task objectives. First, the input task description text is processed by the task encoder. The encoder uses a bidirectional LSTM structure and inputs the task description sequence ,in Indicates The hidden state update formula of the encoder is: ; ; in and Represent the hidden states of the forward and backward LSTMs, respectively. and Represent forward and backward LSTM processing respectively; Represents the word-unit input at the tth time step in the task description sequence.

[0057] By concatenating the bidirectional hidden states and performing a linear transformation, we get the vector representation of the task: ; in, Vectorized representation of tasks, capturing the semantic information of task description; is the learnable weight matrix, h is the LSTM hidden layer dimension, is the bias term, Represents the concatenated last hidden state of the forward LSTM and the first hidden state of the backward LSTM, which contains the complete semantic information of the sequence.

[0058] After obtaining the task vector representation, the present invention calculates the relevance between features and tasks by constructing a multi-level attention network. The network first performs local context encoding on the feature sequence, using a sliding window mechanism with a window size of k. , extract its local context features: ;

[0059] in, Represents a one-dimensional convolution operation, which is used to capture local feature dependencies; For the i local context features; is the i-th feature vector in the feature sequence, representing the original feature information processed by the model.

[0060] Then, the semantic matching degree between the features and the tasks is calculated through the multi-head attention mechanism: ; ; ; ; In the formula, Q , K , V They are query, key, and value respectively; , , are the transformation matrices of query, key, and value respectively, is the dimension of the attention head; For the j The output of an attention head; is the activation function; It is the matrix product of the query vector and the key vector, which is used to calculate the attention weight distribution.

[0061] The outputs of multiple attention heads are concatenated and linearly transformed to obtain the attention score : ; in is the number of attention heads. is the linear transformation matrix of the multi-head attention output, which is used to map the concatenated attention head output to the final representation space; is the corresponding bias vector.

[0062] Finally, the attention score is adjusted through the position weighting mechanism, and the adjusted attention score : ;

[0063] in is the position encoding function, which is used to introduce the influence of position information. In this way, a scoring sequence reflecting the relevance of the task is obtained : .

[0064] 3.2 Feature Information Scoring Module The feature information scoring module quantitatively evaluates the value of features from the perspective of information theory. This module first builds a probability model of feature distribution, and then measures the importance of features through statistics such as information entropy and mutual information.

[0065] The specific implementation process is as follows: In order to accurately estimate the probability distribution of features, the present invention adopts an adaptive kernel density estimation method. , and its probability density estimation formula is: ;

[0066] In the formula, is the number of observed samples, indicating the total number of feature vectors contained in the feature sequence; is the bandwidth parameter, which controls the smoothness of the kernel function and determines the smoothness of the probability density estimation; is the dimension of the feature space, indicating the dimension of each feature vector; is the feature vector of the jth sample, which is used to estimate the reference point of the probability density.

[0067] is a multivariate Gaussian kernel function: ;

[0068] H is the bandwidth matrix, which is determined adaptively by the Silverman criterion: ;

[0069] in, is the empirical covariance matrix of the features.

[0070] This adaptive bandwidth selection strategy can dynamically adjust the shape of the kernel function according to the local density characteristics of the data and improve the accuracy of probability estimation.

[0071] After obtaining the probability distribution estimate of the feature, the system calculates the conditional entropy of the feature. In order to reduce the computational complexity, the feature sequence is divided into several overlapping subsequences, each with a length of l and an overlapping length of l / 2. , and its conditional entropy calculation formula is: ; in express The context window of feature vectors. Conditional probability Estimation by Bayesian formula: ; In order to capture the interdependence between features, the system calculates the mutual information between subsequences. and , and the mutual information calculation formula is: ; Considering the computational efficiency, the mutual information estimation method based on k nearest neighbors is adopted: ; in, K 1 for kThe number of nearest neighbors in the nearest neighbor algorithm; is the digamma function, and Respectively and In their respective marginal spaces The number of neighbors, is the total number of samples.

[0072] In order to comprehensively consider the local and global importance of features, the system introduces a multi-scale feature importance scoring mechanism. At the local scale, the normalized conditional entropy of the feature is calculated: ; In the formula, Represents a characteristic subsequence The normalized conditional entropy of ; Represents a characteristic subsequence In the conditions The conditional entropy under reflects the characteristic subsequence local importance in a specific conditional context; Represents a characteristic subsequence In the conditions The conditional entropy under reflects the characteristic subsequence local importance in a particular context; Indicates that the conditional entropy in the brackets takes the maximum value; On a global scale, calculate the average mutual information between a feature and other features: ; In the formula, Represents a characteristic subsequence The weighted average mutual information with all other features; Represents a characteristic subsequence and characteristic subsequence The mutual information measure between is the decay weight based on sequence distance, expressed as: ; is the temperature parameter, which is used to control the rate of distance attenuation; It is an exponential function with the natural constant e as its base.

[0073] The final feature information score is obtained through weighted combination: ; In the formula, Represents a characteristic subsequence The comprehensive information score of is the learnable weight parameter, is the feature diversity metric function: ; In the formula, Represents a characteristic subsequence and characteristic subsequence The cosine similarity between them; this multi-scale feature information evaluation mechanism can fully capture the information value of the features and provide reliable importance guidance for subsequent feature optimization.

[0074] 3.3 Gradient scale scoring module The gradient scale scoring module evaluates the importance of features by analyzing the gradient behavior during the optimization process. This module not only considers the first-order gradient information, but also introduces the second-order curvature information to build a complete gradient tracking and analysis system. The specific implementation process is as follows: During model training, for each feature vector , the system first calculates its first-order gradient relative to the loss function L: ; In the formula, is the partial derivative; in order to reduce the noise impact caused by random gradient, the present invention designs a dynamic gradient accumulation mechanism. This mechanism maintains an exponential moving average (EMA) gradient statistic: ; in is the momentum coefficient, represents the current time step; and Respectively l Parameters in time step t and time step t The exponential moving average of the gradient is -1; For the i The actual gradient value of the parameter at the current time step. In order to eliminate the deviation in the early stage of training, the EMA is corrected for the deviation: ; In the formula, is the exponential moving average of the gradient after bias correction; is the momentum coefficient of t At the same time, the system also tracks the second-order moment estimate of the gradient: ; ; in is the decay rate of the second-order moment estimate; for t to the power of and Respectively i Parameters in time step t and t The exponential moving average of the squared gradient of -1; For the i parameters at the current time step t The actual gradient value of is the bias-corrected estimate of the second-order moment of the gradient. Based on these statistics, the system calculates the adaptive learning rate: ; in, For the Adaptive learning rate for parameters; is the basic learning rate, is a numerical stability constant.

[0075] In order to more accurately evaluate the optimization difficulty of the feature, the present invention uses the finite difference method to approximate the diagonal elements of the Hessian matrix: ; Where δ is a small perturbation. In order to reduce the computational overhead, the present invention adopts a random sampling strategy, and only calculates the second-order derivatives of some features in each small batch.

[0076] Based on the first-order gradient and second-order curvature information, the present invention designs an adaptive feature importance scoring function: ; In the formula, The first i The gradient sensitivity of each feature; , , Weight coefficients to balance the contributions of different items; is the Sigmoid function; The first-order gradient The exponential moving average gradient statistic of The deviation correction amount; It is the diagonal element of the Hessian matrix approximated by the finite difference method, which is used to evaluate the optimization difficulty of the feature; is the second-order moment estimate.

[0077] In order to capture the dynamic change characteristics of features during the optimization process, the present invention maintains a feature update history record: ; in represents the update amount of the feature, For the iThe update history set of features at time step t, and Represent the feature update amount at time step tk and time step t-1 respectively. Based on the update history, calculate the stability index of the feature: ; In the formula, For the i The stability index of each feature; Indicates i The set of historical updates of features; To find the standard deviation function, is the averaging function.

[0078] The final gradient scale score is obtained by combining multiple indicators: ; in is the stability weight, is the direction consistency weight, is a measure of the consistency of the gradient direction: ; In the formula, For the i Gradient direction consistency measure of features; For the i Features at time step t The average gradient vector of K is the size of the historical time window, indicating the number of historical gradients considered; It is the cosine similarity function, which is used to measure the similarity of the directions of two gradient vectors. The closer the index value is to 1, the more consistent the gradient directions are. For the i Features at time step t - k The average gradient vector of .

[0079] After the evaluation of the above three scoring modules, the comprehensive scoring vector of the feature is obtained: ; in is a learnable combined weight matrix, softmax is the activation function, , , They are feature stability scoring vector, feature information scoring vector and feature gradient scoring vector. Based on the comprehensive scoring, the present invention makes adaptive adjustments to the features: ; in represents element-wise multiplication, is the global adjustment step size, It is a layer normalization operation used to stabilize the scale of features. is the feature representation after adaptive adjustment; is the original feature representation;

[0080] In addition, the present invention also implements a feedback adjustment mechanism to dynamically adjust the weight of the scoring module by monitoring the performance indicators of downstream tasks: ; in is the regularization coefficient, which is used to prevent the weight from being overly concentrated in a certain scoring dimension. is the total loss function adjusted by feedback; It is the main loss function for downstream tasks; It is an L2 norm operation. Through this adaptive scoring and optimization mechanism, it is possible to accurately identify and enhance the most valuable feature representation for the task, significantly improving the performance of the model.

[0081] This multi-dimensional feature evaluation and optimization framework can not only accurately capture the importance of features, but also dynamically adjust the optimization strategy according to task requirements, providing a high-quality feature foundation for subsequent cross-modal interaction modeling.

[0082] 4. Cross-modal interaction modeling module The cross-modal interaction modeling module mainly includes three key evaluation dimensions: modal alignment, information complementarity and synergy gain, which are used to comprehensively evaluate and optimize the interaction quality between different modalities.

[0083] 4.1 Modal Alignment Scoring Module Modal alignment is used to measure the matching degree of different modal features in the semantic space. Given the feature representation of two modalities and , and its alignment is calculated as follows: (1) Feature normalization preprocessing to obtain preprocessing results and : ; .

[0084] (2) Calculate the semantic similarity matrix between modalities: ; in, for and The semantic similarity matrix between .

[0085] (3) Introducing local alignment metrics : ; In the formula, N is the characteristic sequence length; is the similarity matrix Row all elements; Represents the maximum value operation, which is used to find the The similarity of a feature with the best matching feature in the other modality.

[0086] (4) Global alignment calculate: ; In the formula, To find the average function; Used to measure the Euclidean distance between the average representations of two modal features.

[0087] (5) Final modal alignment score : ; In the formula, and is the balance coefficient; It is an exponential function with the natural constant e as its base.

[0088] 4.2 Information complementarity scoring module Information complementarity measures the amount of complementary information carried by different modalities, and is specifically implemented as follows: (1) Calculate the modality-specific information entropy: ; ; In the formula, For modal i Information entropy of For modal j Information entropy of is the probability distribution function; For modal i The eigenvalues ​​in ; For modal j The characteristic values ​​in .

[0089] It should be noted that the eigenvalues ​​in the mode ( and ) refers to the specific values ​​or states that may appear in a particular mode i or j.

[0090] The "modal characteristics" is a broader concept, referring to the characteristics that can describe or characterize the overall properties of the modality, such as information entropy. It is a modal feature.

[0091] (2) Calculate conditional entropy: ; ; In the formula, For known modes j In case of modal i The conditional entropy of is the joint probability distribution; is the conditional probability distribution.

[0092] (3) Calculation of mutual information : ; (4) Therefore, the information complementarity score For: .

[0093] 4.3 Synergy Gain Scoring Module The synergy gain is used to evaluate the synergy effect after the fusion of different modal features, and is quantified by measuring the performance gain after fusion. The specific implementation is as follows: (1) Basic performance evaluation: For any two modal features and , first evaluate its independent performance and : ; ; in It is a performance evaluation function, which can be an evaluation indicator of a specific task (such as classification accuracy, retrieval accuracy, etc.).

[0094] (2) Collaborative feature generation: building a multi-level collaborative feature fusion network : ; ; ; in represents feature concatenation, W 1 ~ W 3 and b 1 ~ b 3 are learnable parameters. and They are the first and second layer networks respectively; is the Sigmoid function.

[0095] (3) Fusion performance calculation: .

[0096] (4) Relative gain evaluation: Calculate the synergistic gain based on basic performance and fusion performance.

[0097] ; ; In the formula, is the absolute synergy gain, which indicates the absolute improvement of the fusion feature performance relative to the best performance of a single modality; is the relative synergy gain, which indicates the relative improvement rate of the fusion feature performance relative to the best performance of a single modality; Indicates the maximum value operation.

[0098] (5) Timing stability assessment: Introducing the timing window , calculate the stability of the gain: ; In the formula, Indicates that at time step t The relative synergy gain value, ; To find the standard deviation function, is the averaging function.

[0099] (6) Final synergistic gain: After comprehensively considering relative gain and stability, we get: ; in To balance the parameters, they are dynamically adjusted according to the task characteristics.

[0100] 5. Adaptive fine-tuning strategy based on dual scoring After completing the unimodal multi-granularity adaptive fine-tuning and cross-modal interaction modeling, the system obtains two key scoring results. In order to make full use of these scoring information to guide feature optimization, the present invention designs an adaptive fine-tuning strategy framework based on dual scoring. The framework selects the optimal fine-tuning strategy for different feature states by comprehensively considering the unimodal scoring and cross-modal interaction scoring.

[0101] In terms of unimodal scoring, its score Scored by Task Relevance , Feature Quality Score , and gradient scale scores composition: ; The weight coefficient Determined by grid search on the validation set, the weight coefficients are set to 0.4, 0.3, and 0.3, respectively; It can also be determined by grid search on the validation set.

[0102] In terms of cross-modal interaction scoring, the system uses modal alignment ( ), information complementarity ( ) and synergy gain ( ) These indicators measure the semantic consistency, information complementarity and synergy effect between different modal features. The comprehensive score of cross-modal interaction is calculated as follows: ; In the formula, and are the weight coefficients of the single modality score and modality alignment score in the deep adjustment demand index respectively; and are the weight coefficients of the information complementarity score and the synergy gain score in the interaction intensity demand index; The modality alignment component in scoring cross-modal interactions; to score the information complementarity component in cross-modal interactions; The synergy gain component in scoring cross-modal interactions.

[0103] Based on these two types of scores, the present invention constructs a single-modal deep adjustment demand index and multimodal interaction intensity requirement index : ; ; In the formula, and are the weight coefficients of the single modality score and modality alignment score in the deep adjustment demand index respectively; and They are the weight coefficients of the information complementarity score and the synergy gain score in the interaction intensity demand index respectively.

[0104] The above two indicators determine the depth and interaction strength of feature optimization respectively. The optimized threshold and The optimization threshold of the sample across modalities. At this time, for each sample, the optimization process of the present invention will adaptively select one of the following four strategies: (1) Deep fine-tuning + strong interaction strategy: and This strategy indicates that the quality of the unimodal features of the current sample needs to be significantly improved, and there is a strong complementary relationship between the cross-modal features.

[0105] (2) Deep fine-tuning + weak interaction strategy: and This strategy indicates that the single-modal features of the current sample need to be deeply optimized, but the feature information between the modalities has good alignment.

[0106] (3) Shallow fine-tuning + strong interaction strategy: and This strategy indicates that the unimodal feature quality of the current sample is high, but the cross-modal feature complementary information needs to be further strengthened.

[0107] (4) Shallow fine-tuning + weak interaction strategy: and This strategy is used when . This strategy indicates that the quality of the single-modal features of this sample is already good and the demand for interaction between modalities is low.

[0108] In order to ensure the stability and adaptability of strategy selection, the system uses a sliding window mechanism to dynamically update the strategy selection threshold. and Specifically, after each training cycle, the change in the feature optimization effect , the system will adjust the threshold accordingly. This adaptive mechanism ensures that the strategy selection can be continuously optimized as the training progresses, thereby achieving better feature optimization results.

[0109] 6. Optimization mechanism Weight optimization update: Use gradient ascent method to optimize weight parameters: ; In the formula, is the objective function J For the weight parameter w The gradient of Represents partial derivatives; concatenation vectors ; is a regularization term used to prevent overfitting; is the regularization coefficient, which is used to control the regularization strength; For the The weight parameter at the iteration; is the learning rate, which is used to control the step size of each parameter update.

[0110] Termination condition judgment: Design a dual termination criterion. When any of the following conditions is met, the performance of the multimodal basic large model is judged to be qualified: ; ; In the formula, and Respectively and The score value of the iteration; is the score improvement threshold. When the score difference between two consecutive iterations is less than the threshold, the model performance is considered to be stable. is the gradient norm threshold. When the L2 norm of the gradient is less than the threshold, the model is considered to have reached the local optimum. is the L2 norm, which is used to measure the size of the gradient vector.

[0111] The reason for designing the double termination criteria is that: ①, Condition 1 focuses on the convergence of performance indicators. If the performance improvement is already very small (Condition 1 is satisfied), even if the gradient is not small enough, further optimization may only bring about minor improvements. ②, Condition 2 focuses on the stability of the parameter space. If the gradient is already very small (Condition 2 is satisfied), even if the difference between the previous and subsequent scores is not small enough, but the model is already close to the local optimal point, further optimization may lead to overfitting.

[0112] Feedback Regulation Mechanism: Introducing Performance Feedback Loop ; ; In the formula, is the performance change; and Respectively and Model performance indicators for iterations; and Respectively and The learning rate used in this iteration; is the learning rate adjustment coefficient; is a symbolic function.

[0113] Through this multi-level scoring and optimization mechanism, the present invention can adaptively adjust the modal interaction strategy to maximize the synergistic gain effect while ensuring modal alignment. At the same time, the feedback adjustment mechanism can dynamically adjust the optimization strategy according to the actual performance to ensure the stability and convergence of the system. The whole process ensures high-quality semantic alignment, information complementarity and synergistic enhancement between different modal features. This multi-dimensional evaluation framework can not only comprehensively measure the quality of modal interaction, but also guide the system to perform targeted optimization and improve the effect of multimodal fusion.

[0114] Experimental / comparative verification: In order to comprehensively evaluate the performance of this method on different modal combinations and task types, this embodiment selects multiple public multimodal datasets for experiments. Among them, MSCOCO is an image-text multimodal dataset, containing more than 120,000 images and corresponding 5 description texts, which is widely used in image description and visual question-answering tasks. CMU-MOSEI is a large-scale text-speech-video trimodal sentiment analysis dataset, which contains 23,453 annotated video clips from online video websites, covering 250 different topics and 1,000 different speakers. HowTo100M is a video-text dataset containing more than 1.36 million instructional videos and their corresponding subtitle texts. MIMIC-CXR is a multimodal dataset in the medical field, containing more than 227,000 chest X-rays and their corresponding radiology reports. This embodiment is divided into training set, validation set and test set in a ratio of 8:1:1.

[0115] This embodiment selects the current mainstream multi-modal basic large model as the optimization object, including: 1. LLaVA-1.5: A visual-language model developed based on Vicuna-13B and CLIP ViT-L / 14, supporting image understanding and visual dialogue.

[0116] 2. BLIP-2: A visual-language model using the Q-Former architecture, based on FlanT5-XXL, capable of connecting visual encoders and large language models.

[0117] ImageBind: A multimodal model that supports six modalities (image, text, audio, depth, thermal, and IMU data), developed by the Meta AI research team.

[0118] 4. KOSMOS-2.5: A large-scale visual-language model developed by Microsoft that excels in processing interleaved image and text inputs.

[0119] The experimental environment is configured with 8×NVIDIA A800 (80GB) GPUs and implemented based on PyTorch 2.5.1. The batch size is 32, the initial value of the learning rate is set to 1e-5, and the cosine annealing strategy is adopted. All experiments are repeated 10 times and the average is taken to eliminate the influence of randomness. This embodiment compares the adaptive multi-level fine-tuning method of the present invention with the mainstream fine-tuning methods, including full parameter fine-tuning, LoRA (Low-Rank Adaptation), P-Tuning v2, Adapter-based fine-tuning, and QLoRA.

[0120] This example first compares the overall performance of the method of the present invention with the mainstream fine-tuning method on various data sets. All experiments were repeated 10 times, and the average and standard deviation were reported. The performance comparison was performed using the LLaVA-1.5 model on the MSCOCO data set, and the results are shown in Table 1.

[0121] Table 1: Results of fine-tuning the LLaVA-1.5 model on MSCOCO using different methods ; On the MSCOCO dataset, the accuracy of the proposed method is 1.47 percentage points higher than the best baseline method (full parameter fine-tuning), and the standard deviation is lower, indicating more stable performance. It is particularly noteworthy that the proposed method improves the generation quality indicators BLEU-4 and CIDEr by 0.92 and 1.75 points respectively, indicating that the model's expressive ability in visual-language generation tasks has been enhanced.

[0122] The ImageBind model is used to compare the performance on the CMU-MOSEI multimodal sentiment analysis dataset. The results are shown in Table 2.

[0123] Table 2: Results of fine-tuning the ImageBind model on CMU-MOSEI using different methods ; On the CMU-MOSEI dataset, this method performs well in the multimodal sentiment analysis task, with an F1 score improvement of 1.82 percentage points and a mean absolute error (MAE) reduction of 0.022 over the best baseline. This shows that this method can effectively integrate the sentiment information of text, speech, and video modalities, and improve the accuracy and stability of sentiment recognition.

[0124] The performance of the BLIP-2 model was compared on the MIMIC-CXR medical imaging dataset. The results are shown in Table 3.

[0125] Table 3: Results of fine-tuning the BLIP-2 model on MIMIC-CXR using different methods ; In the field of medical imaging, this method shows certain advantages, with the diagnostic accuracy increased by 1.71 percentage points and the AUC increased by 0.013. It is worth noting that in terms of standard deviation, this method is 26.4% lower than the average of other methods, indicating that it has higher reliability and stability in high-demand scenarios such as medicine.

[0126] The performance comparison is performed using the KOSMOS-2.5 model on the HowTo100M video-text dataset. The results are shown in Table 4.

[0127] Table 4: Results of fine-tuning the KOSMOS-2.5 model on HowTo100M using different methods ; In the video understanding task, this method improves the action recognition accuracy by 1.62 percentage points and the retrieval R@5 by 1.67 percentage points. This shows that this method has certain advantages in processing time series data and long sequence features.

[0128] Computational resource consumption comparison is performed on the LLaVA-1.5 (13B) model. The results are shown in Table 5. Table 5: Computational resource consumption comparison results on the LLaVA-1.5 (13B) model ; The proposed method not only has certain advantages in performance, but also performs well in resource efficiency. Compared with full parameter fine-tuning, the proposed method reduces the training time by 70.5% and the GPU memory consumption by 73.0%. Compared with other efficient parameter fine-tuning methods, the proposed method reduces the training time by 9.7% and the GPU memory consumption by 8.7% on average.

[0129] It is particularly noteworthy that the proportion of trainable parameters of this method is dynamically adjusted, ranging from 0.41% to 5.25%, which reflects the adaptability of the method - dynamically selecting the optimal parameter adjustment strategy according to different samples and task stages.

[0130] In order to verify the contribution of each component of the present invention, this embodiment conducts a detailed ablation experiment. Table 6 below shows the performance changes after removing different components on the LLaVA-1.5 model.

[0131] Table 6: Performance changes after removing different components on the LLaVA-1.5 model ; The ablation experiment results show that: 1. The cross-modal interaction score is one of the most critical components. After its removal, the accuracy dropped by 2.36 percentage points, indicating that accurately capturing the interaction between modalities is important for multimodal models.

[0132] 2. Although the performance degradation of the adaptive strategy selection mechanism is relatively small (1.65 percentage points), it has a greater impact on computational efficiency. After its removal, the training time increases by 12.53%.

[0133] 3. The gradient scale score and synergy gain score each contributed approximately 0.95 and 1.30 percentage points of performance improvement.

[0134] 4. The sliding window dynamic threshold mechanism improves the performance by 0.77 percentage points compared to the fixed threshold, while reducing the training time by 8.34%.

[0135] This example further analyzes the contributions of the three subcomponents of the cross-modal interaction score, and the results are shown in Table 7.

[0136] Table 7: Contributions of the three subcomponents of the cross-modal interaction score .

[0137] The results show that the importance of the three scoring mechanisms varies in different task types: the modal alignment score contributes more to the image-text description task; the information complementarity score is more critical in multimodal sentiment analysis; and the synergy gain score performs better in medical diagnosis and video understanding tasks that require subtle feature fusion.

[0138] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An optimization method for adaptive multi-level fine-tuning of a multi-modal large model, characterized in that: include: S1. Construct a multimodal dataset including multiple groups of samples, each group of samples contains different types of modal data; S2. Encode the modal data in a single group of samples based on a pre-trained multimodal basic large model to generate standardized feature representations; S3. Calculate the task relevance score, feature information score and gradient scale score of each modal data feature to generate a single modal score; the task relevance score is used to measure the semantic association strength between the sample feature and the task goal, the feature information score is used to measure the information richness of the sample feature, and the gradient scale score is used to measure the importance of the sample feature; S4. Calculate the modal alignment score, information complementarity score and synergy gain score between different modalities to generate a cross-modal interaction score; the modal alignment score is used to measure the matching degree of sample features of different modalities in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between sample features of different modalities, and the synergy gain score is used to measure the synergy effect generated by the fusion of sample features of different modalities; S5. Based on the single-modal comprehensive score and the cross-modal interaction score, construct the depth adjustment demand index and the interaction intensity demand index, dynamically select the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction according to the indicator values, and adaptively optimize the multimodal basic large model.

2. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 1 is characterized in that: Also includes: S6. Verify whether the performance of the multimodal basic large model meets the requirements. If yes, complete fine-tuning. Otherwise, return to S2 to iteratively optimize the model using the next set of samples until the requirements are met. In step S3, the calculation method of the gradient scale score includes: Calculate the first-order gradient of each feature of the feature sequence relative to the loss function, and calculate the second-order moment estimate of the first-order gradient to calculate the gradient sensitivity index: ； In the formula, The first i The gradient sensitivity of each feature; , , Weight coefficients to balance the contributions of different items; is the Sigmoid function; The first-order gradient The exponential moving average gradient statistic of The deviation correction amount; It is the diagonal element of the Hessian matrix approximated by the finite difference method, which is used to evaluate the optimization difficulty of the feature; is the second-order moment estimate; Based on the update history of the feature, calculate the stability index of the feature: ； In the formula, For the i The stability index of each feature; Indicates i The set of historical updates of features; To find the standard deviation function, To find the average function; Calculate the consistency metric of the gradient direction: ； In the formula, For the i Gradient direction consistency measure of features; For the i Features at time step t The average gradient vector of K is the size of the historical time window, indicating the number of historical gradients considered; is the cosine similarity function, which is used to measure the similarity between the directions of two gradient vectors; For the i Features at time step t - k The average gradient vector of Based on the gradient sensitivity, the stability index and the gradient direction consistency index, the gradient scale score of the feature is calculated by weighting: ； In the formula, For the i Gradient scale score of each feature; is the stability weight; is the direction consistency weight.

3. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 1 is characterized in that: In step S4, the calculation method of the synergy gain score includes: Evaluate any two modal features and Basic performance on task-specific evaluation metrics and : ； ； In the formula, is the performance evaluation function; Constructing a multi-level collaborative feature fusion network : ； ； ； In the formula, and They are the first and second layer networks respectively; is the Sigmoid function; Represents feature splicing; W 1~ W 3 and b 1~ b 3 is a learnable parameter; Evaluating Fusion Performance , ; Based on the basic performance and the fusion performance, the preliminary synergy gain is calculated: ； ； In the formula, is the absolute synergy gain, which indicates the absolute improvement of the fusion feature performance relative to the best performance of a single modality; is the relative synergy gain, which indicates the relative improvement rate of the fusion feature performance relative to the best performance of a single modality; Indicates the maximum value operation; Introducing Timing Windows , calculate the stability of the gain : ； In the formula, Indicates that at time step t The relative synergy gain value, ; To find the standard deviation function, To find the average function; Calculate the synergy gain score : ； In the formula, To balance the parameters, they are dynamically adjusted according to the task characteristics.

4. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 1 is characterized in that: In step S3, the calculation method of the task relevance score includes: A task encoder with a bidirectional LSTM structure is used to convert the task description text encoding into a task vector representation; Construct a multi-level attention network and use a sliding window mechanism to encode the local context of the feature sequence; The semantic matching degree between the local context features of each position and the task is calculated through a multi-head attention mechanism. Then, the outputs of multiple attention heads are concatenated and linearly transformed to obtain the attention score. The attention score is then adjusted through a position weighting mechanism to obtain a score sequence that reflects the relevance of the task. The calculation method of the feature information score includes: Divide the feature sequence into several overlapping subsequences; On a local scale, the normalized conditional entropy of the subsequence is calculated; on a global scale, the average mutual information between subsequences is calculated; According to the normalized conditional entropy and the average mutual information, a feature diversity metric function is introduced and a feature information quantity score is calculated through weighted calculation.

5. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 1 is characterized in that: In step S4, the method for calculating the modality alignment score includes: For any two modal features and Perform normalization preprocessing to obtain the preprocessing results and ; Calculate the semantic similarity matrix between two modalities : ； In the formula, d is the dimension of the feature space; superscript T represents transpose; Calculate local alignment : ； In the formula, N is the characteristic sequence length; is the similarity matrix Row all elements; Represents the maximum value operation, which is used to find the The similarity between a feature and the best matching feature in the other modality; Calculate global alignment : ； In the formula, To find the average function; is the L2 norm, which is used to measure the Euclidean distance between the average representations of two modal features; Calculating the modal alignment score : ； In the formula, and is the balance coefficient; is an exponential function with the natural constant e as base; The calculation method of the information complementarity score includes: Calculate the mode-specific information entropy: ； ； In the formula, For modal i Information entropy of For modal j Information entropy of is the probability distribution function; For modal i The eigenvalues ​​in ; For modal j The eigenvalues ​​in ; Calculate the conditional entropy: ； ； In the formula, For known modes j In case of modal i The conditional entropy of is the joint probability distribution; is the conditional probability distribution; Calculate mutual information: ； Calculating information complementarity scores : 。 6. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 1 is characterized in that: The calculation formulas for the unimodal score and the cross-modal interaction score are respectively as follows: ； ； In the formula, and They are unimodal scoring and cross-modal interaction scoring respectively; , and They are task relevance score, feature information score and gradient scale score; weight coefficient and All are determined by grid search on the validation set; , and They are modal alignment score, information complementarity score and synergy gain score respectively.

7. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 6 is characterized in that: In step S5, the depth adjustment demand index and interaction intensity requirement index The calculation formulas are as follows: ； ； In the formula, and are the weight coefficients of the single modality score and modality alignment score in the deep adjustment demand index respectively; and are the weight coefficients of the information complementarity score and the synergy gain score in the interaction intensity demand index; The modality alignment component in scoring cross-modal interactions; to score the information complementarity component in cross-modal interactions; The synergy gain component in scoring cross-modal interactions; Among them, by comparison , The preset strategy selection threshold , The size between them is selected, and one of the following four strategies is selected as the adaptive optimization direction: when and When using deep fine-tuning + strong interaction strategy; when and When , we adopt the deep fine-tuning + weak interaction strategy; when and When using the shallow fine-tuning + strong interaction strategy; when and When , shallow fine-tuning + weak interaction strategy is adopted; Among them, the sliding window mechanism is used to dynamically update the strategy selection threshold and .

8. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 7 is characterized in that: In the process of adaptive optimization of the multimodal basic large model, the gradient ascent method is used to optimize the weight parameters, and the expression formula is: ； In the formula, is the objective function J For the weight parameter w The gradient of Represents partial derivatives; concatenation vectors ; is a regularization term used to prevent overfitting; is the regularization coefficient, which is used to control the regularization strength; For the The weight parameter at the iteration; is the learning rate, which is used to control the step size of each parameter update; In step S6, when any of the following conditions is met, the performance of the multimodal basic large model is judged to be qualified: ； ； In the formula, and Respectively and The score value of the iteration; Improved thresholds for scoring; is the gradient norm threshold; is the L2 norm; The following feedback regulation mechanisms are also used: ； ； In the formula, is the performance change; and Respectively and Model performance indicators for iterations; and Respectively and The learning rate used in this iteration; is the learning rate adjustment coefficient; is a symbolic function.

9. The optimization method of the adaptive multi-level fine-tuning multi-modal large model according to claim 1 is characterized in that: The multimodal dataset is represented as: ； In the formula, Indicates i samples, , m Indicates the number of supported modes; is a set of labels, where , R is the set of real numbers, C is the number of label categories, n is the total number of samples; The types of modalities include text modality, image modality, video modality and audio modality. In step S1, different modality data are also preprocessed, specifically including: Text modality processing: The SentencePiece tokenizer is combined with the BPE algorithm to convert the input text into a standardized text sequence with a vocabulary size of 32K. Special tags are configured, among which the CLS tag is used for sequence representation, the SEP tag is used to separate different text segments, the MASK tag is used for masked language modeling, and the PAD tag is used for sequence alignment. A hybrid strategy of sinusoidal positional encoding and learnable positional encoding is adopted. Sliding window segmentation is performed on ultra-long text sequences, and short sequences are padded with PAD tags. Image modality processing: Adopt an adaptive strategy to select the resolution, dynamically adjust it according to the original resolution of the input image and the task requirements, and use the bicubic interpolation algorithm during the adjustment process; perform multi-level enhancement processing on the image; perform pixel-level standardization on the image; Video modality processing: The input video is sequentially sampled, spatial feature processed, and motion feature extracted, thereby converting the input video into a standardized spatiotemporal feature representation; Audio modal processing: basic preprocessing of raw audio, including unifying the audio sampling rate, normalizing the volume of audio signals, removing silent segments with energy below a preset threshold, segmenting long audio into specific duration units, and retaining a set percentage of overlapping areas between adjacent segments; converting the audio time domain signal into a spectrogram through short-time Fourier transform, mapping the linear frequency to the Mel scale through a Mel filter group, logarithmically compressing the spectrogram, and enhancing audio features and data augmentation.

10. Adaptive multi-level fine-tuning multi-modal large model optimization system, characterized by: The optimization method of the adaptive multi-stage fine-tuning multi-modal large model as claimed in any one of claims 1 to 9 is applied; the system comprises: A data acquisition module is used to construct a multimodal data set, which includes multiple groups of samples, each group of samples contains different types of modal data; An encoding module, which is used to encode the modal data in a single group of samples based on a pre-trained multimodal basic large model to generate a standardized feature representation; The unimodal multi-granularity adaptive fine-tuning module is used to calculate the task relevance score, feature information score and gradient scale score of each modal data feature to generate a unimodal score; the task relevance score is used to measure the semantic association strength between the feature and the specific task goal, the feature information score is used to measure the information richness of the feature; the gradient scale score is used to measure the importance of the feature; The cross-modal interaction modeling module is used to calculate the modal alignment score, information complementarity score and synergy gain score between different modalities to generate a cross-modal interaction score; the modal alignment score is used to measure the matching degree of different modal features in the semantic space, the information complementarity score is used to measure the amount of complementary information carried between different modalities, and the synergy gain score is used to measure the synergy effect generated after the fusion of different modal features; Fine-tuning strategy implementation module, which is used to construct deep adjustment demand indicators and interaction intensity demand indicators based on single-modal comprehensive scores and cross-modal interaction scores, dynamically select the combination strategy of deep fine-tuning / shallow fine-tuning and strong interaction / weak interaction according to the indicator values, and adaptively optimize the multi-modal basic large model; The verification feedback module is used to verify whether the performance of the multimodal basic large model meets the requirements. If yes, fine-tuning is completed; otherwise, the encoding module is returned to use the next set of samples to iteratively optimize the model until the requirements are met.

Images