Large-model multi-mode capability adaptive tuning system

Abstract

The invention relates to the technical field of large models, and particularly discloses a large model multi-modal capability adaptive tuning system, which comprises a multi-modal interaction interface module, a modal adapter group and a cross-modal dynamic coordinator which are connected in sequence, the system further comprises a modal capability state evaluator connected with the cross-modal dynamic coordinator and the modal adapter group, and a dynamic balance tuning engine connected with the cross-modal dynamic coordinator and the modal capability state evaluator. The modular design and the self-adaptive tuning mechanism can be compatible with various modal data such as texts, images and voices, large-scale customized development does not need to be carried out for specific scenes, multi-modal interaction requirements in different fields can be flexibly met, the application landing cost is remarkably reduced, and the method has a wide application range.

Description

Technical Field

[0001] This invention relates to the field of large model technology, specifically to an adaptive tuning system for the multimodal capabilities of large models. Background Technology

[0002] With the development of artificial intelligence technology, large-scale models are increasingly widely used in multimodal interaction scenarios, covering the collaborative processing of various data types such as text, images, voice, and video. They have become a core support for fields such as intelligent customer service, autonomous driving, and intelligent interactive terminals. The core requirement of multimodal large-scale models is to achieve effective fusion and adaptive optimization of data from different modalities to accurately respond to complex task requirements.

[0003] However, existing multimodal technologies still have many limitations: the features of different modal data are significantly heterogeneous, and traditional modality fusion methods often use fixed rules or simple weighting strategies, which are difficult to dynamically adapt to changes in the dependencies between modalities, resulting in poor fusion effects and failing to fully leverage the information value of each modality; during modality tuning, the parameter updates of each modality are often coupled with each other, lacking independent and targeted tuning mechanisms, which easily leads to overfitting of some modalities and underfitting of others, affecting the overall model performance; there is a lack of real-time monitoring and evaluation mechanisms for the learning state of each modality, making it impossible to accurately perceive the learning efficiency and stability of the modality, resulting in blind tuning directions and difficulty in achieving a balanced improvement in modality capabilities; cross-modal fusion weights are mostly statically set and cannot be dynamically adjusted according to the task context, resulting in insufficient adaptability; in the online service phase, there is a lack of effective user feedback closed-loop optimization mechanisms, making it difficult to continuously iterate and improve system performance based on feedback from actual usage scenarios.

[0004] These issues mean that the existing multimodal large models need to be improved in terms of interaction accuracy, adaptability, and user experience in complex scenarios, which limits their promotion and implementation in practical applications. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a large-scale multimodal capability adaptive tuning system, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a large-scale multimodal capability adaptive tuning system, the system comprising a multimodal interaction interface module, a modal adapter group, and a cross-modal dynamic coordinator connected in sequence, further comprising a modal capability state evaluator connected to the cross-modal dynamic coordinator and the modal adapter group respectively, and a dynamic balance tuning engine connected to both the cross-modal dynamic coordinator and the modal capability state evaluator; The multimodal interaction interface module is used to receive user query commands and, in response to the query commands, collect multimodal input data from at least two different types of data sources. The modal adapter group is used to perform feature extraction and representation alignment on the input data of each modality through independent and parameter-tunable modal adapters, and output the first modality. Modality-specific embedding vectors of each modality ; The cross-modal dynamic coordinator is used to receive modality-specific embedding vectors from all modalities. Based on the current task context, modal correlation weights representing intermodal dependencies are dynamically calculated to generate an adaptive fusion coefficient vector that guides multimodal information fusion. ; The modal capability state evaluator is used to monitor and analyze the performance index changes of each modality adapter in real time during the training process. Based on historical gradient information and the evolution law of validation loss, it generates modal state vectors that characterize the current learning efficiency and stability of each modality. ; The dynamic balancing optimization engine is used to integrate adaptive fusion coefficient vectors. With modal state vector Construct a comprehensive optimization objective function that combines task objectives with modal capability balancing objectives. By dynamically and differentially updating the parameters of each modal adapter through the backpropagation algorithm, a collaborative and adaptive improvement of multimodal representation capabilities is achieved.

[0007] Preferably, the modal adapters in the modal adapter group have independent structures and their parameter update processes are decoupled from each other. Each modal adapter adopts a low-rank adaptive structure or a prefix tuning structure. During the tuning process, only a small number of parameters introduced by the modal adapter itself are updated, while the backbone parameters of the underlying large language model remain frozen; wherein: When the modal adapter is a low-rank adaptive structure, low-rank matrix increments are injected into the weights of the fully connected layers of the underlying large language model. To update the parameters, the specific steps are as follows: S1.1 Initialize the low-rank matrix and The Xavier initialization method is used to ensure that the parameters follow a uniform distribution. for Low-rank matrix for Low-rank matrix For the weight dimension of the fully connected layer, It is a low-rank dimension and ; S1.2 Calculate the increment of the low-rank matrix ,pass and Obtained by matrix multiplication; S1.3 Inject incremental update weights, Original weights of the fully connected layer of the underlying large language model By superimposing the results, we obtain the updated weights. ; S1.4 Verify the update effect by calculating the feature extraction accuracy of the modal adapter. If the accuracy improvement is less than the preset threshold of [0.5%, 2%], then adjust... The value is taken and S1.1-S1.3 is repeated until the performance requirements are met; When the modal adapter is a prefix tuning structure, a trainable prefix vector sequence is added before the input sequence to the underlying large language model. To update the parameters, the specific steps are as follows: S1.5 Adaptive determination of prefix length Text modality Value range [10, 30], image and speech modalities The value range is [20, 50]. For the first Each prefix vector has the same dimension as the input dimension of the underlying large language model. S1.6 Initialize the prefix vector sequence Random normal initialization is used; S1.7 Concatenate the input sequence, Add to the original modal input sequence The front end forms a new input sequence. ; S1.8 iteratively updates the prefix parameters, calculates the output loss through forward propagation of the underlying large language model, and updates the parameters based on the loss value through backpropagation. The parameters are set, the core parameters of the underlying model are kept frozen, the number of iterations ranges from [50, 200], and the update is terminated if the loss value no longer decreases.

[0008] Preferably, the cross-modal dynamic coordinator dynamically calculates modal correlation weights and generates an adaptive fusion coefficient vector. The process specifically includes: Step 1, Calculate dimensional intermodal correlation matrix ,matrix The Middle Line number Column elements Indicates the first The modality pair of the first The dependency correlation of each modality is determined by the following steps: S2.1 Modality embedding vector preprocessing, for all Perform L2 normalization to obtain Eliminate scale differences; S2.2 Initialize the trainable associative projection matrix The He initialization method is used. for 3D matrix ( With claim 1 (The dimensions are consistent) S2.3 Calculate the original correlation degree, through Obtain the original correlation degree in scalar form; S2.4 Association normalization is performed using the Sigmoid activation function. (expression )Will Map to the interval (0,1); S2.5 calculation The calculation formula is: ; Step 2, based on the intermodal correlation matrix Generate an adaptive fusion coefficient vector through column-dimensional aggregation operations. , For the first The fusion weight coefficients for each modality are determined by the following steps: S2.6 Set temperature parameters Adjust the weighted intensity of the index; S2.7 Calculate the first We obtain the weighted sum of the columns. ; S2.8 Calculate the global normalization factor and obtain ; S2.9 generation The contribution of a mode in multimodal information fusion is represented by the ratio of the weighted sum to the global normalization factor.

[0009] Preferably, the fusion weight coefficients in the adaptive fusion coefficient vector By analyzing the intermodal correlation matrix The The weighted pooling calculation is obtained from the column, and the calculation formula is as follows: ; in, The natural exponential function serves as the intermodal dependency correlation degree. The index-weighted term; For the first The modality pair of the first The dependency correlation of each modality; the denominator is the global normalization factor, ensuring that all fusion weight coefficients satisfy... ,make It conforms to the characteristics of a probability distribution.

[0010] Preferably, the modal capability state estimator includes a gradient feature extraction unit, a loss trend analysis unit, and a state vector synthesis unit, which collaboratively generate modal state vectors. The specific process is as follows: The gradient feature extraction unit is used to calculate the gradient feature of each modal adapter in the most recent The gradient vector changes of trainable parameters during secondary parameter update iterations yield gradient feature values ​​that characterize modality learning activity. The specific steps are as follows: S3.1 Collects gradient vectors and records the most recent... In the nth iteration The gradient vector of the trainable parameters of each modal adapter ( ); S3.2 Calculate the gradient L2 norm to obtain the gradient norm sequence. ; S3.3 Calculate the exponential moving average to obtain the gradient eigenvalues. ; The loss trend analysis unit is used to calculate the task loss for each modality in the most recent time frame based on the validation set data. The evolutionary features within each validation window yield the loss feature values ​​characterizing the modality learning efficiency. The specific steps are as follows: S3.4 Collects loss values; each verification window contains... Each training batch collects the most recent data. Loss value of each verification window ; S3.5 calculates the rate of change of loss, and obtains ( ); S3.6 Calculate the mean rate of change to obtain the loss characteristic value. ; The state vector synthesis unit is used to synthesize gradient feature values. With loss eigenvalues Perform normalized splicing to obtain the first... Modal state vectors of each mode The specific steps are as follows: S3.7 uses the min-max normalization method to... Normalized to the [0,1] interval, we get , , These are the minimum and maximum values ​​of the gradient eigenvalues ​​for all modes, respectively. Similarly, S3.8 Normalization yields ; S3.9 splicing and The modal state vector is obtained. ,in , They are respectively , The eigenvalues ​​after min-max normalization.

[0011] Preferably, the gradient feature value For the most recent The exponential moving average of the gradient vector L2 norm in the next iteration, and the loss eigenvalues. For the most recent The mean of the rate of change of loss for each validation window, and the formulas for calculating them are as follows: ; ; in, For the first The gradient feature values ​​of each modality represent the modality learning activity; This is the exponential moving average coefficient, with a value range of [0.85, 0.95], used to balance the weights of historical gradients and the current gradient. The larger the gradient eigenvalue, the more stable the change in the gradient eigenvalue; For the first The gradient eigenvalues ​​of the next iteration; For the first During the nth iteration The gradient vector of the trainable parameters of each modal adapter; It is an L2 norm; For the first The loss feature value of each modality characterizes the modality learning efficiency; To verify the number of windows; For the first The first verification window Each modality corresponds to a task loss value; For the first The first verification window Each modality corresponds to a task loss value, with the numerator representing the loss decrease and the denominator used for normalization to obtain the loss change rate.

[0012] Preferably, the comprehensive optimization objective function constructed by the dynamic balancing tuning engine The weighted sum of the task's main loss and the modal capability balance loss is calculated using the following formula: ; in, To comprehensively optimize the objective function; The main loss function is oriented towards specific downstream tasks; To balance the hyperparameters, the values ​​are set to [0.1, 0.5]. This is the loss for modal capability balancing.

[0013] Preferably, the modal capability balance loss The deviation between the modal state vector and the ideal modal state target vector is calculated using the following formula: ; in, This is the loss for modal capability balancing; For the first Balanced penalty weights for each modality; For the first Modal state vectors of 1 modality; The target vector is the preset ideal modal state. For ideal gradient eigenvalues, These are the ideal loss feature values, all of which are constants set empirically. It is an L1 norm.

[0014] Preferably, the large model multimodal capability adaptive tuning system further includes an online feedback optimization module. This online feedback optimization module is connected to both the dynamic balancing tuning engine and the cross-modal dynamic coordinator, and is used to collect user satisfaction feedback on the system output and quantify it into an instant reward value. ( (For feedback time steps); the dynamic balancing optimization engine, during the online service phase, calculates based on the historical real-time reward value sequence. Adjusting the intermodal correlation projection matrix in the cross-modal dynamic coordinator The update step size, the specific steps are as follows: S4.1 Collect user satisfaction ratings. Obtain user ratings through the interactive interface (range [1,5], where 1 point indicates extreme dissatisfaction and 5 points indicates extreme satisfaction). S4.2 quantifies instant reward value, through Rating Mapping to the [0,1] interval, we get ; S4.3 calculates historical cumulative rewards, through get( (This is a discount factor, with a value range of [0.9, 0.99]). S4.4 Dynamically adjust and update the step size, calculate ( The update step size is based on the base value, with a range of [0.001, 0.01]. and Positive correlation; S4.5 fine-tunes the cross-modal coordinator parameters, using the policy gradient method to calculate the parameter gradients. ( (The parameters are trainable for the cross-modal dynamic coordinator and are updated using the gradient ascent algorithm); S9.6 verifies the fine-tuning effect. The average reward value is calculated after every 10 parameter fine-tuning iterations. If the improvement is lower than the preset threshold of [1%, 3%], the reward is reduced. Repeat step S4.5 until the average reward value reaches the preset target of [0.7, 0.9]. Update step size The positive correlation with historical cumulative reward values ​​allows the system to assign higher fusion weight coefficients to modal combinations that have previously received high rewards, and the policy gradient method is used to fine-tune all trainable parameters of the cross-modal dynamic coordinator with the goal of maximizing cumulative rewards.

[0015] Preferably, the multimodal input data includes any two or more of text, images, speech, video, and structured tabular data; the multimodal interaction interface module supports multiple data acquisition methods such as direct text input, speech recognition, image parsing, video frame extraction, and structured data parsing, and can adapt to the input interface protocols of different devices. The acquired multimodal input data is standardized and then transmitted to the modal adapter group. The specific standardization steps are as follows: S5.1 unifies the data format, converting text data into word embedding tensors, image data into pixel value tensors normalized to the [0,1] interval, speech data into Mel spectrum tensors, video data into frame sequence tensors, and structured table data into feature tensors; S5.2 Aligns data dimensions by adjusting the feature dimensions of all modal data using zero-padding or pruning. ; S5.3 processes abnormal data, detects and removes abnormal values ​​related to noisy pixels, silent segments, and invalid characters, and uses interpolation to fill in missing data; S5.4 classifies and transmits data, transmitting the standardized input data to the corresponding modal adapter in the modal adapter group according to the modal type, ensuring the accuracy and real-time performance of data transmission.

[0016] This invention provides an adaptive tuning system for the multimodal capabilities of large models, which has the following advantages: 1. Solving the problem of heterogeneous modal fusion and improving fusion adaptability: This invention uses a modal adapter group to extract and align the features of each modality data independently, effectively eliminating the feature heterogeneity of different modalities; combined with a cross-modal dynamic coordinator to dynamically calculate modal correlation weights and generate an adaptive fusion coefficient vector, the fusion weights can be flexibly adjusted according to the task context, significantly improving the accuracy and adaptability of multimodal information fusion.

[0017] 2. Achieve independent modal tuning and ensure targeted tuning: Each modal adapter adopts an independent structural design, and the parameter update process is decoupled from each other. It can be targeted to the characteristics of different modalities, avoiding interference between modal tuning. At the same time, only a small number of parameters of the adapter itself are updated, without changing the core parameters of the underlying large language model. It balances the tuning effect and efficiency, and effectively solves the problem of overfitting or underfitting of some modalities.

[0018] 3. Real-time monitoring of modal status and optimization of tuning direction: The modal capability status evaluator can capture the learning efficiency and stability of each modality in real time, generate vectors representing the modal status, provide accurate decision-making basis for the dynamic balance tuning engine, and transform the tuning process from blind iteration to targeted optimization, ensuring balanced improvement of each modality's capabilities and avoiding waste of tuning resources.

[0019] 4. Balancing multiple objectives to improve overall system performance: The dynamic balancing optimization engine constructs a comprehensive optimization objective function that combines task objectives with modal capability balancing objectives. Through the backpropagation algorithm, parameters are dynamically and differentially updated, which not only ensures the system's response accuracy to downstream tasks, but also achieves synergistic improvement in multimodal representation capabilities, thus balancing task performance and system stability.

[0020] 5. Construct a closed-loop optimization mechanism to adapt to real-world application scenarios: The online feedback optimization module can collect user satisfaction feedback and quantify it into reward signals, dynamically adjust cross-modal fusion and optimization strategies, and form a closed loop of data input, model optimization, service output and feedback iteration, enabling the system to continuously adapt to changes in real-world application scenarios and continuously improve user experience.

[0021] In summary, the modular design and adaptive optimization mechanism of this invention is compatible with multiple modal data such as text, images, and voice, eliminating the need for extensive customized development for specific scenarios. It can flexibly adapt to the multimodal interaction needs of different fields, significantly reducing the cost of application implementation and having a wide range of applications. Attached Figure Description

[0022] Figure 1 This is a block diagram illustrating the principle of a large-model multimodal capability adaptive tuning system according to the present invention. Figure 2 This is a block diagram illustrating the modal capability state evaluator of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figures 1-2 As shown, the present invention provides a technical solution: a large model multimodal capability adaptive tuning system, the system including a multimodal interaction interface module, a modal adapter group, and a cross-modal dynamic coordinator connected in sequence, and a modal capability state evaluator connected to the cross-modal dynamic coordinator and the modal adapter group respectively, and a dynamic balance tuning engine connected to both the cross-modal dynamic coordinator and the modal capability state evaluator. The multimodal interaction interface module is used to receive user query commands and, in response to the query commands, collect multimodal input data from at least two different types of data sources; the modality adapter group is used to perform feature extraction and representation alignment on the input data of each modality through independent and parameter-adjustable modality adapters, and output the first... Modality-specific embedding vectors of each modality ( , for 3D column vector, The preset feature embedding dimension has a value range of [256, 1024]); the cross-modal dynamic coordinator is used to receive modality-specific embedding vectors from all modalities. Based on the current task context, modal correlation weights representing intermodal dependencies are dynamically calculated to generate an adaptive fusion coefficient vector that guides multimodal information fusion. ( for A 3D column vector, with elements ranging from [0,1], and The modal capability state evaluator is used to monitor and analyze the performance index changes of each modal adapter in real time during the training process. Based on historical gradient information and the evolution law of validation loss, it generates modal state vectors that characterize the current learning efficiency and stability of each modality. ( (This is a 2D column vector containing two features: gradient activity and loss improvement rate). The dynamic balancing tuning engine is used to integrate adaptive fusion coefficient vectors. With modal state vector Construct a comprehensive optimization objective function that combines task objectives with modal capability balancing objectives. By dynamically and differentially updating the parameters of each modal adapter through the backpropagation algorithm, a collaborative and adaptive improvement of multimodal representation capabilities is achieved.

[0025] In embodiments of the present invention, the modal adapter structures in the modal adapter group are independent of each other and their parameter update processes are decoupled from each other. Each modal adapter adopts a low-rank adaptive structure or a prefix tuning structure. During the tuning process, only a small number of parameters introduced by the modal adapter itself are updated, while the backbone parameters of the underlying large language model remain frozen; wherein: When the modal adapter is a low-rank adaptive structure, low-rank matrix increments are injected into the weights of the fully connected layers of the underlying large language model. To update the parameters, the specific steps are as follows: S1.1 Initialize the low-rank matrix and The Xavier initialization method is used to ensure that the parameters follow a uniform distribution. for Low-rank matrix for Low-rank matrix For the weight dimension of the fully connected layer, It is a low-rank dimension and ; S1.2 Calculate the increment of the low-rank matrix ,pass and Obtained by matrix multiplication; S1.3 Inject incremental update weights, Original weights of the fully connected layer of the underlying large language model By superimposing the results, we obtain the updated weights. ; S1.4 Verify the update effect by calculating the feature extraction accuracy of the modal adapter. If the accuracy improvement is less than the preset threshold of [0.5%, 2%], then adjust... The value is taken and S1.1-S1.3 is repeated until the performance requirements are met; When the modal adapter is a prefix tuning structure, a trainable prefix vector sequence is added before the input sequence to the underlying large language model. To update the parameters, the specific steps are as follows: S1.5 Adaptive determination of prefix length Text modality Value range [10, 30], image and speech modalities The value range is [20, 50]. For the first prefix vectors ( The dimension is consistent with the input dimension of the underlying large language model; S1.6 Initialize the prefix vector sequence The data were initialized using a random normal distribution (mean 0, variance 0.01). S1.7 Concatenate the input sequence, Add to the original modal input sequence The front end forms a new input sequence. ; S1.8 iteratively updates the prefix parameters, calculates the output loss through forward propagation of the underlying large language model, and updates the parameters based on the loss value through backpropagation. The parameters are set, the core parameters of the underlying model are kept frozen, the number of iterations ranges from [50, 200], and the update is terminated if the loss value no longer decreases.

[0026] Furthermore, the modal adapter group adopts a structurally independent and parameter-decoupled design. Its purpose is to achieve targeted feature extraction and adaptation optimization for multimodal data, while avoiding disruption of the pre-trained knowledge of the underlying large language model. Each modal adapter operates independently, and parameter updates do not interfere with each other. It can select low-rank adaptation or prefix tuning structures based on the characteristics of different modalities such as text, image, and speech, updating only a small number of parameters it introduces (low-rank matrix or prefix vector). The core parameters of the underlying large language model remain frozen, reducing the computational cost of tuning while preserving the model's original general language knowledge and generalization ability. The low-rank adaptation structure injects low-rank matrix increments into the fully connected layers. It accurately adapts to the mapping requirements of modality-specific features, and with a closed-loop mechanism of initialization, calculation, update and verification, it can dynamically adjust the low-rank dimension. To ensure feature extraction accuracy meets targets and avoid overfitting or underfitting, the prefix optimization structure adaptively matches different modalities of input format and semantic features by adding a trainable prefix vector before the input sequence. The prefix length... By setting differentiating parameters according to modal type and combining iterative updates and loss monitoring, we can ensure efficient adaptation between modal input and model input layer, and ultimately achieve synergistic improvement in the feature representation capabilities of each modality, providing high-quality basic features for subsequent cross-modal fusion.

[0027] In this embodiment, taking a bimodal input scenario of text + image as an example, the working process and implementation effect of the modal adapter are explained: For the text modality (such as the user query command "Recommend historical and cultural attractions in City A on weekends"), the modal adapter adopts a prefix tuning structure, which adaptively determines the prefix length according to the characteristics of the text modality. (Within the range [10, 30]), a prefix vector sequence is generated through random normal initialization (mean 0, variance 0.01). Each prefix vector has the same dimension as the input dimension of the underlying large language model (set to 768 dimensions), and then... With the original text input sequence splicing to form a new input sequence The text feature extraction loss is calculated through forward propagation of the underlying large language model, and the feature is updated based on backpropagation of the loss value. The parameters of the underlying backbone remain frozen. After 120 iterations, the loss value tends to stabilize, and the updates stop. For image modalities (such as real-life photos of scenic spots in City A uploaded by users), the modality adapter adopts a low-rank adaptive structure, with the weight dimension set for the fully connected layer of the underlying large language model. Initial low-rank dimension ( Initialize using the Xavier initialization method. (2048×32 dimensions) and (2048×32 dimensional), calculate the increment of a low-rank matrix using matrix multiplication. ,Will With the original weights of the fully connected layer The updated weights are obtained by superposition. The image feature extraction accuracy was calculated. The initial accuracy improvement of 1.0% was lower than the upper limit of the preset threshold [0.5%, 2%]. Therefore, adjustments were made. The initialization, calculation, and update process was repeated to verify that the accuracy improved by 1.5%, meeting the threshold requirement, and then the adjustment was stopped.

[0028] After implementation, the accuracy of text modality feature extraction improved by 1.8% compared to before optimization, and the accuracy of image modality feature extraction improved by 1.2%. Furthermore, the optimization processes for the two modalities did not interfere with each other, and the generalization ability of the underlying large language model's backbone parameters remained unaffected. This enabled the generation of accurate modality-specific embedding vectors for the cross-modal dynamic coordinator. and This ensures the effectiveness of subsequent cross-modal fusion.

[0029] In an embodiment of the present invention, the cross-modal dynamic coordinator dynamically calculates modal correlation weights and generates an adaptive fusion coefficient vector. The process specifically includes: Step 1, Calculate dimensional intermodal correlation matrix ,matrix The Middle Line number Column elements Indicates the first The modality pair of the first The dependency correlation of each modality is determined by the following steps: S2.1 Modality embedding vector preprocessing, for all Perform L2 normalization to obtain Eliminate scale differences; S2.2 Initialize the trainable associative projection matrix The He initialization method is used. for 3D matrix ( With claim 1 (The dimensions are consistent) S2.3 Calculate the original correlation degree, through Obtain the original correlation degree in scalar form; S2.4 Association normalization is performed using the Sigmoid activation function. (expression )Will Map to the interval (0,1); S2.5 calculation The calculation formula is: ; Step 2, based on the intermodal correlation matrix Generate an adaptive fusion coefficient vector through column-dimensional aggregation operations. , For the first The fusion weight coefficients for each modality are determined by the following steps: S2.6 Set temperature parameters (Value range [0.5, 2.0]), adjust the weighting intensity of the index; S2.7 Calculate the first We obtain the weighted sum of the columns. ; S2.8 Calculate the global normalization factor and obtain ; S2.9 generation The contribution of a mode in multimodal information fusion is represented by the ratio of the weighted sum to the global normalization factor.

[0030] Furthermore, the cross-modal dynamic coordinator aims to overcome the limitations of fixed fusion weights in traditional multimodal models by accurately calculating the intermodal dependencies and dynamically generating adaptive fusion coefficient vectors. It achieves efficient collaborative fusion of multimodal features, providing a scientific and reasonable weight basis for subsequent dynamic balance optimization, while ensuring that the fusion process is highly adapted to the task context, significantly improving the utilization rate of multimodal information and the accuracy of fusion; its calculation process is rigorously designed through multiple steps, forming a complete closed loop from modal embedding vector preprocessing to fusion coefficient generation.

[0031] In step 1, the modality embedding vector is... L2 normalization effectively eliminates scale differences between different modal features, avoiding errors in correlation calculation caused by inconsistent feature magnitudes, and thus providing a basis for subsequent correlation matrix calculations. This lays the foundation for precise calculations; trainable associative projection matrices Using the He initialization method ensures the rationality of matrix initialization parameters, avoids gradient vanishing or exploding problems, and guarantees the original correlation degree. The stability and accuracy of the calculation; the application of the Sigmoid activation function can map the original correlation degree to the (0,1) interval, making the correlation degree more interpretable and facilitating the quantification of the intermodal dependency strength, while The normalization calculation further standardizes the numerical range of the correlation matrix, ensuring that each element in the matrix can truly reflect the relative dependencies between the corresponding modes.

[0032] Temperature parameters in step 2 The settings allow for flexible adjustment of the exponential weighting strength, which can amplify the contribution of highly correlated modes while avoiding the dominance of a single mode in the fusion process. The column-dimensional aggregation operation generates fusion coefficients by using the ratio of the weighted sum to the global normalization factor. This allows the fusion weights of each modality to be precisely matched with its contribution to the task, ultimately achieving adaptive fusion of multimodal features. This effectively solves the technical pain points of poor adaptability and unsatisfactory fusion results of traditional static weight fusion, providing core support for improving the overall performance of the system.

[0033] In this embodiment, the bimodal input scenario of text + image is used as described above, combined with the accurate modal embedding vector output by the modal adapter. and This explains how the cross-modal dynamic coordinator dynamically calculates the fusion coefficient vector. Implementation process: The number of modes (Text modality is modality 1, image modality is modality 2), modality embedding vector dimension (Consistent with the input dimension of the underlying large language model), the specific implementation steps are as follows: Step 1, calculate the 2×2-dimensional intermodal correlation matrix. S2.1 and Perform L2 normalization to obtain , To eliminate the scale difference between the two, S2.2 uses the He initialization method to initialize the associated projection matrix. Its dimension is 768×768, ensuring that the initialization parameters are reasonable; S2.3 calculates the original correlation degree respectively. , , , This yields the original correlation degree in scalar form; S2.4 uses the Sigmoid activation function. Mapping the original correlation scores to the (0,1) intervals yields the following results: , , , S2.5 Calculate the correlation matrix The elements, among which , , , Finally, a 2×2 dimension correlation matrix is ​​obtained. ; Step 2, Generate adaptive fusion coefficient vector S2.6 Set temperature parameters (Within the range [0.5, 2.0]), adjust the weighted index strength; S2.7 calculates the weighted sum of each column, the weighted sum of the first column is... The weighted sum in column 2 is S2.8 Calculate the global normalization factor, i.e. , is the weighted sum of columns 1 and 2; S2.9 generates the fusion coefficient. , Calculated , ,satisfy ,in The text modality has a slightly higher weight, which is in line with the task requirement that user query instructions are mainly text and supplemented by images in this scenario. After implementation, the accuracy of multimodal fusion is 2.3% higher than that of traditional fixed-weight fusion, which fully verifies the effectiveness and rationality of the calculation process.

[0034] In an embodiment of the present invention, the fusion weight coefficients in the adaptive fusion coefficient vector By analyzing the intermodal correlation matrix The The weighted pooling calculation is obtained from the column, and the calculation formula is as follows: ; in, The natural exponential function serves as the intermodal dependency correlation degree. The exponential weighting term serves to amplify the contribution weight of highly correlated modes and suppress the interference of low-correlation modes. For the first The modality pair of the first Dependency of each mode (taken from the inter-modal correlation matrix) The denominator is the global normalization factor, ensuring that all fusion weight coefficients satisfy... ,make It conforms to the characteristics of probability distribution, which facilitates the weighted fusion calculation of subsequent multimodal information.

[0035] Furthermore, adaptive fusion coefficients The calculation formula serves as the core support for generating fusion weights in the cross-modal dynamic coordinator, and its role is to improve the inter-modal correlation matrix. No. The columns are weighted pooled to achieve precise binding between modal fusion weights and intermodal dependencies, fundamentally solving the technical pain points of subjective weight setting and inability to adapt to changes in task context in traditional multimodal fusion, and ensuring the scientific and rational nature of multimodal information fusion.

[0036] Natural exponential function in the formula As intermodal dependency correlation The exponential weighting term effectively amplifies the contribution weight of highly correlated modes to the target mode while significantly suppressing the interference of low-correlation modes, making the fusion weights more discriminative and accurately capturing the core roles of different modes in specific tasks. The denominator, as a global normalization factor, is globally summed and normalized over the weighted correlation of all modes to ensure that all fusion weight coefficients are consistent. satisfy ,make Strictly adhering to probability distribution characteristics not only facilitates subsequent weighted fusion calculations of multimodal information, avoiding fusion effect deviations caused by weight imbalances, but also makes the fusion contribution of each mode interpretable. This provides a quantifiable and comparable weight basis for the dynamic balancing optimization engine, further improving the accuracy of multimodal fusion and the overall stability of the system. Simultaneously, it complements the intermodal correlation matrix mentioned earlier. The computational logic forms a closed loop, ensuring the coherence and rigor of the entire cross-modal fusion process.

[0037] In this embodiment, the dual-modal input scenario of text + image is used as described above, combined with the 2×2-dimensional intermodal correlation matrix calculated above. (in , , , ), fusion weight coefficient The calculation and implementation process, number of modes Specifically, the calculation formula is as follows: The numerator is the correlation matrix. No. The sum of all elements in the column after exponential weighting, with the denominator being the global normalization factor; Calculate the weighted sum of column 1 (the column corresponding to the text modality): ; Calculate the weighted sum of the second column (the column corresponding to the image modalities): ; Calculate the global normalization factor: Calculate the fusion weight coefficients separately. , Verification yields This conforms to the characteristics of a probability distribution; In this calculation result, the text modality fusion weights Higher than image modality This approach highly matches the task requirement of "text query as the main method and image as the auxiliary method" in this scenario. Applying this weight to multimodal fusion results in a higher fusion accuracy compared to traditional fixed-weight methods (such as...). The increase of 2.1% fully verifies the effectiveness, rationality, and practicality of the calculation formula.

[0038] In an embodiment of the present invention, the modal capability state estimator includes a gradient feature extraction unit, a loss trend analysis unit, and a state vector synthesis unit, which collaboratively generate modal state vectors. The specific process is as follows: The gradient feature extraction unit is used to calculate the gradient feature of each modal adapter in the most recent The gradient vector changes of trainable parameters during secondary parameter update iterations yield gradient feature values ​​that characterize modality learning activity. The specific steps are as follows: S3.1 Collects gradient vectors and records the most recent... The next iteration ( The value range [10, 50] is the first... The gradient vector of the trainable parameters of each modal adapter ( ); S3.2 Calculate the gradient L2 norm to obtain the gradient norm sequence. ; S3.3 Calculate the exponential moving average to obtain the gradient eigenvalues. ; The loss trend analysis unit is used to calculate the task loss for each modality in the most recent time frame based on the validation set data. The evolutionary features within each validation window yield the loss feature values ​​characterizing the modality learning efficiency. The specific steps are as follows: S3.4 Collects loss values; each verification window contains... training batches ( (Value range [5, 20]), collect the most recent Verification windows ( Loss value (range [5, 15]) ; S3.5 calculates the rate of change of loss, and obtains ( ); S3.6 Calculate the mean rate of change to obtain the loss characteristic value. ; The state vector synthesis unit is used to synthesize gradient feature values. With loss eigenvalues Perform normalized splicing to obtain the first... Modal state vectors of each mode The specific steps are as follows: S3.7 uses the min-max normalization method to... Normalized to the [0,1] interval, we get , , These are the minimum and maximum values ​​of the gradient eigenvalues ​​for all modes, respectively. Similarly, S3.8 Normalization yields ; S3.9 splicing and The modal state vector is obtained. ,in , They are respectively , The eigenvalues ​​after min-max normalization.

[0039] Furthermore, the modal capability state evaluator, through the three-level collaborative operation of the gradient feature extraction unit, loss trend analysis unit, and state vector synthesis unit, aims to overcome the technical bottleneck of traditional multimodal tuning that "lacks accurate state perception and blind tuning direction," and provide comprehensive and quantitative modal learning state basis for the dynamic balance tuning engine, realizing the transformation of the tuning process from experience-driven to data-driven. The gradient feature extraction unit generates gradient feature values ​​by collecting the gradient vectors of the most recent T iterations, calculating the L2 norm and exponential moving average. It can accurately characterize modality learning activity, capturing dynamic changes in parameter updates and smoothing gradient noise through exponential moving averages to avoid interference from single abnormal gradients in state judgment, thus ensuring the stability of activity assessment. The loss trend analysis unit, based on loss data from multiple validation windows, calculates loss feature values ​​by measuring the loss change rate and mean. This can intuitively reflect the modality learning efficiency. The setting of the verification window effectively avoids the impact of single-batch loss fluctuations, and the normalization of the loss change rate quantifies the improvement speed of modality learning, providing a key basis for judging whether the modality is overfitting or underfitting. The state vector synthesis unit uses the min-max normalization method to eliminate... and The scale difference is used to concatenate the two into a modal state vector of a unified dimension. This approach ensures the comparability of state representations and provides standardized input for the dynamic balancing tuning engine to calculate the modal capability balancing loss. The three work together to form a complete closed loop of activity perception, efficiency evaluation, and state integration, ensuring that tuning resources are tilted towards modalities with insufficient learning efficiency, thereby achieving a balanced improvement in the capabilities of each modality.

[0040] In this embodiment, the dual-modal input scenario of text + image is used, and combined with the modal adapter tuning parameters mentioned above, three-level units collaboratively generate... Process: Setting core parameters (Number of iterations) (Number of verification windows) (Training batches per window) (Exponential moving average coefficient); Gradient feature extraction unit: S3.1 collects text modal prefix vectors Image modal low-rank matrix and Gradient vectors of the last 30 iterations ( to ), ( to S3.2 Calculate the gradient L2 norm to obtain the sequence. and S3.3 is calculated using the exponential moving average. (Text Modal) (Image Modality); Loss Trend Analysis Unit: S3.4 collects the loss values ​​of the most recent 10 validation windows. (text), (Image); S3.5 Calculate the rate of change of loss ( to ), ( to S3.6 Calculate the average value (text), (Image); State vector synthesis unit: S3.7 min-max normalization, , ,have to , Similarly, S3.8 , ,have to , S3.9 splicing , This clearly reflects that the learning activity and efficiency of the image modality are lower than those of the text modality.

[0041] After the dynamic balancing optimization engine increased the optimization efforts on image modalities based on this state vector, its feature extraction accuracy was further improved by 0.4%, verifying the effectiveness and rationality of the three-level unit collaboration.

[0042] In an embodiment of the present invention, the gradient feature value For the most recent The exponential moving average of the gradient vector L2 norm in the next iteration, and the loss eigenvalues. For the most recent The mean of the rate of change of loss for each validation window, and the formulas for calculating them are as follows: ; ; in, For the first The gradient feature values ​​of each modality represent the modality learning activity; This is the exponential moving average coefficient, with a value range of [0.85, 0.95], used to balance the weights of historical gradients and the current gradient. The larger the gradient eigenvalue, the more stable the change in the gradient eigenvalue; For the first The gradient eigenvalues ​​of the next iteration; For the first During the nth iteration The gradient vector of the trainable parameters of each modal adapter; The L2 norm (Euclidean norm) is used to quantify the magnitude of the gradient vector; For the first The loss feature value of each modality characterizes the modality learning efficiency; To verify the number of windows; For the first The first verification window Each modality corresponds to a task loss value; For the first The first verification window Each modality corresponds to a task loss value, with the numerator representing the loss decrease and the denominator used for normalization to obtain the loss change rate.

[0043] Furthermore, gradient eigenvalues With loss eigenvalues The calculation formula, as the core technical support for the three-level unit collaboration of the modal capability state evaluator, plays a role in accurately quantifying the activity and efficiency of modal learning, solving the technical pain points of traditional modal state evaluation such as "no unified quantitative standard and subjective ambiguity of evaluation results", providing a scientific and traceable calculation basis for state vector synthesis and subsequent dynamic optimization, and ensuring the accuracy, stability and interpretability of modal state evaluation. in The exponential moving average formula, by introducing a coefficient... (Value range [0.85, 0.95]) Balances historical gradient features with current gradient information. A reasonable value can make the gradient eigenvalue It can capture the dynamic changes of current parameter updates and suppress the interference caused by single gradient anomalies through the smoothing effect of historical gradients, avoiding drastic fluctuations in evaluation results and accurately representing the true state of modal learning activity. The mean calculation formula effectively avoids the impact of single-batch loss fluctuations on learning efficiency evaluation by averaging the loss change rate of the most recent S validation windows. At the same time, it normalizes by the ratio of the loss decrease to the previous loss value, making the loss change rate of different modalities and different magnitudes comparable, accurately quantifying the level of modal learning efficiency, and clarifying whether the modality is in an efficient learning state. The two formulas work together to quantify the modal learning state from the two core dimensions of activity and efficiency, respectively. Together with the min-max normalization and splicing operation of the state vector synthesis unit, they form a complete technical loop, providing standardized and quantifiable state input for the dynamic balancing optimization engine. This ensures that the optimization strategy can accurately adapt to the learning state of each modality, achieving a reasonable allocation of optimization resources and a balanced improvement of the capabilities of each modality.

[0044] In this embodiment, the dual-modal input scenario of text + image is used as described above, combined with the pre-set core parameters ( , , , ),illustrate and The calculation process verifies the effectiveness and rationality of the formula: First, calculate the gradient eigenvalues. Taking the text modality (Modal 1) as an example, S3.1 collects the most recent 30 iterations ( to gradient vector S3.2 Calculate the L2 norm of the gradient vector in each iteration. Substitute S3.3 into the formula ,by Initial gradient eigenvalues Using this as a baseline, the gradient eigenvalues ​​are obtained after 30 iterations. Similarly, the image modality (second modality) uses... Based on this, iterative calculations yielded the following results. The results are consistent with those in the previous paper, verifying the stability of the formula.

[0045] Recalculate the loss eigenvalues Text modality S3.4 collects the loss values ​​of the most recent 10 validation windows. , , , , , , , , , S3.5 calculates the rate of change of loss for each window (e.g., Substitute S3.6 into the formula Calculated Image modality acquisition loss value , , , , , , , , , Similarly, we can calculate... .

[0046] The calculation results are required for the synthesis of the modal state vectors mentioned above. , A perfect match can be generated by substituting it into min-max normalization. , This provides a reliable basis for dynamic optimization and fully verifies the rationality, accuracy and practicality of the two calculation formulas.

[0047] In an embodiment of the present invention, the comprehensive optimization objective function constructed by the dynamic balancing tuning engine The weighted sum of the task's main loss and the modal capability balance loss is calculated using the following formula: ; in, To comprehensively optimize the objective function, a scalar value is used; the smaller the value, the better the system's task performance and modal balance performance. The main loss function is determined by the task scenario and task objective (cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks). To balance the hyperparameters, the value range is [0.1, 0.5], which is used to adjust the weight ratio of the modal capability balance loss; The modal capability balancing loss is used to constrain the learning efficiency and stability of each modal adapter to tend towards equilibrium, thereby avoiding modal capability imbalance.

[0048] Furthermore, comprehensively optimize the objective function. As the core decision-making basis of the dynamic balance optimization engine, its role is to break through the limitations of the traditional multimodal optimization "single task orientation", build a dual-objective optimization mechanism of task performance and modal balance, fundamentally solve the technical pain points of some modal overfitting, some modal underfitting, and modal capability imbalance, and ensure that the system can achieve synergistic improvement of multimodal representation capabilities while accurately responding to downstream tasks. in As the main loss function for specific tasks, it directly anchors the core business performance of the system (cross-entropy loss is used to ensure classification accuracy for classification tasks, and mean squared error loss is used to ensure prediction accuracy for regression tasks), ensuring that the optimization does not deviate from the actual application requirements. As a modal capability balancing loss, by constraining the deviation of each modal state from the ideal state, it forces modes with insufficient learning efficiency to obtain more tuning resources, avoiding capability imbalance caused by the excessive consumption of resources by dominant modes; balancing hyperparameters (Value range [0.1, 0.5]) The weight ratio of the two objectives can be flexibly adjusted to ensure the core position of task performance while also achieving a reasonable balance. The chosen values ​​avoid excessive interference of modal balance loss on the main task loss, enabling the two to work together to form an optimization loop that balances task accuracy and capability. This provides a clear and scientific optimization objective for the backpropagation algorithm, guides the dynamic and differentiated updates of parameters for each modal adapter, and ultimately achieves a stable improvement in the overall system performance.

[0049] In this embodiment, the bimodal input scenario of text + image is used as described above (the downstream task is scenic spot recommendation intent classification, which belongs to classification task). Combined with the generated modal state vectors, fusion coefficients, and other parameters, a detailed explanation is provided. Calculation and implementation results: First, set the core parameters, (within the range of [0.1, 0.5]) Cross-entropy loss is used, with a pre-defined ideal modal state target vector. ( , ); Known text modality (first modality) fusion coefficients Modal state vector Image modality (second modality) fusion coefficient Modal state vector ; Calculate the modal capability balance loss First, obtain the balance penalty weights. , Then through the formula Calculated , ,therefore Assuming the classification task uses cross-entropy loss... Substituting into the comprehensive optimization objective function, we get .

[0050] After updating the parameters through backpropagation based on this objective function, the overlearning trend of the text modality is suppressed, and the optimization of the image modality is enhanced. The final task classification accuracy is improved by 3.1% compared to before optimization, and the state vectors of the two modalities are optimized as follows: , It is closer to the ideal state, and the modal capability balance is significantly improved, which fully verifies the rationality and effectiveness of the function.

[0051] In an embodiment of the present invention, the modal capability balance loss The deviation between the modal state vector and the ideal modal state target vector is calculated using the following formula: ; in, This is the modal capability balance loss, a scalar value used to quantify the deviation of each modal state from the ideal state; For the first The balance penalty weight of the first mode, and the first mode mentioned above. Fusion weight coefficients for each modality Negative correlation; the lower the fusion weight, the greater the penalty weight, forcibly increasing its optimization intensity. For the first Modal state vectors of 1 modality; The target vector is the preset ideal modal state. The ideal gradient eigenvalues ​​(range [0.4, 0.6]) are the eigenvalues. These are the ideal loss feature values ​​(ranging from [0.05, 0.15]), all of which are empirically set constants; The L1 norm (Manhattan norm) is used to calculate the sum of absolute deviations of vectors. Compared to the L2 norm, it is less sensitive to outliers, ensuring the stability of the balance loss.

[0052] Furthermore, modal capability balance loss As a core component of the comprehensive optimization objective function, its role is to construct a modal capability balancing constraint mechanism, fundamentally solving the imbalance problem of "dominant modes excessively consuming resources and weak modes insufficiently optimizing" in multimodal tuning, and ensuring the coordinated improvement of the representation capabilities of each modality; among which, balancing penalty weights With fusion weight coefficient Negative correlation results in modes with lower fusion weights receiving larger penalty weights, forcing the dynamic balancing optimization engine to allocate optimization resources to modes with insufficient learning efficiency. This avoids a single mode dominating the optimization process and achieves balanced optimization by complementing the weak and supporting the strong. A preset ideal modal state target vector is also included. ( , Setting unified and reasonable learning objectives for each modality clarifies the direction of modal state optimization, avoiding aimless tuning. The application of the L1 norm (Manhattan norm) calculates the absolute deviation of the vector and quantifies the deviation between the modal state and the ideal state. Compared to the L2 norm, it is less sensitive to outliers, effectively avoiding interference from single modal state fluctuations on the balance loss calculation, ensuring the stability and reliability of the loss value. The synergistic effect of these three factors makes... It can accurately quantify the degree of modal capability imbalance and provide clear guidance for equilibrium optimization, in line with the main task loss. By forming a dual-objective optimization loop that achieves both task precision and capability balance, the system can improve the performance of downstream tasks while avoiding the polarization of modal capabilities, thus significantly enhancing the stability and generalization ability of multimodal fusion.

[0053] In this embodiment, the bimodal input scenario of text + image (downstream task is scenic spot recommendation intent classification) is used as described above, and the core parameters are explained in detail. Calculation and implementation results: Setting the target vector of ideal modal state ( , (All values ​​are within the empirical range), known text modality (first modality) fusion coefficients. Modal state vector Image modality (second modality) fusion coefficient Modal state vector First, calculate the balance penalty weights. , This conforms to the constraint logic of "the lower the fusion weight, the greater the penalty weight"; then, the deviation between the modal state and the ideal state is calculated using the L1 norm. , Substituting into the formula, we get .

[0054] After substituting the balance loss into the comprehensive optimization objective function, the dynamic balance tuning engine increases the parameter update intensity of the image modality and suppresses the overlearning trend of the text modality. After iterative tuning, the text modality state vector is optimized as follows: The image modal state vector is optimized as follows: Both are closer to the ideal state, with a 40% improvement in modal capability balance and a 3.1% improvement in task classification accuracy compared to before optimization, fully validating the results. The rationality, effectiveness, and practicality of it.

[0055] In an embodiment of the present invention, the large model multimodal capability adaptive tuning system further includes an online feedback optimization module. This online feedback optimization module is connected to both the dynamic balancing tuning engine and the cross-modal dynamic coordinator, and is used to collect user satisfaction feedback on the system output and quantify it into an instant reward value. ( (For feedback time steps); the dynamic balancing optimization engine, during the online service phase, calculates based on the historical real-time reward value sequence. Adjusting the intermodal correlation projection matrix in the cross-modal dynamic coordinator The update step size, the specific steps are as follows: S4.1 Collect user satisfaction ratings. Obtain user ratings through the interactive interface (range [1,5], where 1 point indicates extreme dissatisfaction and 5 points indicates extreme satisfaction). S4.2 quantifies instant reward value, through Rating Mapping to the [0,1] interval, we get ; S4.3 calculates historical cumulative rewards, through get( (This is a discount factor, with a value range of [0.9, 0.99]). S4.4 Dynamically adjust and update the step size, calculate ( The update step size is based on the base value, with a range of [0.001, 0.01]. and Positive correlation; S4.5 fine-tunes the cross-modal coordinator parameters, using the policy gradient method to calculate the parameter gradients. ( (The parameters are trainable for the cross-modal dynamic coordinator and are updated using the gradient ascent algorithm); S9.6 verifies the fine-tuning effect. The average reward value is calculated after every 10 parameter fine-tuning iterations. If the improvement is lower than the preset threshold of [1%, 3%], the reward is reduced. Repeat step S4.5 until the average reward value reaches the preset target of [0.7, 0.9]. Update step size The positive correlation with historical cumulative reward values ​​allows the system to assign higher fusion weight coefficients to modal combinations that have previously received high rewards, and the policy gradient method is used to fine-tune all trainable parameters of the cross-modal dynamic coordinator with the goal of maximizing cumulative rewards.

[0056] Furthermore, the online feedback optimization module breaks through the technical limitations of the traditional multimodal model training phase being disconnected from actual applications. It directly transforms real user experience into quantitative signals for model tuning, enabling continuous iterative improvement of system performance and ensuring that multimodal fusion and tuning strategies always meet the needs of actual scenarios. Specifically, S4.1-S4.2 maps user satisfaction scores of [1,5] to instant reward values ​​in the [0,1] range. This approach preserves the diversity of user feedback while standardizing reward signals, providing an intuitive and calculable basis for subsequent optimization; S4.3 uses discount factors... (Value range [0.9, 0.99]) Calculate historical cumulative rewards It emphasizes both the timeliness of recent user feedback and the stability of long-term high-quality modal combinations, avoiding optimization fluctuations caused by single, accidental feedback; S4.4 dynamically adjusts the update step size. This ensures a positive correlation between step size and cumulative reward, allowing modal combinations that have previously gained high user approval to receive more tuning resources and strengthening high-quality fusion strategies. S4.5 employs a policy gradient method to fine-tune the cross-modal coordinator parameters, aiming to maximize cumulative reward and ensuring that the tuning direction is precisely aligned with core user needs. S4.6 verifies the fine-tuning effect and dynamically adjusts the base step size. This avoids overfitting or insufficient optimization, ensuring the stability and effectiveness of the optimization process. The entire module forms a complete closed loop of user feedback, reward quantification, cumulative evaluation, step size adjustment, parameter fine-tuning, and effect verification, enabling the system to adapt to changes in scenarios and user preferences in real time, and continuously improve the user experience and system usability of multimodal interaction.

[0057] In this embodiment, the bimodal input scenario of text + image is used as described above (the downstream task is to classify the intent of scenic spot recommendation). Based on the determined core parameters, the implementation process and effects of the module are explained in detail: setting key parameters and discount factors. Basic update step size (All within the preset value range), preset average reward value target [0.7, 0.9]; S4.1 Collect user ratings for 100 system recommendation results through the interactive interface, distributed as follows: 5 points (35 times), 4 points (40 times), 3 points (15 times), 2 points (8 times), 1 point (2 times); S4.2 Use formula Quantified reward value, 5 points correspond to 4 points correspond 3 points correspondence 2 points correspond 1 point corresponds to S4.3 calculates the historical cumulative reward for the 100th feedback. Calculated S4.4 Dynamically adjust update step size S4.5 uses the policy gradient method to calculate the cross-modal coordinator parameters (intermodal correlation projection matrix). gradient of ) Updated using gradient ascent algorithm The average reward value after the first 10 fine-tunings of S4.6 was 0.65, which did not meet the target, so it was reduced. The value was repeatedly fine-tuned to 0.003; after 3 rounds of parameter updates totaling 30 times, the average reward value reached 0.76, meeting the preset requirements.

[0058] After implementation, the fusion weights assigned to high-reward modal combinations by the system will be adjusted to... (text), (Images) are now more aligned with user preferences, and user satisfaction with the recommended attractions has increased by 18% compared to before the optimization, fully validating the effectiveness and practicality of the module.

[0059] In embodiments of the present invention, the multimodal input data includes any two or more of text, images, speech, video, and structured tabular data; the multimodal interaction interface module supports multiple data acquisition methods such as direct text input, speech recognition, image parsing, video frame extraction, and structured data parsing, and can adapt to the input interface protocols of different devices, standardizing the acquired multimodal input data before transmitting it to the modal adapter group. The specific standardization steps are as follows: S5.1 unifies the data format, converting text data into word embedding tensors, image data into pixel value tensors normalized to the [0,1] interval, speech data into Mel spectrum tensors, video data into frame sequence tensors, and structured table data into feature tensors; S5.2 Aligns data dimensions by adjusting the feature dimensions of all modal data using zero-padding or pruning. ; S5.3 processes abnormal data, detects and removes abnormal values ​​related to noisy pixels, silent segments, and invalid characters, and uses interpolation to fill in missing data; S5.4 classifies and transmits data, transmitting the standardized input data to the corresponding modal adapter in the modal adapter group according to the modal type, ensuring the accuracy and real-time performance of data transmission.

[0060] Furthermore, the standardization steps (S5.1-S5.4) of the multimodal interaction interface module are designed to address the core pain points of multimodal data, such as "heterogeneous format, inconsistent dimensions, and varying quality," providing "unified, standardized, and high-quality" input data for feature extraction and cross-modal fusion of the subsequent modal adapter group, thus ensuring the overall adaptability and stability of the system from the source.

[0061] Specifically, S5.1 completely eliminates the format heterogeneity of different modal data by converting different types of data, such as text, images, and speech, into unified tensor formats such as word embedding tensors and pixel value tensors, thus enabling each modal data to have the basic conditions for computation and fusion; S5.2 uniformly adjusts the feature dimensions of all modal data to a unified format through zero-padding or pruning. To address the inconsistency in dimensionality caused by differences in modal characteristics, S5.3 ensures that the modal adapter group can perform unified feature extraction on data from various modalities, avoiding fusion bias caused by dimensionality mismatch; S5.3 effectively improves data quality by detecting and removing noise and invalid data and interpolating to fill missing values, reducing the interference of abnormal data on model tuning and ensuring the reliability of feature extraction and fusion results; S5.4 transmits data according to modal type, ensuring that the standardized modal data accurately matches the corresponding adapter in the modal adapter group, avoiding data transmission confusion, and ensuring real-time transmission, laying the foundation for the efficient operation of subsequent modules; The entire standardized process forms a complete closed loop of unified format, aligned dimensions, optimized quality, and accurate transmission, enabling the system to be compatible with multiple modal data and different device input protocols. It eliminates the need for customized development for specific data formats or devices, significantly improving the system's compatibility and ease of use, and reducing application deployment costs.

[0062] In this embodiment, taking a trimodal input scenario of text + image + structured table as an example (the downstream task is tourism product recommendation, the text is the user query "recommend family tourism package in City A", the image is a real-life photo of the attractions included in the package, and the structured table is a table of information such as package price and number of days of trip), the implementation process of the standardization steps is explained in detail: First, the core parameters and feature dimensions are set. (Consistent with the input dimensions of the underlying large language model); S5.1 Unified data format: Text data is converted into a 768-dimensional word embedding tensor through a pre-trained word embedding model; image data is scaled to 224×224 pixels and normalized to the [0,1] interval to obtain a 224×224×3 pixel value tensor; structured table data is converted into a 768-dimensional feature tensor after one-hot encoding and numerical normalization; S5.2 Aligned data dimensions: Text and structured table tensors are already 768-dimensional and do not require adjustment; image pixel value tensors are compressed to 768 dimensions through global average pooling to ensure consistency with the underlying large language model input dimensions. Consistent; S5.3 Handling Abnormal Data: Three noisy pixels (pixel values ​​deviating from the normal range) were detected in the image and removed, and filled in by interpolation of neighboring pixels; Two invalid special characters were contained in the text and were directly filtered; One missing travel day value in the structured table was filled by interpolation using the average travel day value of the same type of package; S5.4 Classifying and Transmitting Data: The standardized text tensor was transmitted to the prefix tuning structure adapter, the image tensor was transmitted to the low-rank adaptive structure adapter, and the structured table tensor was transmitted to the low-rank adaptive structure adapter (selected according to data characteristics). The transmission delay was controlled within 50ms to ensure real-time performance.

[0063] After implementation, all three modalities of data met the input requirements of subsequent modules. The feature extraction accuracy improved by 3.5% compared to the unstandardized processing, and the data transmission error rate dropped to below 0.1%, fully verifying the effectiveness and rationality of the standardization steps.

[0064] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A large-scale multimodal capability adaptive tuning system, characterized in that, The system includes a multimodal interaction interface module, a modal adapter group, and a cross-modal dynamic coordinator connected in sequence. It also includes a modal capability status evaluator connected to the cross-modal dynamic coordinator and the modal adapter group, and a dynamic balance tuning engine connected to both the cross-modal dynamic coordinator and the modal capability status evaluator. The multimodal interaction interface module is used to receive user query commands and, in response to the query commands, collect multimodal input data from at least two different types of data sources. The modal adapter group is used to perform feature extraction and representation alignment on the input data of each modality through independent and parameter-tunable modal adapters, and output the first modality. Modality-specific embedding vectors of each modality ; The cross-modal dynamic coordinator is used to receive modality-specific embedding vectors from all modalities. Based on the current task context, modal correlation weights representing intermodal dependencies are dynamically calculated to generate an adaptive fusion coefficient vector that guides multimodal information fusion. ; The modal capability state evaluator is used to monitor and analyze the performance index changes of each modality adapter in real time during the training process. Based on historical gradient information and the evolution law of validation loss, it generates modal state vectors that characterize the current learning efficiency and stability of each modality. ; The dynamic balancing optimization engine is used to integrate adaptive fusion coefficient vectors. With modal state vector Construct a comprehensive optimization objective function that combines task objectives with modal capability balancing objectives. By dynamically and differentially updating the parameters of each modal adapter through the backpropagation algorithm, a collaborative and adaptive improvement of multimodal representation capabilities is achieved.

2. The adaptive tuning system for large-scale multimodal capabilities according to claim 1, characterized in that, The modal adapters in the modal adapter group are structurally independent and their parameter update processes are decoupled. Each modal adapter adopts a low-rank adaptive structure or a prefix tuning structure. During the tuning process, only a small number of parameters introduced by the modal adapter itself are updated, while the backbone parameters of the underlying large language model remain frozen. When the modal adapter is a low-rank adaptive structure, low-rank matrix increments are injected into the weights of the fully connected layers of the underlying large language model. To update the parameters, the specific steps are as follows: S1.1 Initialize the low-rank matrix and The Xavier initialization method is used to ensure that the parameters follow a uniform distribution. for Low-rank matrix for Low-rank matrix For the weight dimension of the fully connected layer, It is a low-rank dimension and ; S1.2 Calculate the increment of the low-rank matrix ,pass and Obtained by matrix multiplication; S1.3 Inject incremental update weights, Original weights of the fully connected layer of the underlying large language model By superimposing the results, we obtain the updated weights. ; S1.4 Verify the update effect by calculating the feature extraction accuracy of the modal adapter. If the accuracy improvement is less than the preset threshold of [0.5%, 2%], then adjust... The value is taken and S1.1-S1.3 is repeated until the performance requirements are met; When the modal adapter is a prefix tuning structure, a trainable prefix vector sequence is added before the input sequence to the underlying large language model. To update the parameters, the specific steps are as follows: S1.5 Adaptive determination of prefix length Text modality Value range [10, 30], image and speech modalities The value range is [20, 50]. For the first Each prefix vector has the same dimension as the input dimension of the underlying large language model. S1.6 Initialize the prefix vector sequence Random normal initialization is used; S1.7 Concatenate the input sequence, Add to the original modal input sequence The front end forms a new input sequence. ; S1.8 iteratively updates the prefix parameters, calculates the output loss through forward propagation of the underlying large language model, and updates the parameters based on the loss value through backpropagation. The parameters are set, the core parameters of the underlying model are kept frozen, the number of iterations ranges from [50, 200], and the update is terminated if the loss value no longer decreases.

3. The adaptive tuning system for large-scale multimodal capabilities according to claim 2, characterized in that, The cross-modal dynamic coordinator dynamically calculates modal correlation weights and generates an adaptive fusion coefficient vector. The process specifically includes: Step 1, Calculate dimensional intermodal correlation matrix ,matrix The Middle Line number Column elements Indicates the first The modality pair of the first The dependency correlation of each modality is determined by the following steps: S2.1 Modality embedding vector preprocessing, for all Perform L2 normalization to obtain Eliminate scale differences; S2.2 Initialize the trainable associative projection matrix The He initialization method is used. for 3D matrix ( With claim 1 (The dimensions are consistent) S2.3 Calculate the original correlation degree, through Obtain the original correlation degree in scalar form; S2.4 Association normalization is performed using the Sigmoid activation function. (expression )Will Map to the interval (0,1); S2.5 calculation The calculation formula is: ； Step 2, based on the intermodal correlation matrix Generate an adaptive fusion coefficient vector through column-dimensional aggregation operations. , For the first The fusion weight coefficients for each modality are determined by the following steps: S2.6 Set temperature parameters Adjust the weighted intensity of the index; S2.7 Calculate the first We obtain the weighted sum of the columns. ; S2.8 Calculate the global normalization factor and obtain ; S2.9 generation The contribution of a mode in multimodal information fusion is represented by the ratio of the weighted sum to the global normalization factor.

4. The adaptive tuning system for large-scale multimodal capabilities according to claim 3, characterized in that, The fusion weight coefficient in the adaptive fusion coefficient vector By analyzing the intermodal correlation matrix The The weighted pooling calculation is obtained from the column, and the calculation formula is as follows: ； in, The natural exponential function serves as the intermodal dependency correlation degree. The index-weighted term; For the first The modality pair of the first The dependency correlation of each modality; the denominator is the global normalization factor, ensuring that all fusion weight coefficients satisfy... ,make It conforms to the characteristics of a probability distribution.

5. The adaptive tuning system for large-scale multimodal capabilities according to claim 4, characterized in that, The modal capability state estimator includes a gradient feature extraction unit, a loss trend analysis unit, and a state vector synthesis unit, which generate modal state vectors through the collaborative operation of the three-level units. The specific process is as follows: The gradient feature extraction unit is used to calculate the gradient feature of each modal adapter in the most recent The gradient vector changes of trainable parameters during secondary parameter update iterations yield gradient feature values ​​that characterize modality learning activity. The specific steps are as follows: S3.1 Collects gradient vectors and records the most recent... In the nth iteration The gradient vector of the trainable parameters of each modal adapter ( ); S3.2 Calculate the gradient L2 norm to obtain the gradient norm sequence. ; S3.3 Calculate the exponential moving average to obtain the gradient eigenvalues. ; The loss trend analysis unit is used to calculate the task loss for each modality in the most recent time frame based on the validation set data. The evolutionary features within each validation window yield the loss feature values ​​characterizing the modality learning efficiency. The specific steps are as follows: S3.4 Collects loss values; each verification window contains... Each training batch collects the most recent data. Loss value of each verification window ; S3.5 calculates the rate of change of loss, and obtains ( ); S3.6 Calculate the mean rate of change to obtain the loss characteristic value. ; The state vector synthesis unit is used to synthesize gradient feature values. With loss eigenvalues Perform normalized splicing to obtain the first... Modal state vectors of each mode The specific steps are as follows: S3.7 uses the min-max normalization method to... Normalized to the [0,1] interval, we get , , These are the minimum and maximum values ​​of the gradient eigenvalues ​​for all modes, respectively. Similarly, S3.8 Normalization yields ; S3.9 splicing and The modal state vector is obtained. ,in , They are respectively , The eigenvalues ​​after min-max normalization.

6. The adaptive tuning system for large model multimodal capabilities according to claim 5, characterized in that, The gradient eigenvalues For the most recent The exponential moving average of the gradient vector L2 norm in the next iteration, and the loss eigenvalues. For the most recent The mean of the rate of change of loss for each validation window, and the formulas for calculating them are as follows: ； ； in, For the first The gradient feature values ​​of each modality represent the modality learning activity. This is the exponential moving average coefficient, with a value range of [0.85, 0.95], used to balance the weights of historical gradients and the current gradient. The larger the gradient eigenvalue, the more stable the change in the gradient eigenvalue; For the first The gradient eigenvalues ​​of the next iteration; For the first During the nth iteration The gradient vector of the trainable parameters of each modal adapter; It is an L2 norm; For the first The loss feature value of each modality characterizes the modality learning efficiency; To verify the number of windows; For the first The first verification window Each modality corresponds to a task loss value; For the first The first verification window Each modality corresponds to a task loss value, with the numerator representing the loss decrease and the denominator used for normalization to obtain the loss change rate.

7. The adaptive tuning system for large model multimodal capabilities according to claim 6, characterized in that, The comprehensive optimization objective function constructed by the dynamic balancing tuning engine The weighted sum of the task's main loss and the modal capability balance loss is calculated using the following formula: ； in, To comprehensively optimize the objective function; The main loss function is oriented towards specific downstream tasks; To balance the hyperparameters, the values ​​are set to [0.1, 0.5]. This is the loss for modal capability balancing.

8. The adaptive tuning system for large model multimodal capabilities according to claim 7, characterized in that, The modal capability balance loss The deviation between the modal state vector and the ideal modal state target vector is calculated using the following formula: ； in, This is to balance the modal capability loss; For the first Balanced penalty weights for each modality; For the first Modal state vectors of 1 modality; The target vector is the preset ideal modal state. For ideal gradient eigenvalues, These are the ideal loss feature values, all of which are constants set empirically. It is an L1 norm.

9. The adaptive tuning system for large-scale multimodal capabilities according to claim 8, characterized in that, The large-model multimodal capability adaptive tuning system also includes an online feedback optimization module. This online feedback optimization module is connected to the dynamic balancing tuning engine and the cross-modal dynamic coordinator, and is used to collect user satisfaction feedback on the system output and quantify it into an instant reward value. ( (For feedback time steps); the dynamic balancing optimization engine, during the online service phase, calculates based on the historical real-time reward value sequence. Adjusting the intermodal correlation projection matrix in the cross-modal dynamic coordinator The update step size, the specific steps are as follows: S4.1 Collect user satisfaction ratings. Obtain user ratings through the interactive interface (range [1,5], where 1 point indicates extreme dissatisfaction and 5 points indicates extreme satisfaction). S4.2 quantifies instant reward value, through Rating Mapping to the [0,1] interval, we get ; S4.3 calculates historical cumulative rewards, through get( (This is a discount factor, with a value range of [0.9, 0.99]). S4.4 dynamically adjusts and updates the step size, calculating... ( The update step size is based on the base value, with a range of [0.001, 0.01]. and Positive correlation; S4.5 fine-tunes the cross-modal coordinator parameters, using the policy gradient method to calculate the parameter gradients. ( (The parameters are trainable for the cross-modal dynamic coordinator and are updated using the gradient ascent algorithm); S9.6 verifies the fine-tuning effect. The average reward value is calculated after every 10 parameter fine-tuning iterations. If the improvement is lower than the preset threshold of [1%, 3%], the reward is reduced. Repeat step S4.5 until the average reward value reaches the preset target of [0.7, 0.9]. Update step size The positive correlation with historical cumulative reward values ​​allows the system to assign higher fusion weight coefficients to modal combinations that have previously received high rewards, and the policy gradient method is used to fine-tune all trainable parameters of the cross-modal dynamic coordinator with the goal of maximizing cumulative rewards.

10. The adaptive tuning system for large-scale multimodal capabilities according to claim 9, characterized in that, The multimodal input data includes any two or more of the following: text, image, voice, video, and structured tabular data. The multimodal interaction interface module supports multiple data acquisition methods, including direct text input, voice recognition, image parsing, video frame extraction, and structured data parsing. It can also adapt to the input interface protocols of different devices, standardizing the acquired multimodal input data before transmitting it to the modal adapter group. The specific standardization steps are as follows: S5.1 unifies the data format, converting text data into word embedding tensors, image data into pixel value tensors normalized to the [0,1] interval, speech data into Mel spectrum tensors, video data into frame sequence tensors, and structured table data into feature tensors; S5.2 Aligns data dimensions by adjusting the feature dimensions of all modal data using zero-padding or pruning. ; S5.3 processes abnormal data, detects and removes abnormal values ​​related to noisy pixels, silent segments, and invalid characters, and uses interpolation to fill in missing data; S5.4 classifies and transmits data, transmitting the standardized input data to the corresponding modal adapter in the modal adapter group according to the modal type, ensuring the accuracy and real-time performance of data transmission.

Images