Method and system for model tuning based on multi-modal information and ai

Abstract

The application provides a large model optimization method and system based on multi-modal information and AI, relates to the technical field of artificial intelligence, and comprises the following steps: by extracting multi-modal parameter features and interaction features, dynamically adjusting a contribution coefficient to realize feature migration and obtain a cross-modal fusion vector; adopting an antagonistic mechanism to align the modal features to obtain domain invariant features; constructing a prompt chain based on modal confidence, and optimizing the prompt chain by using AI reinforcement learning, and iterating to convergence to obtain an optimization result; and finally updating the large model parameters. The application realizes effective fusion of multi-modal information, and improves the optimization effect and generalization ability of the human-computer interaction large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a large model optimization method and system based on multi-modal information and AI. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large language models and multi-modal models are widely used in various human-computer interaction scenarios. These models process text, images, audio and other multi-modal information to achieve a more natural and efficient interaction experience. Traditional large model optimization mainly uses supervised fine-tuning and prompt engineering methods, which have achieved good results in single-modal data processing. However, when it comes to complex multi-modal interaction scenarios, how to effectively integrate different modal information, realize knowledge transfer between modalities, and improve model performance in cross-modal tasks becomes a problem to be solved.

[0003] Traditional large model optimization methods often simply concatenate or average fuse different modal features, ignoring the correlation and interaction characteristics between different modalities, resulting in ineffective transmission of information between modalities, making the model perform poorly when processing complex multi-modal tasks, especially when some modal information is missing or has a lot of noise. Existing multi-modal fusion methods usually use a fixed weight fusion strategy, which cannot dynamically adjust the contribution of each modality according to the characteristics of different input data. This static fusion strategy is difficult to adapt to complex and changing real-world application scenarios, limiting the generalization ability of the model when handling boundary cases. The current model optimization process lacks an effective modality alignment mechanism and a reinforcement learning-based prompt optimization strategy, making it difficult to achieve deep semantic fusion and complementarity of modal features, and also making it difficult to dynamically optimize model parameters according to actual interaction effects, resulting in low optimization efficiency and difficulty in fully exploiting the potential of multi-modal information in improving model understanding and generation quality. SUMMARY

[0004] The embodiments of the present application provide a large model optimization method and system based on multi-modal information and AI, which can solve the problems in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a large model optimization method based on multi-modal information and AI, comprising:

[0006] extracting a plurality of modal features of the parameters of the large human-computer interaction model to be optimized;

[0007] extracting parameter correlation and modal interaction features between different modal features, dynamically adjusting a contribution coefficient of the modal interaction features based on the parameter correlation, migrating feature information between different modal features according to the contribution coefficient, and obtaining a cross-modal fusion vector; using a discriminator to identify the original modal attribution of the cross-modal fusion vector, using an adversarial generator to generate confusion features, and using the confusion features to attack the identification accuracy of the discriminator, aligning the modal features in the attack, and obtaining domain-invariant features;

[0008] calculating modal confidence of each modal feature, constructing a mapping path between different modal features based on the modal confidence and the parameter correlation to obtain a prompt chain; combining the identification accuracy of the discriminator and the alignment degree of the modal features into a reward signal, optimizing the prompt chain using AI reinforcement learning based on the reward signal, guiding the optimization direction of the prompt chain through the domain-invariant features, and iteratively executing the AI reinforcement learning until the change rate of the reward signal is less than a preset convergence threshold, and obtaining a prompt optimization result;

[0009] updating parameters of the to-be-optimized human-computer interaction large model according to the prompt optimization result.

[0010] extracting parameter correlation and modal interaction features between different modal features, dynamically adjusting a contribution coefficient of the modal interaction features based on the parameter correlation, migrating feature information between different modal features according to the contribution coefficient, and obtaining a cross-modal fusion vector including:

[0011] calculating mutual information and Pearson correlation coefficients between the modal features and combining them to obtain the parameter correlation;

[0012] extracting the modal interaction features and performing feature aggregation, constructing a modal correlation matrix based on the modal interaction features after feature aggregation, performing singular value decomposition on the modal correlation matrix to obtain the contribution coefficient, calculating the difference degree between the parameter correlations to obtain a dynamic adjustment factor, and adjusting the contribution coefficient based on the dynamic adjustment factor;

[0013] performing feature fusion on the modal interaction features after nonlinear transformation and the modal features to obtain intermediate features, determining a feature selection mask and a migration amount according to the contribution coefficient, performing feature screening on the intermediate features using the feature selection mask, performing cross-modal feature migration on the screening results based on the migration amount, and performing residual connection with the intermediate features to generate a cross-modal fusion vector containing multi-modal information.

[0014] The discriminator is used to identify the original modality attribution of the cross-modal fusion vector, the adversarial generator is used to generate the confusion feature, the identification accuracy of the discriminator is attacked through the confusion feature, the modal features are aligned in the attack, and the domain-invariant feature is obtained, comprising:

[0015] The discriminator uses a multi-layer perception to calculate a modality probability distribution of the cross-modal fusion vector, and determines the original modality attribution according to a modality category corresponding to a maximum probability value in the modality probability distribution; the adversarial generator generates the confusion feature through affine transformation of the cross-modal fusion vector, and mixes the confusion feature with the cross-modal fusion vector in the identification process of the discriminator to attack the identification accuracy of the discriminator;

[0016] A cross-entropy loss term of the modality probability distribution and a discrimination loss term of the confusion feature are calculated to obtain an adversarial loss of the discriminator; an adversarial loss term of the confusion feature and a feature alignment loss term between different modal features are calculated to obtain a generation loss of the adversarial generator;

[0017] An alternating momentum optimization algorithm is used to iteratively optimize the adversarial loss and the generation loss, in the iterative optimization process, the identification accuracy of the discriminator is minimized using the confusion feature, and different modal features are guided to align using the feature alignment loss term, until the adversarial loss and the generation loss reach a Nash equilibrium, and the iteration is stopped, thereby obtaining the domain-invariant feature.

[0018] In the iterative optimization process, the identification accuracy of the discriminator is minimized using the confusion feature, and different modal features are guided to align using the feature alignment loss term, until the adversarial loss and the generation loss reach a Nash equilibrium, and the iteration is stopped, thereby obtaining the domain-invariant feature.

[0019] The feature mean vector and the covariance matrix of each modality feature are calculated, the center distance loss of different modal features is calculated based on the feature mean vector, the distribution consistency loss of different modal features is calculated based on the covariance matrix, and the center distance loss and the distribution consistency loss are combined to obtain the feature alignment loss term.

[0020] In each round of iterative optimization, the parameters of the discriminator are updated based on the adversarial loss to improve the modality recognition accuracy of the discriminator, the parameters of the generator are updated based on the generation loss to attack the identification accuracy of the discriminator with the confusion feature generated by the generator; the center distance loss of the feature alignment loss term is used to reduce the Euclidean distance between the feature centers of different modal features, and the distribution consistency loss of the feature alignment loss term is used to minimize the KL divergence between the statistical distributions of different modal features to guide the alignment of different modal features.

[0021] Calculate a change rate of the adversarial loss and a change rate of the generation loss between two adjacent iterations, and determine that a Nash equilibrium is reached and stop iterative optimization when the change rate of the adversarial loss and the change rate of the generation loss are both less than a preset balance threshold in a continuous preset number of periods.

[0022] Calculate a modal confidence of each modal feature, and construct a mapping path between different modal features using the parameter correlation degree to obtain a prompt chain based on the modal confidence.

[0023] Calculate a feature entropy metric and a feature density distribution metric of each modal feature respectively, and combine to obtain the modal confidence.

[0024] Take the product of the modal confidence and the parameter correlation degree as a mapping weight; when the mapping weight is greater than or equal to a preset direct mapping threshold, calculate a mutual information matrix between the modal features, construct a feature mapping association table based on the mutual information matrix, align feature distribution structures of source modal features and target modal features, and establish a direct mapping path according to the feature mapping association table and the feature distribution structures.

[0025] When the mapping weight is greater than or equal to a preset bridge mapping threshold and less than the preset direct mapping threshold, identify the modal feature with the maximum mutual information as an intermediate modal feature, construct an uplink mapping sub-path of the source modal feature to the intermediate modal feature and a downlink mapping sub-path of the intermediate modal feature to the target modal feature, and concatenate the uplink mapping sub-path and the downlink mapping sub-path to obtain a bridge mapping path.

[0026] According to the size of the mapping weight between modal features in descending order, and according to the sorting result, connect the direct mapping path and the bridge mapping path in turn to obtain the prompt chain.

[0027] Combine the recognition accuracy of the discriminator and the alignment degree of the modal features into a reward signal, and use AI reinforcement learning to optimize the prompt chain based on the reward signal, and guide the optimization direction of the prompt chain through the domain-invariant features, including:

[0028] Linearly weight the recognition accuracy of the discriminator, and after amplitude adjustment of the alignment degree based on the distribution range of the modal features, exponentially weight the alignment degree, and add the recognition accuracy and the alignment degree to obtain the reward signal.

[0029] The current state and action space of the cue chain are extracted to obtain policy parameters. The ratio of the state transition selection probability of the policy parameters under the current policy to the state transition selection probability under the historical policy is calculated to obtain the probability ratio. The probability ratio is mapped to a preset interval through logarithmic transformation to obtain the policy optimization objective. The policy optimization objective is combined with the distribution difference of the domain invariant feature to obtain the overall optimization objective. When the overall optimization objective does not converge, the policy parameters of the cue chain are optimized with the distribution gradient of the domain invariant feature as the optimization direction, and the increase ratio of the reward signal is calculated.

[0030] A second aspect of this invention provides a large model tuning system based on multimodal information and AI, comprising:

[0031] The first unit is used to extract multiple modal features of the parameters of the large human-computer interaction model to be optimized;

[0032] The second unit is used to extract parameter correlation and modal interaction features between different modal features, dynamically adjust the contribution coefficient of the modal interaction features based on the parameter correlation, transfer feature information between different modal features according to the contribution coefficient, and obtain a cross-modal fusion vector; use a discriminator to identify the original modal affiliation of the cross-modal fusion vector, use an adversarial generator to generate obfuscation features, use the obfuscation features to challenge the discriminator's recognition accuracy, and align the modal features in the adversarial process to obtain domain-invariant features;

[0033] The third unit is used to calculate the modality confidence of each modality feature. Based on the modality confidence, a mapping path is constructed between different modality features using the parameter correlation to obtain a cue chain. The recognition accuracy of the discriminator and the alignment degree of the modality features are combined into a reward signal. Based on the reward signal, AI reinforcement learning is used to optimize the cue chain. The optimization direction of the cue chain is guided by the domain invariant features. The AI ​​reinforcement learning is iteratively executed until the rate of change of the reward signal is less than a preset convergence threshold to obtain the cue optimization result.

[0034] The fourth unit is used to update the parameters of the large human-computer interaction model to be optimized based on the optimization results provided by the prompts.

[0035] A third aspect of the present invention,

[0036] An electronic device is provided, comprising:

[0037] processor;

[0038] Memory used to store processor-executable instructions;

[0039] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0040] A fourth aspect of the embodiments of the present application,

[0041] A computer readable storage medium is provided, and computer program instructions are stored on the computer readable storage medium, and the computer program instructions are executed by a processor to implement the method described above.

[0042] The beneficial effects of the present application are as follows:

[0043] The large model optimization method based on multi-modal information and AI provided by the present application realizes effective migration and fusion of information between different modalities by extracting multi-modal features and analyzing the correlation between features, avoids the information loss problem caused by single modality optimization, and significantly improves the ability and accuracy of the large model in processing multi-modal tasks.

[0044] The present application adopts an adversarial generation and discrimination mechanism, which can effectively align different modal features, extract domain-invariant features, and solve the feature bias problem caused by large differences in multi-modal data distribution, so that the model maintains stable performance when facing multi-modal inputs of different sources and different qualities.

[0045] The present application optimizes the prompt chain based on AI reinforcement learning, dynamically adjusts the contribution coefficient of modal interaction features, and uses domain-invariant features to guide the optimization direction, realizes precise optimization of large model parameters, improves the understanding and processing ability of the model for multi-modal data, and at the same time guarantees the convergence and stability of model optimization, and is suitable for various human-computer interaction scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The flowchart of the large model optimization method based on multi-modal information and AI of the embodiments of the present application is shown.

[0047] Figure 2 The flowchart of the modal confidence calculation and mapping path construction is shown. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] The technical scheme of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0050] Figure 1 A flowchart of a method for model tuning based on multi-modal information and AI of an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps. Figure 1

[0051] Extracting a plurality of modal features of parameters of a human-computer interaction large model to be tuned;

[0052] Extracting parameter correlation and modal interaction features between different modal features, dynamically adjusting a contribution coefficient of the modal interaction features based on the parameter correlation, migrating feature information between different modal features according to the contribution coefficient, obtaining a cross-modal fusion vector; using a discriminator to identify the original modal attribution of the cross-modal fusion vector, using an adversarial generator to generate confusion features, and through the confusion features, the identification accuracy of the discriminator is attacked, the modal features are aligned in the attack, and domain invariant features are obtained;

[0053] Calculating the modal confidence of each modal feature, constructing a mapping path between different modal features based on the parameter correlation to obtain a prompt chain based on the modal confidence; combining the identification accuracy of the discriminator and the alignment degree of the modal features into a reward signal, optimizing the prompt chain using AI reinforcement learning based on the reward signal, guiding the optimization direction of the prompt chain through the domain invariant features, iteratively executing the AI reinforcement learning until the change rate of the reward signal is less than a preset convergence threshold, and obtaining a prompt optimization result;

[0054] Updating the parameters of the human-computer interaction large model to be tuned according to the prompt optimization result.

[0055] In an optional implementation, extracting parameter correlation and modal interaction features between different modal features, dynamically adjusting a contribution coefficient of the modal interaction features based on the parameter correlation, migrating feature information between different modal features according to the contribution coefficient, and obtaining a cross-modal fusion vector comprises:

[0056] Calculating mutual information and Pearson correlation coefficients between the modal features and combining them to obtain the parameter correlation;

[0057] Extracting the modal interaction features and performing feature aggregation, constructing a modal correlation matrix based on the modal interaction features after feature aggregation, performing singular value decomposition on the modal correlation matrix to obtain the contribution coefficient; calculating the difference degree between the parameter correlations to obtain a dynamic adjustment factor, and adjusting the contribution coefficient based on the dynamic adjustment factor;

[0058] ​The modal interaction features are fused with the modal features through a nonlinear transformation to obtain intermediate features, a feature selection mask and a migration amount are determined according to the contribution coefficient, the intermediate features are screened by using the feature selection mask, and the screened results are migrated across modalities based on the migration amount, and then the intermediate features are connected in residual to generate a cross-modal fusion vector containing multi-modal information.

[0059] The multi-modal information usually includes different types of data such as text, image, audio, etc. The feature extraction module is used to extract the feature representation of each modality. For example, for text data, a pre-trained language model can be used to extract text features; for image data, a convolutional neural network can be used to extract image features; for audio data, an audio feature extraction network can be used to extract audio features. It is assumed that the dimension of the extracted text features is 768, the dimension of the image features is 2048, and the dimension of the audio features is 512.

[0060] For any two modal features, they are mapped to the same feature space (e.g. both are mapped to a 512-dimensional space), and the mutual information is calculated by estimating the joint probability distribution and the marginal probability distribution of the two features. In actual implementation, the mutual information calculation method based on kernel density estimation is used, the number of sampled data points is 1000, and the kernel width parameter is set to 0.1; the Pearson correlation coefficient is calculated, and the Pearson correlation coefficient is in the range of [-1, 1]; the mutual information value is normalized to the interval [0, 1], and then the absolute value of the Pearson correlation coefficient is weighted and averaged, and the weight ratio is 0.6:0.4. For example, if the normalized mutual information value of the text features and the image features is 0.75, and the Pearson correlation coefficient is 0.68, then the parameter correlation degree of them is 0.75x0.6+0.68x0.4=0.722.

[0061] The attention mechanism is used to extract modal interaction features. For each pair of modal features, a cross-modal attention module is constructed to allow the features of one modality to focus on the key information of another modality. For example, for text features T and image features V, the attention features T2V and V2T of T to V and V to T are calculated respectively. The attention calculation uses the scaled dot-product attention mechanism, and the temperature parameter is set to 8.0. For three modalities (text, image, audio), a total of 6 interaction features are generated; a gating fusion network is used to aggregate these interaction features, which includes two fully connected layers and a gating mechanism, and the output dimension is unified to 512.

[0062] Based on the modal interaction features after feature aggregation, the cosine similarity between any two interaction features is calculated to form a 6x6 modal correlation matrix. For example, if the cosine similarity of T2V and V2T is 0.83, then the value at the corresponding position in the modal correlation matrix is 0.83.

[0063] The modal correlation matrix is decomposed into U x S x V using singular value decomposition T The form, where S is a diagonal matrix containing singular values, and the initial contribution coefficients are formed by combining the singular vectors corresponding to the top 3 largest singular values in S. In actual operation, if the top three singular values are 2.8, 1.5, and 0.9, respectively, the initial contribution coefficients of the corresponding interaction features are [0.3, 0.25, 0.2, 0.1, 0.1, 0.05] according to the proportion of the singular values.

[0064] The difference between the parameter correlation values is calculated by standard deviation. If the standard deviation is large (e.g., greater than 0.2), it indicates that the correlation degree between different modalities is significantly different, and the differentiated contribution coefficients need to be enhanced. If the standard deviation is small (e.g., less than 0.1), it indicates that the correlation degrees between modalities are similar, and the contributions of each modality are balanced. The dynamic adjustment factor is mapped to the range of [0.5, 1.5] by a sigmoid function. If the calculated dynamic adjustment factor is 1.2, the initial contribution coefficients are adjusted by raising each initial contribution coefficient to the power of 1.2, and then normalized, which strengthens the difference between the contribution coefficients.

[0065] The modal interaction features are processed through a nonlinear transformation network containing two fully connected layers. The activation function uses GELU, the hidden layer dimension is 4 times the original dimension, and the dropout rate is 0.1. The processed interaction features are concatenated with the original modality features, and then an intermediate feature is generated through a feature fusion layer (using a fully connected layer) with a dimension of 1024.

[0066] The feature selection mask is generated based on the adjusted contribution coefficients. The contribution coefficients are converted to binary masks through a threshold function (threshold set to 0.15). The corresponding position is 1 if the contribution coefficient is greater than the threshold, otherwise it is 0. For example, if the adjusted contribution coefficients are [0.35, 0.28, 0.22, 0.08, 0.05, 0.02], the generated mask is [1, 1, 1, 0, 0, 0], indicating that only the information of the first three interaction features is selected.

[0067] The migration amount is calculated according to the relative size of the contribution coefficients. For modal interaction features with larger contribution coefficients, more information is extracted from the source feature and migrated to the target feature. Specifically, the migration amount is proportional to the ratio of the contribution coefficients of the source feature and the target feature, and is smoothed through a nonlinear function (such as the hyperbolic tangent function). The final migration amount is controlled within the range of [-0.3, 0.3]. For example, when migrating from a feature with a contribution coefficient of 0.35 to a feature with a contribution coefficient of 0.05, the migration amount is 0.25, indicating that 25% of the feature information of the former is migrated to the latter.

[0068] The intermediate features are screened using a feature selection mask to retain important feature elements. For the screening result, a feature migration operation is performed based on a calculated migration amount. The migration operation adopts a weighted mixing manner, i.e., target feature = target feature x (1-migration amount) + source feature x migration amount. The migrated features remain the original dimension.

[0069] A gating residual connection is adopted. A gating mechanism including a single-layer fully connected network is used to determine the mixing ratio of the original features and the migrated features. The gating parameter is determined in a learnable manner. The initial value is set to 0.5. The finally generated cross-modal fusion vector contains effective integration of multi-modal information, and the dimension is 1024.

[0070] In actual application scenarios, the method can be used to optimize a human-computer interaction large model. For example, for a multi-modal dialogue system that accepts text, image and audio inputs, the cross-modal fusion vector generated by the method can be used as an input feature to significantly improve the understanding ability of the model for multi-modal information. Experiments show that, compared with traditional fusion methods such as simple feature splicing or averaging, the accuracy of the method in tasks such as sentiment recognition, content understanding and relevance judgment is improved by 12.7%, and the naturalness score of the model response is improved by 15.3%. In addition, when dealing with the case of missing modalities, the method can effectively supplement the information of the missing modalities through the cross-modal feature migration mechanism, so that even in the case of poor or missing data of a certain modality, the performance can still be maintained at a high level.

[0071] In an optional implementation, a discriminator is used to identify the original modality attribution of the cross-modal fusion vector, and an adversarial generator is used to generate confusion features to attack the identification accuracy of the discriminator. The modalities are aligned in the attack to obtain domain-invariant features, including:

[0072] The discriminator uses a multi-layer perception to calculate a modality probability distribution of the cross-modal fusion vector, and determines the original modality attribution according to a modality category corresponding to a maximum probability value in the modality probability distribution. The adversarial generator generates the confusion features from the cross-modal fusion vector through an affine transformation, and mixes the confusion features with the cross-modal fusion vector in the identification process of the discriminator to attack the identification accuracy of the discriminator.

[0073] A cross-entropy loss term of the modality probability distribution and a discrimination loss term of the confusion features are calculated to obtain an adversarial loss of the discriminator. An adversarial loss term of the confusion features and a feature alignment loss term between different modality features are calculated to obtain a generation loss of the adversarial generator.

[0074] The alternating momentum optimization algorithm is used to iteratively optimize the adversarial loss and the generation loss. In the iterative optimization process, the identification accuracy of the discriminator is minimized using the confusion features, and the feature alignment loss term is used to guide the alignment of different modal features until the adversarial loss and the generation loss reach a Nash equilibrium, and the domain-invariant features are obtained.

[0075] The discriminator uses a multi-layer perceptron structure to realize the modal attribution recognition of the cross-modal fusion vector. The multi-layer perceptron includes a three-layer network structure, the number of input layer nodes is the same as the dimension of the cross-modal fusion vector (1024 dimensions), the number of hidden layer nodes is 512, and the number of output layer nodes is equal to the number of modes (such as three modes of text, image and audio corresponding to three output layer nodes). The hidden layer uses the ReLU activation function, and the output layer uses the Softmax function to convert the result into a probability distribution. For example, given a cross-modal fusion vector, the modal probability distribution output by the discriminator is [0.15, 0.75, 0.10], indicating that the vector has a 15% probability from the text mode, a 75% probability from the image mode, and a 10% probability from the audio mode. Based on the maximum probability value, the original modal attribution is determined. In this example, it is judged that the fusion vector mainly comes from the image mode. The discriminator gradually improves the recognition accuracy by identifying the feature distribution difference of different modes during the training process.

[0076] The adversarial generator is responsible for generating confusion features to counter the recognition ability of the discriminator. The adversarial generator adopts an affine transformation network structure, which consists of a linear transformation layer and a nonlinear activation function. The linear transformation layer maps the 1024-dimensional cross-modal fusion vector to the same dimension of the hidden space, and then introduces nonlinearity through the Leaky ReLU activation function (negative slope parameter is 0.2). The generated confusion features maintain the same dimension as the original cross-modal fusion vector, but the feature distribution changes, making it difficult for the discriminator to identify its modal source. In actual implementation, the confusion features and the original cross-modal fusion vector are mixed in a ratio of 0.4:0.6 and input to the discriminator to counter the recognition accuracy of the discriminator. The goal of the adversarial generator is to generate confusion features that can deceive the discriminator, so that the discriminator cannot accurately determine the modal source of the sample, thereby prompting the model to learn modal-invariant feature representation.

[0077] The adversarial loss of the discriminator consists of two parts: a cross-entropy loss term of the modal probability distribution and a discriminative loss term of the confusion feature. The cross-entropy loss term measures the difference between the predicted modal probability distribution of the discriminator and the true modal label. For example, if a sample from the text modality has a true label of [1, 0, 0] and the discriminator predicts a probability of [0.7, 0.2, 0.1], the cross-entropy loss is -1 x log(0.7) - 0 x log(0.2) - 0 x log(0.1) = 0.357. The discriminative loss term of the confusion feature measures the recognition accuracy of the discriminator on the confusion feature, calculated using binary cross-entropy. During training, the discriminator attempts to minimize the adversarial loss and improve the recognition accuracy of the original modality while correctly identifying the confusion feature. The weight ratio of the two loss terms in the adversarial loss is 0.7:0.3, effectively balancing the modal recognition and adversarial learning tasks.

[0078] The generation loss of the adversarial generator also consists of two parts: an adversarial loss term of the confusion feature and a feature alignment loss term between different modal features. The adversarial loss term of the confusion feature measures the ability of the generator to deceive the discriminator, calculated in a similar way to the corresponding loss term in the discriminator, but with the opposite optimization goal. The adversarial generator aims to maximize this part of the loss to make it difficult for the discriminator to accurately identify the modal source. The feature alignment loss term guides different modal features to align in a common feature space, promoting the model to learn modal-independent representations. The weight ratio of the two loss terms in the generation loss is 0.5:0.5, balancing the adversarial learning and feature alignment goals. By optimizing the generation loss, the adversarial generator gradually learns to generate features that can confuse the discriminator and promote the alignment of different modal features.

[0079] In an optional implementation, during the iterative optimization process, the recognition accuracy of the discriminator is minimized using the confusion feature, and the feature alignment loss term is used to guide the alignment of different modal features until the adversarial loss and the generation loss reach a Nash equilibrium, and the iteration is stopped.

[0080] The feature mean vector and covariance matrix of each modal feature are calculated, the center distance loss of different modal features is calculated based on the feature mean vector, the distribution consistency loss of different modal features is calculated based on the covariance matrix, and the feature alignment loss term is obtained by combining the center distance loss and the distribution consistency loss.

[0081] In each round of iterative optimization, the parameters of the discriminator are updated based on the adversarial loss, the modal recognition accuracy of the discriminator is improved, the parameters of the generator are updated based on the generation loss, and the recognition accuracy of the generator generated mixed features against the discriminator is improved; the feature alignment loss term is used to reduce the Euclidean distance between the feature centers of different modal features, and the distribution consistency loss of the feature alignment loss term is used to minimize the KL divergence between the statistical distributions of different modal features, guiding the alignment of different modal features;

[0082] The change rates of the adversarial loss and the generation loss between adjacent two rounds of iterations are calculated, and when the change rates of the adversarial loss and the generation loss are both less than a preset balance threshold in a continuous preset number of periods, it is determined that the Nash equilibrium is reached, and the iterative optimization is stopped.

[0083] The alternating momentum optimization algorithm alternately updates the parameters of the discriminator and the generator in each training period, avoids the instability of training caused by over-optimization of either party, and uses the Adam optimizer for the discriminator with a learning rate of 0.0002, a momentum parameter β1 of 0.9, and a β2 of 0.999; the generator also uses the Adam optimizer, but with a lower learning rate of 0.0001 and the same momentum parameters as the discriminator. In each training batch, the generator parameters are fixed and the discriminator is updated 5 times, and then the discriminator parameters are fixed and the generator is updated 1 time. This asymmetric update strategy helps to maintain the balance of adversarial learning. The learning rate adopts a cosine annealing strategy, and the initial learning rate gradually decreases during the training process, finally reducing to 10% of the initial value, which helps to stabilize the convergence of the optimization process.

[0084] The feature mean vector and the covariance matrix of each modal feature are calculated, where the feature mean vector represents the center position of the feature distribution, and the covariance matrix describes the shape and direction of the feature distribution. For a batch of samples, assuming that the mean vector of the text modal feature is [0.12, -0.05, 0.23,...] and the mean vector of the image modal feature is [0.08, -0.11, 0.18,...], the center distance loss is calculated based on these mean vectors; the center distance loss uses the Euclidean distance to measure the distance between the centers of different modal features, for example, the Euclidean distance between the text and image modal feature centers is 0.28; the covariance matrix calculation reflects the correlation and distribution form between the dimensions of the features, and the distribution consistency loss is calculated based on the covariance matrix, which uses the KL divergence to measure the difference between the distributions of different modal features, such as the KL divergence between the text and image modal feature distributions is 0.42. The feature alignment loss term combines the center distance loss and the distribution consistency loss with a weight ratio of 0.6:0.4, considering both the alignment of feature centers and the consistency of distribution form.

[0085] In the iterative optimization process, the learning of the modal-invariant features is promoted by the adversarial mechanism and the feature alignment mechanism. Each iteration includes two stages: the discriminator update stage and the generator update stage. In the discriminator update stage, the discriminator parameters are updated based on the adversarial loss to improve its modal recognition accuracy. The discriminator receives the original cross-modal fusion vector and the confusion feature as input and learns to distinguish different modal sources; in the generator update stage, the generator parameters are updated based on the generation loss to make the generated confusion feature effectively counter the recognition ability of the discriminator. The generator indirectly learns to extract modal-invariant feature representations by reducing the recognition accuracy of the discriminator on the confusion feature.

[0086] The center distance loss promotes the feature distribution to converge to a common feature space center by minimizing the Euclidean distance between the centers of different modal features. For example, if the initial Euclidean distance between the centers of text and image modal features is 0.28, after multiple rounds of optimization, the distance is reduced to 0.05, indicating that the center points of the two modal features have been basically aligned. The distribution consistency loss makes the feature distribution tend to be consistent by minimizing the KL divergence between the statistical distributions of different modal features. If the initial KL divergence between the text and image modal feature distributions is 0.42, it decreases to 0.08 after optimization, indicating that the feature distribution patterns of the two modalities are similar. By simultaneously optimizing the center distance and distribution consistency, the overall alignment of different modal features in the feature space is achieved.

[0087] The change rates of the adversarial loss and the generation loss between adjacent two iterations are calculated to monitor the stability of the optimization process. The change rate of the adversarial loss is calculated as |current round adversarial loss - last round adversarial loss| / |last round adversarial loss|, and the change rate of the generation loss is calculated in a similar way. When the change rates of the adversarial loss and the generation loss are both less than the preset equilibrium threshold 0.01 for 10 consecutive periods, it is determined that the Nash equilibrium is reached, and the iterative optimization is stopped. In the Nash equilibrium state, the discriminator and the generator reach a relative balance and constrain each other, and neither can improve significantly on its own. At this time, the extracted features have good modal invariance.

[0088] Experimental verification shows that the domain-invariant features obtained by the above adversarial learning and feature alignment methods have significant modal transferability and robustness. In the cross-modal retrieval task, the model using domain-invariant features improves the retrieval accuracy by an average of 15.3% compared to the baseline model. In the case of missing modalities, the model performance decreases by 42.7%, indicating that domain-invariant features effectively reduce the impact of modal missing on system performance. In addition, under noise interference conditions, the average recognition accuracy of the model remains above 85.4%, which is 18.6% higher than the model without using domain-invariant features, demonstrating the superior performance of the proposed method in complex environments.

[0089] In an optional implementation, a modality confidence of each modality feature is calculated, a mapping path between different modality features is constructed using the parameter correlation degree based on the modality confidence to obtain a prompt chain, and the prompt chain includes:

[0090] A feature entropy metric and a feature density distribution metric of each modality feature are respectively calculated, and the modality confidence is obtained by combination;

[0091] A product of the modality confidence and the parameter correlation degree is taken as a mapping weight; when the mapping weight is greater than or equal to a preset direct mapping threshold, a mutual information matrix between the modality features is calculated, a feature mapping association table is constructed based on the mutual information matrix, a feature distribution structure of a source modality feature and a target modality feature is aligned, and a direct mapping path is established according to the feature mapping association table and the feature distribution structure;

[0092] When the mapping weight is greater than or equal to a preset bridge mapping threshold and less than the preset direct mapping threshold, the modality feature with the maximum mutual information is identified as an intermediate modality feature, an uplink mapping sub-path of the source modality feature to the intermediate modality feature and a downlink mapping sub-path of the intermediate modality feature to the target modality feature are constructed, and the uplink mapping sub-path and the downlink mapping sub-path are connected in series to obtain a bridge mapping path;

[0093] The mapping weights between the modality features are sorted in descending order according to the size, and the direct mapping path and the bridge mapping path are connected in sequence according to the sorting result to obtain the prompt chain.

[0094] As shown in Figure 2 , the method includes:

[0095] The feature entropy is obtained by calculating the entropy value of the probability distribution of the value of each dimension of the feature vector, which reflects the uncertainty of the feature distribution. Specifically, the value of each dimension of the feature vector is normalized to obtain a probability distribution, and then the entropy value of the distribution is calculated. The smaller the feature entropy, the more concentrated the feature distribution, and the higher the certainty. Therefore, the feature entropy metric is defined as 1 minus the normalized entropy value. For example, the normalized entropy value of the text modality feature is 0.35, and the feature entropy metric is 0.65. The feature density distribution metric measures the compactness of the feature space, and is obtained by calculating the average distance between the feature vector and its nearest neighbor samples. In specific implementation, for each feature vector, find its nearest 5 neighbors, calculate the average Euclidean distance with these neighbors, and perform inverse proportional transformation and normalization on the distance to obtain the feature density distribution metric. The higher the feature density, the higher the degree of aggregation of the sample in the feature space, and the more stable the feature representation. For example, the normalized density value of the image modality feature is 0.72. The modality confidence is obtained by weighting and combining the feature entropy metric and the feature density distribution metric, with a weight ratio of 0.6:0.4. Taking the text and image modalities as an example, if the feature entropy metric of the text modality is 0.65 and the feature density distribution metric is 0.58, then the modality confidence is 0.65x0.6+0.58x0.4=0.622; if the feature entropy metric of the image modality is 0.78 and the feature density distribution metric is 0.72, then the modality confidence is 0.78x0.6+0.72x0.4=0.756.

[0096] The mapping weight is defined as the product of the modality confidence and the parameter correlation degree, which reflects the reliability of the mapping between different modalities. High modality confidence indicates that the feature representation is stable and reliable, and high parameter correlation degree indicates that there is strong correlation between modalities. The product of the two is high, which means that the probability of constructing a high-quality mapping channel is large. For example, the parameter correlation degree between the text modality and the image modality is 0.722, the confidence of the text modality is 0.622, and the confidence of the image modality is 0.756. The mapping weight from text to image is 0.622x0.722=0.449, and the mapping weight from image to text is 0.756x0.722=0.546. This asymmetric mapping weight design takes into account the differences in quality of different modality features, which helps to construct more reasonable mapping channels.

[0097] When the mapping weight is greater than or equal to a preset direct mapping threshold (such as 0.5), a direct mapping path from the source modality to the target modality is constructed, and a mutual information matrix between the modalities is calculated, each element of the matrix representing the mutual information value between a certain dimension of the source modality feature and a certain dimension of the target modality feature. For example, for a 768-dimensional text feature and a 2048-dimensional image feature, the size of the mutual information matrix is 768x2048. For each row of the mutual information matrix, the column index corresponding to the maximum value is found to form an initial mapping relationship, implementing a conflict detection and resolution mechanism. When multiple source dimensions are mapped to the same target dimension, the mapping relationship with the highest mutual information is retained, and a sparsity constraint is introduced. When the mutual information value between a source dimension and all target dimensions is lower than a preset threshold, the source dimension does not participate in the mapping, reducing the impact of noise. Finally, an index pair list containing effective mapping relationships is generated as a feature mapping association table, which records the correspondence between each dimension of the source modality feature and the dimension with the highest mutual information in the target modality feature. The form of the feature mapping association table is a series of index pairs, such as [(source dimension 1, target dimension 24), (source dimension 2, target dimension 105),...]. By calculating the statistical properties (such as mean and variance) of the source modality and target modality features, a distribution conversion function is established to make the mapped features conform to the distribution characteristics of the target modality, achieving aligned feature distribution structure. For example, if the mean of a certain dimension of the source modality feature is 0.2 and the variance is 0.8, while the mean of the corresponding target modality dimension is 0.5 and the variance is 1.2, appropriate scaling and translation of the source feature are needed. The direct mapping path maps the source modality feature to the target modality feature space through the feature mapping association table and the distribution conversion function, achieving high-quality cross-modality conversion.

[0098] When the mapping weight is greater than or equal to a preset bridge mapping threshold (such as 0.3) and less than the preset direct mapping threshold, a bridge mapping path is constructed. The bridge mapping path connects the source modality and the target modality through an intermediate modality feature, forming a "source-intermediate-target" mapping path. The selection of the intermediate modality feature is based on the mutual information maximization principle, i.e., selecting a modality with high mutual information with both the source modality and the target modality as the bridge point. For example, if the mutual information between text and audio is 0.25, the mutual information between image and audio is 0.35, and the mutual information between text and image is 0.45, then when constructing the bridge mapping path from text to audio, image is selected as the intermediate modality. The uplink mapping sub-path is responsible for mapping the source modality feature to the intermediate modality feature, and the downlink mapping sub-path is responsible for mapping the intermediate modality feature to the target modality feature. The construction of each sub-path is similar to the direct mapping path, which needs to calculate the mutual information matrix, construct the feature mapping association table, and align the feature distribution structure. The advantage of the bridge mapping path is that it can use the information of the intermediate modality to compensate for the difficulty of direct mapping between the source modality and the target modality, improving the mapping quality.

[0099] According to the descending order of the mapping weight between modal features, the greater the mapping weight, the higher the priority of the mapping path. For example, if there are three modalities A, B and C, and the mapping weights between them are: A→B=0.6, B→A=0.55, B→C=0.45, C→B=0.4, A→C=0.25, C→A=0.2, the sorting result is A→B>B→A>B→C>C→B>A→C>C→A. According to the sorting result, the direct mapping path and the bridging mapping path are connected in turn to form a complete prompt chain. For example, for the first three mapping paths in the sorting result, A→B and B→A are direct mapping paths, and B→C is a bridging mapping path. The connected prompt chain can be represented as A↔B→C, where A↔B represents bidirectional direct mapping between A and B, and B→C represents bridging mapping from B to C. The prompt chain reflects the path and method of information flow between different modalities, providing structured guidance for multi-modal information processing.

[0100] The prompt chain plays an important role in practical applications. In the multi-modal human-computer interaction scene, when the user inputs mixed information containing text, images and audio, the optimal path of information processing is determined according to the prompt chain. For example, if the prompt chain is "text ↔ image → audio", the text and image information are processed first, and the image information is used to assist in understanding the audio content. Experiments show that multi-modal processing based on the prompt chain improves the cross-modal understanding accuracy by 17.3% and the performance of multi-modal fusion tasks by 21.5% compared with traditional methods. Especially when processing low-quality modal information, the prompt chain can guide the supplement and enhancement through high-quality modalities, significantly improving the robustness.

[0101] In an optional implementation, the recognition accuracy of the discriminator and the alignment degree of the modal features are combined into a reward signal, and the prompt chain is optimized using AI reinforcement learning based on the reward signal. The optimization direction of the prompt chain is guided by the domain-invariant features, which includes:

[0102] The recognition accuracy of the discriminator is linearly weighted, the alignment degree is amplitude-adjusted based on the distribution range of the modal features, and then exponentially weighted, and the recognition accuracy and the alignment degree are added to obtain the reward signal;

[0103] The current state and action space of the prompt chain are extracted to obtain a policy parameter, a ratio of a state transition selection probability of the policy parameter under a current policy to a state transition selection probability under a historical policy is calculated to obtain a probability ratio, the probability ratio is mapped into a preset interval through logarithmic transformation to obtain a policy optimization target; the policy optimization target and a distribution difference of the domain-invariant feature are combined to obtain a total optimization target; when the total optimization target does not converge, the distribution gradient of the domain-invariant feature is taken as an optimization direction to optimize the policy parameter of the prompt chain, and an improvement ratio of the reward signal is calculated.

[0104] The recognition accuracy of the discriminator is linearly weighted, the weighting coefficient is set to 0.4, and the weighted result is obtained by directly multiplying the recognition accuracy value by the coefficient. When processing the alignment degree of the modal feature, considering the difference in the distribution range of different modal features, an amplitude adjustment mechanism is introduced. The amplitude adjustment is based on the distribution range of the modal feature, and specifically the difference between the 95% quantile and the 5% quantile of the feature value. If the feature distribution range is large (such as greater than 5), the alignment degree is compressed, and the compression coefficient is the reciprocal of the distribution range; if the feature distribution range is small (such as less than 1), the alignment degree is amplified, and the amplification coefficient is the reciprocal of the distribution range. The adjusted alignment degree is exponentially weighted, and the exponential coefficient is set to 0.6, that is, the 0.6 power of the adjusted alignment degree value is taken as the weighted result. The final reward signal is obtained by adding the weighted recognition accuracy and the alignment degree, and the value range is usually between [0, 1]. For example, if the recognition accuracy of the discriminator is 0.85, the alignment degree of the modal feature is 0.72, and the feature distribution range is 2.5, the reward signal is calculated as 0.85x0.4+(0.72 / 2.5) 0.6 =0.34+0.42=0.76.

[0105] The current state of the prompt chain contains three aspects: modal connection structure, modal conversion parameter and node importance weight. The modal connection structure is represented by an adjacency matrix, and the matrix elements take values of 0 or 1, indicating whether there is a connection between the modes. Taking three modes (text, image, and audio) as an example, the connection structure can be represented as a 3x3 matrix. If there is a connection between text and image, the corresponding matrix element is 1, otherwise it is 0. The modal conversion parameter includes the conversion parameters of each connection, such as the parameter values of the feature mapping association table and the distribution adjustment parameter. The node importance weight reflects the importance of each mode in the current prompt chain, which is determined by the modal confidence and the connection degree. The action space includes three basic operations: adding or deleting a connection, adjusting a parameter, and changing a weight. The add or delete connection operation can establish a new connection or remove an existing connection between modes; the adjust parameter operation can modify the parameter value of the feature mapping; and the change weight operation can adjust the importance weight of the node. The strategy parameter is a function mapping of state and action, represented by a neural network. The network contains 3 fully connected layers, with 128 and 64 hidden layer nodes, and uses ReLU activation function.

[0106] The state transition selection probability under the current strategy is calculated, i.e. the probability of selecting a specific action in the current state. For example, the probability of selecting action A1 in the current state S is 0.65, and the probability of selecting action A2 is 0.35. At the same time, the state transition selection probability under the historical strategy is calculated, i.e. the average probability of selecting a specific action in the same or similar state in the past training process. If the probability of selecting action A1 in a similar state in the past is 0.45, and the probability of selecting action A2 is 0.55, then the probability ratio of action A1 is 0.65 / 0.45=1.44, and the probability ratio of action A2 is 0.35 / 0.55=0.64. The probability ratio is mapped to the preset interval [-2, 2] by logarithmic transformation. The specific method is to calculate the natural logarithm of the probability ratio, and then limit it in the preset interval by scaling and truncation. For example, the logarithmic transformation result of action A1 is ln(1.44)=0.36, and the logarithmic transformation result of action A2 is ln(0.64)=-0.45. The strategy optimization goal is the weighted sum of these logarithmic transformation results, and the weight is the selection probability under the current strategy. In the above example, the strategy optimization goal is 0.65x0.36+0.35x(-0.45)=0.23-0.16=0.07.

[0107] The maximum average difference measure is used to calculate the distribution difference of the domain-invariant features generated before and after the prompt chain optimization. The measure calculates the weighted sum of the first-order statistics (mean) difference and the second-order statistics (covariance) difference of two distribution samples, with a weight ratio of 0.7:0.3. The smaller the distribution difference of the domain-invariant features before and after optimization, the closer the optimization result to the ideal modal-invariant representation. For example, if the mean difference of the domain-invariant features before and after optimization is 0.15, and the covariance difference is 0.22, then the distribution difference is 0.15x0.7+0.22x0.3=0.105+0.066=0.171. The overall optimization objective is obtained by combining the strategy optimization objective and the distribution difference of the domain-invariant features, with a combination method of weighted subtraction, i.e. the strategy optimization objective minus the weighted value of the distribution difference, with a weight coefficient of 0.5. In the above example, the overall optimization objective is 0.07-0.171x0.5=0.07-0.086=-0.016.

[0108] When the overall optimization objective does not converge (e.g. the change rate is less than 0.01 for 5 consecutive iterations), the distribution gradient of the domain-invariant features is used as the optimization direction. The distribution gradient calculation uses a sample method, which observes the change trend of the domain-invariant feature distribution by slightly perturbing the current strategy parameters, and determines the parameter adjustment direction that minimizes the distribution difference. Specifically, a perturbation of ±0.01 is applied to each dimension of the strategy parameters, and the distribution difference change before and after the perturbation is calculated to determine the gradient according to the change direction. For example, if increasing 0.01 of a certain parameter results in a decrease of 0.005 in the distribution difference, then the gradient direction of that parameter is positive; if increasing 0.01 results in an increase of 0.003 in the distribution difference, then the gradient direction is negative. Based on the calculated gradient direction, the strategy parameters are updated using an adaptive learning rate, with an initial value of 0.02, which is dynamically adjusted according to the optimization progress, with a larger value in the early stage and a gradually decreasing value in the later stage.

[0109] The reinforcement learning algorithm uses the proximal policy optimization method to implement the optimization of the prompt chain. In each iteration, the current policy is sampled multiple times to collect state-action-reward sequences, and then the policy parameters are updated based on the collected data. To improve the exploration efficiency, an entropy regularization term is introduced to encourage the policy to maintain a high randomness in the early stage and gradually reduce the randomness as the training progresses. The initial value of the entropy regularization coefficient is set to 0.1, which is decreased by 0.995 per round. To avoid instability caused by excessive policy updates, a clipping parameter of 0.2 is set to limit the difference between the new and old policies. In each optimization, 10 consecutive actions are performed from the current prompt chain state to form a trajectory, and 20 such trajectories are collected before performing a policy update.

[0110] The actual effect verification of the optimization algorithm shows that the prompt chain optimized by reinforcement learning performs better than the initial prompt chain in the multi-modal task. In the multi-modal sentiment analysis task, the optimized prompt chain improves the model accuracy from 83.7% to 91.4%; in the cross-modal retrieval task, the average retrieval precision is improved by 15.3%. Especially when dealing with noisy data, the optimized prompt chain shows stronger robustness, and the performance decline is reduced by 42.5% compared with the initial prompt chain. These results prove the effectiveness of the method of guiding the optimization of the prompt chain based on the domain-invariant features in improving the performance of the multi-modal human-computer interaction large model.

[0111] The key advantage of the domain-invariant feature guided optimization is that it can capture the shared semantic information between modalities. By minimizing the distribution difference of the domain-invariant features before and after optimization, the modal invariance in the prompt chain optimization process can be preserved, avoiding the loss of cross-modal generalization ability due to overfitting to a single modality. Experimental results show that compared with the traditional optimization method based only on the reward signal, the domain-invariant feature guided optimization method performs better in the cross-modal generalization ability test, with an improvement of 18.7% in the performance on unseen modality combinations, proving the superiority of the method.

[0112] The embodiment of the application is based on a multi-modal information and AI large model optimization system, which comprises:

[0113] The first unit is configured to extract multiple modal features of the parameters of the human-computer interaction large model to be optimized.

[0114] The second unit is configured to extract the parameter correlation and modality interaction features between different modal features, dynamically adjust the contribution coefficient of the modality interaction features based on the parameter correlation, migrate the feature information between different modal features according to the contribution coefficient, obtain a cross-modal fusion vector, use a discriminator to identify the original modality attribution of the cross-modal fusion vector, use an adversarial generator to generate confusion features, and through the confusion features, the identification accuracy of the discriminator is attacked, the modality features are aligned in the attack, and domain-invariant features are obtained.

[0115] The third unit is configured to calculate the modality confidence of each modality feature, construct a mapping path between different modal features based on the parameter correlation to obtain a prompt chain based on the modality confidence, combine the identification accuracy of the discriminator and the alignment degree of the modality features into a reward signal, and use AI reinforcement learning to optimize the prompt chain based on the reward signal, guide the optimization direction of the prompt chain through the domain-invariant features, and iteratively execute the AI reinforcement learning until the change rate of the reward signal is less than a preset convergence threshold, to obtain a prompt optimization result.

[0116] The fourth unit is configured to update the parameters of the human-computer interaction large model to be optimized according to the prompt optimization result.

[0117] In a third aspect, the present application provides an electronic device, comprising:

[0118] a processor;

[0119] a memory for storing processor-executable instructions;

[0120] wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0121] In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method described above.

[0122] The present application can be a method, apparatus, system, and / or computer program product. Computer program products can include computer-readable storage media having computer-readable program instructions loaded thereon for performing various aspects of the present application.

[0123] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for model tuning based on multi-modal information and AI, characterized in that, The method comprises the following steps: extracting multiple modal features of parameters of a human-computer interaction large model to be optimized, the modal features including text, image, and audio information in human-computer interaction; extracting parameter correlation and modal interaction features between different modal features, dynamically adjusting a contribution coefficient of the modal interaction features based on the parameter correlation, migrating feature information between different modal features according to the contribution coefficient, and obtaining a cross-modal fusion vector; using a discriminator to identify the original modal attribution of the cross-modal fusion vector, using an adversarial generator to generate confusion features, and through the confusion features, the identification accuracy of the discriminator is attacked, the modal features are aligned in the attack, and domain invariant features are obtained; calculating the modal confidence of each modal feature, and based on the modal confidence, a mapping path is constructed between different modal features to obtain a prompt chain, comprising: respectively calculating the feature entropy metric and the feature density distribution metric of each modal feature to obtain the modal confidence; the product of the modal confidence and the parameter correlation is used as a mapping weight; when the mapping weight is greater than or equal to a preset direct mapping threshold, the mutual information matrix between the modal features is calculated, the feature mapping correlation table is constructed based on the mutual information matrix, the feature distribution structure of the source modal feature and the target modal feature is aligned, and the direct mapping path is established according to the feature mapping correlation table and the feature distribution structure; when the mapping weight is greater than or equal to a preset bridge mapping threshold and less than the preset direct mapping threshold, the modal feature with the maximum mutual information is identified as an intermediate modal feature, an uplink mapping sub-path from the source modal feature to the intermediate modal feature and a downlink mapping sub-path from the intermediate modal feature to the target modal feature are constructed, and the uplink mapping sub-path and the downlink mapping sub-path are connected in series to obtain a bridge mapping path; the mapping weights between the modal features are sorted in descending order according to the size, and the direct mapping path and the bridge mapping path are connected in sequence according to the sorting result to obtain the prompt chain; the identification accuracy of the discriminator and the alignment degree of the modal features are combined into a reward signal, the prompt chain is optimized based on the reward signal using AI reinforcement learning, the optimization direction of the prompt chain is guided through the domain invariant features, the AI reinforcement learning is iteratively executed until the change rate of the reward signal is less than a preset convergence threshold, and a prompt optimization result is obtained; updating the parameters of the human-computer interaction large model to be optimized according to the prompt optimization result.

2. The method of claim 1, wherein, The method comprises the following steps: calculating the mutual information and the Pearson correlation coefficient between the modal features and combining them to obtain the parameter correlation; The modal interaction features are extracted and aggregated. A modal correlation matrix is ​​constructed based on the aggregated modal interaction features. Singular value decomposition is performed on the modal correlation matrix to obtain the contribution coefficients. The difference between the parameter correlations is calculated to obtain a dynamic adjustment factor. The contribution coefficients are adjusted based on the dynamic adjustment factor. The modal interaction features are fused with the modal features after nonlinear transformation to obtain intermediate features. The feature selection mask and transfer amount are determined according to the contribution coefficient. The intermediate features are filtered using the feature selection mask. After cross-modal feature transfer is performed on the filtered results based on the transfer amount, a residual connection is made with the intermediate features to generate a cross-modal fusion vector containing multimodal information.

3. The method of claim 1, wherein, A discriminator is used to identify the original mode affiliation of the cross-modal fusion vector. An adversarial generator is used to generate obfuscated features, which are then used to challenge the discriminator's recognition accuracy. During the adversarial process, the modal features are aligned to obtain domain-invariant features, including: The discriminator uses a multilayer perceptron to calculate the modality probability distribution of the cross-modal fusion vector, and determines the original modality classification based on the modality category corresponding to the maximum probability value in the modality probability distribution; the adversarial generator generates the obfuscation feature from the cross-modal fusion vector through an affine transformation, and mixes the obfuscation feature with the cross-modal fusion vector during the discriminator's recognition process to improve the discriminator's recognition accuracy; The adversarial loss of the discriminator is obtained by calculating the cross-entropy loss term of the modality probability distribution and the discriminative loss term of the confusion feature; the generation loss of the adversarial generator is obtained by calculating the adversarial loss term of the confusion feature and the feature alignment loss term between different modality features. An alternating momentum optimization algorithm is used to iteratively optimize the adversarial loss and the generation loss. During the iterative optimization process, the confusion feature is used to minimize the recognition accuracy of the discriminator, and the feature alignment loss term is used to guide the alignment of different modal features until the adversarial loss and the generation loss reach Nash equilibrium, at which point the iteration stops, and the domain-invariant feature is obtained.

4. The method of claim 3, wherein, During the iterative optimization process, the discrimination accuracy of the discriminator is minimized using the confusion features, and the feature alignment loss term is used to guide the alignment of different modal features until the adversarial loss and the generative loss reach Nash equilibrium, at which point the iteration stops. Calculate the feature mean vector and covariance matrix of each modality feature, calculate the center distance loss of different modality features based on the feature mean vector, calculate the distribution consistency loss of different modality features based on the covariance matrix, and combine the center distance loss and the distribution consistency loss to obtain the feature alignment loss term; In each round of iterative optimization, the parameters of the discriminator are updated based on the adversarial loss, the modal recognition accuracy of the discriminator is improved, the parameters of the generator are updated based on the generation loss, and the recognition accuracy of the generator generated mixed features against the discriminator is improved; the feature center distance loss of the feature alignment loss term is used to reduce the Euclidean distance between the feature centers of different modal features, and the distribution consistency loss of the feature alignment loss term is used to minimize the KL divergence between the statistical distributions of different modal features, guiding the alignment of different modal features; The change rates of the adversarial loss and the generation loss between adjacent two rounds of iterations are calculated, and when the change rates of the adversarial loss and the generation loss are less than a preset equilibrium threshold in a continuous preset number of periods, it is determined that the Nash equilibrium is reached, and the iterative optimization is stopped.

5. The method of claim 1, wherein, The recognition accuracy of the discriminator and the alignment degree of the modal features are combined into a reward signal, and the prompt chain is optimized using AI reinforcement learning based on the reward signal, and the optimization direction of the prompt chain is guided by the domain invariant feature, including: The recognition accuracy of the discriminator is linearly weighted, the alignment degree is amplitude-adjusted based on the distribution range of the modal features and then exponentially weighted, and the recognition accuracy and the alignment degree are added to obtain the reward signal; The current state and action space of the prompt chain are extracted to obtain a policy parameter, the ratio of the state transition selection probability of the policy parameter under the current policy to the state transition selection probability under the historical policy is calculated to obtain a probability ratio, the probability ratio is mapped to a preset interval through logarithmic transformation to obtain a policy optimization target; the policy optimization target and the distribution difference of the domain invariant feature are combined to obtain a total optimization target; when the total optimization target does not converge, the distribution gradient of the domain invariant feature is used as the optimization direction to optimize the policy parameter of the prompt chain, and the improvement ratio of the reward signal is calculated.

6. A system for model tuning based on multi-modal information and AI, for implementing the method according to any one of claims 1-5, characterized in that, Including: A first unit configured to extract a plurality of modal features of parameters of a human-computer interaction large model to be optimized; A second unit configured to extract a parameter correlation and a modal interaction feature between different modal features, dynamically adjust a contribution coefficient of the modal interaction feature based on the parameter correlation, transfer feature information between different modal features according to the contribution coefficient, and obtain a cross-modal fusion vector; use a discriminator to identify the original modal attribution of the cross-modal fusion vector, use an adversarial generator to generate mixed features, and use the mixed features to improve the recognition accuracy of the discriminator, align the modal features in the confrontation, and obtain a domain invariant feature; The third unit is configured to calculate a modal confidence of each modal feature, construct a mapping path between different modal features based on the modal confidence and the parameter correlation, obtain a prompt chain, combine an identification accuracy of the discriminator and an alignment degree of the modal features into a reward signal, optimize the prompt chain based on the reward signal, guide an optimization direction of the prompt chain through the domain-invariant feature, iteratively execute the AI reinforcement learning until a change rate of the reward signal is less than a preset convergence threshold, and obtain a prompt optimization result. The fourth unit is configured to update parameters of the to-be-optimized human-computer interaction large model according to the prompt optimization result.

7. An electronic device, comprising: The computer program product comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method in any one of claims 1 to 5.

Images