A large language model fusion method, device and medium based on multimodal data

Abstract

The present invention discloses a large language model fusion method, device and medium based on multimodal data, which specifically relates to the technical field of data fusion, and is used to solve the problem that the importance of maintaining the uniqueness of each modality is ignored in the existing multimodal fusion process. By extracting high-dimensional semantic and visual features from text and image data, using an autoencoder to decouple semantic features and visual features to distinguish common features from modality-specific features, and by calculating the mutual information between modalities and analyzing the complementarity between modalities, the degree of dependence between modalities and the contribution of unique information are determined; after allowing fusion, the fused potential representation generated in the variational autoencoder framework captures the comprehensive information of each modality, and the information loss is evaluated by the KL divergence; through the comprehensive analysis of the variability of the information entropy and the information loss after fusion, the performance of the fusion model in maintaining the uniqueness of the modality is comprehensively evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and more specifically, to a large language model fusion method, device and medium based on multimodal data. Background Art

[0002] With the rapid development of artificial intelligence, multimodal data processing has become an important direction for improving the understanding and generation capabilities of large language models. Multimodal data fusion involves the integration of information from different perceptual channels, such as text, images, audio, etc., where each data modality carries rich and unique information. Especially in the fusion of text and image data, the information carried by these two modalities is essentially different: text expresses semantics through serialized symbols, while images display visual information through spatially distributed pixels.

[0003] Existing multimodal fusion technologies mostly adopt primary fusion strategies, such as simple feature concatenation or linear combination. Although this method can achieve information integration between modalities to a certain extent, it often ignores the importance of maintaining the uniqueness of each modality, which may lead to information redundancy or mutual interference between modalities, making it impossible to effectively utilize the unique expression characteristics of each modality, and failing to maximize their respective contributions to the performance of the large language model.

[0004] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a large language model fusion method, device and medium based on multimodal data to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A large language model fusion method based on multimodal data includes the following steps:

[0008] S1: Use pre-trained natural language processing models to extract high-dimensional semantic features from text data; use pre-trained convolutional neural networks to extract visual features from image data;

[0009] S2: Use autoencoders to decouple semantic features from visual features and distinguish between common features and modality-specific features;

[0010] S3: Determine the degree of dependence between different modalities by calculating the mutual information between modalities; determine the complementarity between modalities by analyzing the unique information provided by each modality and its potential contribution to the fusion task;

[0011] S4: Based on the analysis results of the degree of dependence between different modalities and the complementarity between modalities, decide whether to perform modal fusion;

[0012] S5: When modal fusion is allowed: generate a fused latent representation by encoding the text input and the image input, and use KL divergence to evaluate the information loss between the fused latent representation and the original modality; evaluate the degree of preservation of unique information during modal fusion by calculating the information entropy of each modal feature and combining it with the fusion variability;

[0013] S6: Comprehensively analyze the information loss between the fused latent representation and the original modality and the degree of preservation of uniqueness during the fusion process to assess the potential risk of poor modality uniqueness during fusion;

[0014] By calculating the mutual information between the modes, the degree of dependence between different modes is determined, specifically:

[0015] Let the random variable is the text feature representation, and the random variable is the image feature representation; the mutual information value is calculated by the following formula: ;in, is the mutual information value, and Represent the possible values ​​of text features and image features respectively, is the joint probability distribution, and They are the marginal probability distribution of text features and the marginal probability distribution of image features respectively;

[0016] Setting the Threshold , judge the degree of dependence according to the mutual information value: when When , it is determined that the dependence between modes is strong; when When , it is judged that the dependence between modes is weak;

[0017] By analyzing the unique information provided by each modality and its potential contribution to the fusion task, the complementarity between the modalities is determined, specifically:

[0018] Generate embedding vectors and embedding vector ; embed the vector and embedding vector Mapped into the same high-dimensional feature space;

[0019] Calculate the Mahalanobis distance of text modal features; Calculate the Mahalanobis distance of image modal features;

[0020] Calculate the distribution difference score, the expression is: ;in, is the distribution difference score, It is The Mahalanobis distance of the text modality features, It is The Mahalanobis distance of the image modality features, is the total number of features, Represents the index number of the feature vector;

[0021] Setting Thresholds Used to distinguish the strength of modal complementarity: When , it is determined that the text modality and the image modality have strong complementarity; when When , it is determined that the complementarity between the text modality and the image modality is weak;

[0022] By calculating the information entropy of each modal feature and combining it with the variability after fusion, the degree of preservation of unique information in the modal fusion process is evaluated, specifically:

[0023] Assume that the text modal feature is represented as a random variable , the image modality features are represented as random variables , the fused latent representation is a random variable ;

[0024] Calculate the information entropy of text modal features ; Calculate the information entropy of image modality features ; Calculate the information entropy of the fused potential representation ;

[0025] Calculate the variability of modal information entropy, the formula is as follows: ;in, Represents the variability of modal information entropy;

[0026] A comprehensive analysis is conducted on the information loss between the fused latent representation and the original modality and the degree of preservation of uniqueness during the fusion process to assess the potential risk of poor modality uniqueness during fusion, specifically:

[0027] The uniqueness loss index and the modal information entropy variability are normalized, and the normalized uniqueness loss index and modal information entropy variability are weighted summed to calculate the modal uniqueness hidden danger coefficient;

[0028] Setting preset thresholds , the modal uniqueness hidden danger coefficient and the preset threshold Compare:

[0029] when When , it is determined that the potential risk of poor modality uniqueness during fusion is normal; when When , it is determined that the potential risk of poor modality uniqueness during fusion is high;

[0030] is the modal uniqueness risk coefficient.

[0031] In a preferred embodiment, an autoencoder is used to decouple semantic features from visual features to distinguish between common features and modality-specific features, specifically including:

[0032] S201: Input text semantic features and image visual features into the pre-trained autoencoder model;

[0033] S202: Use the encoder part of the autoencoder to map text semantic features and image visual features into low-dimensional latent representations, respectively, to capture shared and specific features between modalities;

[0034] S203: Separate the encoded latent representation into common features and modality-specific features through a decoupling module;

[0035] S204: using the decoder part of the autoencoder, reconstructing the common features and the modality-specific features into high-dimensional representations respectively;

[0036] S205: Store the reconstructed general features and modality-specific features separately.

[0037] In a preferred embodiment, based on the analysis results of the degree of dependence between different modalities and the complementarity between modalities, it is decided whether to perform modal fusion, specifically:

[0038] When the dependency between modalities is strong and the text modality and the image modality are highly complementary, it is determined to perform modal fusion; otherwise, it is determined not to perform modal fusion temporarily.

[0039] In a preferred embodiment, the text input and the image input are encoded to generate a fused latent representation, and the KL divergence is used to evaluate the information loss between the fused latent representation and the original modality, specifically:

[0040] Set text modal input and image modal input ;set up is the potential representation after fusion;

[0041] Fusion of latent representations Through the encoding process Combined with text modal input and image modal input The input is generated as: ;in, is the probability distribution of the encoded latent representation, For text modal input, is the image modality input;

[0042] The reconstructed output is and , the calculation method is as follows: ;in, Representing a given latent representation The conditional probability distribution of text and image data is and They are the reconstructed text output and the reconstructed image output respectively;

[0043] Use KL divergence to evaluate latent representation With conditional input and The information loss between: ;in, is the uniqueness loss index, represents the Kullback-Leibler divergence, and They represent the probability distribution of the latent representation when only text is considered and the probability distribution of the latent representation when only images are considered, respectively.

[0044] On the other hand, the present invention provides an electronic device, comprising: a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a large language model fusion method based on multimodal data.

[0045] On the other hand, the present invention provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, a large language model fusion method based on multimodal data is implemented.

[0046] The technical effects and advantages of the large language model fusion method, device and medium based on multimodal data of the present invention are as follows:

[0047] 1. By using pre-trained natural language processing models and convolutional neural networks, high-dimensional semantic and visual features can be effectively extracted from text and image data, ensuring that the richest and most expressive information is obtained from each modality. Secondly, the use of autoencoders to decouple semantic features from visual features can not only distinguish between common features and modality-specific features, but also maintain the uniqueness of each modality, thereby avoiding information redundancy and inter-modal interference common in traditional fusion methods. By calculating the mutual information between modalities and analyzing the complementarity between modalities, the degree of dependence between modalities and the contribution of unique information can be accurately determined, which is crucial to determining whether to perform modal fusion.

[0048] 2. After allowing fusion, the fused latent representation generated in the variational autoencoder framework not only captures the comprehensive information of each modality, but also evaluates the information loss through KL divergence to ensure that important modal features are not lost during the fusion process; finally, through a comprehensive analysis of the variability of information entropy and information loss after fusion, the performance of the fusion model in maintaining the uniqueness of the modality is comprehensively evaluated, which is of great significance for optimizing the model structure and improving the processing capabilities of large language models. It not only improves the model's understanding and generation capabilities, but also enhances the model's robustness and efficiency in processing complex multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of a large language model fusion method based on multimodal data according to the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example

[0051] Figure 1 The present invention provides a large language model fusion method based on multimodal data, which includes the following steps:

[0052] S1: Use pre-trained natural language processing models to extract high-dimensional semantic features from text data; use pre-trained convolutional neural networks to extract visual features from image data.

[0053] S2: Use autoencoders to decouple semantic features from visual features and distinguish between common features and modality-specific features.

[0054] S3: Determine the degree of dependence between different modalities by calculating the mutual information between modalities; determine the complementarity between modalities by analyzing the unique information provided by each modality and its potential contribution to the fusion task.

[0055] S4: Based on the analysis results of the degree of dependence between different modalities and the complementarity between modalities, decide whether to perform modal fusion.

[0056] S5: When modal fusion is allowed: generate a fused latent representation by encoding the text input and the image input, and use KL divergence to evaluate the information loss between the fused latent representation and the original modality; evaluate the degree of retention of unique information during modal fusion by calculating the information entropy of each modal feature and combining it with the variability after fusion.

[0057] S6: Comprehensively analyze the information loss between the fused latent representation and the original modality and the degree of preservation of uniqueness during the fusion process to evaluate the potential risk of poor modality uniqueness during fusion.

[0058] Use a pre-trained natural language processing model to extract high-dimensional semantic features from text data, specifically:

[0059] First, text data needs to be standardized, including removing stop words, clearing punctuation marks, converting to lowercase, etc., to reduce noise and unify the data format.

[0060] Select a pre-trained natural language processing model, such as BERT (Bidirectional Encoder Representations from Transformers). The BERT model uses the bidirectional encoder representation of Transformer to understand the meaning of words in context and is suitable for extracting high-dimensional semantic features from text.

[0061] The preprocessed text is input into the BERT model. The BERT model outputs a series of vectors, each of which corresponds to a word or token in the input text. These vectors capture the deep semantic features of the text data for subsequent modal fusion.

[0062] Use a pre-trained convolutional neural network to extract visual features from image data, specifically:

[0063] The image data first needs to be size-standardized and color-normalized. For example, all images can be resized to a uniform resolution (such as 224x224 pixels) and the color channels can be normalized to eliminate the effects of lighting and color deviation.

[0064] Choose a pre-trained convolutional neural network, such as ResNet-50. ResNet-50 uses a residual learning framework to simplify network training, can effectively extract visual features from images, and is suitable for deep visual understanding tasks.

[0065] The preprocessed image data is input into the ResNet-50 model. The model processes the image through a series of convolutional layers, activation layers, and pooling layers, and outputs a set of feature maps. These feature maps represent the image visual features of the image, such as edges, textures, shapes, etc., and are suitable for deep learning analysis of visual information.

[0066] The autoencoder is used to decouple semantic features from visual features and distinguish between common features and modality-specific features, including:

[0067] S201: Input text semantic features and image visual features into the pre-trained autoencoder model:

[0068] An autoencoder is a neural network model for unsupervised learning that can learn low-dimensional latent representations of data and achieve data compression and reconstruction. The autoencoder model is pre-trained and can capture the correlation between text and image features well. Text features come from high-dimensional semantic vectors such as BERT, and image features come from feature maps such as ResNet. These high-dimensional features are standardized through the feature preprocessing module and input into the autoencoder model to ensure the distribution consistency of the input features.

[0069] S202: Use the encoder part of the autoencoder to map text semantic features and image visual features into low-dimensional potential representations, capturing shared and specific features between modalities:

[0070] The encoder part of the autoencoder consists of a series of convolutional layers and fully connected layers. Its purpose is to map high-dimensional text and image features into low-dimensional latent space representations. In the first part of the encoder, text features and image features are processed through independent network paths to ensure that the uniqueness of each modality is initially retained. Next, through the shared network layer, the encoder extracts common features between modalities and captures possible correlation information between text and images, such as the correlation between text descriptions and image visual content. Through this encoding method, it is ensured that the low-dimensional latent representation contains both unique features of the modality and common features shared across modalities.

[0071] The encoding process can also prevent the model from overfitting and improve the effect of feature decoupling through regularization methods (such as L2 regularization and Dropout). The activation function of the encoder layer selects ReLU to ensure that the integrity of nonlinear information can be maintained when extracting potential representations.

[0072] S203: Through the decoupling module, the encoded latent representation is separated into common features and modality-specific features to ensure information complementarity and uniqueness:

[0073] The decoupling module uses a special structure to distinguish between common features and modality-specific features. Common features refer to the shared information between text and image modalities, while modality-specific features capture the independent characteristics of text and image modalities respectively.

[0074] To achieve decoupling, the decoupling module uses a method based on conditional variational autoencoder (CVAE), that is, the decoupling module first decouples information through conditional variables (such as text categories or main content features of images), and the common features are processed through a shared network path, while the modality-specific features are processed separately through independent network paths, ensuring the separation of features so that information from different modalities will not be confused.

[0075] S204: Use the decoder part of the autoencoder to reconstruct the common features and modality-specific features into high-dimensional representations to verify the effectiveness of the decoupling:

[0076] The decoder part of the autoencoder is used to reconstruct the encoded low-dimensional latent representation into the original high-dimensional features. This process is to verify the effectiveness of feature decoupling. When passing through the decoder, the common features and modality-specific features will be reconstructed into high-dimensional representations close to the original input features. The decoder is also composed of multiple layers of fully connected layers and convolutional layers. Its goal is to restore the high-dimensional features as much as possible and verify whether the decoupled latent representation completely retains the original information.

[0077] In order to ensure that the decoder can accurately restore the original features, reconstruction loss (such as MSE, mean square error) is used as the main loss function. At the same time, L1 regularization is introduced to further enhance the sparsity of the model, ensuring that the model only captures the most important information during reconstruction to prevent overfitting.

[0078] The decoded features are compared with the input features, and the reconstruction error is calculated to judge the effectiveness of feature decoupling. A low reconstruction error means that the model successfully captures the key common features and modality-specific features, and the decoupling of these features is effective.

[0079] S205: The reconstructed general features and modality-specific features are stored separately as input for subsequent fusion and evaluation:

[0080] After reconstruction, the model stores the common features and modality-specific features separately for use in subsequent modality fusion and evaluation. The storage of these features can adopt different processing strategies:

[0081] Common features: are stored as information that can be shared in subsequent steps (such as modality fusion). These features will play a key role in modality fusion to ensure the complementarity of cross-modal information.

[0082] Modality-specific features: are stored as separate features to maintain the uniqueness of text and image.

[0083] Efficient data compression and storage algorithms are used in the feature storage process to reduce the consumption of computing resources and ensure fast responsiveness of data reading.

[0084] By calculating the mutual information between the modes, the degree of dependence between different modes is determined, specifically:

[0085] Mutual information is used to measure the dependency between two random variables. Through mutual information, the degree of information sharing between different modalities can be evaluated. In the scenario of multimodal data fusion, mutual information can be used to quantify the degree of dependency between text modality and image modality.

[0086] Let the random variable is the text feature representation, and the random variable is the image feature representation. The mutual information value is calculated by the following formula: ;in, is the mutual information value, and Represent the possible values ​​of text features and image features respectively; It is a joint probability distribution, which indicates the probability of text features and image features occurring simultaneously; and They are the marginal probability distribution of text features and the marginal probability distribution of image features, respectively, indicating the probability of each modal feature occurring independently.

[0087] In order to determine the strength of the dependence between different modes, set the threshold , the degree of dependence is determined according to the mutual information value, as follows:

[0088] when When , it is determined that the dependence between modalities is strong, and there is a strong dependence between text and image features.

[0089] when When , it is judged that the dependence between the modes is weak, indicating that the two modal features are relatively independent and the fusion effect may be limited.

[0090] The preset threshold The cross-validation method can be used to test and optimize on multiple training data sets. The setting of depends on the specific application scenario: if the fusion task requires higher complementarity, a higher threshold can be set; if the task requires broad dependence, the threshold can be appropriately lowered.

[0091] By analyzing the unique information provided by each modality and its potential contribution to the fusion task, the complementarity between the modalities is determined, specifically:

[0092] In multimodal fusion tasks, the difference in feature distribution of different modalities is an important indicator for evaluating modal complementarity.

[0093] Text features are embedded into vectors using natural language processing models such as BERT , image features generate embedding vectors through convolutional neural networks (such as ResNet) .

[0094] Embedding vector and embedding vector Mapped into the same high-dimensional feature space.

[0095] For the text modality, its feature distribution in the embedding space can be measured by the Mahalanobis distance: ;in, is the Mahalanobis distance of the text modality feature, which is used to measure the distribution of the text modality feature in the embedding space; Represents a text feature embedding vector; The mean vector representing the text modality features; The covariance matrix representing the text modality feature embedding vector reflects the correlation of text modality features in different dimensions; Represents the transpose of a matrix.

[0096] When calculating the Mahalanobis distance, the product of a vector and a matrix is ​​involved, where the vector is a column vector, and is its transpose, which becomes a row vector. Through the transpose operation, matrix multiplication with the covariance matrix can be performed.

[0097] For the image modality, its feature distribution in the embedding space is measured by the same Mahalanobis distance: ;in, The Mahalanobis distance representing the image modality features is used to measure the distribution of the image modality features in the embedding space; Represents an image feature embedding vector; Represents the mean vector of the image modality features; Represents the covariance matrix of the image modality feature embedding vector.

[0098] The distribution difference score between the text modality and the image modality is calculated as: ;in, is the distribution difference score, It is The Mahalanobis distance of the text modality features, It is The Mahalanobis distance of the image modality features, is the total number of features, Indicates the index number of the feature vector.

[0099] The distribution difference score calculates the relative distribution difference between the two modalities. The greater the difference, the more unique and complementary the information between the two modalities is, and the stronger the complementarity is. The higher the distribution difference score, the less duplication of information between the modalities is, and the more complementary information can be provided to each other.

[0100] Based on the calculated distribution difference score, the complementarity between the modalities is judged. Set the threshold Used to distinguish the strength of modal complementarity, as follows:

[0101] when , it is determined that the text modality and the image modality have strong complementarity, which means that they provide significantly different information and are suitable for modality fusion.

[0102] when When , it is determined that the complementarity between the text modality and the image modality is weak, indicating that there is a lot of overlap in the information provided by the two, and the fusion effect may be limited or not significant.

[0103] Among them, the threshold The setting of is usually done through cross-validation and experimental adjustment. In different data sets and application scenarios, different thresholds are tested using cross-validation methods. , to determine its best impact on model performance. Specifically, the distribution difference score can be calculated on multiple validation sets, and a threshold that makes the model perform best can be selected based on the performance indicators of the fused model (such as accuracy, recall, etc.). Usually, the threshold When it is higher, it means stricter complementarity judgment criteria, which is suitable for fusion tasks with high difference requirements.

[0104] Based on the analysis results of the degree of dependence between different modes and the complementarity between modes, it is decided whether to perform modal fusion, specifically:

[0105] When the dependency between modalities is strong and the complementarity is high, modal fusion can improve the overall performance of the model by integrating the unique information of different modalities. Strong dependency indicates that there is a large information correlation between text and image modalities, which means that the interaction between the two will bring meaningful enhancements. In addition, high complementarity means that the information provided by each modality does not overlap, and after fusion, it can reduce information redundancy, maximize the use of multimodal characteristics, and improve the processing ability of complex tasks. On the contrary, when the dependency is weak and the complementarity is insufficient, modal fusion may introduce noise or add unnecessary complexity, so not fusing it for the time being can avoid performance degradation or unnecessary computational overhead.

[0106] Therefore: when the dependency between modalities is strong and the text modality and the image modality are highly complementary, it is determined to perform modal fusion; otherwise, it is determined not to perform modal fusion temporarily.

[0107] The fused latent representation is generated by encoding the text input and the image input, and the KL divergence is used to evaluate the information loss between the fused latent representation and the original modality, specifically:

[0108] In multimodal data processing, it is crucial to ensure that the fused representation can preserve the unique information of the original modalities as much as possible.

[0109] Set two modal inputs: text modal input and image modal input ; Set the potential representation after fusion to .

[0110] The variational autoencoder will evaluate the uniqueness loss by the following steps:

[0111] Fusion of latent representations Through the encoding process , combined with text modal input and image modal input The input is generated as: ;in, is the probability distribution of the encoded latent representation.

[0112] in, is a key probability distribution function, which is called the posterior distribution or encoding distribution in the variational autoencoder (VAE) framework. This distribution describes and Under this condition, the potential representation after fusion The probability distribution of .

[0113] The reconstructed output is and , and They are the reconstructed text output and the reconstructed image output respectively, and the calculation method is as follows: ;in, Representing a given latent representation The conditional probability distribution of text and image data.

[0114] Measures the difference between the reconstruction error and the original data, and uses KL divergence to evaluate the potential representation With conditional input and The information loss between: ;in, is the uniqueness loss index, represents the Kullback-Leibler divergence, and They represent the probability distribution of the latent representation when only text is considered and the probability distribution of the latent representation when only images are considered, respectively.

[0115] The larger the uniqueness loss index is, the greater the information loss between the fused latent representation and the original modality is, which means that the difference between the fused latent representation and the original modality is more significant. This is because the KL divergence measures the difference between two probability distributions. The larger the value is, the greater the distribution deviation between the reconstructed latent representation and the original modality input is, indicating that the unique information of the original modality is lost more in the fusion process, resulting in increased information loss.

[0116] By calculating the information entropy of each modal feature and combining it with the variability after fusion, the degree of preservation of unique information in the modal fusion process is evaluated, specifically:

[0117] Information entropy represents the magnitude of uncertainty in a system. If the information entropy of a feature changes significantly after modal fusion, it indicates that the uniqueness of the modality may be destroyed.

[0118] Assume that the text modal feature is represented as a random variable , the image modality features are represented as random variables , the fused latent representation is a random variable .

[0119] Calculate the information entropy of text modal features It measures the uncertainty of text modal features, and the formula is as follows: ;in, is the probability distribution of text modality features, Represents a specific instance or observation of a single text modality feature.

[0120] Similarly, the information entropy of image modality features Represents the entropy of the image modality feature, and the formula is: ;in, is the probability distribution of the image modality features, Represents a specific instance or observation of a single image modality feature.

[0121] Information entropy of the fused latent representation It represents the uncertainty of the overall system after fusion, and the calculation formula is as follows: ;in, is the probability distribution of the fused features, Represents a specific instance or observation of the fused latent representation.

[0122] The modal information entropy variation is calculated to measure the change in information entropy of text and image modalities before and after fusion. The formula is as follows: .

[0123] in, It represents the variability of modal information entropy, that is, the total variability of information entropy of each modality before and after modal fusion. If the sum of information entropy is greater than the information entropy after fusion, it indicates that some modal uniqueness is lost during the fusion process; on the contrary, if the total entropy is less than the entropy after fusion, it may mean that more information redundancy is introduced during the fusion process.

[0124] The greater the variability of the modal information entropy, the lower the degree of preservation of uniqueness during the fusion process. This means that the fusion process may lead to the loss of modal uniqueness, the information is compressed or fused too tightly, and the unique information of each modality cannot be preserved.

[0125] A comprehensive analysis is conducted on the information loss between the fused latent representation and the original modality and the degree of preservation of uniqueness during the fusion process to assess the potential risk of poor modality uniqueness during fusion, specifically:

[0126] The uniqueness loss index and modal information entropy variability are normalized and converted to the same scale for better weighting.

[0127] The normalized uniqueness loss index is denoted as ; The normalized modal information entropy variability is marked as .

[0128] The normalized uniqueness loss index and modal information entropy variability are weighted and summed to calculate the modal uniqueness hidden danger coefficient to reflect the different weights of the two on the fusion uniqueness: .

[0129] is the modal uniqueness hidden danger coefficient, and They are and The weight factor of Used to balance the influence of the two.

[0130] A large modal uniqueness hidden danger coefficient indicates that the modal uniqueness loss is large during the fusion process, and the system needs to re-evaluate the fusion strategy to avoid excessive information loss or excessive modal compression.

[0131] The small modal uniqueness hidden danger coefficient indicates that the modal characteristics are well preserved during the fusion process, the fusion strategy is effective, the information loss is small, and the system performance is stable.

[0132] That is, the larger the modal uniqueness hidden danger coefficient is, the greater the hidden danger of poor modal uniqueness during fusion.

[0133] Setting preset thresholds It is used as a criterion for judging the fusion effect and is determined based on experiments and actual needs. The settings will be based on historical data or cross-validation results.

[0134] Preset Threshold The setting of is usually based on historical data analysis and experimental testing, aiming to optimize model performance. By cross-validating on different data sets, the threshold that can maximize or best balance key performance indicators (such as accuracy, recall, etc.) is determined. The setting of this threshold also takes into account specific needs in practical applications, such as the importance of preserving modality uniqueness and the need to prevent excessive compression or loss of information.

[0135] when When , it is determined that the potential risk of poor modal uniqueness during fusion is normal, indicating that the modal uniqueness is well preserved, the information loss is small, and the fused representation significantly improves the overall performance of the model. At this time, continue the current modal fusion strategy without adjustment. In this case, it means that the complementarity between the modalities is good, the information of each modality can be effectively retained after fusion, and there is no excessive information redundancy or loss.

[0136] when When , it is determined that the potential risk of poor modal uniqueness during fusion is high, indicating that the uniqueness loss is large during modal fusion, and the information entropy has changed significantly, resulting in excessive compression or loss of modal features. At this time, the following measures can be taken:

[0137] By adjusting the weights of different modal inputs during the fusion process, the model can better preserve the uniqueness of each modality. Increase the weights of modalities that lose more uniqueness, or reduce the weights of modalities that have too much influence on the fusion result.

[0138] Improve the modal feature extraction process to ensure that the features of each modality can express its original information as completely as possible.

[0139] Through modal decoupling, the dependencies between modes are reduced and more uniqueness within the modes is retained.

[0140] Multi-task learning can help the model learn one task while retaining information that is beneficial to other tasks.

[0141] Dynamically adjust the fusion strategy to adjust the fusion process based on real-time feedback to adapt to different data characteristics and needs. Implement adaptive fusion mechanisms, such as online learning or incremental learning strategies, to adjust the fusion strategy in real time based on model performance feedback.

[0142] By introducing regularization terms to limit the model from overfitting to certain features, the excessive compression of information can be reduced. Regularization terms such as L1 and L2 regularization can be added to the loss function, or dropout technology can be used to reduce the excessive influence of specific modes.

[0143] By implementing the above measures, the problem of poor modal uniqueness in the fusion process can be effectively addressed, thereby improving the overall performance and robustness of the multimodal fusion model.

[0144] Example 2

[0145] An electronic device includes: a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, a large language model fusion method based on multimodal data is implemented.

[0146] Example 3

[0147] A computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, a large language model fusion method based on multimodal data is implemented.

[0148] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0149] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0150] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0153] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0155] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0156] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0157] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A large language model fusion method based on multimodal data, characterized in that: The steps include: S1: Use pre-trained natural language processing models to extract high-dimensional semantic features from text data; use pre-trained convolutional neural networks to extract visual features from image data; S2: Use autoencoders to decouple semantic features from visual features and distinguish between common features and modality-specific features; S3: Determine the degree of dependence between different modalities by calculating the mutual information between modalities; determine the complementarity between modalities by analyzing the unique information provided by each modality and its potential contribution to the fusion task; S4: Based on the analysis results of the degree of dependence between different modalities and the complementarity between modalities, decide whether to perform modal fusion; S5: When modal fusion is allowed: generate a fused latent representation by encoding the text input and the image input, and use KL divergence to evaluate the information loss between the fused latent representation and the original modality; evaluate the degree of preservation of unique information during modal fusion by calculating the information entropy of each modal feature and combining it with the fusion variability; S6: Comprehensively analyze the information loss between the fused latent representation and the original modality and the degree of preservation of uniqueness during the fusion process to assess the potential risk of poor modality uniqueness during fusion; By calculating the mutual information between the modes, the degree of dependence between different modes is determined, specifically: Let the random variable is the text feature representation, and the random variable is the image feature representation; the mutual information value is calculated by the following formula: ;in, is the mutual information value, and Represent the possible values ​​of text features and image features respectively, is the joint probability distribution, and They are the marginal probability distribution of text features and the marginal probability distribution of image features respectively; Setting the Threshold , judge the degree of dependence according to the mutual information value: when When , it is determined that the dependence between modes is strong; when When , it is judged that the dependence between modes is weak; By analyzing the unique information provided by each modality and its potential contribution to the fusion task, the complementarity between the modalities is determined, specifically: Generate embedding vectors and embedding vector ; embed the vector and embedding vector Mapped into the same high-dimensional feature space; Calculate the Mahalanobis distance of text modal features; Calculate the Mahalanobis distance of image modal features; Calculate the distribution difference score, the expression is: ;in, is the distribution difference score, It is The Mahalanobis distance of the text modality features, It is The Mahalanobis distance of the image modality features, is the total number of features, Represents the index number of the feature vector; Setting Thresholds Used to distinguish the strength of modal complementarity: When , it is determined that the text modality and the image modality have strong complementarity; when When , it is determined that the complementarity between the text modality and the image modality is weak; By calculating the information entropy of each modal feature and combining it with the variability after fusion, the degree of preservation of unique information in the modal fusion process is evaluated, specifically: Assume that the text modal feature is represented as a random variable , the image modality features are represented as random variables , the fused latent representation is a random variable ; Calculate the information entropy of text modal features ; Calculate the information entropy of image modality features ; Calculate the information entropy of the fused potential representation ; Calculate the variability of modal information entropy, the formula is as follows: ;in, Represents the variability of modal information entropy; A comprehensive analysis is conducted on the information loss between the fused latent representation and the original modality and the degree of preservation of uniqueness during the fusion process to assess the potential risk of poor modality uniqueness during fusion, specifically: The uniqueness loss index and the modal information entropy variability are normalized, and the normalized uniqueness loss index and modal information entropy variability are weighted summed to calculate the modal uniqueness hidden danger coefficient; Setting preset thresholds , the modal uniqueness hidden danger coefficient and the preset threshold Compare: when When , it is determined that the potential risk of poor modality uniqueness during fusion is normal; when When , it is determined that the potential risk of poor modality uniqueness during fusion is high; is the modal uniqueness risk coefficient.

2. The large language model fusion method based on multimodal data according to claim 1, characterized in that: The autoencoder is used to decouple semantic features from visual features and distinguish between common features and modality-specific features, including: S201: Input text semantic features and image visual features into the pre-trained autoencoder model; S202: Use the encoder part of the autoencoder to map text semantic features and image visual features into low-dimensional latent representations, respectively, to capture shared and specific features between modalities; S203: Separate the encoded latent representation into common features and modality-specific features through a decoupling module; S204: using the decoder part of the autoencoder, reconstructing the common features and the modality-specific features into high-dimensional representations respectively; S205: Store the reconstructed general features and modality-specific features separately.

3. The large language model fusion method based on multimodal data according to claim 2, characterized in that: Based on the analysis results of the degree of dependence between different modes and the complementarity between modes, it is decided whether to perform modal fusion, specifically: When the dependency between modalities is strong and the text modality and the image modality are highly complementary, it is determined to perform modal fusion; otherwise, it is determined not to perform modal fusion temporarily.

4. The large language model fusion method based on multimodal data according to claim 3 is characterized in that: The fused latent representation is generated by encoding the text input and the image input, and the KL divergence is used to evaluate the information loss between the fused latent representation and the original modality, specifically: Set text modal input and image modal input ; set up is the potential representation after fusion; Fusion of latent representations Through the encoding process Combined with text modal input and image modal input The input is generated and expressed as: ;in, is the probability distribution of the encoded latent representation, For text modal input, is the image modality input; The reconstructed output is and , the calculation method is as follows: ;in, Representing a given latent representation The conditional probability distribution of text and image data is and They are the reconstructed text output and the reconstructed image output respectively; Use KL divergence to evaluate latent representation With conditional input and The information loss between: ;in, is the uniqueness loss index, represents the Kullback-Leibler divergence, and They represent the probability distribution of the latent representation when only text is considered and the probability distribution of the latent representation when only images are considered, respectively.

5. An electronic device, characterized in that: The electronic device includes: a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, a large language model fusion method based on multimodal data is implemented as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, it implements a large language model fusion method based on multimodal data as described in any one of claims 1-4.

Images