A multimodal large language model establishment method, medium and system

Abstract

The application provides a multimodal large language model establishing method, medium and system, belongs to computer model technical field, multimodal large language model establishing method proposed by the application, first, text, image, audio data are standardized pretreated, then the feature extraction structure containing bidirectional long short time memory network, residual network and one-dimensional convolution network is constructed. The feature extraction effect is evaluated by the alignment function, the split index is calculated based on the alignment component matrix, and the mode contribution distribution model is constructed to evaluate the feature importance. Finally, the feature fusion is realized by using the attention mechanism, and the model training is completed by optimizing the classifier layer and the loss function. The method realizes efficient multimodal feature fusion through feature alignment optimization and contribution value distribution, and solves the technical problem of poor feature fusion effect caused by insufficient multimodal feature alignment in the prior art.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer models, and in particular relates to a multi-modal large language model establishing method, medium and system. BACKGROUND

[0002] Multi-modal large language models are an important research direction in the field of artificial intelligence, mainly used for fusion analysis tasks of text, image, audio and other multi-modal data. Traditional multi-modal feature fusion methods mainly include simple concatenation, weighted average and attention mechanism. The simple concatenation method directly connects the feature vectors of different modalities into longer vectors, which is simple to implement but ignores the correlation between modalities. The weighted average method combines different modalities by setting fixed weights, but it is difficult to adapt to the dynamic changes of the importance of different modalities in different scenarios. The method based on attention mechanism can adaptively adjust the feature weights, but due to the lack of explicit measurement of feature alignment degree, it is easy to cause information loss in the feature fusion process.

[0003] The existing multi-modal feature fusion technology mainly has the following problems: first, different modal data has different feature distribution and statistical characteristics, and direct feature fusion can easily lead to misalignment of information between modalities. Second, existing methods often ignore the interaction between modal features, and cannot fully utilize the complementary information of multi-modal data. Third, for time series data, there is a lack of effective mechanism to model the influence of historical features on current prediction. In addition, in the feature extraction process, the contribution of different levels of features to the final prediction result also lacks effective evaluation methods.

[0004] To solve these problems, a method is needed to effectively measure and optimize the alignment degree of features, and on this basis to achieve better feature fusion. Especially in processing long sequence, multi-scale multi-modal data, how to ensure the effectiveness of feature extraction and the accuracy of feature fusion is a key problem to be solved. However, due to the lack of effective measurement and optimization mechanism for feature alignment degree, the existing technology is difficult to realize high-quality feature fusion, which limits the performance of the model. That is, there is a technical problem of insufficient multi-modal feature alignment leading to poor feature fusion effect in the existing technology. SUMMARY

[0005] Therefore, the present application provides a multi-modal large language model establishing method, medium and system, which can solve the technical problem of insufficient multi-modal feature alignment leading to poor feature fusion effect in the existing technology.

[0006] The application is implemented in the following manner: a first aspect of the application provides a multimodal large language model establishment method comprising the following steps: obtaining text data, image data and audio data as training input data and performing preprocessing to obtain standardized training data; constructing a multi-layer convolutional neural network structure, setting a text feature extraction layer, an image feature extraction layer and an audio feature extraction layer; calculating an alignment function between the convolution kernel parameters of each layer and the standardized training data, constructing an alignment component matrix and a non-alignment component matrix and calculating a splitting index; constructing a modal contribution allocation model including a feature contribution function, an interaction influence function and a time decay function, the modal contribution allocation model being used to calculate the contribution value of each modal feature to the model prediction result; calculating attention weights based on the contribution value and performing feature fusion to obtain a trained multimodal large language model.

[0007] In the step of preprocessing the training input data, the text data is subjected to word segmentation processing and converted into a 768-dimensional vector representation through word embedding technology; the image data is uniformly adjusted to a size of 224x224 pixels and subjected to normalization processing to scale the pixel values to between 0 and 1; the audio data is subjected to resampling to uniformly adjust the sampling rate to 16000Hz and extract mel spectrum features.

[0008] In the multi-layer convolutional neural network structure, the text feature extraction layer adopts a 3-layer bidirectional long short-term memory layer with 512 hidden units per layer; the image feature extraction layer adopts a 152-layer residual network containing 4 residual blocks; and the audio feature extraction layer adopts 6 convolutional layers with a kernel size of 3 and a channel number increasing from 64 to 512 layer by layer.

[0009] In the step of calculating the alignment function, the mutual information between the feature maps output by each layer of the network and the input data is calculated; a cosine similarity matrix between the feature maps is calculated; and the mutual information and the similarity matrix are combined to obtain the alignment function value.

[0010] In the modal contribution allocation model, the feature contribution function adopts a weighted attention mechanism, the feature importance is weighted according to the historical prediction accuracy; the interaction influence function quantifies the interaction between modalities by calculating the mutual information and conditional mutual information between features; and the time decay function adopts an exponential decay form, the decay rate being adaptively adjusted according to the time interval.

[0011] The calculation of the attention weights adopts an 8-head attention mechanism, the attention scores are calculated through dot product attention, and the final attention weights are obtained by normalizing using a softmax function.

[0012] The process also includes the step of constructing a classifier layer, which contains three fully connected layers with hidden layer dimensions of 768, 384, and 192 respectively. Each fully connected layer is followed by a batch normalization layer and a ReLU activation function.

[0013] The training process employs a combination of cross-entropy loss and auxiliary loss, with the auxiliary loss including L2 regularization and adversarial training loss. Training is terminated when the loss function value is less than 0.01 or the validation set accuracy reaches 95%.

[0014] A second aspect of the present invention provides a computer-readable storage medium storing program instructions, which, when executed in a computer, are used to perform the above-described method for establishing a multimodal large language model.

[0015] A third aspect of the present invention provides a multimodal large language model building system, comprising the aforementioned computer-readable storage medium, wherein the system is any one of a computer, a server, or a microcontroller, the computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor for executing program instructions stored in the computer-readable storage medium.

[0016] Compared with existing technologies, this invention provides a method, medium, and system for building a multimodal large language model. This invention proposes a multimodal feature fusion method based on an alignment function and modal contribution allocation. This method quantifies the effect of feature extraction by introducing an alignment function, constructs aligned and unaligned component matrices, and evaluates the alignment degree of features through a splitting index. Simultaneously, a modal contribution allocation model incorporating a feature contribution function, an interaction effect function, and a temporal decay function is designed, enabling accurate evaluation of the importance of different modal features.

[0017] By introducing an alignment function and a splitting exponent, this invention can accurately evaluate and optimize the alignment effect during feature extraction, effectively solving the problem of feature misalignment. The modal contribution allocation model, by considering the feature's own contribution, interaction effects, and temporal effects, achieves a comprehensive evaluation of feature importance, overcoming the shortcomings of existing methods that neglect feature interactions and temporal effects. Furthermore, the attention weighting mechanism based on contribution values ​​makes the feature fusion process more accurate and interpretable.

[0018] The present invention effectively solves the problem of poor feature fusion results caused by insufficient multimodal feature alignment through innovative designs such as feature alignment measurement and optimization, modal contribution allocation, and attention fusion based on contribution values. This solution not only improves the quality of feature extraction but also enhances the accuracy of feature fusion, enabling the model to better handle complex multimodal data analysis tasks. In summary, the present invention solves the technical problem of poor feature fusion results caused by insufficient multimodal feature alignment in the prior art. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0021] like Figure 1 The diagram shown is a flowchart of a method for establishing a multimodal large language model according to the first aspect of the present invention. This method includes the following steps:

[0022] S01. Obtain text data, image data, and audio data as training input data, and preprocess the training input data to obtain standardized training data;

[0023] S02. Construct a multi-layer convolutional neural network structure, and set a text feature extraction layer, an image feature extraction layer and an audio feature extraction layer in the multi-layer convolutional neural network structure, wherein the text feature extraction layer adopts a bidirectional long short-term memory network structure, the image feature extraction layer adopts a residual neural network structure, and the audio feature extraction layer adopts a one-dimensional convolutional neural network structure.

[0024] S03. Calculate the alignment function between the convolution kernel parameters of each layer in the text feature extraction layer, the image feature extraction layer, and the audio feature extraction layer and the standardized training data;

[0025] S04. Construct an aligned component matrix and an unaligned component matrix based on the alignment function, and calculate the splitting index between the aligned component matrix and the unaligned component matrix;

[0026] S05. Establish the mapping relationship between the splitting index and the network parameters of each layer in the multi-layer convolutional neural network structure;

[0027] S06. Construct a modality contribution allocation model between the multi-layer convolutional neural network structure and the attention mechanism layer. The modality contribution allocation model includes a feature contribution function, an interaction influence function, and a time-series decay function.

[0028] S07. Optimize the parameters of the multi-layer convolutional neural network structure according to the mapping relationship to obtain the optimized network parameters;

[0029] S08. Use the optimized network parameters to extract features from the standardized training data to obtain multimodal features;

[0030] S09. Input the multimodal features into the modal contribution allocation model and calculate the contribution value of each modal feature to the model prediction result;

[0031] S10. Construct an attention mechanism layer and calculate the attention weights between different modal features based on the contribution values;

[0032] S11. The multimodal features are weighted and fused based on the attention weights to obtain a fused feature vector;

[0033] S12. Construct a classifier layer and input the fused feature vector into the classifier layer for training;

[0034] S13. Calculate the loss function value between the model output and the labeled data;

[0035] S14. Optimize and update the parameters of the multi-layer convolutional neural network structure, the modality contribution allocation model, the attention mechanism layer, and the classifier layer based on the loss function value;

[0036] S15. Repeat steps S08 to S14 until the loss function value is less than a preset threshold to obtain a trained multimodal large language model.

[0037] The specific structure of the modal contribution allocation model is as follows:

[0038] The feature contribution function is used to calculate the self-contribution value of a single modal feature. The input includes a single modal feature in the multimodal features and the historical prediction accuracy of the single modal feature. The output is the feature contribution score of the single modal feature.

[0039] The interaction influence function is used to calculate the degree of mutual influence between different modal features. The input includes any two modal features and the joint prediction accuracy of the two modal features. The output is the interaction contribution score between the two modal features.

[0040] The time-series decay function is used to calculate the degree of influence of historical features on the current prediction. The input includes the modal features at the current time and the modal feature sequence at the historical time. The output is the feature influence decay coefficient based on time distance.

[0041] The modal contribution allocation model calculates the final modal feature contribution value by comprehensively considering the feature contribution score, the interaction contribution score, and the feature influence attenuation coefficient.

[0042] The specific implementation methods of the above steps are described in detail below.

[0043] The specific implementation of step S01 is as follows: Standardized preprocessing is performed on three different types of training input data. First, text data undergoes word segmentation, using the Jieba word segmentation algorithm to split the text into word sequences. Each word is then converted into a fixed-dimensional vector representation using word embedding technology, with the word embedding dimension set to 768 dimensions. Stop words are removed, and lexical reconstruction is performed. For image data, all images are uniformly adjusted to 224×224 pixels, normalized to scale pixel values ​​to between 0 and 1, and data augmentation is performed, including random cropping, horizontal flipping, and brightness and contrast adjustments. The augmented images are then standardized to meet a distribution with a mean of 0 and a variance of 1. For audio data, resampling is first performed to uniformly adjust the sampling rate to 16000Hz. Then, Mel spectral features are extracted, with a frame length of 25 milliseconds, a frame shift of 10 milliseconds, and 80 Mel filter banks. Finally, the features are normalized. Through these preprocessing steps, the raw data of different modalities are converted into standardized feature representations, laying the foundation for subsequent feature extraction and model training.

[0044] The specific implementation of step S02 is as follows: A multi-layer convolutional neural network structure is constructed, containing three parallel feature extraction branches. The text feature extraction layer adopts a bidirectional long short-term memory network structure, containing three bidirectional long short-term memory layers, with 512 hidden units in each layer. Residual connections are added between each layer to alleviate the gradient vanishing problem, and a normalization and dropout layer is added after each layer, with the dropout rate set to 0.1. The image feature extraction layer adopts a residual neural network structure, specifically a 152-layer residual network, containing four residual blocks. Each residual block contains multiple 3×3 convolutional layers and a 1×1 bottleneck layer, and residual learning is achieved through shortcut connections. The audio feature extraction layer adopts a one-dimensional convolutional neural network structure, containing six convolutional layers, with a kernel size of 3 and a stride of 1 in each layer. The number of channels increases from 64 to 512 layer by layer. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, and a max pooling layer is used for downsampling. This multi-branch network structure can effectively extract feature representations of different modalities.

[0045] The specific implementation of step S03 is as follows: The alignment degree between the network parameters of each layer and the training data is calculated, and an alignment function is introduced to measure the effectiveness of feature extraction. First, the mutual information between the feature map output by each network layer and the input data is calculated using a mutual information calculation method based on kernel density estimation. Then, the similarity matrix between feature maps is calculated, and cosine similarity is used to measure the relationship between features. Finally, the mutual information and the similarity matrix are combined to obtain the alignment function value; the larger the function value, the more effective the feature extraction. For the text feature extraction layer, the alignment degree between the word vector sequence and the output of the Long Short-Term Memory network is calculated. For the image feature extraction layer, the alignment degree between the feature map output by each residual block and the original image is calculated. For the audio feature extraction layer, the alignment degree between the feature map output by each convolutional layer and the Mel-spectral features is calculated.

[0046] The specific implementation of step S04 is as follows: An aligned component matrix and an unaligned component matrix are constructed based on the alignment function values. The aligned component matrix contains features whose alignment function values ​​are greater than a threshold of 0.7. The unaligned component matrix contains other features. The feature space of these two matrices is analyzed using singular value decomposition (SVD), the singular values ​​corresponding to the main eigenvectors are calculated, and a splitting index is defined based on the ratio of the singular values. Specifically, the splitting index is the ratio of the sum of the first k largest singular values ​​of the aligned component matrix to the sum of the first k largest singular values ​​of the unaligned component matrix, where k is set to 20% of the matrix rank. A larger splitting index indicates better alignment of the extracted features.

[0047] The specific implementation of step S05 is as follows: A mapping relationship is established between the splitting exponent and network parameters, using a gradient-based optimization method. First, the gradient of the splitting exponent with respect to the network parameters of each layer is calculated. Then, the parameters are adjusted according to the gradient direction and magnitude, specifically using an adaptive moment estimation algorithm for parameter updates. For the text feature extraction layer, the main optimizations are to the weight matrix and bias terms in the long short-term memory network; for the image feature extraction layer, the optimizations are to the convolution kernel parameters in the residual blocks; and for the audio feature extraction layer, the optimizations are to the convolution kernel parameters of the one-dimensional convolutional layer. In this way, the optimization objective of the splitting exponent is transformed into the optimization problem of network parameters.

[0048] The specific implementation of step S06 is as follows: A modal contribution allocation model is constructed, comprising three core functions. The feature contribution function employs a weighted attention mechanism, weighting feature importance based on historical prediction accuracy. The accuracy is calculated using an exponential moving average, with a smoothing factor set to 0.9. The interaction influence function quantifies intermodal interactions by calculating mutual information and conditional mutual information between features, estimating mutual information using a nonparametric method based on kernel density estimation. The temporal decay function adopts an exponential decay form, with the decay rate adaptively adjusted according to the time interval; the longer the interval, the faster the decay. Specifically, it uses the negative power form of the exponential function. The outputs of these three functions are combined through a weighted summation to obtain the final modal feature contribution value.

[0049] The specific implementation of step S07 is as follows: Optimize network parameters based on the established mapping relationship using a gradient-based iterative optimization method. First, calculate the splitting exponent under the current parameters, then calculate the gradient of the splitting exponent with respect to the parameters of each layer. Update the parameters using an adaptive moment estimation algorithm, initially setting the learning rate to 0.001, and adjusting the learning rate using a cosine annealing strategy. During the optimization process, parameter regularization constraints are also considered, employing L2 regularization to prevent overfitting, with the regularization coefficient set to 0.0001. Different learning rates and regularization coefficients can be set for different feature extraction layers to adapt to the characteristics of different modalities.

[0050] The specific implementation of step S08 is as follows: Features are extracted from the standardized training data using optimized network parameters to obtain multimodal feature representations. Text features are extracted using a bidirectional long short-term memory network to obtain sequence features with a dimension of 512; image features are extracted using a residual network to obtain spatial features with a dimension of 2048; and audio features are extracted using a one-dimensional convolutional network to obtain time-frequency features with a dimension of 1024. To unify the dimensions of different modal features, a fully connected layer is used to map all features to the same feature space, uniformly set to 768 dimensions. During feature extraction, batch processing is used to improve computational efficiency, with a batch size of 32.

[0051] The specific implementation of step S09 is as follows: Multimodal features are input into the modality contribution allocation model, and the contribution value of each modality to the prediction result is calculated. First, the self-contribution score of each modality feature is calculated using the feature contribution function, considering the discriminative ability and historical performance of the features. Then, the synergistic effect between modalities is calculated using the interaction effect function, quantifying the mutual reinforcement or inhibition between features. Finally, the influence of temporal information is considered using the temporal decay function to attenuate the contribution of historical features. The outputs of the three functions are weighted and averaged to obtain the final contribution value. The weights are determined through grid search and set to 0.4, 0.4, and 0.2, respectively.

[0052] The specific implementation of step S10 is as follows: An attention mechanism layer is constructed, and attention weights are calculated based on modal contribution values. A multi-head attention mechanism is adopted, with 8 heads, each independently learning the correlation between features of different modalities. The calculation of attention weights considers three components: query vector, key vector, and value vector. The query vector and key vector are generated from the features at the current time step, and the value vector contains historical information. The attention score is obtained through dot product attention calculation, and the final attention weights are obtained by normalization using the softmax function.

[0053] The specific implementation of step S11 is as follows: Multimodal features are weighted and fused based on attention weights to obtain a fused feature vector. First, features from different modalities are projected onto the same feature space through a linear transformation, and then a weighted sum is calculated based on the attention weights. To enhance the expressive power of the features, a gating mechanism is added during the fusion process, using the sigmoid function to control the information flow. Finally, the final fused feature vector is obtained through residual connections and layer normalization, maintaining a dimension of 768.

[0054] The specific implementation of step S12 is as follows: A classifier layer is constructed using a multilayer perceptron structure. The classifier contains three fully connected layers with hidden layer dimensions of 768, 384, and 192 respectively. The output dimension of the last layer is equal to the number of classes. Each fully connected layer is followed by a batch normalization layer and a ReLU activation function, and dropout is used to prevent overfitting, with a dropout rate set to 0.1. Label smoothing is employed during training, with a smoothing factor set to 0.1 to improve the model's generalization ability.

[0055] The specific implementation of step S13 is as follows: Calculate the loss function value between the model output and the labeled data. The loss function adopts a combination of cross-entropy loss and auxiliary loss, where cross-entropy loss is used to measure classification accuracy, and auxiliary loss includes L2 regularization and adversarial training loss. Adversarial training improves the robustness of the model by adding perturbations, the size of which is controlled by the parameter ε and set to 0.01. The weights of each part in the total loss function are determined by grid search on the validation set.

[0056] The specific implementation of step S14 is as follows: Optimize the parameters of each component of the model based on the loss function value. Adaptive Moment Estimation (IME) is used for backpropagation, with an initial learning rate set to 0.001. To handle the optimization needs of different components, different learning rates are used for different parts: the learning rate for the feature extraction layer is 0.1 times the baseline learning rate, while the attention layer and classifier layer use the baseline learning rate. Gradient clipping is used to prevent gradient explosion, with a clipping threshold set to 1.0. Model performance is evaluated on the validation set after each epoch. If performance does not improve after 5 consecutive epochs, the learning rate is reduced by a factor of 0.1.

[0057] The specific implementation of step S15 is as follows: Repeat the processes of feature extraction, contribution value calculation, attention fusion, classification prediction, and parameter optimization until the loss function value is less than a preset threshold. The preset threshold is set to 0.01, or training can be terminated early when the accuracy on the validation set reaches 95%. During training, the order of the training data is randomly shuffled in each epoch, and model checkpoints are saved periodically. When the model converges, the checkpoint with the best performance on the validation set is selected as the final model. The trained multimodal large language model can effectively integrate information from different modalities to achieve accurate inference.

[0058] The equations or mathematical models involved in this invention will be explained in detail below.

[0059] The word embedding vector is calculated as follows:

[0060] E = WV + b;

[0061] In the formula, E is the word embedding vector; W is the weight matrix; V is the one-hot encoded vector; and b is the bias term. Specifically, W has a dimension of 768 × vocabulary size, obtained through pre-training; and b has a dimension of 768 × 1, initialized to 0.

[0062] The principle behind the word embedding vector calculation equation above is as follows: First, a vocabulary mapping matrix is ​​constructed, representing each word as a one-hot vector with a dimension equal to the vocabulary size. Then, a weight matrix W is introduced for linear transformation, mapping the high-dimensional sparse vector to a low-dimensional dense space. Finally, a bias term b is added to increase the model's flexibility. The weight matrix W is obtained through pre-training on a large-scale corpus and optimized using algorithms such as word2vec or GloVe by minimizing the prediction loss of context words. The effect of this equation is to encode the semantic information of words into a fixed-dimensional vector space, making semantically similar words closer together in the vector space.

[0063] The calculation for image normalization is represented as follows:

[0064]

[0065] In the formula, I norm I is the normalized image; μ is the original image; σ is the image mean; ∈ is a small value to prevent division by zero, which is 1e-5.

[0066] The principle behind the image normalization equation is based on the statistical concept of standardization. First, the mean μ and standard deviation σ of the image pixel values ​​are calculated, where the mean reflects the overall brightness level and the standard deviation reflects the contrast level. By subtracting the mean and dividing by the standard deviation, the data distribution is adjusted to a standard distribution with a mean of 0 and a variance of 1. The ε term is added to prevent numerical instability when the standard deviation is close to 0. The effect of this equation is to eliminate differences in image brightness and contrast, making the feature distribution of different images more consistent.

[0067] The computational representation of audio Mel-spectral feature extraction is as follows:

[0068] M = H·|STFT(x)| 2 ;

[0069] In the formula, M is the Mel spectral characteristic; H is the Mel filter bank matrix; STFT(x) is the short-time Fourier transform; and x is the audio signal.

[0070] The principle behind the above audio Mel-frequency spectral feature extraction equation is as follows: First, a short-time Fourier transform (STFT) is performed on the audio signal to obtain a time-frequency representation; then, the power spectrum is obtained by squaring the spectrum; finally, a weighted sum is performed using a Mel-filter bank H. The Mel-filter bank is designed based on the characteristics of human hearing, exhibiting higher resolution in the low-frequency region and lower resolution in the high-frequency region. The effect of this equation is to convert the audio signal into spectral features that better match the perceptual characteristics of the human ear.

[0071] The forward computation representation of a bidirectional long short-term memory network is as follows:

[0072] f t =σ(W f ·[h t-1 x t ]+b f );

[0073] i t =σ(W i ·[h t-1 x t ]+b i );

[0074]

[0075] o t =σ(W o ·[h t-1 x t ]+b o );

[0076] h t =o t ·tanh(C t );

[0077] In the formula, f t For the Gate of Oblivion; i t For input gate; o t For output gate; C t For unit state; h t In hidden state; W f W i W C W o b is the weight matrix; f b i b C b o σ is the bias term; σ is the sigmoid function; x t The input vector.

[0078] The above-mentioned bidirectional Long Short-Term Memory (LSTM) network equations are established based on a recurrent neural network structure, incorporating three gating mechanisms: a forget gate, an input gate, and an output gate. The forget gate controls the degree of forgetting of historical information, the input gate controls the degree of input of new information, and the output gate controls the degree of output. Each gate maps its output to a range of 0 to 1 using a sigmoid function, achieving soft gating. The unit state C is updated through a combination of the forget gate and the input gate, while the hidden state h is generated through a combination of the output gate and the unit state. The weight matrix W and the bias term b are optimized using the backpropagation algorithm. The effect of this equation set is that it can effectively process long-sequence data and capture long-term dependencies.

[0079] The computational representation of residual blocks in a residual network is as follows:

[0080] y = F(x, W) i )+x;

[0081] In the formula, y represents the residual block output; F(x, W) i ) is the residual mapping function; x is the input; W i Let be the weight matrix of the i-th layer.

[0082] The principle behind establishing the residual block equation in the aforementioned residual network is as follows: Shortcut connections are introduced on top of a traditional convolutional network, directly adding the input to the output of the nonlinear transformation F(x). F(x) comprises a combination of multiple convolutional layers, batch normalization layers, and activation functions. The effect of this equation is to alleviate the vanishing gradient problem in deep networks, making deep networks easier to train.

[0083] The computational representation of a one-dimensional convolutional layer is as follows:

[0084]

[0085] In the formula, y j For the j-th output feature map; xi Let k be the i-th input feature map; i，j For convolution kernel; b j is the bias term; M is the number of input channels; * indicates the convolution operation.

[0086] The principle behind establishing the above one-dimensional convolutional layer equation is as follows: based on the definition of convolution, a sliding window operation is performed on the input feature map. At each location, the inner product of the local region and the convolution kernel is calculated, and a bias term is added. The convolution kernel parameters are optimized through backpropagation, and the bias term is used to adjust the overall offset of the feature map. The effect of this equation is to extract local feature patterns and it possesses translation invariance.

[0087] The alignment function is calculated as follows:

[0088] A(F,D)=αMI(F,D)+(1-α)S(F);

[0089] In the formula, V(F, D) is the alignment function value; F is the feature map; D is the input data; MI(F, D) is the mutual information; S(F) is the feature similarity; and α is the weight coefficient, which ranges from 0 to 1.

[0090] The calculation of mutual information is represented as follows:

[0091]

[0092] In the formula, p(x, y) is the joint probability distribution; p(x) and p(y) are the marginal probability distributions.

[0093] Feature similarity is calculated as follows:

[0094]

[0095] In the formula, F is the characteristic matrix; ||·|| represents the Frobenius norm of the matrix.

[0096] The alignment function equation described above is established by combining two metrics: mutual information and feature similarity, and then weighting them by a weighted average with α. Mutual information measures the statistical correlation between features and input data, while feature similarity measures the structural consistency of the feature space. α is determined through a grid search on the validation set. This equation effectively evaluates the overall effectiveness of feature extraction.

[0097] The construction representations of aligned and unaligned component matrices are as follows:

[0098] M aligned =[F i |A(F i ,D)>θ];

[0099] M unaligned =[Fi |A(F i [, D)≤θ];

[0100] In the formula, M aligned For aligning component matrices; M unaligned θ is the unaligned component matrix; θ is the threshold, set to 0.7.

[0101] The calculation of the split index is expressed as follows:

[0102]

[0103] In the formula, SI is the split index; To align the i-th singular value of the component matrix; is the i-th singular value of the unaligned component matrix; k is the number of singular values ​​in the top 20%.

[0104] The characteristic contribution function is calculated as follows:

[0105] C f (x)=w·x·acc hist ;

[0106] In the formula, C f (x) represents the feature contribution score; x is the feature vector; w is the attention weight; acc hist The historical prediction accuracy is calculated using an exponential moving average.

[0107] The principle behind establishing the feature contribution function equation is as follows: the feature vector is multiplied by the attention weights and the historical accuracy. The attention weights are calculated using the softmax function, and the historical accuracy is updated using an exponential moving average. The effect of this equation is to dynamically adjust the contribution of the feature based on its importance and historical performance.

[0108] The interaction effect function is calculated as follows:

[0109] C i (x1, x2) = MI(x1, x2|y);

[0110] In the formula, C i (x1, x2) represents the interaction contribution score; MI(x1, x2|y) represents the conditional mutual information; and y represents the predicted label.

[0111] The principle behind establishing the above interaction effect function equation is: using conditional mutual information to measure the interaction between features of different modalities, and considering the conditional probability distribution of the predicted labels. The effect of this equation is to quantify the synergistic effect between features.

[0112] The calculation of the time-series decay function is expressed as follows:

[0113] D(t) = e-λt ;

[0114] In the formula, D(t) is the attenuation coefficient; λ is the attenuation rate, which is positively correlated with the time interval t.

[0115] The principle behind establishing the above time-series decay function equation is as follows: it adopts an exponential function form, and the decay rate λ is positively correlated with the time interval. The equation simulates the natural decay process of the characteristic influence over time.

[0116] The final modal contribution value is calculated as follows:

[0117] C total =β1C f +β2C i +β3D;

[0118] In the formula, C total The final contribution value is β1, β2, and β3, which are combined weights determined through grid search.

[0119] The computational representation of the multi-head attention mechanism is as follows:

[0120]

[0121] In the formula, Q is the query matrix; K is the key matrix; V is the value matrix; d k is the dimension of the key vector.

[0122] The principle behind establishing the multi-head attention mechanism equation is as follows: The query, key, and value matrices are scaled and dot-producted for attention calculation, and the attention weights are obtained by normalization using the softmax function. (Scaling factor) This is used to prevent the gradient of the softmax function from vanishing due to an excessively large dot product result. The effect of this equation is to learn the dynamic relationships between features.

[0123] The total loss function is calculated as follows:

[0124] L total =L CE +λ1L L2 +λ2L adv ;

[0125] In the formula, L total Total loss; L CE For cross-entropy loss; L L2 L2 regularization loss; L adv To counteract training loss; λ1 and λ2 are weight coefficients.

[0126] The calculation of adversarial training loss is expressed as follows:

[0127] L adv =max||δ||≤∈ L(x+δ, y);

[0128] In the formula, δ represents the adversarial perturbation; ∈ represents the upper limit of the perturbation size, set to 0.01; x represents the input sample; and y represents the true label.

[0129] The principle behind establishing the above total loss function equation is to combine cross-entropy loss, L2 regularization loss, and adversarial training loss, with the weight coefficients adjusted based on validation set performance. Cross-entropy loss optimizes classification accuracy, L2 regularization prevents overfitting, and adversarial training improves model robustness. The effect of this equation is to achieve multi-objective optimization and balance the various performance metrics of the model.

[0130] The above formulas fully consider the characteristics and requirements of each processing stage. They employ power relations to represent the nonlinear mapping of features, division relations for normalization, exponential relations to express the decay effect, and summation relations to integrate the contributions of multiple features. The use of logarithmic relations in mutual information calculation aligns with the basic principles of information theory, while the use of dot product and softmax in the attention mechanism reflects the probabilistic characteristics of attention distribution. Each formula includes adjustable parameters and necessary constraints to improve the model's adaptability and robustness. Compared to existing technologies, this scheme innovates in feature alignment measurement, modality contribution allocation, and loss function design, enabling it to better handle the fusion of multimodal data.

[0131] A second aspect of the present invention provides a computer-readable storage medium storing program instructions, which, when executed in a computer, are used to perform the above-described method for establishing a multimodal large language model.

[0132] A third aspect of the present invention provides a multimodal large language model building system, comprising the aforementioned computer-readable storage medium, wherein the system is any one of a computer, a server, or a microcontroller, the computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor for executing program instructions stored in the computer-readable storage medium.

[0133] Specifically, the principle of this invention is as follows: First, an alignment function is introduced as a quantitative indicator to evaluate the feature extraction effect. This function combines two dimensions: mutual information and feature similarity, comprehensively reflecting the correspondence between features and original data. By calculating the mutual information between the feature map and the input data, the degree to which the features retain the original information can be measured; by calculating the similarity matrix between feature maps, the structural consistency of the feature space can be evaluated.

[0134] Secondly, based on the alignment function, the aligned and unaligned component matrices are used to calculate the splitting index through singular value decomposition, providing a clear optimization objective for the feature extraction process. A larger splitting index indicates better alignment, providing clear guidance for optimizing network parameters.

[0135] Furthermore, the modal contribution allocation model fully considers multiple dimensions of feature influence. The feature contribution function, by combining attention weights and historical performance, achieves dynamic evaluation of feature importance; the interaction influence function characterizes the interaction between features through conditional mutual information; and the temporal decay function simulates the natural decay of feature influence over time. This multi-dimensional contribution evaluation mechanism makes the feature fusion process more accurate and reasonable.

[0136] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0137] The specific implementation of step S01 is as follows: Standardization preprocessing is performed on three different types of training input data, where text data is represented by word embedding vectors. The specific calculation process is as follows:

[0138] E = WV + b;

[0139] In the formula, E is the word embedding vector with a dimension of 768; W is the weight matrix with a dimension of 768 × vocabulary size, obtained through pre-training on a large-scale corpus using the word2vec algorithm; V is the one-hot encoded vector with a dimension of vocabulary size × 1; b is the bias term with a dimension of 768 × 1, initialized to 0. The standardization processing for image data is calculated as follows:

[0140]

[0141] In the formula, I norm The image is normalized; I is the original image; μ is the image mean; σ is the standard deviation; ∈ is a delimited value to prevent division by zero, taken as 1e-5. For audio data, Mel-spectral feature extraction is calculated as follows:

[0142] M = H·|STFT(x)| 2 ;

[0143] In the formula, M represents the Mel spectral feature; H represents the Mel filter bank matrix; STFT(x) represents the short-time Fourier transform; and x represents the audio signal. During audio feature extraction, the sampling rate is first uniformly adjusted to 16000Hz, then the frame length is set to 25 milliseconds, the frame shift to 10 milliseconds, and the number of Mel filter banks is set to 80. The main purpose of this step is to convert the raw data from different modalities into standardized feature representations, laying the foundation for subsequent feature extraction and model training.

[0144] The specific implementation of step S02 is as follows: A multi-layer convolutional neural network structure is constructed, and the text feature extraction layer adopts a bidirectional long short-term memory network structure. The calculation process is as follows:

[0145] f t =σ(W f ·[h t-1 x t ]+b f );

[0146] i t =σ(W i ·[h t-1 x t ]+b i );

[0147]

[0148] o t =σ(W o ·[h t-1 x t ]+b o );

[0149] h t =o t ·tanh(C t );

[0150] In the formula, f t For the Gate of Oblivion; i t For input gate; o t For output gate; C t For unit state; h t In hidden state; W f W i W C W o b is the weight matrix; f b i b C b o σ is the bias term; σ is the sigmoid function; x t The input vector is used. The image feature extraction layer adopts a residual network structure, and the calculation process of the residual block is as follows:

[0151] y = F(x, W) i )+x;

[0152] In the formula, y represents the residual block output; F(x, W) i ) is the residual mapping function; x is the input; W i Let be the weight matrix of the i-th layer. The audio feature extraction layer adopts a one-dimensional convolutional neural network structure, and its calculation process is as follows:

[0153]

[0154] In the formula, y j For the j-th output feature map; x i Let k be the i-th input feature map; i，j For convolution kernel; b j is the bias term; M is the number of input channels; * indicates a convolution operation. The main function of this step is to extract deep feature representations of each modality of data through a deep neural network.

[0155] The specific implementation of step S03 is as follows: Calculate the alignment function between the convolution kernel parameters of each layer in the feature extraction layer and the standardized training data. The calculation process is as follows:

[0156] A(F,D)=αMI(F,D)+(1-α)S(F);

[0157] In the formula, A(F, D) is the alignment function value; F is the feature map; D is the input data; MI(F, D) is the mutual information; S(F) is the feature similarity; α is the weight coefficient, ranging from 0 to 1. The specific calculation of mutual information is as follows:

[0158]

[0159] In the formula, p(x, y) is the joint probability distribution; p(x) and p(y) are the marginal probability distributions. Feature similarity is calculated as follows:

[0160]

[0161] In the formula, F is the feature matrix; ||·|| represents the Frobenius norm of the matrix. The main purpose of this step is to evaluate the effectiveness of feature extraction and ensure that the extracted features retain the key information of the original data.

[0162] The specific implementation of step S04 is as follows: An aligned component matrix and an unaligned component matrix are constructed based on the alignment function. The construction process is as follows:

[0163] M aligned =[F i |A(F i ,D)>θ];

[0164] M unaligned =[F i |A(F i [, D)≤θ];

[0165] In the formula, M aligned For aligning component matrices; M unalignedThis is the unaligned component matrix; θ is the threshold, set to 0.7. Then, the splitting index is calculated:

[0166]

[0167] In the formula, SI is the split index; To align the i-th singular value of the component matrix; is the i-th singular value of the unaligned component matrix; k is the number of singular values ​​to be selected from the top 20%. The main purpose of this step is to quantify the alignment of the features, providing a basis for subsequent parameter optimization.

[0168] The specific implementation of step S05 is as follows: Establish a mapping relationship between the splitting index and network parameters using a gradient-based optimization method. The process of establishing the mapping relationship includes calculating the gradient of the splitting index with respect to the parameters of each layer:

[0169]

[0170] In the formula, W represents the network parameters. Parameter updates employ an adaptive moment estimation algorithm:

[0171] m t =β1m t-1 +(1-β1)g t ;

[0172]

[0173] In the formula, m t and v t Let β1 and β2 be the first and second order momentum, respectively; β1 and β2 be the momentum decay rates; η be the learning rate; and ∈ be the numerical stability constant. The main function of this step is to improve the alignment effect of feature extraction by optimizing network parameters.

[0174] The specific implementation of step S06 is as follows: Construct a modal contribution allocation model, including a characteristic contribution function, an interaction influence function, and a time-series decay function. The calculation process of the characteristic contribution function is as follows:

[0175] C f (x)=w·x·acc hist ;

[0176] In the formula, C f (x) represents the feature contribution score; x is the feature vector; w is the attention weight; acc hist This represents the historical prediction accuracy. The historical prediction accuracy is calculated using an exponential moving average:

[0177] acc hist =βacc hist-1 +(1-β)acc t ;

[0178] In the formula, β is the smoothing factor, set to 0.9; acc t This represents the prediction accuracy at the current moment. The calculation process for the interaction effect function is as follows:

[0179] C i (x1, x2) = MI(x1, x2|y);

[0180] In the formula, C i (x1, x2) represents the interaction contribution score; MI(x1, x2|y) represents the conditional mutual information; and y represents the predicted label. The calculation process of the time-series decay function is as follows:

[0181] D(t) = e -λt ;

[0182] In the formula, D(t) is the attenuation coefficient; λ is the attenuation rate, which is positively correlated with the time interval t. The final modal contribution value is calculated as follows:

[0183] C total =β1C f +β2C i +β3D;

[0184] In the formula, C total The final contribution value is β1, β2, and β3; these are the combined weights, determined through grid search. The main purpose of this step is to evaluate the contribution of different modal features to the prediction results.

[0185] The specific implementation of step S07 is as follows: The parameters of the multi-layer convolutional neural network structure are optimized based on the established mapping relationship, using a gradient-based iterative optimization method. The objective function for parameter optimization is:

[0186] L opt =I SI +γL reg ;

[0187] In the formula, L opt To optimize the objective function; I SI For the loss based on the split index; L reg γ is the regularization term; γ is the regularization coefficient, set to 0.0001. Parameter updates employ an adaptive learning rate adjustment strategy:

[0188]

[0189] In the formula, η t Let η be the learning rate at time t; η0 be the initial learning rate, set to 0.001; and T be the total number of iterations. The main purpose of this step is to optimize the network parameters and improve the feature extraction effect.

[0190] The specific implementation of step S08 is as follows: Feature extraction is performed on the standardized training data using the optimized network parameters to obtain multimodal features. The calculation process for text feature extraction is as follows:

[0191]

[0192] In the formula, For text features; E t Word embedding sequence; W lstm These are the parameters for the Long Short-Term Memory (LSTM) network. The calculation process for image feature extraction is as follows:

[0193]

[0194] In the formula, Image features; I t Image input; W res These are the parameters of the residual network. The calculation process for audio feature extraction is as follows:

[0195]

[0196] In the formula, For audio features; M t Mel spectrum characteristics; W conv These are the parameters for the convolutional network. The main purpose of this step is to obtain the feature representations of each modality.

[0197] The specific implementation of step S09 is as follows: input the multimodal features into the modal contribution allocation model and calculate the feature contribution values. The contribution value calculation includes three aspects: self-contribution, interaction contribution, and temporal influence. The calculation process for self-contribution is as follows:

[0198]

[0199] In the formula, C self The value represents its own contribution; n is the number of features; w i These are the feature weights. The calculation process for the interaction contribution is as follows:

[0200]

[0201] In the formula, C inter The interaction contribution value. The calculation process for the time series impact is as follows:

[0202]

[0203] In the formula, C temp The value represents the temporal contribution; T represents the size of the historical time window. The main purpose of this step is to quantify the influence of each modal feature on the prediction result.

[0204] The specific implementation of step S10 is as follows: Construct an attention mechanism layer and calculate attention weights based on contribution values. The calculation process of the multi-head attention mechanism is as follows:

[0205]

[0206] In the formula, Q is the query matrix; K is the key matrix; V is the value matrix; d k Let be the dimension of the key vector. The calculation process for attention weights is as follows:

[0207]

[0208] In the formula, α ij For attention weights; e ij Attention is scored. The integration process of multi-head attention is as follows:

[0209] MultiHead(Q,K,V)=Concat(head1,...,head h W O ;

[0210] In the formula, head i W is the output of the i-th attention head; O This step outputs a mapping matrix. Its main purpose is to learn the dynamic relationships between features of different modalities.

[0211] The specific implementation of step S11 is as follows: Multimodal features are weighted and fused based on attention weights to obtain a fused feature vector. The calculation process for feature weighted fusion is as follows:

[0212]

[0213] In the formula, h fusion To fuse feature vectors; α i h is the attention weight for the i-th feature; i Let be the i-th modal feature. To enhance the expressive power of the features, a gating mechanism is introduced:

[0214] g i =σ(W g h i +b g );

[0215] h gated =g i ⊙h i ;

[0216] In the formula, g i W is the gate value. g b is the gated weight matrix; gThis represents the bias term; ⊙ indicates element-wise multiplication. Finally, residual joins and layer normalization are performed:

[0217] h final =LayerNorm(h fusion +h gated );

[0218] In the formula, h final This is the final fused feature vector. The main purpose of this step is to effectively combine features from different modalities into a unified representation.

[0219] The specific implementation of step S12 is as follows: A classifier layer is constructed using a multilayer perceptron structure. The forward propagation calculation process of the classifier is as follows:

[0220] z1 = ReLU(W1h) final +b1);

[0221] z2 = ReLU(W2z1 + b2);

[0222] z3 = W3z2 + b3;

[0223] In the formula, z1, z2, and z3 are the outputs of each layer; W1, W2, and W3 are the weight matrices; and b1, b2, and b3 are the bias terms. The calculation process for label smoothing is as follows:

[0224] y smooth = (1-∈)y+∈ / K;

[0225] In the formula, y smooth y represents the smoothed label; ∈ represents the original label; K represents the smoothing factor, set to 0.1; and K represents the number of classes. The main function of this step is to map the fused features to the class space, achieving the final classification prediction.

[0226] The specific implementation of step S13 is as follows: Calculate the loss function value between the model output and the labeled data. The calculation process for cross-entropy loss is as follows:

[0227]

[0228] In the formula, y i This is a real label; To predict probabilities, the L2 regularization loss is calculated as follows:

[0229]

[0230] In the formula, λ is the regularization coefficient; N is the number of parameters. The calculation process for adversarial training loss is as follows:

[0231] L adv =max ||δ||≤∈L(x+δ, y);

[0232] In the formula, δ represents the adversarial perturbation; ∈ represents the upper limit of the perturbation magnitude, set to 0.01. The total loss function is calculated as follows:

[0233] L total =L CE +λ1L L2 +λ2L adv ;

[0234] In the formula, λ1 and λ2 are weight coefficients. The main purpose of this step is to evaluate the training effect of the model and provide guidance for parameter optimization.

[0235] The specific implementation of step S14 is as follows: The parameters of each component of the model are optimized and updated based on the loss function value. The calculation process of the parameter gradient is as follows:

[0236]

[0237] In the formula, g t W is the gradient at time t; t These are the model parameters. The gradient clipping calculation process is as follows:

[0238]

[0239] In the formula, θ is the clipping threshold, set to 1.0. The calculation process for the learning rate adjustment is as follows:

[0240]

[0241] In the formula, η0 is the initial learning rate; γ is the decay factor, set to 0.1; and s is the decay step size. The main function of this step is to optimize the model parameters through gradient descent.

[0242] The specific implementation of step S15 is as follows: Repeat the processes of feature extraction, contribution value calculation, attention fusion, classification prediction, and parameter optimization until convergence. The convergence judgment calculation process is as follows:

[0243] ΔL=|L t -L t-1 |<∈;

[0244] In the formula, ΔL represents the change in the loss function; ∈ is the convergence threshold, set to 0.01. The calculation process for validation set performance evaluation is as follows:

[0245]

[0246] In the formula, Acc val Let N be the validation set accuracy; N be the number of samples; and I(·) be the indicator function. The criteria for saving the model are:

[0247]

[0248] In the formula, Let be the validation set accuracy at time t. The main purpose of this step is to obtain the optimal model parameters through iterative training.

[0249] To better understand and implement this invention, a specific application scenario is provided below as Example 2: A research team, in developing a new generation of multilingual interactive assistants, adopted the multimodal large language model establishment method of this invention for technical implementation. This assistant needs to simultaneously process user-input text commands, image scenes, and voice dialogues. To achieve efficient multimodal information understanding and response, the research team collected a training set containing 100,000 multimodal interaction data points. The text data includes user commands and system responses, with an average length of 50 words; the image data consists of scene images, uniformly adjusted to 224×224 pixels; and the audio data is user speech, with a sampling rate of 16000Hz.

[0250] The research team first preprocessed the training data. For text data, they used the Jieba word segmentation tool for segmentation and converted each word into a 768-dimensional vector representation using a pre-trained word embedding model. For image data, they performed normalization to scale pixel values ​​to between 0 and 1 and used data augmentation methods such as random cropping and horizontal flipping to expand the training samples. For audio data, they extracted Mel spectral features from 80 Mel filter banks, set the frame length to 25 milliseconds, and the frame shift to 10 milliseconds.

[0251] In the construction of the feature extraction network, the text feature extraction layer uses a 3-layer bidirectional long short-term memory network with 512 hidden units per layer and a dropout rate of 0.1. The image feature extraction layer uses a 152-layer residual network containing 4 residual blocks, each containing multiple 3×3 convolutional layers and a 1×1 bottleneck layer. The audio feature extraction layer uses a 6-layer one-dimensional convolutional network with a kernel size of 3, and the number of channels increases from 64 to 512 layer by layer.

[0252] The research team evaluated the feature extraction performance and calculated the alignment function values ​​shown in Table 1 below:

[0253] Table 1 Alignment Function Values

[0254] Modality type First layer Second layer Third layer Fourth layer Fifth layer Sixth layer Text 0.82 0.79 0.75 - - - Image 0.85 0.81 0.78 0.73 - - Audio 0.83 0.80 0.76 0.72 0.69 0.65

[0255] Aligned component matrices and unaligned component matrices are constructed based on the alignment function values, with an alignment threshold of 0.7. The calculated splitting indexes are 1.85 for text features, 1.92 for image features, and 1.78 for audio features.

[0256] In the modal contribution allocation model, the feature contribution function employs a weighted attention mechanism, and the smoothing factor for historical prediction accuracy is set to 0.9. The interaction effect function quantifies the interaction between modes by calculating conditional mutual information, and the decay rate of the time-series decay function is set to 0.2. The final contribution values ​​of the three modes are distributed as shown in Table 2 below:

[0257] Table 2. Distribution of Contribution Values

[0258] Interaction scenario Text contribution value Image contribution value Audio contribution value Instruction understanding 0.65 0.20 0.15 Scenario description 0.25 0.60 0.15 Voice dialogue 0.30 0.15 0.55

[0259] The attention mechanism layer employs an 8-head attention structure, with each attention head having a dimension of 96. During feature fusion, a gating mechanism is used to control information flow, with the hidden layer dimension of the gating network set to 384. The classifier layer consists of three fully connected layers with dimensions of 768, 384, and 192 respectively. Each layer is followed by a batch normalization layer and a ReLU activation function, with a dropout rate set to 0.1.

[0260] During training, a batch size of 32 was used, with an initial learning rate of 0.001, and cosine annealing was employed to adjust the learning rate. In the loss function, the cross-entropy loss weight was 1.0, the L2 regularization coefficient was 0.0001, and the adversarial training perturbation was set to 0.01. The training process data on the validation set is shown in Table 3 below.

[0261] Table 3 Training Data Table

[0262] Training round Loss value Accuracy rate 10 0.42 0.85 20 0.35 0.89 30 0.28 0.92 40 0.22 0.94 50 0.18 0.95

[0263] Model training is complete when the validation set accuracy reaches 95%. Compared to traditional multimodal fusion methods, the research team previously used a simple feature concatenation and fixed-weight weighted averaging method, which only achieved 88% accuracy on the same validation set. Traditional methods mainly suffer from the following problems: the feature extraction process lacks alignment evaluation, resulting in inconsistent feature quality across different modalities; the feature fusion process uses fixed weights, which cannot adapt to the dynamic changes in modality importance under different scenarios; and it does not consider the interactions and temporal effects between features, resulting in insufficient information utilization.

[0264] This invention achieves quality assessment and optimization of the feature extraction process by introducing an alignment function and a splitting index; it realizes dynamic evaluation of feature importance through a modal contribution allocation model; and it achieves more accurate feature fusion through multi-head attention and gating mechanisms. These innovative designs effectively solve the problem of poor fusion results caused by insufficient feature alignment, resulting in a significant improvement in the model's performance on multimodal interaction understanding tasks.

[0265] In practical applications, this multimodal interaction assistant can accurately understand users' text commands, image scenes, and voice dialogues, and make reasonable responses based on the importance of each modality in different scenarios. Especially in complex multimodal interaction scenarios, the model demonstrates strong robustness and adaptability. This improved performance is attributed to the innovative design of this invention in feature alignment optimization and modal contribution allocation.

[0266] It should be noted that the variables involved in this invention are explained in detail in Table 4 below.

[0267] Table 4. Variable Explanation Table

[0268]

[0269]

[0270] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for establishing a multimodal large language model, characterized in that, Includes the following steps: S01. Obtain text data, image data, and audio data as training input data, and preprocess the training input data to obtain standardized training data; S02. Construct a multi-layer convolutional neural network structure, and set a text feature extraction layer, an image feature extraction layer and an audio feature extraction layer in the multi-layer convolutional neural network structure, wherein the text feature extraction layer adopts a bidirectional long short-term memory network structure, the image feature extraction layer adopts a residual neural network structure, and the audio feature extraction layer adopts a one-dimensional convolutional neural network structure. S03. Calculate the alignment function between the convolution kernel parameters of each layer in the text feature extraction layer, the image feature extraction layer, and the audio feature extraction layer and the standardized training data; S04. Construct an aligned component matrix and an unaligned component matrix based on the alignment function, and calculate the splitting index between the aligned component matrix and the unaligned component matrix; S05. Establish the mapping relationship between the splitting index and the network parameters of each layer in the multi-layer convolutional neural network structure; S06. Construct a modality contribution allocation model between the multi-layer convolutional neural network structure and the attention mechanism layer. The modality contribution allocation model includes a feature contribution function, an interaction influence function, and a time-series decay function. S07. Optimize the parameters of the multi-layer convolutional neural network structure according to the mapping relationship to obtain the optimized network parameters; S08. Use the optimized network parameters to extract features from the standardized training data to obtain multimodal features; S09. Input the multimodal features into the modal contribution allocation model and calculate the contribution value of each modal feature to the model prediction result; S10. Construct an attention mechanism layer and calculate the attention weights between different modal features based on the contribution values; S11. The multimodal features are weighted and fused based on the attention weights to obtain a fused feature vector; S12. Construct a classifier layer and input the fused feature vector into the classifier layer for training; S13. Calculate the loss function value between the model output and the labeled data; S14. Optimize and update the parameters of the multi-layer convolutional neural network structure, the modality contribution allocation model, the attention mechanism layer, and the classifier layer based on the loss function value; S15. Repeat steps S08 to S14 until the loss function value is less than a preset threshold to obtain a trained multimodal large language model. The steps for calculating the alignment function are as follows: calculating the mutual information between the feature map output by each network layer and the input data; calculating the cosine similarity matrix between the feature maps; combining the mutual information and the similarity matrix to obtain the alignment function value; in the modality contribution allocation model, the feature contribution function adopts a weighted attention mechanism, which weights the feature importance according to the historical prediction accuracy. The interaction effect function quantifies the interaction between modes by calculating the mutual information and conditional mutual information between features; the time decay function adopts an exponential decay form, and the decay rate is adaptively adjusted according to the time interval.

2. The method for establishing a multimodal large language model according to claim 1, characterized in that, The preprocessing steps for the training input data are as follows: the text data is segmented and converted into 768-dimensional vector representations using word embedding technology; the image data is uniformly adjusted to 224×224 pixels and normalized to scale the pixel values ​​to between 0 and 1; the audio data is resampled, the sampling rate is uniformly adjusted to 16000Hz and Mel-spectral features are extracted.

3. The method for establishing a multimodal large language model according to claim 1, characterized in that, In the multi-layer convolutional neural network structure, the text feature extraction layer adopts a 3-layer bidirectional long short-term memory layer with 512 hidden units in each layer; the image feature extraction layer adopts a 152-layer residual network containing 4 residual blocks. The audio feature extraction layer uses 6 convolutional layers, with each layer having a kernel size of 3, and the number of channels increases from 64 to 512 layer by layer.

4. The method for establishing a multimodal large language model according to claim 1, characterized in that, Attention weights are calculated using an 8-head attention mechanism. Attention scores are obtained through dot product attention calculations, and the final attention weights are obtained by normalization using the softmax function.

5. The method for establishing a multimodal large language model according to claim 1, characterized in that, It also includes the step of constructing a classifier layer, which contains three fully connected layers with hidden layer dimensions of 768, 384, and 192 respectively. Each fully connected layer is followed by a batch normalization layer and a ReLU activation function.

6. The method for establishing a multimodal large language model according to claim 1, characterized in that, During training, a combination of cross-entropy loss and auxiliary loss is used, where the auxiliary loss includes L2 regularization and adversarial training loss. Training is terminated when the loss function value is less than 0.01 or the accuracy of the validation set reaches 95%.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions, which, when executed in a computer, are used to perform the multimodal large language model establishment method according to any one of claims 1-6.

8. A multimodal large language model building system, characterized in that, The system includes the computer-readable storage medium of claim 7, wherein the system is any one of a computer, a server, or a microcontroller, the computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor that executes the program instructions stored in the computer-readable storage medium.

Images