A multimodal sentiment analysis method and system based on pre-training and online learning

Abstract

The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method and system based on pre-training and online learning. The method comprises obtaining sentiment analysis multimodal data, constructing a sentiment analysis model, including constructing a multi-head attention mechanism to encode text, image and audio modalities respectively, and splicing and fusing the encoded modal features; using a feedforward neural network and a multi-head attention mechanism to mine the interactive relationship between the modalities in the fused features; evaluating the contribution of each modality in the joint feature to the sentiment analysis task by constructing a fully connected neural network; fusing the joint feature and the weighted features of each modality to achieve feature supplementation and optimization, and outputting the sentiment prediction result using the trained model. The present invention constructs a multi-head attention mechanism to encode each modality, splices and fuses the encoded modal features, thereby significantly improving the accuracy of sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method and system based on pre-training and online learning. Background Art

[0002] With the rapid development of smart terminals and Internet technology, multimodal data (text, images, audio) on social platforms has brought opportunities for sentiment analysis, but the lack of uncertain modalities has highlighted challenges. Early sentiment analysis mostly relied on a single modality. After the rise of multimodality, there are many existing multimodal sentiment analysis (MSA) models, but they assume that multiple modalities are complete, such as the low-rank multimodal fusion algorithm (LMF), which has achieved certain results in improving network effects. However, these models are generally built on the assumption that the three modalities of text, image and audio are always complete and available. However, in real scenarios, there are many uncontrollable factors, such as occlusion of camera equipment that makes it impossible to collect image modalities, and sudden strong noise interference that causes the loss of audio modalities. This directly leads to many MSA models based on the complete modality assumption being paralyzed in practical applications and unable to play the expected role.

[0003] At present, there are two types of methods to solve the missing modalities: generative learning and joint learning. In the category of generative learning, the modal translation method attempts to fill the modal gap by generating new data that matches the existing modal observation distribution. In the joint learning approach, the similar modality completion method is committed to finding similar modalities from the data set and working with the existing modalities to complete the missing modality completion task. However, most of these methods are heavily dependent on a large amount of full-modal data. Generative learning such as modal translation and joint learning such as similar modality completion often require a large amount of full-modal data pre-training, which is difficult to achieve in practice due to privacy, heterogeneity, and collection costs. In addition, any slight change in the model or data requires retraining, which is time-consuming and labor-intensive, and the pre-trained model has low practicality. At this stage, a multimodal sentiment analysis method and system based on pre-training and online learning is needed. Summary of the invention

[0004] In order to solve the problem that traditional models cannot perform effective multimodal sentiment analysis when the three modalities of text, image and audio are incomplete, the present invention provides a multimodal sentiment analysis method and system based on pre-training and online learning.

[0005] In a first aspect, the present invention provides a multimodal sentiment analysis method based on pre-training and online learning, which adopts the following technical solution:

[0006] A multimodal sentiment analysis method based on pre-training and online learning, including:

[0007] Acquire multimodal data for sentiment analysis, including image, audio, and text modalities, and mark missing data;

[0008] Construct a sentiment analysis model, including building a multi-head attention mechanism to encode text, image, and audio modalities respectively, and concatenate and fuse the encoded features of each modality to obtain fused features;

[0009] The feedforward neural network and multi-head attention mechanism are used to mine the interactive relationship between the modes in the fusion features to obtain the joint features;

[0010] The contribution of each modality in the joint feature to the sentiment analysis task is evaluated by constructing a fully connected neural network, and each modality is weighted according to its contribution.

[0011] The joint features and weighted features of each modality are fused to achieve feature supplementation optimization, and finally the optimized modalities are fused to complete the construction of the sentiment analysis model;

[0012] The sentiment analysis model is trained based on a combination of meta-learning and online learning, and the trained model is used to output the sentiment prediction results.

[0013] Furthermore, the multi-head attention mechanism is constructed to encode the text, image and audio modalities respectively, including converting the multimodal data into a unified input matrix after preprocessing, and performing a tight linear transformation of the input matrix by the matrix dimension parameters of the query, key and value in each head, and performing attention calculation using the scaled dot product operation to obtain the feature matrix output by each head.

[0014] Furthermore, the encoded features of each modality are concatenated and fused, including inputting the fused features into a feedforward neural network, performing deep feature extraction on the fused features by element-by-element nonlinear transformation to obtain feature representations of three modalities, and concatenating the features of each modality using a multi-head attention mechanism to obtain a fused feature matrix. The fused feature calculation formula is:

[0015] ,

[0016] in, Expressed as a fusion feature matrix, They are the text, audio, and image modality features after encoding and deep feature extraction, Represented as sequential concatenation of modal feature operations.

[0017] Furthermore, the use of the feedforward neural network and the multi-head attention mechanism to mine the interactive relationship between the modes in the fusion feature includes using the multi-head attention mechanism and the feedforward neural network to process the fusion feature matrix, and then obtain the joint feature. The calculation formula of the joint feature is:

[0018] ,

[0019] ,

[0020] in, Represented as a fusion feature matrix, including text, image, and audio modality features, Represented as a multi-head attention mechanism, It is represented as a feedforward neural network, and Z is represented as the result of the interaction after self-attention. Represented as joint features.

[0021] Furthermore, the weighting of each modality according to the contribution includes constructing a fully connected neural network architecture containing multiple neurons as a weight generation network, setting the number of input layer neurons according to the dimension of the joint feature, setting the number of output layer neurons according to the number of modalities, inputting the joint feature into the hidden layer to perform a linear transformation operation, obtaining an intermediate result vector and performing nonlinear activation using a hyperbolic tangent function, and the weight generation network calculates the weight formula of each modality as follows:

[0022] ,

[0023] in, It is represented as a modality weight matrix, which indicates the relative contribution of each modality in the fusion feature. Each column represents the weight corresponding to each modality. Expressed as an activation function, namely the hyperbolic tangent function, Represented as the hidden layer weight matrix, It is represented as the bias vector.

[0024] Furthermore, the joint feature and the weighted features of each modal are fused to achieve feature supplement optimization, including multiplying the weight of each modality with its corresponding modal feature to obtain the weighted modal feature, fusing the joint feature with each weighted single modal feature, and performing deep interaction through a fully connected layer to achieve modal content optimization.

[0025] Furthermore, the sentiment analysis model is trained in a manner based on a combination of meta-learning and online learning, including setting an initial learning rate and calculating the current learning rate of the model according to a learning rate decay formula, calculating the current batch gradient of the model based on the current learning rate using a gradient accumulation formula, and updating parameters by accumulating N batches of gradients and combining them with the current learning rate.

[0026] Furthermore, the sentiment analysis model is trained based on a combination of meta-learning and online learning, and further includes calculating the loss again and obtaining the meta-gradient after each batch of data completes the parameter update according to the loss, accumulating the meta-gradients of multiple batches, and when the meta-gradients accumulate to a preset number, entering an outer loop, using the meta-optimizer to update the global parameters according to the formula based on the accumulated multiple meta-gradients, and resetting the accumulated gradients to zero after the update.

[0027] Furthermore, the sentiment analysis model is trained based on a combination of meta-learning and online learning, and further includes performing self-attention calculations based on different modalities of the model and obtaining corresponding loss functions, adjusting the parameters in the self-attention mechanism of each modality through an optimization function, calculating the cross entropy loss within each modality using a multi-head self-attention mechanism, and finally determining the optimal weight of each modality during fusion by calculating the loss of modality weight evaluation.

[0028] In the second aspect, a multimodal sentiment analysis system based on pre-training and online learning includes:

[0029] The data acquisition module is configured to: acquire multimodal data for sentiment analysis, including acquiring image, audio and text modalities, and marking missing data;

[0030] The model module is configured to: construct a sentiment analysis model, including constructing a multi-head attention mechanism to encode text, image and audio modalities respectively, and concatenate and fuse the encoded features of each modality to obtain fused features;

[0031] The feature module is configured to: use a feedforward neural network and a multi-head attention mechanism to mine the interactive relationship between the modalities in the fusion feature to obtain the joint feature;

[0032] The weight module is configured to: evaluate the contribution of each modality in the joint feature to the sentiment analysis task by constructing a fully connected neural network, and assign a weight to each modality according to the contribution;

[0033] The feature extraction module is configured to: fuse the joint feature and the weighted features of each modality to achieve feature supplement optimization, and finally fuse the optimized modalities to complete the construction of the sentiment analysis model;

[0034] The training module is configured to: train the sentiment analysis model based on a combination of meta-learning and online learning, and output the sentiment prediction results using the trained model.

[0035] In summary, the present invention has the following beneficial technical effects:

[0036] 1. The present invention encodes each modality by constructing a multi-head attention mechanism, which can deeply mine the emotional feature associations within and between text, image and audio modalities, splice and fuse the encoded modal features, and use a feedforward neural network and a multi-head attention mechanism to further mine the interactive relationship between modalities in the fused features to obtain joint features. This multi-level, multi-stage feature processing method can comprehensively and deeply extract emotional information from multimodal data, thereby significantly improving the accuracy of sentiment analysis and providing users with more accurate sentiment judgment results.

[0037] 2. The present invention adopts a fully connected neural network method to evaluate the contribution of each modality in the joint feature to the sentiment analysis task and assign weights, so that the model can dynamically determine the importance of each modality according to the actual data situation. Through precise weight allocation, the model can more reasonably focus on key modal information when performing sentiment analysis, reduce the interference of irrelevant or minor modal information, optimize the allocation of computing resources, and further improve the efficiency and accuracy of sentiment analysis.

[0038] 3. The present invention integrates the joint features and weighted features of each modal to achieve the step of feature supplementation and optimization, which effectively integrates the advantages of multimodal information. The weighted modal features are integrated into the joint features according to their importance, which not only strengthens the emotional feature expression of important modalities, but also makes up for the defects that may be caused by the lack of modality or insufficient single modal information.

[0039] 4. The present invention is based on a training method that combines meta-learning with online learning, which enables the model to have strong adaptability and learning ability. By setting the initial learning rate and dynamically adjusting the learning rate according to the learning rate decay formula, the model can quickly adapt to new user data or data distribution changes in the early stage of training. As the training progresses, the parameters are gradually refined to avoid falling into the local optimal solution.

[0040] 5. The present invention adopts a meta-learning mechanism to accumulate sample meta-gradient information through an inner loop, and uses a meta-optimizer in an outer loop to update global parameters. It can optimize the overall performance of the model between different tasks or data batches, so that the model can quickly learn and transfer knowledge when facing new sentiment analysis tasks or changes in data distribution, effectively improving the generalization ability of the model, reducing dependence on large-scale labeled data, reducing training costs, and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the connection of sentiment analysis models in a multimodal sentiment analysis method based on pre-training and online learning in Example 1 of the present invention.

[0042] Figure 2 It is a schematic diagram of a fully connected neural network in a multimodal sentiment analysis method based on pre-training and online learning in Example 1 of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0044] Example 1

[0045] Reference Figure 1 , a multimodal sentiment analysis method based on pre-training and online learning, including:

[0046] Acquire multimodal data for sentiment analysis, including image, audio, and text modalities, and mark missing data;

[0047] Construct a sentiment analysis model, including building a multi-head attention mechanism to encode text, image, and audio modalities respectively, and concatenate and fuse the encoded features of each modality to obtain fused features;

[0048] The feedforward neural network and multi-head attention mechanism are used to mine the interactive relationship between the modes in the fusion features to obtain the joint features;

[0049] The contribution of each modality in the joint feature to the sentiment analysis task is evaluated by constructing a fully connected neural network, and each modality is weighted according to its contribution.

[0050] The joint features and weighted features of each modality are fused to achieve feature supplementation optimization, and finally the optimized modalities are fused to complete the construction of the sentiment analysis model;

[0051] The sentiment analysis model is trained based on a combination of meta-learning and online learning, and the trained model is used to output the sentiment prediction results.

[0052] Specifically, a multimodal sentiment analysis method based on pre-training and online learning includes the following steps:

[0053] S1. Obtain multimodal data for sentiment analysis, including image, audio, and text modalities, and mark missing data;

[0054] This embodiment aims to perform multimodal sentiment analysis on user-posted content on social media platforms. The sentiment analysis multimodal data covers image (represented by I), audio (represented by A) and text (represented by T) modalities. If there is a missing modality, Indicates the missing modality. For example, when the image modality is missing, the multimodal feature can be recorded as ( , A, T), the core of the research is to conduct user sentiment analysis based on such multimodal data with uncertain modal missing, and to represent the multimodal data set with uncertain modal missing.

[0055] S2. Construct an emotion analysis model, including constructing a multi-head attention mechanism to encode the text, image, and audio modalities respectively, and splicing and fusing the encoded features of each modality to obtain fused features;

[0056] As Figure 1 shown, in the scenario of emotion analysis of multi-modal movie and TV reviews, first collect user review data from movie and TV review websites. These data include text reviews, pictures related to movies and TV shows uploaded by users, and some audio reviews recorded by users. For the text modality, perform text cleaning operations, remove stop words (such as common words without actual emotional meaning like "of", "is", "in", etc.), punctuation marks, and convert the text into word vectors. Then, concatenate these word vector sequences in the order of the reviews into a two-dimensional matrix as the initial feature matrix of the text modality. The multi-modal data for emotion analysis includes three modalities:

[0057] ,

[0058] Among them, , and represent the image, audio, and text modalities respectively. For the image modality , use an image feature extraction algorithm (such as a convolutional neural network based on deep learning, such as the ResNet model) to process the image. Input the image into the ResNet model and extract the feature vector output at a specific intermediate layer (such as the second-to-last layer). The dimension of this feature vector is 2048, and use it as the initial feature representation of the image modality. Then, normalize the feature vectors of all images so that their numerical range is between [-1, 1] to form the feature matrix of the image modality.

[0059] For the audio modality , use an audio processing library (such as Librosa) to extract the acoustic features of the audio, such as calculating the Mel-frequency cepstral coefficients (MFCC) of the audio to obtain a two-dimensional MFCC feature matrix, where each frame of audio corresponds to a row of MFCC feature vectors, and the vector dimension is 13 (the extracted feature parameters can be adjusted according to actual needs).

[0060] Integrate the above-mentioned feature matrices of the text, image, and audio modalities to construct a unified input matrix after preprocessing and transformation of the multi-modal data. Assume that the dimension of the text feature matrix is [M, 300] (M is the number of reviews), the dimension of the image feature matrix is [M, 2048], and the dimension of the audio feature matrix is [M, 13]. Then, by concatenating these matrices in the column direction, obtain the dimension of the unified input matrix.

[0061] A multi-head attention mechanism is constructed to encode each modality, and the number of heads of the multi-head attention mechanism is set to 8. For each head, the query (Q), key (K), and value (V) matrix dimension parameters are all set to 64 dimensions. Taking the text modality as an example, the part of the unified input matrix corresponding to the text modality (i.e., the first 300 columns) is combined with the query matrix parameter of the text modality , key matrix parameters , value matrix parameters Perform linear transformation to obtain the query matrix of text mode , where X is the unified input matrix, key matrix Sum Matrix ,Then, the attention is calculated using the scaled dot product operation, and the calculation formula is:

[0062] ,

[0063] in, Expressed as The text modality feature matrix output by each head is obtained by concatenating the text modality feature matrices output by the 8 heads in the column direction and integrating them through a linear transformation matrix to obtain the encoded text modality feature matrix Similarly, the above multi-head attention mechanism encoding operation is performed on the image modality and audio modality to obtain the encoded image modality feature matrix and the audio modal feature matrix ,During the calculation process, the image modality and the audio modality use their corresponding ,query, key, and value matrix parameters respectively. , , and , , .

[0064] The fusion feature matrix is ​​input into the feedforward neural network (FFN), where the fusion matrix is ​​expressed as:

[0065] ,

[0066] in, They are the text, audio, and image modality features after encoding and deep feature extraction, It is expressed as a sequential concatenation of modal feature operations. FFN uses a multi-layer perceptron structure, which includes two hidden layers. The number of neurons in the first hidden layer is 1024, and the number of neurons in the second hidden layer is 512. The number of neurons in the output layer is the same as the dimension of the input feature matrix. In FFN, the input fusion feature matrix Perform element-by-element nonlinear transformation. For the first hidden layer, the input x of each neuron is linearly transformed. ,in, is the weight matrix of the first hidden layer, is the bias vector), and then the ReLU activation function is used for nonlinear activation to obtain the output of the first hidden layer. This output is used as the input of the second hidden layer, and the above linear transformation and activation operations are repeated to obtain the output of the second hidden layer. Finally, the output of the second hidden layer is passed through a linear transformation layer to obtain the feature representation of the three modalities. This feature representation integrates the deep information of each modality encoded by the multi-head attention mechanism and the complex interaction features between modalities mined by the feedforward neural network to a certain extent.

[0067] S3. Use feedforward neural network and multi-head attention mechanism to mine the interaction relationship between each modality in the fusion feature to obtain the joint feature;

[0068] According to the fusion matrix obtained by S2, the interactive relationship between each modality in the fusion feature is mined using the feedforward neural network and the multi-head attention mechanism, including using the multi-head attention mechanism and the feedforward neural network to process the fusion feature matrix, and then obtain the joint feature. The calculation formula of the joint feature is:

[0069] ,

[0070] ,

[0071] in, Represented as a fusion feature matrix, including text, image, and audio modality features, Represented as a multi-head attention mechanism, It is represented as a feedforward neural network, and Z is represented as the result of the interaction after self-attention. Represented as joint features, the final joint features are obtained through residual connection In multimodal sentiment analysis, data of different modalities (text, image, audio) contain rich and complex sentiment features. As the depth of the model increases, such as in the multi-layer nonlinear transformation process of FFN, gradient vanishing or gradient explosion problems may occur, making it difficult for the model to learn deep features. Residual connections allow the model to directly learn the difference between input and output, allowing information to be transmitted more smoothly in the network. When fusing multimodal features, residual connections retain the original fused features. information, preventing the loss of important modal information in the complex feature extraction process, helping the model to better capture the interactive relationship between modalities and improve the accuracy of sentiment analysis. Finally, layer normalization operations are added in data preprocessing and calculations at each layer within the model. The features of multimodal data have different scales and distributions. For example, the dimension and numerical range of text word vectors, the size of image feature vectors, and the value range of audio features are different. If normalization is not performed, during model training, different modal features will have different degrees of influence on the update of model parameters, resulting in unstable training and difficulty in converging to the optimal solution. Layer normalization maps the data to a uniform scale by standardizing the input data of each layer, so that the model can more effectively learn the relationship between different modal features.

[0072] S4. Evaluate the contribution of each modality in the joint feature to the sentiment analysis task by constructing a fully connected neural network, and assign weights to each modality according to the contribution;

[0073] like Figure 2 As shown in the figure, a fully connected neural network is constructed as a weight generation network. The network has an input layer, a hidden layer, and an output layer. The dimension sets the number of neurons in the input layer to ensure that the joint feature information is fully received. The number of neurons in the output layer matches the number of modalities, and the corresponding neuron output weights are set for each modality. When the network is initialized, the weight matrix and bias vector of each layer are initialized according to a specific distribution (such as random normal distribution) to prepare for subsequent learning.

[0074] Input Joint Features To the weight generation network, perform linear transformation operations on the hidden layer. According to the formula Calculate, where is the hidden layer weight matrix, whose dimension is Determined with the network architecture, the element value represents the initial connection strength between each dimension of the joint feature and the modal weight; The bias vector is used to provide a translation to assist the network in fitting data. The intermediate result vector is obtained through linear transformation, and the linear mapping relationship between the joint features and the modal weights is preliminarily mined.

[0075] Apply the hyperbolic tangent function to the intermediate result vector for nonlinear activation, according to the formula calculate. The function maps the input to the interval (-1, 1), introduces nonlinear factors, explores the deep, nonlinear mapping relationship between joint features and modal weights, accurately captures the impact of complex emotional features of multimodal data on modal weights, such as processing the impact of text metaphor semantics, special visual elements of images, and complex rhythm of audio on weights, improves the accuracy and rationality of weight calculation, and has a great impact on economics. The result after function processing is normalized by Softmax function according to the formula:

[0076] ,

[0077] in, It is represented as a modality weight matrix, which indicates the relative contribution of each modality in the fusion feature. Each column represents the weight corresponding to each modality. Expressed as an activation function, namely the hyperbolic tangent function, Represented as the hidden layer weight matrix, Represented as a bias vector, the Softmax function converts the input into a modal weight vector W with probability values ​​in the interval (0, 1) and a sum of 1, which intuitively presents the relative contribution of each modality in the multimodal sentiment analysis task. The model reasonably allocates attention and computing resources in subsequent operations based on this weight vector, focuses on key modal information, improves the overall sentiment analysis effect, and adapts to the needs of various multimodal sentiment scenarios. For example, it accurately weighs modal weights in multimodal sentiment analysis applications such as social media, film and television reviews, and intelligent customer service, thereby enhancing sentiment analysis performance and reliability.

[0078] S5, fusing the joint features and the weighted features of each modality to achieve feature supplementation optimization, and finally fusing the optimized modalities to complete the construction of the sentiment analysis model;

[0079] The joint feature and the weighted features of each modal are fused to achieve feature supplement optimization, including multiplying the weight of each modal with its corresponding modal feature to obtain the weighted modal feature, fusing the joint feature with each weighted single modal feature, and performing deep interaction through the fully connected layer to achieve modal content optimization. In order to make up for the information loss caused by the missing modality and expand the information of the non-missing modality, it is proposed to use a joint feature module to optimize each modality.

[0080] First, the features of each modality are multiplied by the corresponding weights obtained from the neural network evaluation to obtain the weighted modality features, so that the modality with greater contribution and more relevant to the current emotional task can play a greater role. The weighted formula for each modality is expressed as:

[0081] ,

[0082] ,

[0083] ,

[0084] in, They are the text, audio, and image modality features after encoding and deep feature extraction, , and Indicates the corresponding weights of text, audio, and image modalities.

[0085] Then, the joint feature is fused with each weighted single-modal feature, and the modal content is optimized through deep interaction through the fully connected layer. The optimization formula is:

[0086] ,

[0087] ,

[0088] ,

[0089] in, , and Respectively represent the weighted text, audio, and image modality features, It is represented as a joint feature. Finally, the optimized modes are fused to further explore the interaction of each mode information.

[0090] ,

[0091] in, , and Respectively represent the optimized text, audio, and image modality features, It is represented as the joint features that are spliced ​​and fused again. This step is a further integration of the previous modalities after fusion and optimization with the joint features. It comprehensively summarizes the emotional information of each modality after weighting, fusion and optimization, realizes the deep interaction and coordination of each modal information, and constructs a final feature representation that integrates the advantages of multimodality, condenses the rich emotional clues of each modality, and has deep correlations between modalities. This final feature can provide a high-discrimination, high-information-density feature basis for subsequent sentiment analysis tasks (such as sentiment classification, sentiment scoring, etc.), more effectively capture and analyze the emotional information in multimodal data, and improve the overall performance and accuracy of multimodal sentiment analysis to adapt to complex and diverse multimodal sentiment analysis application scenarios, such as accurate analysis of user emotional connotations in scenarios such as social media sentiment monitoring and film and television multimodal evaluation.

[0092] S6. Train the sentiment analysis model based on a combination of meta-learning and online learning, and use the trained model to output sentiment prediction results.

[0093] In the multimodal sentiment analysis task, due to the uncertainty of the data source and the dynamic changes of user preferences, the model needs to have the ability of online learning to cope with the ever-changing data distribution. To meet this requirement, this paper proposes an online learning module based on the combination of meta-learning and online learning. The online learning module includes adaptive learning rate adjustment, gradient accumulation and step size update, improved meta-learning and sample weight mechanism. The following is a specific description of this embodiment.

[0094] First, in each use, the model uses the current batch of data to calculate the loss and dynamically adjusts the learning rate according to the change in the loss, thereby achieving adaptive learning rate adjustment. Specifically, the initial learning rate is set to , It decreases as the number of training steps increases:

[0095] ,

[0096] in, is the attenuation factor of the learning rate, t is the current number of steps, and T is the preset step interval. This formula ensures that the learning rate gradually decreases over time, so that it can quickly adapt to new users in the early stage of use, and can adjust model parameters more accurately in subsequent use. In addition, through the evaluation of the loss, if the current loss fails to improve significantly, the model will appropriately increase the learning rate to allow for greater exploration. When the loss does not improve for several consecutive times, the learning rate will be halved to prevent the model from falling into a local optimum.

[0097] In the gradient accumulation part, considering the real-time requirements of online learning, the model adopts a strategy combining gradient accumulation with step size update, that is, The accumulated gradients are used to update the model parameters in batches. The specific update formula is:

[0098] ,

[0099] in, are model parameters, is the loss function of the i-th batch of data, is the parameter value of the model at the tth iteration, Expressed as learning rate, this gradient accumulation strategy can effectively reduce the fluctuation of each parameter update and improve the stability of online learning.

[0100] In order to further improve the generalization ability and adaptability of the model so that the model can be better continuously trained and updated, an improved meta-learning is proposed. The goal of the improved meta-learning is to optimize the initial parameters of the model so that it can quickly learn on new tasks or data. Meta-learning is divided into an inner loop and an outer loop. In the inner loop, after each batch of data completes the parameter update according to the loss, the loss will be calculated once and the relevant gradient, namely the meta-gradient, will be obtained, but the parameters will no longer be updated, and then the gradients of multiple batches will be accumulated. When the preset number is accumulated, the outer loop is performed, and the meta-optimizer is used to update the global parameters according to the accumulated multiple meta-gradients, and after the update, the accumulated gradients are reset to zero so that new gradients can be accumulated next time. The inner loop focuses on the rapid adaptation of the model to each small batch of samples, while the outer loop is responsible for optimizing the overall performance of the model on multiple tasks. This inner and outer loop structure enables the model to learn quickly in complex tasks and maintain good generalization ability.

[0101] Specifically, assuming that the input data of the model is a multimodal dataset ,in Indicates The sample modal input, Indicates the corresponding sentiment label, represents the number of modes, is the total number of samples, and the meta-loss is obtained through the standard forward propagation process of the model, and its loss function is defined as the cross entropy loss:

[0102] ,

[0103] in, represents the final sentiment classification generated, Represented as the true sentiment label,

[0104] Let's assume that The parameters of the step are , the meta-loss of the current data is , then the meta-gradient is:

[0105] ,

[0106] in, is the meta-loss of the current data, and the formula for accumulating meta-gradients of multiple batches is:

[0107] ,

[0108] in, Represented as meta-gradient, next, the global parameters are updated based on the accumulated gradient :

[0109] ,

[0110] in, is the meta-learning rate. Finally, in order to avoid the catastrophic forgetting problem in online learning, a sample weight mechanism is introduced to assign a weight to each user sample used. , whose initial value is set to 1 and gradually increases over time:

[0111] ,

[0112] in, is the time step at which the model starts online learning, is the total number of training time steps.

[0113] The loss function based on sample weights is:

[0114] ,

[0115] in, It is expressed as the task loss of the sample, so that the new and old samples will have different weight values ​​to represent the different attention of the new and old samples, and the weight values ​​are not too different to better balance the new and old samples. Finally, when updating the parameters, the weights of the new and old samples are integrated to better balance the contributions of the new and old samples, thereby preventing the model from excessively forgetting the previously learned knowledge.

[0116] The total loss of the model consists of multiple modules, including multimodal feature extraction, attention mechanism, and multi-task loss calculation. In the multimodal fusion part of the model, the features of the three modalities of image, audio, and text are used. Each modality is enhanced through the self-attention mechanism and the corresponding loss is calculated. Specifically, the model performs self-attention calculations on images, audio, and text respectively, and its loss can be expressed as:

[0117]

[0118] in, is the loss of the ith mode, is the corresponding weight, which is automatically learned through the soft attention mechanism. Within each modality, we use a multi-head self-attention mechanism to capture the contextual information within the modality. This mechanism generates feature vectors with better semantic expression by performing query, key and value operations on the features of different modalities. The loss of the self-attention mechanism is calculated by cross entropy and is defined as:

[0119]

[0120] in, represents the number of heads of multi-head attention, Indicates In order to reasonably allocate weights during multimodal fusion, this embodiment proposes a neural network evaluation module, whose loss is calculated by the following formula:

[0121]

[0122] in, is the actual weight distribution, is the modality weight predicted by the model. By optimizing this loss, we can obtain the optimal weight of each modality during fusion.

[0123] In the optimization process, in order to prevent overfitting of model parameters, an L2 regularization term is added to the loss function. L2 regularization adds a square sum penalty term to all trainable parameters of the model in the loss function. The specific definition is:

[0124]

[0125] in, Indicates the model trainable parameters, which can prevent the model from overfitting the training data by controlling the strength of the regularization term. Finally, the total loss function of the model is the weighted sum of each sub-loss, that is:

[0126]

[0127] in, is the weight hyperparameter of the regularization term. After a series of processing within the above model, the output layer of the model will eventually generate a sentiment prediction result for each input sample. If it is a sentiment classification task, the output layer uses the softmax function to convert the result calculated by the model into a probability distribution form of different sentiment categories (such as "positive", "negative", "neutral", etc.).

[0128] Example 2

[0129] The difference between this embodiment and embodiment 1 is that this embodiment provides a multimodal sentiment analysis system based on pre-training and online learning, including:

[0130] The data acquisition module is configured to: acquire multimodal data for sentiment analysis, including acquiring image, audio and text modalities, and marking missing data;

[0131] The model module is configured to: construct a sentiment analysis model, including constructing a multi-head attention mechanism to encode text, image and audio modalities respectively, and concatenate and fuse the encoded features of each modality to obtain fused features;

[0132] The feature module is configured to: use a feedforward neural network and a multi-head attention mechanism to mine the interactive relationship between the modalities in the fusion feature to obtain the joint feature;

[0133] The weight module is configured to: evaluate the contribution of each modality in the joint feature to the sentiment analysis task by constructing a fully connected neural network, and assign a weight to each modality according to the contribution;

[0134] The feature extraction module is configured to: fuse the joint feature and the weighted features of each modality to achieve feature supplement optimization, and finally fuse the optimized modalities to complete the construction of the sentiment analysis model;

[0135] The training module is configured to: train the sentiment analysis model based on a combination of meta-learning and online learning, and output the sentiment prediction results using the trained model.

[0136] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method based on pre-training and online learning, characterized in that: include: Acquire multimodal data for sentiment analysis, including acquiring image, audio, and text modalities, and marking missing data; The text, image and audio modalities are encoded and processed separately through the multi-head attention mechanism to generate fusion features; The interactive relationship between each modality in the fusion feature is spliced ​​by using the feedforward neural network and the multi-head attention mechanism to obtain the joint feature, including inputting the fusion feature into the feedforward neural network, performing deep feature extraction on the fusion feature through element-by-element nonlinear transformation, obtaining the feature representation of the three modalities, and using the multi-head attention mechanism to splice the features of each modality to obtain the fusion feature matrix. The calculation formula of the fusion feature matrix is: ， in, Expressed as a fusion feature matrix, They are the text, audio, and image modality features after encoding and deep feature extraction, It is represented as sequential concatenation of modal feature operations; The contribution of each modality in the joint feature to the sentiment analysis task is evaluated by a fully connected neural network, and a weight is assigned to each modality according to the contribution, wherein the weight is assigned to each modality according to the contribution, including constructing a fully connected neural network architecture including multiple neurons as a weight generation network, setting the number of input layer neurons according to the dimension of the joint feature, setting the number of output layer neurons according to the number of modalities, inputting the joint feature into the hidden layer to perform a linear transformation operation, obtaining an intermediate result vector, and performing nonlinear activation using a hyperbolic tangent function; The joint feature and the weighted features of each modality are fused to achieve feature supplementation optimization and generate a sentiment analysis model, including multiplying the weight of each modality with its corresponding modality feature to obtain the weighted modality feature, fusing the joint feature with each weighted single modality feature, and performing deep interaction through the fully connected layer to achieve modality content optimization; The sentiment analysis model is trained based on a combination of meta-learning and online learning, and the trained model is used to output the sentiment prediction results.

2. According to claim 1, a multimodal sentiment analysis method based on pre-training and online learning is characterized in that: The text, image and audio modalities are encoded and processed respectively through a multi-head attention mechanism, including a unified input matrix converted by preprocessing of multimodal data, a tight linear transformation of the input matrix by matrix dimension parameters of queries, keys and values ​​in each head, and attention calculation using a scaled dot product operation to obtain a feature matrix output by each head.

3. The multimodal sentiment analysis method based on pre-training and online learning according to claim 1, characterized in that: The method uses a feedforward neural network and a multi-head attention mechanism to splice the interaction relationship between the modes in the fusion feature, and also includes using the multi-head attention mechanism and the feedforward neural network to process the fusion feature matrix, thereby obtaining a joint feature. The calculation formula of the joint feature is: ， ， in, Represented as a fusion feature matrix, including text, image, and audio modality features, Represented as a multi-head attention mechanism, It is represented as a feedforward neural network, and Z is represented as the result of the interaction after self-attention. Represented as joint features.

4. The multimodal sentiment analysis method based on pre-training and online learning according to claim 1, characterized in that: The sentiment analysis model is trained by combining meta-learning with online learning, including setting an initial learning rate and calculating the current learning rate of the model according to a learning rate decay formula, calculating the current batch gradient of the model based on the current learning rate using a gradient accumulation formula, and updating parameters by accumulating N batches of gradients and combining them with the current learning rate.

5. The multimodal sentiment analysis method based on pre-training and online learning according to claim 4 is characterized in that: The sentiment analysis model is trained in a manner based on a combination of meta-learning and online learning, and further includes calculating the loss again and obtaining a meta-gradient after each batch of data completes parameter update according to the loss, accumulating meta-gradients of multiple batches, entering an outer loop when the meta-gradients accumulate to a preset number, using a meta-optimizer to update global parameters according to a formula based on the accumulated multiple meta-gradients, and resetting the accumulated gradients to zero after the update.

6. The multimodal sentiment analysis method based on pre-training and online learning according to claim 5, characterized in that: The sentiment analysis model is trained by combining meta-learning with online learning, and also includes performing self-attention calculations based on different modes of the model and obtaining corresponding loss functions, adjusting the parameters in the self-attention mechanism of each mode through an optimization function, calculating the cross entropy loss within each mode using a multi-head self-attention mechanism, and finally determining the optimal weight of each mode during fusion by calculating the loss of modality weight evaluation.

7. A multimodal sentiment analysis system based on pre-training and online learning, executing the method as claimed in claim 1, characterized in that: include: The data acquisition module is configured to: acquire multimodal data for sentiment analysis, including acquiring image, audio and text modalities, and marking missing data; The model module is configured to: encode the text, image and audio modalities separately through the multi-head attention mechanism to generate fusion features; The feature module is configured to: use a feedforward neural network and a multi-head attention mechanism to splice the interaction relationship between the modalities in the fusion feature to obtain a joint feature; The weight module is configured to: evaluate the contribution of each modality in the joint feature to the sentiment analysis task through a fully connected neural network, and assign a weight to each modality according to the contribution; The feature extraction module is configured to: fuse the joint feature and the weighted features of each modality to achieve feature supplementation optimization and generate a sentiment analysis model; The training module is configured to: train the sentiment analysis model based on a combination of meta-learning and online learning, and output the sentiment prediction results using the trained model.

Images