A sentiment analysis method and system based on single-modal and multi-modal joint training

Abstract

The present invention discloses a sentiment analysis method and system based on joint training of unimodal and multimodal methods, and relates to the technical field of multimodal sentiment analysis. In order to take into account the interaction and independence of multimodal methods, the present invention proposes a fusion strategy called JTUM, which includes three unimodal tasks and multimodal tasks. In unimodal tasks, a unimodal label generation module is used to obtain independent unimodal supervision and learn unimodal representation. In multimodal tasks, a crossmodal transformer is used to learn the interaction between multimodal sequences. Finally, a self-attention mechanism is used to collect time information to optimize the prediction results. The method of the present invention pays attention to both the characteristics of a single modality and the interaction between cross-modalities, improves the robustness of the prediction model, and can obtain more accurate sentiment analysis results, providing important technical assistance for the application of big data processing technologies such as platform intelligent management and interest recommendation.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a sentiment analysis method and system based on single-modality and multimodal joint training. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the proliferation of smart products and the rapid development of social media, human expression is no longer limited to text. More and more people are using visual and audio to express their emotions, leading to an explosive growth in multimodal data. Multimodal sentiment analysis aims to predict the emotions expressed in multimodal data using multimodal signals such as text, visuals, and audio. Multimodal sentiment analysis has important applications in areas such as social platforms, e-commerce platform evaluation, and interest recommendation. Therefore, the key to multimodal sentiment analysis is how to effectively integrate multiple modalities.

[0004] In previous studies, researchers have proposed some fusion methods from multiple perspectives. Fusion methods based on simple operations such as splicing, concatenation or weighting, fusion methods based on attention mechanisms, and fusion methods based on tensors. However, these methods lack the common attention to multimodal interaction information and single-modal independent information. To address this problem, Hazarika et al. designed a framework that includes two subspaces: modality-invariant and modality-specific representations. The former is used to learn the commonalities of multiple modalities, and the latter is used to learn the unique features of each modality. However, subspace differences cannot fully represent the differences between single modalities. Yu et al. proposed a unimodal label generation module to obtain unimodal modality supervision to learn the unique information of a single modality, but for multimodal fusion, only direct splicing technology was used, and the interaction between multiple modalities was not fully considered.

[0005] Therefore, how to provide a multimodal fusion method that takes into account the differences between modalities and the interactions between modalities has become one of the urgent problems to be solved in the existing multimodal sentiment analysis process. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a sentiment analysis method and system based on single-modal and multi-modal joint training. In order to take into account the interaction and independence of multi-modality, a fusion strategy including three single-modal tasks and multi-modal tasks is proposed, so that the model pays attention to both the characteristics of a single modality and the interaction between modalities, achieving more accurate sentiment analysis and providing important technical assistance for the application of big data processing technologies such as platform intelligent management and interest recommendation.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A first aspect of the present invention provides a sentiment analysis method based on single-modality and multi-modality joint training, comprising the following steps:

[0009] Obtain video clips and extract the three modalities of text, audio, and vision; encode the three modalities separately to obtain separate feature vector representations;

[0010] According to the independence and interactivity of modalities, single-modal tasks and multi-modal tasks are established respectively;

[0011] The feature vectors of the three modalities are input into the unimodal task to obtain the unimodal label of each modality;

[0012] The feature vectors of the three modalities are input into the multimodal task for two-to-two modal interaction to obtain the modal representations after three cross-modal interactions;

[0013] The modal representations after three cross-modal interactions are combined with the self-attention mechanism to output the prediction results of the multimodal task;

[0014] The unimodal task and the multimodal task are jointly trained under the supervision of the unimodal labels and the multimodal labels, and the prediction results output by the multimodal task after training are used as the final results of sentiment analysis.

[0015] Furthermore, the specific steps of encoding the three modes to obtain separate feature vector representations are as follows:

[0016] For text modality, a 12-layer BERT is used, and the head word vector output by the last layer is used as the text modality feature vector representation;

[0017] For the visual and audio modalities, we first use a feature extractor to extract features, then use stacked LSTMs to encode the visual and audio modalities separately. Finally, we use the first word vector of the last layer as the representation of the entire sequence to obtain the visual modality feature vector representation and the audio modality feature vector representation.

[0018] Furthermore, unimodal tasks and multimodal tasks share the encoded feature vector representation.

[0019] Furthermore, the specific steps for inputting the feature vectors of the three modalities into the unimodal task to obtain the unimodal label of each modality are as follows:

[0020] Input the feature vectors of the three modalities into the fully connected layer;

[0021] Obtain unimodal prediction results and unimodal representation through linear regression;

[0022] Use ULGM to get the unimodal label based on the unimodal prediction results.

[0023] Furthermore, the specific steps for using ULGM to obtain unimodal labels based on unimodal prediction results are as follows:

[0024] Calculate the center points of the three single-modal representations and one fusion modality representation of positive and negative samples respectively;

[0025] Calculate the relative distance between each modal representation and the center point of the positive and negative samples;

[0026] The ratio of the single-modal label to the multi-modal label is obtained according to the relative distance;

[0027] The offset is calculated based on the ratio of the unimodal label to the multimodal label to obtain the unimodal label.

[0028] Furthermore, the feature vectors of the three modalities are input into the multimodal task for pairwise modal interaction. The specific steps to obtain the modal representations after the three cross-modal interactions are as follows:

[0029] The feature vectors of the three modalities are transformed through the fully connected layer;

[0030] The transformed feature vector representation is input into the crossmodal transformer to obtain the interaction information between the modalities;

[0031] The outputs of crossmodal transformers are concatenated to obtain three modal representations after cross-modal interaction.

[0032] Furthermore, the specific steps of using the modal representations after three cross-modal interactions combined with the self-attention mechanism to output the prediction results of the multimodal task are as follows:

[0033] The modal representations of the three cross-modal interactions are fused into a joint vector representation using a self-attention mechanism;

[0034] The prediction results of multimodal tasks are obtained through the fully connected joint vector representation.

[0035] Furthermore, the specific steps for fusing the three modal representations after cross-modal interaction into a joint vector representation using the self-attention mechanism are as follows:

[0036] A self-attention mechanism is used to collect temporal information of each modality sequence;

[0037] Extract the last element in each sequence and concatenate them;

[0038] The concatenated results are projected into a lower-dimensional space to obtain a multimodal fusion representation.

[0039] Furthermore, the specific steps for jointly training unimodal tasks and multimodal tasks under the supervision of unimodal labels and multimodal labels are as follows:

[0040] Take loss as the optimization target and set the loss function;

[0041] The difference between unimodal and multimodal labels is used as the weight of the loss function;

[0042] The loss function is solved according to the prediction results of the multimodal task, the multimodal labels, the unimodal prediction results and the unimodal labels, so that the loss function is close to the expected value, thereby completing the joint training process.

[0043] A second aspect of the present invention provides a sentiment analysis system based on single-modality and multi-modality joint training, comprising:

[0044] A modality encoding module is configured to take a video clip and extract the three modalities: text, audio, and visual; and encode each of the three modalities to produce a separate feature vector representation.

[0045] A task division module is configured to establish single-modal tasks and multi-modal tasks according to the independence and interactivity of modalities respectively;

[0046] The unimodal task module is configured to input the feature vectors of the three modalities into the unimodal task to obtain the unimodal label of each modality;

[0047] The multimodal task module is configured to input the feature vectors of the three modalities into the multimodal task for pairwise modal interaction, and obtain the modal representations after the three cross-modal interactions;

[0048] The prediction module is configured to use the modal representations after the three cross-modal interactions combined with the self-attention mechanism to output the prediction results of the multimodal task;

[0049] The joint training module is configured to jointly train unimodal tasks and multimodal tasks under the supervision of unimodal labels and multimodal labels, and use the prediction results output by the multimodal task after training as the final results of sentiment analysis.

[0050] One or more of the above technical solutions have the following beneficial effects:

[0051] The present invention discloses a sentiment analysis method based on joint training of unimodal and multimodal methods. In order to take into account the interaction and independence of multimodal methods, a fusion strategy called JTUM is proposed, which includes three unimodal tasks and multimodal tasks. In unimodal tasks, a unimodal label generation module is used to obtain independent unimodal supervision and learn unimodal representation. In multimodal tasks, a crossmodal transformer is used to learn the interaction between multimodal sequences. Finally, a self-attention mechanism is used to collect temporal information to optimize the prediction results. The method of the present invention pays attention to both the characteristics of a single modality and the interaction between modalities, improves the robustness of the prediction model, and can obtain more accurate sentiment analysis results, providing important technical assistance for the application of big data processing technologies such as existing platform intelligent management and interest recommendation.

[0052] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0054] Figure 1 This is a flow chart of a sentiment analysis method based on single-modality and multi-modality joint training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0056] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "comprising" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations;

[0057] Explanation of terms:

[0058] JTUM: Joint Training Strategy of Unimodal and Multimodal

[0059] ULGM: Unimodal Label Generation Module

[0060] LSTM: long short-term memory, Chinese interpretation: long short-term memory network

[0061] BERT: Bidirectional Encoder Representation from Transformers

[0062] CMTRM: crossmodal transformer, Chinese meaning: cross-modal encoder.

[0063] FC: fully connected layers, Chinese interpretation: fully connected layers

[0064] ReLU: Rectified Linear Unit, Chinese meaning: linear rectification unit

[0065] Leaky ReLU: Leaky version of a Rectified Linear Unit, Chinese interpretation: Leaky linear rectifier unit

[0066] Example 1:

[0067] With the development of society, the ways and means of people's emotional expression have gradually diversified, resulting in the emergence of many industries that utilize sentiment analysis, such as psychological counseling. For example, the emotional polarity of opinions can be analyzed based on people's discussions on a hot topic on social platforms. The advancement of science and technology has made sentiment analysis more intelligent and precise. For example, people's emotions are gradually used as one of the bases in big data processing technologies such as interest recommendation and platform intelligent management. However, human emotions are complex, and their expression methods are also diverse. The existing technology is based on one-sidedness for sentiment analysis, resulting in low accuracy of analysis results. The differences between modalities in the multimodal analysis process make it difficult to fuse multiple modalities, which increases the difficulty of sentiment analysis. Therefore, how to effectively fuse multiple modalities is the key to multimodal sentiment analysis. Based on this, embodiment 1 of the present invention provides a sentiment analysis method based on joint training of single modality and multimodality, such as Figure 1As shown in the figure, given a video clip, including three modalities: text, audio, and vision, the emotional intensity in the video is analyzed through these three modalities. A feature extractor is used to extract vector features of vision and audio. The three modalities are then encoded to obtain corresponding vector representations, which are shared by both multimodal tasks and unimodal tasks in the modal fusion part. In the multimodal task, the three modal representations obtained by encoding are projected, dimensionally transformed, and then modeled by the crossmodal transformer (CMTRM) and projected into a low-dimensional space. Then, the three modal representations are connected to obtain the fusion result. In the unimodal task, the three modal representations are projected and linearly regressed, and the ULGM module guides the training process. Finally, the multimodal and unimodal tasks are jointly learned to make the final sentiment score prediction. Among them, the input of the overall model is a unimodal sequence .in Represents one of the sets, t, v, and a represent the three modalities of text, vision, and audio respectively. represents the length of the modal m sequence, The dimension of the feature vector representing the modality m. The output of the model is a result reflecting the intensity of emotion. ,use To predict the final results.

[0068] The specific steps include:

[0069] Step 1: Obtain a video clip and extract the three modalities of text, audio, and vision; encode the three modalities separately to obtain separate feature vector representations;

[0070] Step 2: Establish single-modal tasks and multi-modal tasks based on the independence and interactivity of modalities respectively;

[0071] Step 3: Input the feature vectors of the three modalities into the unimodal task to obtain the unimodal label of each modality;

[0072] Step 4: Input the feature vectors of the three modalities into the multimodal task for pairwise modal interaction to obtain the modal representations after the three cross-modal interactions;

[0073] Step 5: Use the modal representations after the three cross-modal interactions and combine them with the self-attention mechanism to output the prediction results of the multimodal task;

[0074] In step 6, the unimodal task and the multimodal task are jointly trained under the supervision of the unimodal labels and the multimodal labels, and the prediction results output by the multimodal task after training are used as the final results of sentiment analysis.

[0075] In step 1, the input of each mode Encode and obtain their feature representation :

[0076] For text modality, a 12-layer BERT is used, and the head word vector output by the last layer is used as the text modality feature vector representation ;

[0077] For visual and audio modalities, we first use the feature extractor to extract features, then use the stacked LSTM to encode the visual and audio modalities separately, and finally, use the first word vector of the last layer as the representation of the entire sequence to obtain the visual modality feature vector representation. and audio modal feature vector representation .

[0078] The feature encoding of each modality is expressed as follows:

[0079] .

[0080] in, Indicates the parameters assigned by BERT to t mode, Indicates the parameters assigned by LSTM to the v mode, Indicates the parameters assigned to LSTM for mode a.

[0081] In step 3, in order to guide the training process of the unimodal task, this embodiment uses ULGM to obtain the unimodal label, and learns the unimodal representation through the unimodal label, so as to better learn the unique information of the unimodality and highlight the differences of the unimodality. The unimodal training task and the multimodal interaction task share the encoded feature representation. , , .

[0082] The specific process of obtaining a unimodal label is as follows:

[0083] First, the feature vectors of the three modalities are input into the fully connected layer to reduce the dimensionality difference between the modalities;

[0084] Then, the unimodal prediction results are obtained through linear regression and unimodal representation :

[0085]

[0086] Where u∈{t,v,a}, Eu represents one of the obtained Et, Ev, and Ea. W represents the weight matrix weight, and b represents the bias.

[0087] Finally, ULGM is used to obtain the unimodal label based on the unimodal prediction results. :

[0088] .

[0089] Among them, ULGM contains three inputs, the multimodal labels originally included in the dataset , multimodal fusion representation , unimodal representation Its main purpose is to use multimodal labels and modal representations to generate unimodal supervision values, and to obtain unimodal labels by calculating the offset of unimodal labels relative to multimodal labels.

[0090] The specific steps are:

[0091] Calculate the center points of three single-modal representations and one fusion modality representation of positive and negative samples respectively and ( );

[0092] Calculate Distance to the center of the two polar points , ;

[0093] according to and The relative distance from the modal representation to the positive and negative centers can be obtained ; Unimodal tags With multimodal labels The ratio can be seen as and The ratio of . , ,but , ,

[0094] This results in a unimodal tag:

[0095]

[0096] In step 4, to balance consistency and diversity in multimodal fusion, this example adds a crossmodal transformer to learn pairwise modal interactions in multimodal tasks. Combining the crossmodal transformer with ULGM optimizes multimodal fusion results by jointly training unimodal and multimodal tasks. The crossmodal transformer can focus on long-term dependencies between modalities, learn data interactions between multimodal sequences across different time steps, and perform efficient multimodal fusion.

[0097] The specific steps to obtain the modal representation after three cross-modal interactions are:

[0098] First, in order to adapt to the calculation of the crossmodal transformer, the feature vectors of the three modes are , , The features are transformed through the fully connected layer to obtain the transformed feature vector :

[0099] .

[0100] Then, the converted feature vector representation is input into the crossmodal transformer to obtain the interaction information between the modalities. Specifically, one modality is selected as the target modality each time, and the other two modalities are used as the source modalities. In this way, three groups of crossmodal transformers can be divided. Each group of crossmodal transformers strengthens the target modality by repeatedly using the low-level features of the two source modalities, thereby learning the attention across the two modal features.

[0101] The core of the Crossmodal transformer is the crossmodal attention blocks. For example, text t is used as the target modality and audio a is used as the source modality. Assuming there are L layers of blocks, is the input of the text modality of the i-th layer (i=0,1,2,…,L), which is the output of the i-1-th layer block and then normalized. is the input of the audio modality, which is the source modality feature representation obtained by layer normalization. The crossmodal attention from the source modality a to the target modality t can be expressed as :

[0102]

[0103] The Query, Keys, and Values ​​of the attention mechanism are defined as , , , , , Represents weight.

[0104] express In the multi-head version of the i-th layer, the calculation formula of the crossmodal transformer is:

[0105]

[0106] Similarly, we can also derive the output of the crossmodal transformer when text t is the target modality and vision v is the source modality:

[0107] .

[0108] Finally, the outputs of the crossmodal transformers from the shared target modality are concatenated to obtain the three modal representations after cross-modal interaction:

[0109] .

[0110] In step 5, the specific steps of using the modal representations after three cross-modal interactions combined with the self-attention mechanism to output the prediction results of the multimodal task are as follows:

[0111] After concatenating the outputs of the crossmodal transformers from the shared target modality, the three modal representations after cross-modal interaction are , , A self-attention mechanism is used to fuse the outputs into a joint vector representation for final prediction. This embodiment designs a simple fusion mechanism in which each output performs a transformer-based self-attention. The self-attention mechanism can capture the long-range dependency features in each modal sequence and collect temporal information to aid prediction.

[0112] After using the self-attention mechanism to collect the time information of each modal sequence, the last element in each sequence is extracted And connect them; project the connected results into a lower dimensional space to obtain a multimodal fusion representation :

[0113] .

[0114] The prediction results of multimodal tasks are obtained through the fully connected joint vector representation :

[0115] .

[0116] According to the prediction results The size of the value is used to judge the intensity of the emotion from negative to positive, thereby completing the analysis of the emotion.

[0117] In step 6, in the joint training phase, this embodiment jointly learns the unimodal task and the multimodal task under the supervision of the unimodal label and the multimodal label. It should be noted that the unimodal task only exists in the training phase. Figure 1 The dotted part below only as the final predicted value.

[0118] The loss function is set as the optimization objective; the difference between the unimodal label and the multimodal label is used as the weight of the loss function; the loss function is solved based on the prediction results of the multimodal task, the multimodal label, the unimodal prediction results, and the unimodal label, so that the loss function is close to the expected value, thus completing the joint training process. In order to make the generated unimodal label stable, the unimodal label is dynamically updated. Specifically:

[0119] The L1 loss is used as the optimization target, where N represents the number of training samples and the weight of the loss function is the difference between the unimodal label and the multimodal label. is the weight of the i-th sample of the auxiliary unimodal task, represents the multimodal prediction value of the i-th sample, represents the multimodal label of the i-th sample, represents the unimodal prediction value of the i-th sample, Represents a dynamically updated unimodal label.

[0120] in, ,

[0121] .

[0122] In order to verify the technical effect of the multimodal sentiment analysis method of the present application, this embodiment conducts the following experimental verification:

[0123] The datasets used in the experiment are two public multimodal sentiment analysis datasets.

[0124] The CMU-MOSI dataset collects 93 monologue videos from YouTube, primarily film reviews, from 89 different speakers. Each speaker expresses their views on a particular film or topic. Each video ranges in length from 2 to 5 minutes and is divided into 2,199 segments. Each segment is manually annotated with a sentiment score ranging from -3 to +3, where scores below zero indicate negativity and scores above zero indicate positivity.

[0125] Compared to the CMU-MOSI dataset, the CMU-MOSEI dataset boasts a significantly larger and more diverse sample size and topic diversity. It contains 5,000 monologue videos from 1,000 different speakers on YouTube, covering 250 topics. These are divided into 22,856 annotated video clips, and sentiment scores are similarly labeled from -3 to +3. The specific divisions of the dataset and the percentage of each component are shown in Table 1.

[0126] Table 1 Dataset division

[0127]

[0128] This example conducted classification and regression experiments on the CMU-MOSI and CMU-MOSEI datasets, using five evaluation metrics, which can be categorized as regression and classification metrics. Regression metrics include mean absolute error (MAE) and Pearson correlation coefficient (Corr), while classification metrics include seven-class accuracy (Acc-7), binary accuracy (Acc-2), and F1-Score. Table 2 provides an overview of these five metrics.

[0129] Table 2 Evaluation indicators

[0130]

[0131] In this example, the model uses different hyperparameter values ​​for different modalities. The model training time is set to 25 using the early stopping method, and the experimental environment is a single NVIDIA RTX6000 GPU. Table 3 summarizes the parameter settings for each dataset model during training.

[0132] Table 3: Hyperparameter settings for the dataset

[0133]

[0134] The core of this invention is to combine a crossmodal transformer for multimodal interaction with a label generation module for learning unimodal representations. The model in this embodiment uses LSTM and BERT in the encoding phase, employing the crossmodal transformer to focus on interactions between modalities, and a self-attention mechanism to collect temporal information for prediction. Considering these characteristics, several baseline models with comparable performance were selected to validate the effectiveness of JTUM. These include: RNN-based models (MV-LSTM, BC-LSTM); models using attention mechanisms and transformers (RAVEN, MulT); tensor-based models (TFN, LMF); and models that separate unimodal and multimodal tasks (MISA, Self-MM).

[0135] MV-LSTM: Multi-View LSTM is a variant of LSTM that models view-specific and cross-view dynamics by partitioning the internal representation, thereby better utilizing multi-view representation learning.

[0136] BC-LSTM: Learn the information before and after the utterance by using bidirectional LSTM in the video.

[0137] TFN: Uses tensor fusion method to model the relationship between modalities and triple Cartesian product to simulate unimodal, bimodal, and trimodal interactions.

[0138] LMF: Efficient multimodal fusion by decomposing high-rank weight tensors into modality-specific low-rank factors.

[0139] RAVEN: Dynamically adapting sentiment polarity of word embeddings by modeling non-linguistic multimodal information.

[0140] MulT: Using directional pairwise cross-modal attention to focus on multimodal interactions at different time steps, adapting one modality to another.

[0141] MISA: By dividing the two subspaces into modality-invariant and modality-specific representations, we can learn the commonalities between multiple modalities and the characteristics of a single modality respectively, and better learn modality representations to help fusion.

[0142] Self-MM: Trains unimodal tasks through a label generation module based on self-supervised learning to obtain independent unimodal supervision.

[0143] Table 4 shows the comparison results of the models on the CMU-MOSI and CMU-MOSEI datasets. It can be seen that the model results of this embodiment are better than most baseline results.

[0144] Table 4: Experimental results

[0145]

[0146] (B) indicates language features based on BERT; △: from Hazarika et al.; ◊: from Yu et al.; ↑ / ↓: higher / lower values ​​indicate better model evaluation results. For Acc-2 and F1-Score, the left side of / indicates negative / non-negative, and the right side indicates negative / positive. Bold values ​​indicate the best results, while underlined values ​​indicate suboptimal results.

[0147] To analyze the contributions of the components introduced in the model, this example conducts ablation experiments on the CMU-MOSI and CMU-MOSEI datasets. The experimental results are shown in Table 5.

[0148] Table 5: Ablation learning.

[0149]

[0150] Among them, (-) indicates the removal of the corresponding module. Model (1) describes the influence of the crossmodal transformer, model (2) describes the influence of the self-attention mechanism, and model (3) includes both (1) and (2).

[0151] In general, this example treats the three groups of crossmodal transformers as a whole, verifies the effects of the crossmodal transformer and self-attention respectively, and removes one category each time to analyze their impact on performance.

[0152] Since the two datasets have a large difference in the number of samples, and some indicators are easily affected by the number of samples, we analyze the two datasets separately. On the CMU-MOSI dataset, the results of the three models (1), (2), and (3) show that for the crossmodal transformer and self-attention modules, the performance decreases whether they are removed separately or simultaneously, which shows that both modules play their roles. On the CMU-MOSEI dataset, when both modules are removed simultaneously, the classification results of model (3) are greatly reduced, and when the crossmodal transformer is removed, the regression indicators have the worst results.

[0153] It can be seen that the combination of both modules provides the best results in most indicators, which shows that the model of this embodiment has learned complementary information and collected time information for the final prediction.

[0154] Example 2:

[0155] A second embodiment of the present invention provides a sentiment analysis system based on single-modality and multi-modality joint training, including:

[0156] A modality encoding module is configured to take a video clip and extract the three modalities: text, audio, and visual; and encode each of the three modalities to produce a separate feature vector representation.

[0157] A task division module is configured to establish single-modal tasks and multi-modal tasks according to the independence and interactivity of modalities respectively;

[0158] The unimodal task module is configured to input the feature vectors of the three modalities into the unimodal task to obtain the unimodal label of each modality;

[0159] The multimodal task module is configured to input the feature vectors of the three modalities into the multimodal task for pairwise modal interaction, and obtain the modal representations after the three cross-modal interactions;

[0160] The prediction module is configured to use the modal representations after the three cross-modal interactions combined with the self-attention mechanism to output the prediction results of the multimodal task;

[0161] The joint training module is configured to jointly train unimodal tasks and multimodal tasks under the supervision of unimodal labels and multimodal labels, and use the prediction results output by the multimodal task after training as the final results of sentiment analysis.

[0162] The steps involved in the above-mentioned Example 2 correspond to those in the method example 1. For detailed implementation, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0163] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0164] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A sentiment analysis method based on single-modality and multi-modality joint training, characterized in that: The following steps are involved: Obtain video clips and extract the three modalities of text, audio, and vision; encode the three modalities separately to obtain separate feature vector representations; According to the independence and interactivity of modalities, single-modal tasks and multi-modal tasks are established respectively; The feature vectors of the three modalities are input into the unimodal task to obtain the unimodal label of each modality; The feature vectors of the three modalities are input into the multimodal task for two-to-two modal interaction to obtain the modal representations after three cross-modal interactions; The modal representations after three cross-modal interactions are combined with the self-attention mechanism to output the prediction results of the multimodal task; Under the supervision of unimodal and multimodal labels, the unimodal and multimodal tasks are jointly trained, and the prediction results output by the multimodal task after training are used as the final results of sentiment analysis; The specific steps for inputting the feature vectors of the three modalities into the multimodal task for pairwise modal interaction and obtaining the modal representations after the three cross-modal interactions are as follows: The feature vectors of the three modalities are transformed through the fully connected layer; The transformed feature vector representation is input into the crossmodal transformer to obtain the interaction information between the modalities; The outputs of the crossmodal transformers are concatenated to obtain three modal representations after cross-modal interaction, specifically: in, , , Indicates the modalities after three cross-modal interactions, a represents audio, t represents text, and v represents vision; represents the L-th layer crossmodal transformer output from source modality a to target modality t, Represents the L-th layer crossmodal transformer output from source modality v to target modality t; represents the L-th layer crossmodal transformer output from source modality a to target modality v, Represents the L-th layer crossmodal transformer output from source modality t to target modality v; represents the L-th layer crossmodal transformer output from source modality t to target modality a, Represents the L-th layer crossmodal transformer output from source modality t to target modality a.

2. The sentiment analysis method based on single-modality and multi-modality joint training according to claim 1, characterized in that The specific steps for encoding the three modes to obtain separate feature vector representations are: For the text modality, a 12-layer BERT is used, and the head word vector output by the last layer is used as the text modality feature vector representation; For the visual and audio modalities, we first use a feature extractor to extract features, then use stacked LSTMs to encode the visual and audio modalities separately. Finally, we use the first word vector of the last layer as the representation of the entire sequence to obtain the visual modality feature vector representation and the audio modality feature vector representation.

3. The sentiment analysis method based on single-modality and multi-modality joint training according to claim 1, characterized in that Unimodal tasks and multimodal tasks share the encoded feature vector representation.

4. The sentiment analysis method based on single-modality and multi-modality joint training according to claim 1, characterized in that The specific steps for inputting the feature vectors of the three modalities into the unimodal task to obtain the unimodal label of each modality are as follows: Input the feature vectors of the three modalities into the fully connected layer; Obtain unimodal prediction results and unimodal representation through linear regression; Use ULGM to get the unimodal label based on the unimodal prediction results.

5. The sentiment analysis method based on single-modality and multi-modality joint training according to claim 4, characterized in that: The specific steps of using ULGM to obtain unimodal labels based on unimodal prediction results are: Calculate the center points of the three single-modal representations and one fusion modality representation of positive and negative samples respectively; Calculate the relative distance between each modal representation and the center point of the positive and negative samples; The ratio of the single-modal label to the multi-modal label is obtained according to the relative distance; The offset is calculated based on the ratio of the unimodal label to the multimodal label to obtain the unimodal label.

6. The sentiment analysis method based on single-modality and multi-modality joint training according to claim 1, characterized in that The specific steps of using the modal representations after three cross-modal interactions combined with the self-attention mechanism to output the prediction results of the multimodal task are as follows: The modal representations of the three cross-modal interactions are fused into a joint vector representation using a self-attention mechanism; The joint vector representation is predicted through full connection to obtain the prediction results of the multimodal task.

7. The sentiment analysis method based on single-modality and multi-modality joint training according to claim 6, characterized in that: The specific steps of fusing the modal representations of the three cross-modal interactions into a joint vector representation using the self-attention mechanism are as follows: A self-attention mechanism is used to collect temporal information of modal sequences after cross-modal interaction; Extract the last element in each sequence and concatenate them; The concatenated results are projected into a lower-dimensional space to obtain a multimodal fusion representation.

8. The sentiment analysis method based on single-modality and multi-modality joint training according to claim 1, characterized in that: The specific steps for jointly training unimodal and multimodal tasks under the supervision of unimodal and multimodal labels are: Take loss as the optimization target and set the loss function; The difference between unimodal and multimodal labels is used as the weight of the loss function; The loss function is solved according to the prediction results of the multimodal task, the multimodal labels, the unimodal prediction results and the unimodal labels, so that the loss function is close to the expected value, thereby completing the joint training process.

9. A sentiment analysis system based on single-modality and multi-modality joint training, characterized in that: The sentiment analysis method based on single-modality and multi-modality joint training according to any one of claims 1 to 8 is adopted, comprising: A modality encoding module is configured to take a video clip and extract the three modalities: text, audio, and visual; and encode each of the three modalities to produce a separate feature vector representation. A task division module is configured to establish single-modal tasks and multi-modal tasks according to the independence and interactivity of modalities respectively; The unimodal task module is configured to input the feature vectors of the three modalities into the unimodal task to obtain the unimodal label of each modality; The multimodal task module is configured to input the feature vectors of the three modalities into the multimodal task for pairwise modal interaction, and obtain the modal representations after the three cross-modal interactions; The prediction module is configured to use the modal representations after the three cross-modal interactions combined with the self-attention mechanism to output the prediction results of the multimodal task; The joint training module is configured to jointly train unimodal tasks and multimodal tasks under the supervision of unimodal labels and multimodal labels, and use the prediction results output by the multimodal task after training as the final results of sentiment analysis.

Images