Emotion analysis method and device for promoting multi-modal information fusion

Abstract

The invention discloses an emotion analysis method and device for promoting multi-mode information fusion, and the method comprises the steps: building an acoustic vocabulary and a visual vocabulary based on a K-Means algorithm, querying the features of a non-text mode, obtaining a word index sequence of the non-text mode, and embedding the word index sequence into the initial features of the text mode to build a non-language displacement vector; adding the non-language displacement vector into the initial feature of the text mode for displacement operation to obtain a multi-mode vector; and performing context sensing and sentiment analysis on the multi-modal vector to obtain and output a sentiment analysis result. According to the method, the feature sequence of the non-text mode is converted into the word index sequence which is the same as the text mode through the K-Means algorithm, the distribution gap between the language text mode and the non-language text mode during fusion is reduced, and subsequent feature fusion is facilitated; during feature fusion, the expression of a language text mode is enhanced through a non-language displacement vector, so that the accuracy of sentiment analysis is improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a sentiment analysis method and device for promoting multimodal information fusion. Background Art

[0002] Humans communicate naturally through multimodal signals such as language, vision, and audio. With the surge in the number of human-centered online videos, multimodal sentiment analysis, which uses language text information and non-language text information (video image information and audio information) to identify human emotions in videos, has attracted widespread attention in recent years. Multimodal sentiment analysis uses multimodal signals including spoken language (language text), facial gestures (vision), and sound behavior to identify emotions in videos. In terms of the accuracy of sentiment analysis, language text modality is usually better than non-language modality, but considering that the meaning of a sentence is usually different in different non-language contexts, combining non-language information with language text information representation helps to understand the exact emotion conveyed by the discourse. Therefore, feature extraction of information from each modality and multimodal methods combining information data from multiple modalities have become the main methods of current multimodal sentiment analysis.

[0003] However, this method of combining multimodal information to analyze human sentiment faces two major problems: (1) The feature representation of the initial language text after feature extraction is a word index sequence from the vocabulary, whereas the representation of visual and auditory modal features is a real vector sequence. The distribution gap between language and non-language features is large, which is not conducive to the integration of language text representation and non-language emotional background; (2) How to strengthen the weight of language text modal features in the process of feature fusion to improve the accuracy of sentiment analysis.

[0004] Therefore, existing sentiment analysis methods based on multimodal information have problems such as large differences in multimodal information and a low proportion of text modality in the fusion process, which leads to low accuracy of sentiment analysis methods based on multimodal information. Summary of the invention

[0005] The present invention provides a sentiment analysis method and device for promoting the fusion of multimodal information, which is used to solve the technical problem that the existing sentiment analysis method based on multimodal information has large differences in multimodal information and a low proportion of text modality in the fusion process, resulting in low accuracy of the sentiment analysis method of multimodal information.

[0006] A first aspect of the present invention provides a sentiment analysis method for promoting multimodal information fusion, the method comprising:

[0007] Acquire multimodal data, preprocess each modality data, and obtain initial features of each modality; multimodality includes text / language text modality, audio modality, and video modality;

[0008] Obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establish an acoustic vocabulary table and a visual vocabulary table respectively;

[0009] The initial features of the audio modality and the initial features of the video modality are queried through the acoustic vocabulary and the visual vocabulary respectively to obtain the word index sequence of the non-text modality;

[0010] Embedding a word index sequence of a non-text modality into an initial feature of a text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector;

[0011] The multimodal vector is input into the Transformer model for context perception, and the multimodal perception information is output; the multimodal perception information is input into the sentiment analysis module for sentiment analysis, and the sentiment analysis result is obtained and output.

[0012] Furthermore, the step of acquiring multimodal data, preprocessing each modal data respectively, and obtaining initial features of each modality includes:

[0013] Preprocessing the data of the text modality through the SentiLARE model to obtain initial features of the text modality;

[0014] Preprocessing the data of the audio modality through the COVAREP extraction library to obtain initial features of the audio modality;

[0015] The data of the video modality is preprocessed by using the Py-Feat toolkit or the Facet system to obtain initial features of the video modality.

[0016] Furthermore, the steps of obtaining a training set, clustering the training set based on a K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establishing an acoustic vocabulary table and a visual vocabulary table respectively include:

[0017] Acquire a training set and extract sound segments and visual segments from the training set to obtain an audio modality frame set and a video modality frame set, respectively;

[0018] The K-Means algorithm is used to cluster and group the audio modal frame set and the video modal frame set respectively, and the audio cluster center set and the video cluster center set are obtained by learning;

[0019] Based on the audio cluster center set and the video cluster center set, an acoustic vocabulary and a visual vocabulary are established respectively.

[0020] Furthermore, the calculation formulas for querying the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary and the visual vocabulary are as follows:

[0021]

[0022] Where: Indicates the i-th frame The index tag, represents the features of the i-th frame of the sequence, is the j-th cluster center of mode m.

[0023] Furthermore, the step of embedding the word index sequence of the non-text modality into the initial feature of the text modality to establish a non-language displacement vector, adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation, and outputting a multi-modal vector includes:

[0024] Concatenate the word index sequence of the non-text modality with the initial features of the text modality to obtain an audio concatenation vector and a video concatenation vector;

[0025] Performing weight calculation on the audio splicing vector and the video splicing vector respectively to obtain an audio gate vector and a video gate vector;

[0026] The non-language displacement vector is obtained by multiplying the audio splicing vector and the video splicing vector by the associated gate vector respectively;

[0027] The non-language displacement vector is added to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector.

[0028] Furthermore,

[0029] The audio gate vector The calculation method is expressed as:

[0030]

[0031] Where: represents a nonlinear activation function, represents the weight matrix of the auditory channel, represents the scalar deviation of the audio mode; represents the audio concatenation vector, A sequence representing the initial features of the text modality, a sequence of word indices representing the audio modality;

[0032] The video gate vector The calculation method is expressed as:

[0033]

[0034] Where: represents the weight matrix of the visual channel, scalar deviation representing the video modality; represents the video splicing vector, A sequence of word indices representing the video modality;

[0035] The non-verbal displacement vector The calculation method is expressed as:

[0036]

[0037] Where: and Represent the weight matrices of acoustic information and visual information respectively, represents the deviation vector;

[0038] The multimodal vector The calculation method is expressed as:

[0039]

[0040] Where: Represents the scaling factor.

[0041] Furthermore, the Transformer model consists of a multi-head attention mechanism, a forward network, a residual connection layer and a layer normalization module.

[0042] A second aspect of the present invention provides a sentiment analysis device for promoting multimodal information fusion, the device comprising:

[0043] The initial feature acquisition module is used to acquire multimodal data, preprocess the data of each modality, and obtain the initial features of each modality; multimodality includes text modality, audio modality and video modality;

[0044] The vocabulary building module is used to obtain the training set, cluster the training set based on the K-Means algorithm to learn the acoustic vocabulary and visual vocabulary, and build the acoustic vocabulary and visual vocabulary respectively;

[0045] A word index sequence generation module for non-text modality, used for querying the initial features of the audio modality and the initial features of the video modality through an acoustic vocabulary and a visual vocabulary respectively, to obtain a word index sequence of the non-text modality;

[0046] A feature fusion module is used for embedding a word index sequence of a non-text modality into an initial feature of a text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector;

[0047] The sentiment result output module is used to input the multimodal vector into the Transformer model for context perception and output multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, and obtain and output the sentiment analysis result.

[0048] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-described sentiment analysis methods when executing the computer program.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of any of the above-described sentiment analysis methods are implemented.

[0050] It can be seen from the above technical solutions that the present invention has the following advantages:

[0051] The present invention provides a sentiment analysis method and device for promoting multimodal information fusion. The method comprises: obtaining multimodal data, preprocessing each modal data, and obtaining initial features of each modality; the multimodality comprises text modality, audio modality and video modality; obtaining a training set, clustering the training set based on a K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establishing an acoustic vocabulary table and a visual vocabulary table respectively; respectively querying the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary table and the visual vocabulary table to obtain a word index sequence of a non-text modality; embedding the word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial features of the text modality for shifting operation to obtain a multimodal vector; inputting the multimodal vector into a Transformer model for context perception, and outputting multimodal perception information; inputting the multimodal perception information into a sentiment analysis module for sentiment analysis, and obtaining and outputting a sentiment analysis result.

[0052] In the present invention, the initial feature sequence of the non-text modality is converted into the same word index sequence as the language text modality through the K-Means algorithm, thereby reducing the initial distribution difference between heterogeneous modalities, further narrowing the distribution gap between the language text modality and the non-language text modality features during fusion, and facilitating subsequent feature fusion; at the same time, during multimodal feature fusion, the representation of the language text modality is enhanced by combining the non-language displacement vectors of the video modality and the audio modality, thereby improving the accuracy of the sentiment analysis of the multimodal data, thereby solving the technical problem that the existing sentiment analysis method based on multimodal information has large multimodal information differences and the text modality accounts for a low proportion in the fusion process, resulting in low accuracy of the sentiment analysis method of the multimodal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A flowchart of a sentiment analysis method for promoting multimodal information fusion provided by an embodiment of the present invention;

[0055] Figure 2 A conversion flow chart of a feature conversion module structure diagram provided by an embodiment of the present invention;

[0056] Figure 3 A processing flow chart of a feature fusion module and a Transformer model provided in an embodiment of the present invention;

[0057] Figure 4 A flowchart of the operation of a single Attention in the multi-head attention mechanism provided in an embodiment of the present invention;

[0058] Figure 5 A model structure diagram of a sentiment analysis device for promoting multimodal information fusion provided by an embodiment of the present invention;

[0059] Figure 6 A structural block diagram of a sentiment analysis device for promoting multimodal information fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention provide a sentiment analysis method and device for promoting the fusion of multimodal information, which are used to solve the technical problem that the existing sentiment analysis method based on multimodal information has large differences in multimodal information and a low proportion of text modality in the fusion process, resulting in low accuracy of the sentiment analysis method of multimodal information.

[0061] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See also Figure 1In a first aspect, the present invention provides a sentiment analysis method for promoting multimodal information fusion, the method comprising:

[0063] Step 101, obtain multimodal data, preprocess each modal data, and obtain initial features of each modality; the multimodality includes text modality t, audio modality a and video modality v.

[0064] It can be understood that the present invention obtains multimodal data from the MOSI dataset and the MOSEI dataset. The MOSEI dataset has 23,453 text data and 3,228 videos. The text data of the MOSI dataset contains about 2,191 sentences. The video data contains about 2,219 video clips. Both datasets use [-3, 3] to mark the target's emotional intensity, and are applicable datasets in the direction of multimodal sentiment analysis.

[0065] Step 101 includes the following sub-steps:

[0066] Sub-step 1011, preprocessing the data of the text modality through the SentiLARE model to obtain the initial features of the text modality.

[0067] It should be noted that the SentiLARE model is an emotion-aware language representation model. The purpose of its pre-training is to better capture the emotional information of the text through the pre-training of emotions and sentiments. The SentiLARE model is based on the Transformer architecture and is improved on the basis of the Bert model. In order to obtain the language knowledge of each word, the word is marked with its part-of-speech tag, and the emotional tendency is obtained in SentiWordNet through the context-aware emotional attention mechanism.

[0068] SentiWordNet is a sentiment lexicon based on WordNet that provides sentiment polarity scores for words, including positive, negative, and neutral. These scores reflect the sentiment tendency of words, enabling SentiWordNet to be used for sentiment analysis and sentiment recognition in natural language processing tasks. Part-of-speech (POS) embedding and word-level sentiment polarity embedding can be added to each sentence through the SentiLARE pre-trained model.

[0069] Sub-step 1012, pre-processing the audio modality data through the COVAREP extraction library to obtain the initial features of the audio modality.

[0070] It should be noted that the COVAREP extraction library is used to extract acoustic features related to the emotion and intonation of speech, including 12-Mel frequency cepstral coefficients, pitch, clear / clear segmentation features, glottal source parameters, peak slope parameters, and maximum discrete quotient. The feature dimension of the feature information of the audio part in the MOSI dataset and the MOSEI dataset is 74.

[0071] Sub-step 1013, preprocessing the data of the video modality through the Py-Feat toolkit or the Facet system to obtain the initial features of the video modality.

[0072] It should be noted that for the MOSI dataset, the Py-Feat toolkit can be used to extract facial features for each visual frame, including facial features, 7 facial expressions, and facial action units. Facial landmarks are used to select frames with open eyes by calculating the aspect ratio of the eyes; frames with closed eyes may cause uncertainty in expression analysis and are therefore discarded; finally, 27-dimensional features containing facial expressions and facial action units are formed in each frame with open eyes. For the MOSEI dataset, 35-dimensional visual features containing facial action units and facial postures can be extracted through the Facet system.

[0073] See also Figure 2 , step 102, obtaining a training set, clustering the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establishing an acoustic vocabulary table and a visual vocabulary table respectively.

[0074] It can be understood that the K-Means algorithm is an unsupervised learning method that does not require a pre-labeled training set. K-Means is a clustering algorithm that finds K clusters of a given data set. It can also be called the K-means method. It can find K different clusters, and the center of each cluster is calculated using the mean of the values ​​contained in the cluster.

[0075] Clustering, in simple terms, is to automatically classify data with similar features in a large data set into a cluster. The more similar the objects in the cluster are, the better the clustering effect is. The number of clusters K is specified by the user, and each cluster is described by its centroid, that is, the center of all the points in the cluster. The biggest difference between clustering algorithms and classification algorithms is that the target category of classification is known, while the target category of clustering is unknown.

[0076] The K-Means method has the advantages of low computational complexity and simple calculation. The present invention uses the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and then establish an acoustic vocabulary table and a visual vocabulary table.

[0077] Step 102 includes the following sub-steps:

[0078] Sub-step 1021, obtaining a training set and extracting sound segments and visual segments from the training set to obtain an audio modality frame set and a video modality frame set, respectively.

[0079] It should be noted that the training set may include video clips in various scenes, and audio and visual frames are extracted from each video clip. For example, the visual frame may contain facial information of the characters in the picture, and the audio frame may contain information such as waveforms of various sounds.

[0080] Then, the frame set of non-text mode It is expressed as:

[0081]

[0082] in, A sequence of features representing non-text modalities, represents the number of frames of mode m, Indicates the length of a single mode.

[0083] In sub-step 1022, the K-Means algorithm is used to cluster and group the audio modality frame set and the video modality frame set respectively, and the audio cluster center set and the video cluster center set are learned.

[0084] By using the k-means method The frames in are divided into Km groups as follows:

[0085]

[0086] in, , is the j-th cluster center of mode m.

[0087] Sub-step 1023, based on the audio cluster center set C a and the video cluster center C v Set up acoustic vocabulary and visual vocabulary respectively.

[0088] Step 103 , query the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary and the visual vocabulary respectively, and obtain a word index sequence of the non-text modality.

[0089] Given a non-text feature sequence: , is the length of the sequence, is the feature of the i-th frame of the sequence; by querying the non-linguistic feature vocabulary established by the K-Means algorithm, the corresponding index sequence can be obtained ,for The i-th frame Features, Represents a sequence set in a non-text feature sequence, where the query process is calculated as follows:

[0090]

[0091] in, is the i-th frame The meaning of the above formula is to find an index , so that the feature of the i-th frame To the jth cluster center The Euclidean distance of is minimized; then, the obtained index sequence It is used as the representation of the audio modality or the video modality, thereby converting the extracted non-text vector into a word index sequence that is the same as the text modality representation, reducing the initial distribution difference between different modalities and further narrowing the distribution gap between text modality and non-text modality features during fusion.

[0092] Step 104, embed the word index sequence of the non-text modality into the initial feature of the text modality to establish a non-language displacement vector, and add the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector.

[0093] It is understandable that each word belongs to a certain part of the semantic space, which depends only on the meaning of the word in the language structure (i.e., the sentence). However, non-textual behavior can affect the meaning of words and thus the position of words in the semantic space, that is, textual features and non-textual features jointly determine the new position of words in the semantic space.

[0094] In this field, text modality usually performs better than non-text modality, so strengthening the importance of text modality in multimodal sentiment analysis is a feasible way to improve the recognition accuracy of multimodal sentiment analysis. Based on this, the present invention regards the new position as the addition of the language-only position and the displacement vector, forming a vector with trajectory and size, and moves the language-only position of the word to the new position according to the non-language behavior, thereby strengthening the weight of the language text feature in the fused tensor.

[0095] See also Figure 3 (a), step 104 specifically includes:

[0096] Sub-step 1041, concatenates the word index sequence of the non-text modality with the initial features of the text modality to obtain an audio concatenation vector and a video concatenation vector.

[0097] Among them, the word index sequence of the non-text modality and the initial features of the text modality can be expressed as a triple ), respectively representing the inputs of the i-th word in their respective sequences, where Represents the feature input of the text modality, represents the sequence input of the audio modality, Represents a sequence input of video modality.

[0098] The initial features of the text modality and the word index sequence of the non-text modality are concatenated through the Attention Gate operation to form [ ]and[ ];in, represents the audio concatenation vector, Represents a video splicing vector.

[0099] Sub-step 1042, weight calculation is performed on the audio splicing vector and the video splicing vector respectively to obtain an audio gate vector and a video gate vector.

[0100] Audio Gate Vector The calculation method is expressed as:

[0101]

[0102] Where: represents a nonlinear activation function, represents the weight matrix of the auditory channel, A scalar deviation representing the audio mode.

[0103] Video Gate Vector The calculation method is expressed as:

[0104]

[0105] Where: represents the weight matrix of the visual channel, A scalar deviation representing the video modality.

[0106] Sub-step 1043 , obtain a non-language displacement vector by multiplying the audio splicing vector and the video splicing vector by the associated gate vector respectively.

[0107] In this substep, and Multiply by their respective gate vectors and add together to create a non-linguistic displacement vector :

[0108]

[0109] Where: and Represent the weight matrices of acoustic information and visual information respectively, represents the bias vector.

[0110] Sub-step 1044, adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector.

[0111] Multimodal vector The calculation method is expressed as:

[0112]

[0113] Where: represents a scaling factor which keeps the effect of the nonverbal transformation Hi within a desired range.

[0114] The present invention well integrates non-language feature information into language feature information. While strengthening the weight of language text modal features, it also well adds non-language text modal information to multimodal sentiment analysis, significantly improving the accuracy of multimodal sentiment analysis tasks.

[0115] Step 105: input the multimodal vector into the Transformer model for context perception, and output the multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, and obtain and output the sentiment analysis result.

[0116] It should be noted that before the multimodal vector is input into the Transformer model, it needs to be processed by layer normalization (Layer Norm) and dropout layer; among them, the dropout layer randomly discards some neurons on the output after layer normalization to prevent overfitting. The parameter of the dropout layer is set to 0.3, that is, each neuron has a 30% probability of being abandoned.

[0117] It is understandable that the Transformer model is composed of a multi-head attention mechanism, a feed forward network, a residual connection and a layer normalization module, such as Figure 3 (b) As shown in Figure 2, Transformer can transform the input sequence into a context-aware representation, which contains the information of each position in the input sequence and can capture the relationship between the input sequences.

[0118] Among them, multi-head attention is an extended form of the attention mechanism, which can extract information more effectively when processing sequence data. In multi-head attention, multiple sets of attention weights are used, each set of weights can learn different semantic information, and each set of weights will generate a context vector. Finally, these context vectors are concatenated and then a linear transformation is performed to obtain the final output.

[0119] The multi-head attention mechanism can be seen as multiple parallel self-attention mechanisms (Self-Attention), which consists of three matrices: Q (query), K (key value), and V (value). All three matrices are obtained by linearly transforming the input (fully connected layer), such as Figure 4 As shown. Among them, , W is the weight of the fully connected layer, X is the input tensor, They correspond to their respective weight matrices.

[0120] Perform matrix multiplication on matrices Q and K to calculate the similarity matrix S of the Q and K vectors, then divide S by a specific constant α and activate it through the activation function Softmax function to obtain the attention score matrix S', and finally perform matrix multiplication on S' and V to obtain the output of the attention mechanism. The specific calculation process formula is as follows:

[0121]

[0122] Among them, the calculation of the specific constant α Refers to the feature dimension of the input tensor, and T represents the transpose operation of the matrix.

[0123] like Figure 3 As shown in (b), in the Transformer model, the input and output of the Multi-Head Attention mechanism constitute a residual network. In addition, the input and output of the FeedForward network also constitute a residual network, and then layer normalization is performed after each residual network.

[0124] After the multimodal features are fused, the fusion result, i.e. the new text representation after the fusion of text, vision and acoustic modalities, is input to the Mith layer of the Transformer with a total of M layers. Then, for the Mth layer, its output will be a high-level sentiment representation dominated by language text with visual and auditory information. The output of the M-layer Transformer is the input of the sentiment analysis module. In the sentiment analysis process, two linear transformations are performed to obtain the final sentiment result, such as Figure 5 shown.

[0125] Finally, for the sentiment analysis results, the sentiment intensity of binary classification and regression is mainly used for analysis. For binary classification, the binary classification accuracy (Acc-2) and weighted F1 score (F1) are used; for regression, the recognition performance is mainly evaluated by the mean absolute error (MAE) and Pearson correlation (Corr).

[0126] Please refer to Table 1, where the TFN, MISA, MulT, Self-MM, and MMIM model data are from Han et al. (2021), and the scores measured by the present invention on the MOSI dataset and the MOSEI dataset are shown in the experimental results in the last row of Table 1. Therefore, compared with the mainstream model, the sentiment analysis method provided by the present invention has achieved certain results, can effectively integrate multimodal data, and improve the accuracy of sentiment analysis.

[0127] Table 1 Comparison of experimental results of different sentiment analysis models on MOSI and MOSEI datasets

[0128]

[0129] The present invention provides a sentiment analysis method for promoting multimodal information fusion, which has the following advantages:

[0130] 1. The present invention proposes a feature conversion strategy for converting non-language vectors into indexes. The K-Means unsupervised clustering algorithm can be used to convert the original feature sequences of video images and audio into index sequences similar to text, thereby reducing the initial distribution differences between heterogeneous modalities. This will further narrow the distribution gap between text modality and non-text modality features during fusion, thereby promoting the integration of language text representation and non-language emotional context, thereby improving the accuracy of sentiment analysis.

[0131] 2. When fusing multimodal features, the language text representation can be enhanced by combining the emotional information of visual and auditory modalities, thereby strengthening the weight of the language text features in the fused tensor, and further improving the accuracy of the emotional analysis of multimodal information.

[0132] See also Figure 5 and Figure 6 , the second aspect of the present invention provides a sentiment analysis device for promoting multimodal information fusion, the device comprising: an initial feature acquisition module, a feature conversion module, a feature fusion module and a sentiment result output module;

[0133] The initial feature acquisition module 501 is used to acquire multimodal data, pre-process each modal data, and obtain the initial features of each modality; the multimodality includes text modality, audio modality and video modality;

[0134] The feature conversion module includes a vocabulary building module and a non-text modality word index sequence generation module;

[0135] A vocabulary building module 502 is used to obtain a training set, cluster the training set based on a K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and respectively build an acoustic vocabulary and a visual vocabulary;

[0136] A word index sequence generation module 503 for non-text modality is used to query the initial features of the audio modality and the initial features of the video modality through an acoustic vocabulary and a visual vocabulary respectively to obtain a word index sequence for the non-text modality;

[0137] The feature fusion module 504 is used for embedding the word index sequence of the non-text mode into the initial feature of the text mode to establish a non-language displacement vector, and adding the non-language displacement vector to the initial feature of the text mode to perform a shift operation to obtain a multi-modal vector;

[0138] The sentiment result output module 505 is used to input the multimodal vector into the Transformer model for context perception and output the multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, and obtain and output the sentiment analysis result.

[0139] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above sentiment analysis methods are implemented.

[0140] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above sentiment analysis methods when the computer program is executed by a processor.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0146] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sentiment analysis method for promoting multimodal information fusion, characterized in that: The method comprises: Acquire multimodal data, preprocess each modal data, and obtain initial features of each modality; multimodality includes text modality, audio modality, and video modality; Obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establish an acoustic vocabulary table and a visual vocabulary table respectively; The initial features of the audio modality and the initial features of the video modality are queried through the acoustic vocabulary and the visual vocabulary respectively to obtain the word index sequence of the non-text modality; Embedding a word index sequence of a non-text modality into an initial feature of a text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector; The multimodal vector is input into the Transformer model for context perception, and the multimodal perception information is output; the multimodal perception information is input into the sentiment analysis module for sentiment analysis, and the sentiment analysis result is obtained and output.

2. The sentiment analysis method according to claim 1, characterized in that: The step of acquiring multimodal data, preprocessing each modal data respectively, and obtaining initial features of each modality includes: Preprocessing the data of the text modality through the SentiLARE model to obtain initial features of the text modality; Preprocessing the data of the audio modality through the COVAREP extraction library to obtain initial features of the audio modality; The data of the video modality is preprocessed by using the Py-Feat toolkit or the Facet system to obtain initial features of the video modality.

3. The sentiment analysis method according to claim 1, characterized in that: The steps of obtaining a training set, clustering the training set based on a K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establishing an acoustic vocabulary table and a visual vocabulary table respectively include: Acquire a training set and extract sound segments and visual segments from the training set to obtain an audio modality frame set and a video modality frame set, respectively; The K-Means algorithm is used to cluster and group the audio modality frame set and the video modality frame set respectively, and the audio cluster center set and the video cluster center set are obtained by learning; Based on the audio cluster center set and the video cluster center set, an acoustic vocabulary and a visual vocabulary are established respectively.

4. The sentiment analysis method according to claim 3, characterized in that: The calculation formulas for querying the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary and the visual vocabulary respectively are as follows: Where: Indicates the i-th frame The index tag, represents the features of the i-th frame of the sequence, is the j-th cluster center of mode m.

5. The sentiment analysis method according to claim 1, characterized in that: The step of embedding the word index sequence of the non-text modality into the initial feature of the text modality to establish a non-language displacement vector, adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation, and outputting a multi-modal vector includes: Concatenate the word index sequence of the non-text modality with the initial features of the text modality to obtain an audio concatenation vector and a video concatenation vector; Performing weight calculation on the audio splicing vector and the video splicing vector respectively to obtain an audio gate vector and a video gate vector; The non-language displacement vector is obtained by multiplying the audio splicing vector and the video splicing vector by the associated gate vector respectively; The non-language displacement vector is added to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector.

6. The sentiment analysis method according to claim 5, characterized in that: The audio gate vector The calculation method is expressed as: Where: represents a nonlinear activation function, represents the weight matrix of the auditory channel, represents the scalar deviation of the audio mode; represents the audio concatenation vector, A sequence representing the initial features of the text modality, a sequence of word indices representing the audio modality; The video gate vector The calculation method is expressed as: Where: represents the weight matrix of the visual channel, scalar deviation representing the video modality; represents the video splicing vector, A sequence of word indices representing the video modality; The non-verbal displacement vector The calculation method is expressed as: Where: and Represent the weight matrices of acoustic information and visual information respectively, represents the deviation vector; The multimodal vector The calculation method is expressed as: Where: Represents the scaling factor.

7. The sentiment analysis method according to claim 1, characterized in that: The Transformer model consists of a multi-head attention mechanism, a forward network, a residual connection layer and a layer normalization module.

8. A sentiment analysis device for promoting multimodal information fusion, characterized in that: The device comprises: The initial feature acquisition module is used to acquire multimodal data, preprocess the data of each modality, and obtain the initial features of each modality; multimodality includes text modality, audio modality and video modality; The vocabulary building module is used to obtain the training set, cluster the training set based on the K-Means algorithm to learn the acoustic vocabulary and visual vocabulary, and build the acoustic vocabulary and visual vocabulary respectively; A word index sequence generation module for non-text modality, used for querying the initial features of the audio modality and the initial features of the video modality through an acoustic vocabulary and a visual vocabulary respectively, to obtain a word index sequence of the non-text modality; A feature fusion module is used for embedding a word index sequence of a non-text modality into an initial feature of a text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector; The sentiment result output module is used to input the multimodal vector into the Transformer model for context perception and output multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, and obtain and output the sentiment analysis result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the sentiment analysis method as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the sentiment analysis method as described in any one of claims 1 to 7 are implemented.

Images