A sentiment analysis method and device for promoting multimodal information fusion

Abstract

The present invention discloses a sentiment analysis method and device for promoting multimodal information fusion. The method comprises: establishing an acoustic vocabulary and a visual vocabulary based on the K-Means algorithm and querying the features of the non-text modality, obtaining a word index sequence of the non-text modality and embedding it into the initial features of the text modality to establish a non-language displacement vector; adding the non-language displacement vector to the initial features of the text modality and performing a shift operation to obtain a multimodal vector; performing context perception and sentiment analysis on the multimodal vector to obtain and output a sentiment analysis result. In the present invention, the feature sequence of the non-text modality is converted into a word index sequence identical to that of the text modality through the K-Means algorithm, narrowing the distribution gap between the features of the language text modality and the non-language text modality during fusion, thereby facilitating subsequent feature fusion; and enhancing the representation of the language text modality through the non-language displacement vector during feature fusion, thereby improving the accuracy of the sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a sentiment analysis method and device for promoting multimodal information fusion. Background Art

[0002] Humans communicate naturally through multimodal signals such as language, vision, and audio. With the surge in the number of human-centered online videos, multimodal sentiment analysis, which uses both verbal and non-verbal information (video, image, and audio) to identify human emotions in videos, has attracted widespread attention in recent years. Multimodal sentiment analysis utilizes multimodal signals, including spoken language (verbal), facial gestures (visual), and vocal behavior, to identify emotions in videos. While verbal text modalities generally outperform non-verbal modalities in terms of sentiment analysis accuracy, given that the meaning of sentences often varies in different non-verbal contexts, combining non-verbal information with verbal textual representations helps understand the precise emotion conveyed by the utterance. Therefore, multimodal methods that extract features from each modality and combine data from multiple modalities have become the primary approach to multimodal sentiment analysis.

[0003] However, this method of combining multimodal information to analyze human sentiment faces two major problems: (1) The feature representation of the initial language text after feature extraction is a word index sequence from the vocabulary, whereas the representation of visual and auditory modal features is a real vector sequence. The distribution gap between language and non-language features is large, which is not conducive to the integration of language text representation and non-language emotional background; (2) How to strengthen the weight of language text modal features in the process of feature fusion to improve the accuracy of sentiment analysis.

[0004] Therefore, existing sentiment analysis methods based on multimodal information have problems such as large differences in multimodal information and a low proportion of text modality in the fusion process, resulting in low accuracy of sentiment analysis methods based on multimodal information. Summary of the Invention

[0005] The present invention provides a sentiment analysis method and device for promoting the fusion of multimodal information, which is used to solve the technical problem that the existing sentiment analysis method based on multimodal information has large differences in multimodal information and a low proportion of text modality in the fusion process, resulting in low accuracy of the sentiment analysis method of multimodal information.

[0006] A first aspect of the present invention provides a sentiment analysis method for promoting multimodal information fusion, the method comprising:

[0007] Acquire multimodal data, preprocess each modality data, and obtain initial features of each modality; multimodality includes text / language text modality, audio modality, and video modality;

[0008] Obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establish an acoustic vocabulary table and a visual vocabulary table respectively;

[0009] The initial features of the audio modality and the initial features of the video modality are queried through the acoustic vocabulary and the visual vocabulary respectively to obtain the word index sequence of the non-text modality;

[0010] Embedding a word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial features of the text modality to perform a shift operation to obtain a multimodal vector;

[0011] The multimodal vector is input into the Transformer model for context perception, and multimodal perception information is output; the multimodal perception information is input into the sentiment analysis module for sentiment analysis, and the sentiment analysis result is obtained and output.

[0012] Furthermore, the step of acquiring multimodal data and preprocessing each modal data to obtain initial features of each modality includes:

[0013] Preprocessing the data of the text modality through the SentiLARE model to obtain initial features of the text modality;

[0014] Preprocessing the audio modality data using the COVAREP extraction library to obtain initial features of the audio modality;

[0015] The data of the video modality is preprocessed using the Py-Feat toolkit or the Facet system to obtain initial features of the video modality.

[0016] Furthermore, the steps of obtaining a training set, clustering the training set based on a K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establishing an acoustic vocabulary table and a visual vocabulary table respectively include:

[0017] Acquiring a training set and extracting sound segments and visual segments from the training set to obtain an audio modality frame set and a video modality frame set, respectively;

[0018] The K-Means algorithm is used to cluster the audio modality frame set and the video modality frame set respectively, and the audio cluster center set and the video cluster center set are obtained by learning;

[0019] An acoustic vocabulary and a visual vocabulary are respectively established based on the audio cluster center set and the video cluster center set.

[0020] Furthermore, the calculation formulas for querying the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary and the visual vocabulary are as follows:

[0021]

[0022] Where: Indicates the i-th frame The index tag, represents the features of the i-th frame of the sequence, is the j-th cluster center of mode m.

[0023] Furthermore, the step of embedding the word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, adding the non-language displacement vector to the initial features of the text modality to perform a shift operation, and outputting a multimodal vector includes:

[0024] Concatenate the word index sequence of the non-text modality with the initial features of the text modality to obtain the audio concatenation vector and the video concatenation vector;

[0025] Performing weight calculation on the audio splicing vector and the video splicing vector respectively to obtain an audio gate vector and a video gate vector;

[0026] The non-language displacement vector is obtained by multiplying the audio splicing vector and the video splicing vector by the associated gate vector respectively;

[0027] The non-language displacement vector is added to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector.

[0028] Further,

[0029] The audio gate vector The calculation method is expressed as:

[0030]

[0031] Where: represents a nonlinear activation function, represents the weight matrix of the auditory channel, represents the scalar deviation of the audio mode; represents the audio concatenation vector, A sequence representing the initial features of the text modality, A sequence of word indices representing the audio modality;

[0032] The video gate vector The calculation method is expressed as:

[0033]

[0034] Where: represents the weight matrix of the visual channel, scalar deviation representing the video modality; represents the video splicing vector, A sequence of word indices representing the video modality;

[0035] The non-verbal displacement vector The calculation method is expressed as:

[0036]

[0037] Where: and Represent the weight matrices of acoustic information and visual information respectively, represents the bias vector;

[0038] The multimodal vector The calculation method is expressed as:

[0039]

[0040] Where: Represents the scaling factor.

[0041] Furthermore, the Transformer model consists of a multi-head attention mechanism, a forward network, a residual connection layer and a layer normalization module.

[0042] A second aspect of the present invention provides a sentiment analysis device for promoting multimodal information fusion, the device comprising:

[0043] The initial feature acquisition module is used to obtain multimodal data, preprocess the data of each modality, and obtain the initial features of each modality; multimodality includes text modality, audio modality, and video modality;

[0044] The vocabulary building module is used to obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and build acoustic vocabulary and visual vocabulary respectively;

[0045] A word index sequence generation module for non-text modalities is used to query the initial features of the audio modality and the initial features of the video modality through an acoustic vocabulary and a visual vocabulary respectively to obtain a word index sequence for the non-text modality;

[0046] A feature fusion module is used for multimodal vectors, which embeds the word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, and adds the non-language displacement vector to the initial features of the text modality to perform a shift operation to obtain a multimodal vector;

[0047] The emotion result output module is used to input the multimodal vector into the Transformer model for context perception and output multimodal perception information; input the multimodal perception information into the emotion analysis module for emotion analysis, and obtain and output the emotion analysis result.

[0048] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-described sentiment analysis methods when executing the computer program.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described sentiment analysis methods.

[0050] It can be seen from the above technical solutions that the present invention has the following advantages:

[0051] The present invention provides a sentiment analysis method and device for promoting multimodal information fusion, the method comprising: acquiring multimodal data, preprocessing the data of each modality, and obtaining initial features of each modality; the multimodality includes text modality, audio modality, and video modality; acquiring a training set, clustering the training set based on a K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establishing an acoustic vocabulary table and a visual vocabulary table respectively; querying the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary table and the visual vocabulary table respectively to obtain a word index sequence of a non-text modality; embedding the word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial features of the text modality for a shift operation to obtain a multimodal vector; inputting the multimodal vector into a Transformer model for context perception, and outputting multimodal perception information; inputting the multimodal perception information into a sentiment analysis module for sentiment analysis, and obtaining and outputting a sentiment analysis result.

[0052] In the present invention, the initial feature sequence of the non-text modality is converted into the same word index sequence as the language text modality through the K-Means algorithm, thereby reducing the initial distribution difference between heterogeneous modalities, further narrowing the distribution gap between the language text modality and non-language text modality features during fusion, and facilitating subsequent feature fusion; at the same time, during multimodal feature fusion, the representation of the language text modality is enhanced by combining the non-language displacement vectors of the video modality and the audio modality, thereby improving the accuracy of the sentiment analysis of multimodal data, thereby solving the technical problem that the existing sentiment analysis method based on multimodal information has large multimodal information differences and the text modality accounts for a low proportion in the fusion process, resulting in low accuracy of the sentiment analysis method of multimodal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A flowchart of the steps of a sentiment analysis method for promoting multimodal information fusion provided by an embodiment of the present invention;

[0055] Figure 2 A conversion flow chart of the feature conversion module structure diagram provided by an embodiment of the present invention;

[0056] Figure 3 A processing flow chart of the feature fusion module and Transformer model provided in an embodiment of the present invention;

[0057] Figure 4 This is a flowchart of the operation of a single Attention in the multi-head attention mechanism provided by an embodiment of the present invention;

[0058] Figure 5 A model structure diagram of a sentiment analysis device for promoting multimodal information fusion provided by an embodiment of the present invention;

[0059] Figure 6 A structural block diagram of a sentiment analysis device for promoting multimodal information fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention provide a sentiment analysis method and device for promoting the fusion of multimodal information, which is used to solve the technical problem that the existing sentiment analysis method based on multimodal information has large differences in multimodal information and a low proportion of text modality in the fusion process, resulting in low accuracy of the sentiment analysis method of multimodal information.

[0061] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] See also Figure 1In a first aspect, the present invention provides a sentiment analysis method for promoting multimodal information fusion, the method comprising:

[0063] Step 101: Acquire multimodal data, preprocess each modal data, and obtain initial features of each modality; the multimodality includes text modality t, audio modality a, and video modality v.

[0064] As can be understood, this paper obtains multimodal data from the MOSI and MOSEI datasets. The MOSEI dataset contains 23,453 text items and 3,228 videos. The MOSI dataset's text data contains approximately 2,191 sentences, while the video data contains approximately 2,219 video clips. Both datasets annotate the target's sentiment intensity using the scale [-3, 3], making them suitable for multimodal sentiment analysis.

[0065] Step 101 includes the following sub-steps:

[0066] Sub-step 1011 , preprocessing the text modality data through the SentiLARE model to obtain the initial features of the text modality.

[0067] It should be noted that the SentiLARE model is an emotion-aware language representation model. Its pre-training aims to better capture the emotional information of text by pre-training on sentiment and emotion. The SentiLARE model is based on the Transformer architecture and improves upon the BERT model. To capture the linguistic knowledge of each word, it tags each word with its part-of-speech tag and uses a context-aware sentiment attention mechanism to capture sentiment in SentiWordNet.

[0068] SentiWordNet is a sentiment lexicon based on WordNet that provides sentiment polarity scores for words, including positive, negative, and neutral. These scores reflect the emotional inclination of words, enabling SentiWordNet to be used for sentiment analysis and emotion recognition in natural language processing tasks. Part-of-speech (POS) embeddings and word-level sentiment polarity embeddings are added to each sentence using the SentiLARE pre-trained model.

[0069] Sub-step 1012 , pre-processing the audio modality data using the COVAREP extraction library to obtain initial features of the audio modality.

[0070] It should be noted that the COVAREP extraction library is used to extract acoustic features related to speech emotion and intonation, including 12-Mel frequency cepstral coefficients, pitch, clear / clear segmentation features, glottal source parameters, peak slope parameters, and maximum discrete quotient. The feature dimension of the audio feature information in both the MOSI and MOSEI datasets is 74.

[0071] Sub-step 1013 , pre-processing the video modality data using the Py-Feat toolkit or the Facet system to obtain initial features of the video modality.

[0072] It should be noted that for the MOSI dataset, the Py-Feat toolkit can be used to extract facial features for each visual frame, including facial features, seven facial expressions, and facial action units. Facial landmarks are used to select frames with open eyes by calculating the eye aspect ratio. Frames with closed eyes are discarded as they may cause uncertainty in expression analysis. Finally, a 27-dimensional feature containing facial expressions and facial action units is generated for each frame with open eyes. For the MOSEI dataset, a 35-dimensional visual feature containing facial action units and facial pose is extracted using the Facet system.

[0073] See also Figure 2 ,Step 102, obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establish an acoustic vocabulary table and a visual vocabulary table respectively.

[0074] It can be understood that the K-Means algorithm is an unsupervised learning method that does not require a pre-labeled training set. K-Means is a clustering algorithm that finds K clusters of a given data set. It can also be called the K-means method. It can find K different clusters, and the center of each cluster is calculated using the mean of the values ​​contained in the cluster.

[0075] Clustering, in simple terms, involves automatically grouping data with similar characteristics within a complex dataset into clusters. The more similar the objects within a cluster, the better the clustering effect. The number of clusters, K, is user-specified, and each cluster is described by its centroid—the center of all points within the cluster. The key difference between clustering and classification algorithms is that the target category is known in classification, while the target category is unknown in clustering.

[0076] The K-Means method has the advantages of low computational complexity and simple calculation. The present invention uses the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and then establish an acoustic vocabulary table and a visual vocabulary table.

[0077] Step 102 includes the following sub-steps:

[0078] Sub-step 1021 , obtaining a training set and extracting sound segments and visual segments from the training set to obtain an audio modality frame set and a video modality frame set, respectively.

[0079] It should be noted that the training set can include video clips from various scenes, and audio and visual frames are extracted from each video clip. For example, the visual frame may contain information about the face of the person in the picture, and the audio frame may contain information such as the waveform of various sounds.

[0080] Then, the frame set of non-text mode Expressed as:

[0081]

[0082] in, A sequence of features representing non-text modalities, represents the number of frames of modality m, Indicates the length of a single mode.

[0083] In sub-step 1022, the K-Means algorithm is used to cluster and group the audio modality frame set and the video modality frame set respectively, and the audio cluster center set and the video cluster center set are learned.

[0084] By using the k-means method, The frames in are divided into Km groups as follows:

[0085]

[0086] in, , is the j-th cluster center of mode m.

[0087] Sub-step 1023, based on the audio cluster center set C a and the video cluster center C v Set up acoustic vocabulary and visual vocabulary respectively.

[0088] Step 103 : query the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary and the visual vocabulary respectively to obtain a word index sequence of the non-text modality.

[0089] Given a non-text feature sequence: , is the length of the sequence, is the feature of the i-th frame of the sequence; by querying the non-linguistic feature vocabulary established by the K-Means algorithm, the corresponding index sequence can be obtained ,for Frame i Features, Represents a sequence set in a non-text feature sequence, where the query process is calculated as follows:

[0090]

[0091] in, is the i-th frame The meaning of the above formula is to find an index , so that the feature of the i-th frame To the jth cluster center The Euclidean distance of is minimized; then, the obtained index sequence It is used as the representation of the audio modality or video modality, thereby converting the extracted non-text vector into the same word index sequence as the text modality representation, reducing the initial distribution difference between different modalities and further narrowing the distribution gap between text modality and non-text modality features during fusion.

[0092] Step 104 : embed the word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, and add the non-language displacement vector to the initial features of the text modality to perform a shift operation to obtain a multi-modal vector.

[0093] It is understandable that each word belongs to a certain part of the semantic space, which depends solely on the word's meaning within the language structure (i.e., the sentence). However, non-textual behavior can affect the meaning of words and thus their position in the semantic space. In other words, textual features and non-textual features jointly determine the new position of a word in the semantic space.

[0094] In this field, text modalities generally perform better than non-text modalities. Therefore, strengthening the importance of text modalities in multimodal sentiment analysis is a feasible way to improve the recognition accuracy of multimodal sentiment analysis. Based on this, the present invention regards the new position as the sum of the language-only position and the displacement vector, forming a vector with trajectory and magnitude. The language-only position of the word is moved to the new position based on non-linguistic behavior, thereby strengthening the weight of language and text features in the fused tensor.

[0095] See also Figure 3 (a) Step 104 specifically includes:

[0096] Sub-step 1041 , concatenates the word index sequence of the non-text modality with the initial features of the text modality to obtain an audio concatenation vector and a video concatenation vector.

[0097] Among them, the word index sequence of the non-text modality and the initial features of the text modality can be expressed as a triple ), respectively represent these inputs of the i-th word in their respective sequences, where Represents the feature input of text modality, represents the sequence input of the audio modality, Represents a sequence input of video modality.

[0098] The initial features of the text modality and the word index sequence of the non-text modality are concatenated through the Attention Gate operation to form [ ]and[ ];in, represents the audio concatenation vector, Represents the video splicing vector.

[0099] In sub-step 1042 , weight calculation is performed on the audio splicing vector and the video splicing vector respectively to obtain an audio gate vector and a video gate vector.

[0100] Audio Gate Vector The calculation method is expressed as:

[0101]

[0102] Where: represents a nonlinear activation function, represents the weight matrix of the auditory channel, A scalar deviation representing the audio modality.

[0103] Video Gate Vector The calculation method is expressed as:

[0104]

[0105] Where: represents the weight matrix of the visual channel, A scalar bias representing the video modality.

[0106] In sub-step 1043 , the audio splicing vector and the video splicing vector are multiplied by the associated gate vectors to obtain a non-language displacement vector.

[0107] In this substep, and Multiply by their respective gate vectors and add them together to create a non-linguistic displacement vector :

[0108]

[0109] Where: and Represent the weight matrices of acoustic information and visual information respectively, represents the bias vector.

[0110] Sub-step 1044 , adding the non-language displacement vector to the initial features of the text modality to perform a shift operation to obtain a multi-modal vector.

[0111] Multimodal vector The calculation method is expressed as:

[0112]

[0113] Where: represents a scaling factor that keeps the effect of the nonverbal transformation Hi within a desired range.

[0114] The present invention effectively integrates non-linguistic feature information into linguistic feature information. While strengthening the weight of linguistic text modal features, it also effectively incorporates non-linguistic text modal information into multimodal sentiment analysis, significantly improving the accuracy of multimodal sentiment analysis tasks.

[0115] Step 105: Input the multimodal vector into the Transformer model for context perception and output multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, and obtain and output the sentiment analysis result.

[0116] It should be noted that before the multimodal vector is input into the Transformer model, it needs to be processed by layer normalization (Layer Norm) and dropout layers. The dropout layer randomly discards some neurons in the output after layer normalization to prevent overfitting. The parameter of the dropout layer is set to 0.3, which means that each neuron has a 30% probability of being abandoned.

[0117] It is understandable that the Transformer model is composed of a multi-head attention mechanism, a feed forward network, a residual connection and a layer normalization module, such as Figure 3 (b) Transformer can transform the input sequence into a context-aware representation, which contains the information of each position in the input sequence and can capture the relationship between the input sequences.

[0118] Multi-head attention is an extension of the attention mechanism that can more effectively extract information when processing sequential data. In multi-head attention, multiple sets of attention weights are used, each of which can learn different semantic information and generate a context vector. Finally, these context vectors are concatenated and then subjected to a linear transformation to produce the final output.

[0119] The multi-head attention mechanism can be seen as multiple parallel self-attention mechanisms (Self-Attention), which consists of three matrices: Q (query), K (key value), and V (value). All three matrices are obtained by linearly transforming the input (fully connected layer), such as Figure 4 As shown. Among them, , W is the weight of the fully connected layer, X is the input tensor, They correspond to their respective weight matrices.

[0120] Perform matrix multiplication on matrices Q and K to calculate the similarity matrix S of the Q and K vectors. Then divide S by a specific constant α and activate it through the activation function Softmax function to obtain the attention score matrix S'. Finally, perform matrix multiplication on S' and V to obtain the output of the attention mechanism. The specific calculation process formula is as follows:

[0121]

[0122] Among them, the calculation of the specific constant α Refers to the feature dimension of the input tensor, and T represents the transpose operation of the matrix.

[0123] like Figure 3 As shown in (b), in the Transformer model, the input and output of the multi-head attention mechanism form a residual network. In addition, the input and output of the feedforward network also form a residual network. Then, layer normalization is performed after each residual network.

[0124] After the multimodal features are fused, the fusion result, that is, the new text representation after the fusion of text, visual, and acoustic modalities, is input into the Mith layer of the Transformer with a total of M layers. Then, for the Mth layer, its output will be a high-level emotional representation dominated by language text with visual and auditory information. The output of the M-layer Transformer is the input of the sentiment analysis module. In the sentiment analysis process, the final emotional result is obtained through two linear transformations, such as Figure 5 shown.

[0125] Finally, sentiment analysis results were analyzed using sentiment intensity for binary classification and regression. For binary classification, the accuracy of the binary classification (Acc-2) and the weighted F1 score (F1) were used; for regression, recognition performance was evaluated using mean absolute error (MAE) and Pearson correlation (Corr).

[0126] Please refer to Table 1. The data for the TFN, MISA, MulT, Self-MM, and MMIM models in Table 1 are from Han et al. (2021). The scores of the present invention measured on the MOSI and MOSEI datasets are shown in the experimental results in the last row of Table 1. Therefore, compared with mainstream models, the sentiment analysis method provided by the present invention has achieved certain results, can effectively integrate multimodal data, and improve the accuracy of sentiment analysis.

[0127] Table 1 Comparison of experimental results of different sentiment analysis models on MOSI and MOSEI datasets

[0128]

[0129] The present invention provides a sentiment analysis method for promoting multimodal information fusion, which has the following advantages:

[0130] 1. This paper proposes a feature conversion strategy that converts non-language vectors into indexes. The K-Means unsupervised clustering algorithm can be used to convert the original feature sequences of video images and audio into index sequences similar to text, thereby reducing the initial distribution differences between heterogeneous modalities. This will further narrow the distribution gap between text modality and non-text modality features during fusion, thereby promoting the integration of language text representation and non-language emotional context, thereby improving the accuracy of sentiment analysis.

[0131] 2. When fusing multimodal features, the language text representation can be enhanced by combining the emotional information of visual and auditory modalities, thereby increasing the weight of language text features in the fused tensor and improving the accuracy of sentiment analysis of multimodal information.

[0132] See also Figure 5 and Figure 6 , a second aspect of the present invention provides a sentiment analysis device for promoting multimodal information fusion, the device comprising: an initial feature acquisition module, a feature conversion module, a feature fusion module and a sentiment result output module;

[0133] Initial feature acquisition module 501 is used to acquire multimodal data, preprocess the data of each modality, and obtain initial features of each modality; multimodality includes text modality, audio modality, and video modality;

[0134] The feature conversion module includes a vocabulary building module and a non-text modality word index sequence generation module;

[0135] A vocabulary building module 502 is used to obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and build an acoustic vocabulary and a visual vocabulary respectively;

[0136] A non-text modality word index sequence generation module 503 is configured to query the initial features of the audio modality and the initial features of the video modality using an acoustic vocabulary and a visual vocabulary respectively to obtain a non-text modality word index sequence;

[0137] A feature fusion module 504 is used for embedding a word index sequence of a non-text modality into an initial feature of a text modality to create a non-language displacement vector, and then adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector.

[0138] The sentiment result output module 505 is used to input the multimodal vector into the Transformer model for context perception and output the multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, and obtain and output the sentiment analysis result.

[0139] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above sentiment analysis methods when executing the computer program.

[0140] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above sentiment analysis methods when the computer program is executed by a processor.

[0141] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0143] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0144] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0146] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sentiment analysis method for promoting multimodal information fusion, characterized in that: The method comprises: Acquire multimodal data, preprocess each modality data, and obtain the initial features of each modality; multimodality includes text modality, audio modality, and video modality; Obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establish an acoustic vocabulary table and a visual vocabulary table respectively; The initial features of the audio modality and the initial features of the video modality are queried through the acoustic vocabulary and the visual vocabulary respectively to obtain the word index sequence of the non-text modality; Embedding a word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, and adding the non-language displacement vector to the initial features of the text modality to perform a shift operation to obtain a multimodal vector; Input the multimodal vector into the Transformer model for context perception and output multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, and obtain and output the sentiment analysis result; The steps of embedding the word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, adding the non-language displacement vector to the initial features of the text modality to perform a shift operation, and outputting a multimodal vector include: Concatenate the word index sequence of the non-text modality with the initial features of the text modality to obtain the audio concatenation vector and the video concatenation vector; Performing weight calculation on the audio splicing vector and the video splicing vector respectively to obtain an audio gate vector and a video gate vector; The non-language displacement vector is obtained by multiplying the audio splicing vector and the video splicing vector by the associated gate vector respectively; Adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector; The audio gate vector The calculation method is expressed as: ； Where: represents a nonlinear activation function, represents the weight matrix of the auditory channel, represents the scalar deviation of the audio mode; represents the audio concatenation vector, A sequence representing the initial features of the text modality, A sequence of word indices representing the audio modality; The video gate vector The calculation method is expressed as: ； Where: represents the weight matrix of the visual channel, scalar deviation representing the video modality; represents the video splicing vector, A sequence of word indices representing the video modality; The non-verbal displacement vector The calculation method is expressed as: ； Where: and Represent the weight matrices of acoustic information and visual information respectively, represents the bias vector; The multimodal vector The calculation method is expressed as: ； Where: Represents the scaling factor.

2. The sentiment analysis method according to claim 1, characterized in that The step of acquiring multimodal data, preprocessing each modal data separately, and obtaining initial features of each modality includes: Preprocessing the data of the text modality through the SentiLARE model to obtain initial features of the text modality; Preprocessing the audio modality data using the COVAREP extraction library to obtain initial features of the audio modality; The data of the video modality is preprocessed using the Py-Feat toolkit or the Facet system to obtain initial features of the video modality.

3. The sentiment analysis method according to claim 1, characterized in that The steps of obtaining a training set, clustering the training set based on a K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and establishing an acoustic vocabulary table and a visual vocabulary table respectively include: Acquiring a training set and extracting sound segments and visual segments from the training set to obtain an audio modality frame set and a video modality frame set, respectively; The K-Means algorithm is used to cluster the audio modality frame set and the video modality frame set respectively, and the audio cluster center set and the video cluster center set are obtained by learning; An acoustic vocabulary and a visual vocabulary are respectively established based on the audio cluster center set and the video cluster center set.

4. The sentiment analysis method according to claim 3, characterized in that The calculation formulas for querying the initial features of the audio modality and the initial features of the video modality through the acoustic vocabulary and the visual vocabulary respectively are as follows: ； Where: Indicates the i-th frame The index tag, represents the features of the i-th frame of the sequence, is the j-th cluster center of mode m.

5. The sentiment analysis method according to claim 1, characterized in that The Transformer model consists of a multi-head attention mechanism, a forward network, a residual connection layer and a layer normalization module.

6. A sentiment analysis device for promoting multimodal information fusion, characterized in that: The device comprises: The initial feature acquisition module is used to obtain multimodal data, preprocess the data of each modality, and obtain the initial features of each modality; multimodality includes text modality, audio modality, and video modality; The vocabulary building module is used to obtain a training set, cluster the training set based on the K-Means algorithm to learn acoustic vocabulary and visual vocabulary, and build acoustic vocabulary and visual vocabulary respectively; A word index sequence generation module for non-text modalities is used to query the initial features of the audio modality and the initial features of the video modality through an acoustic vocabulary and a visual vocabulary respectively to obtain a word index sequence for the non-text modality; A feature fusion module is used for multimodal vectors, which embeds the word index sequence of the non-text modality into the initial features of the text modality to establish a non-language displacement vector, and adds the non-language displacement vector to the initial features of the text modality to perform a shift operation to obtain a multimodal vector; The sentiment result output module is used to input the multimodal vector into the Transformer model for context perception and output multimodal perception information; input the multimodal perception information into the sentiment analysis module for sentiment analysis, obtain and output the sentiment analysis result; The feature fusion module is specifically used to: Concatenate the word index sequence of the non-text modality with the initial features of the text modality to obtain the audio concatenation vector and the video concatenation vector; Performing weight calculation on the audio splicing vector and the video splicing vector respectively to obtain an audio gate vector and a video gate vector; The non-language displacement vector is obtained by multiplying the audio splicing vector and the video splicing vector by the associated gate vector respectively; Adding the non-language displacement vector to the initial feature of the text modality to perform a shift operation to obtain a multi-modal vector; The audio gate vector The calculation method is expressed as: ； Where: represents a nonlinear activation function, represents the weight matrix of the auditory channel, represents the scalar deviation of the audio mode; represents the audio concatenation vector, A sequence representing the initial features of the text modality, A sequence of word indices representing the audio modality; The video gate vector The calculation method is expressed as: ； Where: represents the weight matrix of the visual channel, scalar deviation representing the video modality; represents the video splicing vector, A sequence of word indices representing the video modality; The non-verbal displacement vector The calculation method is expressed as: ； Where: and Represent the weight matrices of acoustic information and visual information respectively, represents the bias vector; The multimodal vector The calculation method is expressed as: ； Where: Represents the scaling factor.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the sentiment analysis method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the sentiment analysis method according to any one of claims 1 to 5 are implemented.

Images