A multimodal sentiment classification method, device, equipment and storage medium

Abstract

The present invention is applicable to the field of artificial intelligence technology, and in particular relates to a multimodal sentiment classification method, device, equipment and storage medium, which extracts speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information, uses a preset bidirectional long short-term memory model to obtain a speech hidden feature sequence and a text hidden feature sequence, calculates a target feature sequence, performs feature fusion with corresponding features in the text hidden feature sequence, inputs the fused features into a fully connected layer, obtains a sentiment classification result of the speech information, interactively processes the speech feature sequence and the text feature sequence, obtains attention weights of corresponding features in the speech feature sequence and the text feature sequence, increases the correlation between the speech features and the text features through the attention weights, thereby obtaining more accurate fusion features, thereby improving the accuracy of sentiment classification.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a multimodal emotion classification method, apparatus, device and storage medium. Background Art

[0002] With the continuous development of social networks, the forms of expressing emotions are becoming increasingly diverse. Traditional single emotion recognition models, such as those that solely use the time domain, frequency domain, and related characteristics of speech signals to represent emotions, are no longer sufficient to identify complex emotional information. Therefore, the use of multimodal emotion recognition has become a powerful tool for identifying complex emotions. Multimodal emotion recognition incorporates more information. For example, adding text information to speech signals can convey richer emotions and capture more information that may be contained in the speech.

[0003] Existing multimodal emotion recognition methods typically construct different sub-models to capture the emotional features corresponding to different modal data, and then fuse these features. However, this process weakens the correlation and interaction between different modalities, which affects the accuracy of multimodal emotion classification. Therefore, how to enhance the interaction between different modalities to improve the accuracy of multimodal emotion classification has become an urgent problem to be solved. Summary of the Invention

[0004] Based on this, it is necessary to provide a multimodal sentiment classification method, device, equipment and storage medium to address the above technical problems, so as to solve the problem of low accuracy of multimodal sentiment classification.

[0005] In a first aspect, a multimodal sentiment classification method is provided, the method comprising:

[0006] Extracting speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information; the speech feature sequence includes a feature vector corresponding to at least one segment of speech; and the text feature sequence includes a feature vector corresponding to at least one segment of text;

[0007] Using a preset first bidirectional long short-term memory model to encode the speech feature sequence to obtain a speech hidden feature sequence corresponding to the speech feature sequence, and using a preset second bidirectional long short-term memory model to encode the text feature sequence to obtain a text hidden feature sequence corresponding to the text feature sequence;

[0008] Calculating an attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence, and using the attention weight to update the speech hidden feature sequence to obtain a target feature sequence; the length of the target feature sequence is the same as the length of the text hidden feature sequence;

[0009] The features in the target feature sequence are fused with the corresponding features in the text hidden feature sequence to obtain fused features, and the fused features are input into a fully connected layer to obtain the emotion classification result of the speech information.

[0010] In a second aspect, a multimodal sentiment classification device is provided, the device comprising:

[0011] A speech feature sequence and text feature sequence extraction module is used to extract speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information; the speech feature sequence includes a feature vector corresponding to at least one speech segment; the text feature sequence includes a feature vector corresponding to at least one text segment;

[0012] a hidden feature sequence extraction module, configured to encode the speech feature sequence using a preset first bidirectional long short-term memory model to obtain a speech hidden feature sequence corresponding to the speech feature sequence, and to encode the text feature sequence using a preset second bidirectional long short-term memory model to obtain a text hidden feature sequence corresponding to the text feature sequence;

[0013] an attention weight acquisition module, configured to calculate an attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence, and update the speech hidden feature sequence using the attention weight to obtain a target feature sequence; the target feature sequence has the same length as the text hidden feature sequence;

[0014] The classification module is used to fuse the features in the target feature sequence with the corresponding features in the text hidden feature sequence to obtain fused features, and input the fused features into the fully connected layer to obtain the emotion classification result of the speech information.

[0015] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal-based emotion classification method as described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal-based emotion classification method as described in the first aspect is implemented.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] Extract speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information, wherein the speech feature sequence includes a feature vector corresponding to at least one speech segment; the text feature sequence includes a feature vector corresponding to at least one text segment; use a preset first bidirectional long short-term memory model to encode the speech feature sequence to obtain a speech hidden feature sequence corresponding to the speech feature sequence; use a preset second bidirectional long short-term memory model to encode the text feature sequence to obtain a text hidden feature sequence corresponding to the text feature sequence; calculate the attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence; use the attention weight to update the speech hidden feature sequence to obtain a target feature sequence; the length of the target feature sequence is the same as the length of the text hidden feature sequence; the features in the target feature sequence are fused with the corresponding features in the text hidden feature sequence to obtain a fused feature; the fused feature is input into a fully connected layer to obtain a sentiment classification result of the speech information; the speech feature sequence and the text feature sequence are interactively processed to obtain the attention weights of the corresponding features in the speech feature sequence and the text feature sequence; the correlation between the speech feature and the text feature is increased by the attention weight, thereby obtaining a more accurate fused feature, thereby improving the accuracy of sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 1 is a schematic diagram of an application environment of a multimodal sentiment classification method provided by an embodiment of the present invention;

[0021] Figure 2 This is a flowchart of a multimodal sentiment classification method provided by one embodiment of the present invention;

[0022] Figure 3 1 is a schematic structural diagram of a multimodal emotion classification system provided by an embodiment of the present invention;

[0023] Figure 4 It is a structural diagram of a computer device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0026] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0028] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] It should be understood that the order of execution of the steps in the following embodiments does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0031] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0032] An embodiment of the present invention provides a multimodal sentiment classification method that can be applied in Figure 1 In an application environment, a client communicates with a server. The client includes, but is not limited to, a palmtop computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook computer, a personal digital assistant (PDA), and other computer devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0033] See also Figure 2 , is a flowchart of a multimodal sentiment classification method provided by an embodiment of the present invention. The multimodal sentiment classification method can be applied to Figure 1 The server in the above-mentioned server connects to the corresponding client and provides the client with multi-modal emotion classification management services. Figure 2 As shown, the multimodal sentiment classification method may include the following steps:

[0034] S201: Extracting speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information.

[0035] In step S201, speech features and text features are extracted from the speech information respectively. The speech feature sequence includes a feature vector corresponding to at least one segment of speech; the text feature sequence includes a feature vector corresponding to at least one segment of text. The speech feature sequence includes a feature vector for each speech frame, and the text feature sequence includes a word vector for each word.

[0036] In this embodiment, the speech features in the speech information are extracted by an acoustic model to obtain a speech feature sequence, wherein the acoustic model is a differentiated knowledge representation of acoustic, phonetic, environmental variables, speaker gender, and accent, including acoustic models based on Hidden Markov Model (HMM), such as a mixed Gaussian-Hidden Markov Model (GMM-HMM) and a deep neural network-Hidden Markov Model (DNN-HMM). The Hidden Markov Model is a weighted finite state automaton in a discrete time domain. Of course, end-to-end acoustic models can also be included, such as a continuous time classification-long short-term memory (CTC-LSTM) model and an attention model. Each state of the acoustic model represents the probability distribution of the speech features of a speech unit (such as a word, syllable, and factor, etc.) in that state, and is connected into an ordered state sequence through transitions between states, that is, a speech feature sequence of a segment of speech information is obtained.

[0037] The language model extracts the text features of the speech information to obtain a text feature sequence. The new speech model is encoded and processed to extract the corresponding speech features. Multiple sets of speech features are input into the language model. The language model estimates the probability (or rationality) of each possible speech set and determines the set of feature vectors with the highest probability from the multiple possible speech features. The text sequence corresponding to the set of feature vectors with the highest probability is the most consistent with grammatical rules. Finally, the language model obtains the text sequence corresponding to the optimal feature vector based on the pronunciation dictionary and uses this text sequence as the text feature sequence corresponding to the speech information.

[0038] It should be noted that acoustic models include but are not limited to hybrid acoustic models such as GMM-HMM-based acoustic models, DNN-HMM-based acoustic models, RNN-HMM-based acoustic models, and CNN-HMM-based acoustic models, as well as end-to-end acoustic models such as the continuous time series classification-long short-term memory model CTC-LSTM and the attention model. The embodiment of the present invention does not impose any restrictions on the selection of acoustic models. Language models include but are not limited to statistical language models and neural network language models. Classic statistical language models include N-gram language models. The embodiment of the present invention does not impose any restrictions on the selection of language models.

[0039] Optionally, extracting voice features and text features from the voice information to obtain a voice feature sequence corresponding to the voice information and a text feature sequence corresponding to the voice information includes:

[0040] Extract the time domain features of the speech information and convert the time domain features into frequency domain features to obtain the speech feature sequence of the speech information;

[0041] Use the preset pre-training model to train the text features and obtain the target text feature sequence corresponding to the text features.

[0042] In the present embodiment, when extracting the speech feature sequence, the speech feature sequence is obtained by extracting the time domain features and frequency domain features of the speech information, and the time domain features are converted into the number of frames of a fixed window. For example, the speech information sample related to multiple segments of objects can be subjected to framing processing to obtain multiple speech frames, and these multiple speech frames are arranged in sequence according to time sequence to obtain the speech frame sequence related to the multiple segments of objects. Alternatively, after obtaining multiple speech frames, these multiple speech frames can be sampled based on a predetermined sampling interval, and the sampled speech frames are arranged in time sequence to obtain the speech frame sequence. The number of frames corresponding to the fixed window is determined from the speech frame sequence, wherein the predetermined sampling interval is greater than or equal to 1 frame. By sampling the speech frames, the amount of calculation can be reduced while ensuring training accuracy. This is because the pronunciation of a text usually lasts for multiple speech frames. In the windowing process of the windowing process obtained by the framing processing, the interruptions that occur at the beginning and end of the frame are minimized. After this process, the continuous speech information is converted into windowing frames. These window frames are passed to the discrete Fourier transformer, which converts the windowed frames into frequency domain features, calculates Mel-Frequency Cepstral Coefficients (MFCCs) from the frequency domain features to obtain speech feature vectors, and obtains a speech feature sequence based on the speech feature vectors.

[0043] In this embodiment, text features corresponding to speech information are obtained through a feedforward neural network model, wherein the representation method of historical information in the feedforward neural network model and the N-gram language model use the first n-1 words for representation.

[0044] It should be noted that when acquiring text features, recurrent neural network language models can also be used to extract text features from speech information. The main difference between recurrent neural network language models and feedforward neural network language models lies in the different ways they represent word history information. Recurrent neural network models acquire historical information through continuous looping of hidden layers and learning from them. In recurrent neural networks, the hidden layers can represent historical information beyond just the previous n-1 words. In theory, this model can represent longer-term contextual information.

[0045] In this embodiment, after obtaining the text features corresponding to the voice information, the text features are trained through a preset pre-training model to obtain a text feature sequence. The pre-training model is to convert text features into embedding vectors. In the pre-training model, after the model completes multiple pre-training tasks on a large-scale data set, its ability to extract features is very strong, so that it can adapt to more complex tasks. This stage is called the fine-tuning stage. The fine-tuning stage only requires replacing the classifier of the network for different tasks, which is very convenient. Thanks to the huge pre-training data and the training of multiple pre-training tasks, the model has very strong generalization, which greatly improves the accuracy of the fine-tuning task. The pre-training model is the part of the network that extracts features using a specific target task training network on a large data set. Since most of the parameters in the model no longer need to be trained, the convergence speed of the model in the fine-tuning stage is very fast.

[0046] Optionally, a preset pre-training model is used to train the text features to obtain a target text feature sequence corresponding to the text features, including:

[0047] Use the preset pre-training model to train the text features and obtain the initial text feature sequence;

[0048] Through the mask mechanism, the initial text feature sequence is supplemented and processed to obtain the target text feature sequence.

[0049] In this embodiment, the pre-trained model used is the BERT pre-trained model, which is represented by the bidirectional encoder of Transformer. Bert pre-trains deep bidirectional representations by jointly adjusting the context in all layers, mainly using the Encoder of Transformer. The BERT pre-trained model extracts text information from all layers above, below, left, and right through training on a large amount of corpus to achieve bidirectional representation of text. Since it extracts context, words, sentences, etc. in detail, the word vectors it obtains are dynamic, ensuring that the word vectors obtained for the same word in different language environments are different, which can better express the relationship between words, sentences, and context.

[0050] The input feature representation of the BERT pre-trained model consists of three components: word embedding, fragment embedding, and position embedding. The final input model vector representation is obtained by summing the corresponding positions. For example, the word embedding matrix has dimensions [4,768], the fragment embedding matrix has dimensions [3,768], and the position embedding matrix has dimensions [2,768]. For a character, its one-hot encoding is [1,0,0,0], its position one-hot encoding is [1,0,0], and its fragment one-hot encoding is [1,0]. The combined feature of these three one-hot encodings is [1,0,0,0,1,0,0,1,0]. Passing this through a fully connected layer of dimension [4+3+2,768] = [9,768] yields a word vector equivalent to the sum of the three matrix dimensions. Therefore, the BERT training model uses the word vector obtained by summing these three dimensions as the model input. The BERT pre-training model learns the text semantic information of the fused features, which is beneficial to model training and improves the accuracy of the model.

[0051] Through the BERT pre-training model, the embedded word vectors corresponding to the text features are obtained. In a piece of text, the word vector sequence is composed of word vectors to obtain the initial text feature sequence.

[0052] The length of the initial text sequence is the number of sentences in a conversation. When the number of sentences in a conversation varies, the length of the initial text feature sequence varies. To obtain a fixed-length text feature sequence, the initial text features are processed into variable-length sequences. Shorter sequences in the initial text feature sequence are padded using a masking mechanism. Shorter sequences in the text feature sequence are padded with 0s. First, the longest text feature sequence is obtained from the multiple obtained text feature sequences. The length of this text feature sequence is used as the template length. The remaining text feature sequences are padded with 0s to obtain the target text feature sequence. Suppose there is a vector of length 5: x = [1, 9, 3, 4, 5]. Using the masking mechanism to padded this vector sequence to a vector of length 8, the result is x = [1, 9, 3, 4, 5, 0, 0, 0].

[0053] Optionally, a preset pre-training model is used to train text features to obtain an initial text feature sequence, including:

[0054] Perform word segmentation processing on text features to obtain word segmentation data corresponding to the text features;

[0055] Use the preset pre-training model to train the word segmentation data to obtain the initial text feature sequence.

[0056] In this embodiment, when training text features, generally text features contain multiple sentences, and a sentence contains multiple words. The training model is to train word vectors, so the text features are segmented to obtain corresponding segmented data.

[0057] During word segmentation, for English text, word segmentation can be performed according to its space, and for Chinese text, it is possible to compare the character string in the text content with the word segmentation dictionary. By contrast, if the dictionary contains the character string, the character string and the word match successfully, then the two ends of the character string are segmented, otherwise not segmented. This method is generally based on a very large word segmentation dictionary. If the word segmentation dictionary is insufficient or wrong, the word segmentation effect will be greatly reduced. Mainly containing N-gram grammar model and hidden Markov model based on statistical word segmentation algorithm, they are based on the frequency of adjacent characters in the sentence to work, and judge whether two characters are the same word according to the size of the frequency. This method can segment all characters in the text, and belongs to the method of full segmentation. In the present embodiment, the method of using the two combined is carried out word segmentation to text features to obtain word segmentation data.

[0058] When using a pre-trained model to train word segmentation data, each sentence is distinguished by adding characteristic characters. A special marker [CLS] is added to the beginning of each sentence to represent the entire sentence or sentence pair. Although this special marker [CLS] participates in the sequence encoding, it is ignored in the final output. Therefore, the pre-trained model packages a segment of text features into a single sentence for processing. Pre-training the model can obtain the word vectors corresponding to the text features, thereby obtaining a text feature sequence containing the word vectors.

[0059] S202: Use a preset first bidirectional long short-term memory model to encode the speech feature sequence to obtain a speech hidden feature sequence corresponding to the speech feature sequence, and use a preset second bidirectional long short-term memory model to encode the text feature sequence to obtain a text hidden feature sequence corresponding to the text feature sequence.

[0060] In step S202, a hidden feature sequence is extracted from the speech feature sequence and the text feature sequence. The hidden feature sequence abstracts the speech feature sequence and the text feature sequence, and represents the states of the speech feature sequence and the text feature sequence at different times.

[0061] In this embodiment, a first bidirectional long short-term memory model and a second bidirectional long short-term memory model are used to encode the speech feature sequence and the text feature sequence, respectively. The bidirectional long short-term memory model contains two independent hidden vectors that propagate in opposite directions. Therefore, for the same input data, two hidden outputs are ultimately obtained, that is, two feature vectors related to the input information. The bidirectional long short-term memory model then concatenates or averages the two feature vectors to obtain a single vector, which is then output.

[0062] It should be noted that the bidirectional long short-term memory model uses the BPTT (Back Propagation Through Time) algorithm to train and update all parameters, including the input layer, hidden layer, and output layer. The key to model training is to calculate the gradient of the error with respect to the model parameters and then use gradient descent to learn the optimal parameters. Because two inputs are input in completely opposite directions, information input later in one direction is prioritized in the other direction. This reduces the differential influence of the model input information. The bidirectional long short-term memory model can alleviate the problem of vanishing gradients.

[0063] S203: Calculate the attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence, and use the attention weight to update the speech hidden feature sequence to obtain a target feature sequence.

[0064] In step S203, the length of the target feature sequence is the same as the length of the text hidden feature sequence. The features in the speech hidden feature sequence are aligned with the features in the text hidden feature sequence, and the weights between the features in the speech hidden feature sequence and any features in the text hidden feature sequence are calculated. The weights are the final state attention probability distributions between the features in the speech hidden feature sequence and the features in the text hidden feature sequence. Based on the corresponding weights, the feature vectors in the speech hidden feature sequence and the target features in the speech hidden feature sequence are updated.

[0065] In this embodiment, the different influences between the features in the speech hidden feature sequence and the features in the text hidden feature sequence are calculated through the attention mechanism. The bidirectional long-short-term memory model based on the attention mechanism uses the final state of the speech hidden feature sequence and the text hidden feature sequence at each moment to calculate the attention probability distribution of the final state at each moment, thereby using the attention probability distribution between the features in the speech hidden feature sequence and the features in the text hidden feature sequence to calculate the final state, and obtain the influence of any feature in the speech hidden feature sequence and any text hidden feature sequence, that is, the attention weight. Using the attention weight, the speech hidden feature sequence is updated to obtain the target feature sequence, and the features in the target feature sequence are optimized using the attention weight, as shown in Formula (1).

[0066]

[0067] In formula (1), is the jth speech feature vector in the target feature sequence, d i is the i-th speech feature vector in the speech hidden feature sequence, α j,i is the text feature vector h j In the speech feature vector d i The attention weight on .

[0068] The addition of the attention mechanism can give different weights to the calculation of historical nodes in the bidirectional long-short-term memory model, so that the features can solve the problem of information redundancy to the greatest extent while retaining the most effective information, thereby optimizing speech features and text features and further optimizing feature representation.

[0069] It should be noted that in multimodal features, attention is focused on speech features and text features, so attention is used on both speech hidden feature sequences and text hidden feature sequences to obtain more information.

[0070] Optionally, calculating the attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence includes:

[0071] Use the alignment function to calculate the mapping probability between any feature in the text hidden feature sequence and all features in the speech hidden feature sequence;

[0072] According to the mapping probability, the attention distribution size of any feature in the text hidden feature sequence in each feature in the speech hidden feature sequence is calculated to obtain the attention weight of any feature in the text hidden feature sequence and each feature in the speech hidden feature sequence. In this embodiment, the tanh function is used as the alignment function. Through the attention mechanism, the attention probability distribution between the features in the speech hidden feature sequence and the features in the text hidden feature sequence can be obtained. Based on the probability distribution between the features, the probability between the final state of the speech hidden feature and the text hidden feature is calculated, that is, the attention weight. This is shown in the following formulas (2) and (3).

[0073]

[0074]

[0075] In formula (2), tanh function is the alignment function, d i is the i-th speech feature vector in the speech hidden feature sequence, h j is the jth text feature vector in the text hidden feature sequence, u, v are learnable parameter matrices, a j,i is the speech feature vector d i In the text feature vector h j In formula (3), N is the length of the speech hidden feature sequence, α j,i is the text feature vector h j In the speech feature vector d i The size of attention on , that is, the attention weight.

[0076] By using the attention probability distribution, we can control the influence of speech hidden features on text hidden features, while retaining more valuable information, reducing the impact of irrelevant or weakly related information on the output data.

[0077] It should be noted that when the alignment operation is performed through the alignment model, the correlation between the hidden features of the speech and the hidden features of the text is obtained. When the attention mechanism is used to map the unimodal features, the correlation between the unimodal features and the labels is learned, and the attention distribution of the labels on the unimodal features is learned. In this embodiment, when the attention mechanism is used to map the multimodal features, the correlation between the hidden features of the speech and the hidden features of the text is learned, and the attention distribution of the hidden features of the text on the hidden features of the speech is learned.

[0078] S204: Fusing the features in the target feature sequence with the corresponding features in the text hidden feature sequence to obtain fused features, and inputting the fused features into the fully connected layer to obtain the emotion classification result of the speech information.

[0079] In step S204, the features in the target feature sequence are fused with the corresponding features in the text hidden feature sequence to obtain fused features. The fused features represent the speech features and text features in the speech information. The fused features are passed through a fully connected layer to obtain the emotion classification result of the speech information.

[0080] In this embodiment, the features of different modalities are combined to form a total feature. The total feature represents the features in the hidden feature sequence of speech and also represents the features in the hidden feature vector of text. The total feature is input into the classifier in the fully connected layer to obtain the sentiment classification result.

[0081] Optionally, the features in the target feature sequence are fused with the corresponding features in the text hidden feature sequence to obtain a fusion result, including:

[0082] The features in the target feature sequence are concatenated with the corresponding features in the text hidden feature sequence to obtain a multimodal feature vector sequence;

[0083] Use the preset fusion model to fuse the multimodal feature vector sequence to obtain the fusion feature.

[0084] In this embodiment, the features in the target feature sequence are spliced ​​with the corresponding features in the text hidden feature sequence to obtain a multimodal feature vector sequence. The splicing process links the multimodal feature information to play a role of mutual complementation.

[0085] A multimodal feature vector sequence is fed into a bidirectional gated recurrent model as input features, which is then fused. This model adds a reverse inference mechanism to the classic gated recurrent model, where the last input feature vector is propagated backward along the time step. The vector corresponding to the last feature vector from the forward and reverse learning steps is concatenated and output as the final result, enhancing the model's expressiveness. It can be seen that the use of a bidirectional gated recurrent model can better resolve long-term dependencies between connected features, fully reflecting the relationship between previous and later inputs in the current output.

[0086] Optionally, the fused features are input into a fully connected layer to obtain the emotion classification results of the speech information, including:

[0087] Through the pooling layer, the fusion feature is subjected to the maximum pooling operation to obtain the standard fusion feature;

[0088] The standard fusion features are input into the fully connected layer, and the fusion feature values ​​are classified to obtain the emotion classification results of the speech information.

[0089] In this embodiment, the fused features are subjected to maximum pooling processing, the largest feature is used as the feature expression within the current receptive field, the dimension of the fused features is reduced, and a standard fused feature of a fixed length is obtained. The standard fused feature is input into the fully connected layer for further feature extraction to obtain a feature vector, and the feature vector is sent to the softmax layer for final sentiment classification, wherein the setting of the objective function is often based on the cross entropy loss function.

[0090] Extract speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information, wherein the speech feature sequence includes a feature vector corresponding to at least one speech segment; the text feature sequence includes a feature vector corresponding to at least one text segment, encode the speech feature sequence using a preset first bidirectional long short-term memory model to obtain a speech hidden feature sequence corresponding to the speech feature sequence, encode the text feature sequence using a preset second bidirectional long short-term memory model to obtain a text hidden feature sequence corresponding to the text feature sequence, calculate the attention weights between the features in the speech hidden feature sequence and any features in the text hidden feature sequence, obtain a target feature sequence using the attention weights, fuse the features in the target feature sequence with the corresponding features in the text hidden feature sequence to obtain fused features, input the fused features into a fully connected layer, obtain the sentiment classification result of the speech information, interactively process the speech feature sequence and the text feature sequence to obtain the attention weights of the corresponding features in the speech feature sequence and the text feature sequence, increase the correlation between the speech features and the text features through the attention weights, thereby obtaining more accurate fused features, thereby improving the accuracy of sentiment classification.

[0091] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a multimodal emotion classification device provided by an embodiment of the present invention. In this embodiment, the terminal includes various units for executing Figure 2 Each step in the corresponding embodiment. Please refer to Figure 2 as well as Figure 2 For the sake of convenience, only the parts related to this embodiment are shown.

[0092] See also Figure 3 , the classification device 30 includes:

[0093] The feature extraction module 31 is used to extract speech features and text features from the speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information; the speech feature sequence includes a feature vector corresponding to at least one speech segment; and the text feature sequence includes a feature vector corresponding to at least one text segment;

[0094] The hidden feature determination module 32 is configured to encode the speech feature sequence using a preset first bidirectional long short-term memory model to obtain a speech hidden feature sequence corresponding to the speech feature sequence, and to encode the text feature sequence using a preset second bidirectional long short-term memory model to obtain a text hidden feature sequence corresponding to the text feature sequence;

[0095] A target sequence determination module 33 is configured to calculate an attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence, and update the speech hidden feature sequence using the attention weight to obtain a target feature sequence;

[0096] The classification module 34 is used to fuse the features in the target feature sequence with the corresponding features in the text hidden feature sequence to obtain fused features, and input the fused features into the fully connected layer to obtain the emotion classification result of the speech information.

[0097] Optionally, the feature extraction module 31 includes:

[0098] A speech feature sequence acquisition unit is used to extract the time domain features of the speech information and convert the time domain features into frequency domain features to obtain the speech feature sequence of the speech information;

[0099] The target sequence acquisition unit is used to train the text features using a preset pre-training model to obtain a target text feature sequence corresponding to the text features.

[0100] Optionally, the target sequence acquisition unit includes:

[0101] The initial sequence acquisition subunit uses a preset pre-training model to train text features and obtain an initial text feature sequence;

[0102] The supplementary processing subunit is used to perform supplementary processing on the initial text feature sequence through a mask mechanism to obtain a target text feature sequence.

[0103] Optionally, the target sequence determination module 33 includes:

[0104] An alignment unit, configured to obtain a mapping probability between a feature in a speech hidden feature sequence and any feature in a text hidden feature sequence through an alignment model;

[0105] The attention weight acquisition unit is used to calculate the attention weight between the features in the speech hidden feature sequence and any features in the text hidden feature sequence according to the mapping probability.

[0106] Optionally, the classification module 34 includes:

[0107] A multimodal sequence acquisition unit is used to concatenate features in the target feature sequence with corresponding features in the text hidden feature sequence to obtain a multimodal feature vector sequence;

[0108] Use the preset fusion model to fuse the multimodal feature vector sequence to obtain the fusion feature.

[0109] The pooling processing unit is used to perform a maximum pooling operation on the fusion features through the pooling layer to obtain the standard fusion features;

[0110] The classification result acquisition unit is used to input the standard fusion feature into the fully connected layer, classify the fusion feature value, and obtain the emotion classification result of the speech information.

[0111] Figure 4 This is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, any of the above-mentioned steps of the multimodal-based emotion classification method are implemented.

[0112] The computer device may include, but is not limited to, a processor and a memory. It will be understood by those skilled in the art that Figure 4 The above is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0113] The processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or any conventional processor.

[0114] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the computer device, and in other embodiments, it can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory can also include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, boot loaders (BootLoader), data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or is about to be output.

[0115] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned method embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0116] The present invention may implement all or part of the processes in the above-mentioned method embodiments, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0117] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0118] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0119] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A multimodal sentiment classification method, characterized in that: include: Extracting speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information; the speech feature sequence includes a feature vector corresponding to at least one segment of speech; and the text feature sequence includes a feature vector corresponding to at least one segment of text; Using a preset first bidirectional long short-term memory model to encode the speech feature sequence to obtain a speech hidden feature sequence corresponding to the speech feature sequence, and using a preset second bidirectional long short-term memory model to encode the text feature sequence to obtain a text hidden feature sequence corresponding to the text feature sequence; Calculating an attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence, and using the attention weight to update the speech hidden feature sequence to obtain a target feature sequence; the length of the target feature sequence is the same as the length of the text hidden feature sequence; Concatenating the features in the target feature sequence with the corresponding features in the text hidden feature sequence to obtain a multimodal feature vector sequence; The multimodal feature vector sequence is used as input feature and input into the bidirectional gated recurrent model. The multimodal feature vector sequence is fused by the bidirectional gated recurrent model to obtain fused features. The fused features are input into the fully connected layer to obtain the emotion classification result of the speech information.

2. The multimodal sentiment classification method according to claim 1, wherein: The extracting of speech features and text features from the speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information includes: Extracting time domain features of the voice information and converting the time domain features into frequency domain features to obtain a voice feature sequence of the voice information; The text features are trained using a preset pre-training model to obtain a target text feature sequence corresponding to the text features.

3. The multimodal sentiment classification method according to claim 2, wherein: The method of using a preset pre-training model to train the text features to obtain a target text feature sequence corresponding to the text features includes: Using a preset pre-training model, the text features are trained to obtain an initial text feature sequence; The initial text feature sequence is supplemented by a mask mechanism to obtain a target text feature sequence.

4. The multimodal sentiment classification method according to claim 3, wherein: The method of using a preset pre-training model to train the text features to obtain an initial text feature sequence includes: Performing word segmentation processing on the text feature to obtain word segmentation data corresponding to the text feature; The word segmentation data is trained using a preset pre-training model to obtain an initial text feature sequence.

5. The multimodal sentiment classification method according to claim 1, wherein: The calculating of the attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence includes: Calculate the mapping probability between any feature in the text hidden feature sequence and all features in the speech hidden feature sequence using an alignment function; According to the mapping probability, the attention distribution size of any feature in the text hidden feature sequence in each feature in the speech hidden feature sequence is calculated to obtain the attention weight of any feature in the text hidden feature sequence and each feature in the speech hidden feature sequence.

6. The multimodal sentiment classification method according to claim 1, wherein: Inputting the fusion features into a fully connected layer to obtain a sentiment classification result of the speech information includes: Through the pooling layer, a maximum pooling operation is performed on the fusion feature to obtain a standard fusion feature; The standard fusion feature is input into the fully connected layer, and the fusion feature value is classified to obtain the emotion classification result of the speech information.

7. A multimodal sentiment classification device, characterized in that: The device comprises: A feature extraction module is used to extract speech features and text features from speech information to obtain a speech feature sequence corresponding to the speech information and a text feature sequence corresponding to the speech information; the speech feature sequence includes a feature vector corresponding to at least one speech segment; the text feature sequence includes a feature vector corresponding to at least one text segment; a hidden feature determination module, configured to encode the speech feature sequence using a preset first bidirectional long short-term memory model to obtain a speech hidden feature sequence corresponding to the speech feature sequence, and to encode the text feature sequence using a preset second bidirectional long short-term memory model to obtain a text hidden feature sequence corresponding to the text feature sequence; a target sequence determination module, configured to calculate an attention weight between each feature in the speech hidden feature sequence and any feature in the text hidden feature sequence, and update the speech hidden feature sequence using the attention weight to obtain a target feature sequence; A classification module, configured to concatenate features in the target feature sequence with corresponding features in the text hidden feature sequence to obtain a multimodal feature vector sequence; The multimodal feature vector sequence is used as input feature and input into the bidirectional gated recurrent model. The multimodal feature vector sequence is fused by the bidirectional gated recurrent model to obtain fused features. The fused features are input into the fully connected layer to obtain the emotion classification result of the speech information.

8. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multimodal-based emotion classification method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal-based emotion classification method according to any one of claims 1 to 6 is implemented.

Images