Multi-modal sentiment analysis method and system, electronic equipment and storage medium

Abstract

The invention provides a multi-modal sentiment analysis method and system, electronic equipment and a storage medium. The method comprises the following steps: S1, acquiring user voice and expression pictures; and S2, inputting the user voice and the expression picture into a CRNN-SVM model for feature extraction, merging the extracted features through a vector splicing technology, classifying the features, and judging the emotion polarity. According to the method, information of two modes of voice and expression pictures is combined, sentiment features of more dimensions can be captured, sentiment analysis is carried out by fusing data of different modes, dependence on a single data source is avoided, errors caused by the quality problem of the single data source are reduced, and the accuracy of sentiment analysis is improved. The multi-source data fusion mode enables the model to be more robust, and can adapt to sentiment analysis tasks in different scenes.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal emotion analysis technology such as speech and image, and specifically relates to a multimodal emotion analysis method, system, electronic device and storage medium. Background Art

[0002] Multimodal sentiment analysis is an important research direction in artificial intelligence. It refers to the use of multiple types of input data, such as text, speech, and images, to identify and predict emotional states. Traditional sentiment analysis mainly focuses on text content. Recurrent neural networks (RNNs) and long short-term memory models (LSTMs) are commonly used sentiment analysis models in the past. However, only a single information source cannot enable the model to learn rich features. Multimodal sentiment analysis provides multiple features for deep learning models by integrating visual, auditory and other information. While improving the accuracy of the model, it is also conducive to enhancing the generalization ability of the model, enabling the model to handle different types of downstream sentiment analysis tasks. Summary of the invention

[0003] The purpose of the present invention is to design a multimodal sentiment analysis method that can analyze various types of information, improve the accuracy of sentiment analysis tasks, and be used to analyze the emotional state of users when making voice calls.

[0004] The technical solution of the present invention is as follows:

[0005] A multimodal sentiment analysis method, the method comprising:

[0006] S1 obtains user voice and expression pictures;

[0007] S2 inputs the user's voice and expression picture into the CRNN-SVM model for feature extraction, and merges the extracted features through vector splicing technology for classification to determine the emotional polarity.

[0008] Furthermore, the CRNN-SVM model is composed of a convolutional neural network, a recurrent neural network and a support vector model.

[0009] Furthermore, the S2 is specifically:

[0010] Read the user's voice information, convert the time series signal into a frequency domain signal through Fourier transform, draw the original spectrum, and use convolutional neural network and recurrent neural network to extract the spatial features and time series features of the original spectrum respectively;

[0011] Use convolutional neural networks to extract features of expression images;

[0012] Through vector splicing technology, the extracted spatial features, temporal features and features of expression pictures are merged and input into the support vector model for classification to determine the emotional polarity.

[0013] A multimodal sentiment analysis system, comprising:

[0014] Acquisition module, used to obtain user voice and expression pictures;

[0015] The feature extraction and classification module is used to input the user's voice and expression picture into the CRNN-SVM model for feature extraction, and merge the extracted features through vector splicing technology for classification to determine the emotional polarity.

[0016] Further, the feature extraction and classification module includes:

[0017] The speech feature extraction module is used to read the user's speech information, convert the time series signal into a frequency domain signal through Fourier transform, draw the original spectrum, and use the convolutional neural network and recurrent neural network to extract the spatial features and time series features of the original spectrum respectively;

[0018] Image feature extraction module, used to extract features of expression images using convolutional neural networks;

[0019] The classification module is used to combine the extracted spatial features, temporal features and features of expression pictures through vector splicing technology and input them into the support vector model for classification to determine the emotional polarity.

[0020] The present invention also provides an electronic device, characterized in that it includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-mentioned multimodal sentiment analysis method.

[0021] The present invention also provides a computer-readable storage medium, characterized in that it stores a computer program, and the computer program is the above-mentioned multimodal sentiment analysis method when executed by a processor.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] The present invention combines information from two modalities, voice and expression pictures, to capture emotional features in more dimensions. By fusing data from different modalities for sentiment analysis, it no longer relies on a single data source, thereby reducing errors caused by quality problems of a single data source. This multi-source data fusion approach makes the model more robust and able to adapt to sentiment analysis tasks in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to illustrate the embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.

[0025] Figure 1 The sentiment analysis model framework of the present invention is shown;

[0026] Figure 2 A schematic diagram of the curves showing the variation of the accuracy of the three models with the step length of the present invention is shown. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] The embodiment of the present invention proposes a multimodal sentiment analysis method, the general framework of which is as follows: Figure 1 As shown in the figure. First, the voice of the mobile device is collected on the mobile terminal and transcoded into audio signals. When the user ends the call, these audio signals will be sent to the server by the mobile terminal communication module. When the scheduling module in the server receives requests from the terminal, it will assign IDs to these requests and send them to the idle thread pool. After that, the scheduling module will initiate a recognition request to the voice module. After the voice module responds to the request, it will call the thread pool to read the audio signal, convert the time series signal into a frequency domain signal through Fourier transform (FFT), and draw the original spectrum as the input of the deep learning model. In addition, the mobile terminal will also intercept the user's expression pictures and send them to the server. At that time, the expression processing module in the server will receive these pictures for analysis and processing. Since the convolutional neural network (CNN) is suitable for processing image tasks and can identify local features in the image, for the spectrum, CNN can identify specific frequency patterns and spatial features. RNN is suitable for processing sequence data. The spectrum shows the evolution of the audio signal on the time axis. RNN can capture the contextual dependencies in the time series. Therefore, the present invention will use CNN and RNN to extract the spatial features and temporal features of speech information respectively, and use CNN to extract the features of expression pictures. Finally, vector splicing technology is used to merge the feature vectors and input them into the support vector machine (SVM) model for classification to determine the emotional polarity.

[0029] Example 1

[0030] This paper uses three different datasets: eNTRAFACE'05, RML and AFEW6.0.

[0031] Comparative experiment:

[0032] In order to verify the effectiveness of the sentiment analysis method proposed in this paper, experiments were conducted on CNN-SVM, RNN-SVM, and CRNN-SVM. The experimental results are shown in the figure. Figure 2 shown.

[0033] As can be seen from the figure, the larger the step size, the higher the model accuracy. When the step size is 150, the model accuracy is basically stable. As the step size increases, the number of samples generated by training also increases, and the risk of overfitting of the model also increases. Therefore, a step size of 150 is more appropriate.

[0034] The test sample set is used to calculate the input features of the softmax classifier, and the calculated new features are input into the SVM to complete the final classification. CNN-SVM and RNN-SVM have the same classifier. The experimental results of the three models are shown in Table 1.

[0035] Table 1 Comparison experimental results of three models

[0036]

[0037]

[0038] As can be seen from the table above, CRNN-SVM has the strongest emotion recognition ability, with an accuracy of 93.5% when the step length is 36, which is nearly 20% higher than CNN-SVM and RNN-SVM. This is because CNN is used to extract local features and spatial features, RNN is used to extract temporal features, and finally feature fusion is performed, which allows the model to learn richer feature representations, which is conducive to analyzing multimodal samples. The experimental results verify the effectiveness of the CRNN-SVM model proposed in this paper.

[0039] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: The method comprises: S1 obtains user voice and expression pictures; S2 inputs the user's voice and expression picture into the CRNN-SVM model for feature extraction, and merges the extracted features through vector splicing technology for classification to determine the emotional polarity.

2. The multimodal sentiment analysis method according to claim 1, characterized in that: The CRNN-SVM model is composed of a convolutional neural network, a recurrent neural network and a support vector model.

3. The multimodal sentiment analysis method according to claim 1, characterized in that: The S2 is specifically: Read the user's voice information, convert the time series signal into a frequency domain signal through Fourier transform, draw the original spectrum, and use convolutional neural network and recurrent neural network to extract the spatial features and time series features of the original spectrum respectively; Use convolutional neural networks to extract features of expression images; Through vector splicing technology, the extracted spatial features, temporal features and features of expression pictures are merged and input into the support vector model for classification to determine the emotional polarity.

4. A multimodal sentiment analysis system, characterized in that: include: Acquisition module, used to obtain user voice and expression pictures; The feature extraction and classification module is used to input the user's voice and expression picture into the CRNN-SVM model for feature extraction, and merge the extracted features through vector splicing technology for classification to determine the emotional polarity.

5. The multimodal sentiment analysis system according to claim 4, characterized in that: The feature extraction and classification module includes: The speech feature extraction module is used to read the user's speech information, convert the time series signal into a frequency domain signal through Fourier transform, draw the original spectrum, and use the convolutional neural network and recurrent neural network to extract the spatial features and time series features of the original spectrum respectively; Image feature extraction module, used to extract features of expression images using convolutional neural networks; The classification module is used to combine the extracted spatial features, temporal features and features of expression pictures through vector splicing technology and input them into the support vector model for classification to determine the emotional polarity.

6. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the multimodal sentiment analysis method described in claims 1-3.

7. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the multimodal sentiment analysis method as described in claims 1-3.

Images