Electronic device and multi-modal sentiment classification method based on multi-pass fusion

Abstract

The application provides an electronic device and a multi-modal emotion classification method based on multi-channel fusion. The electronic device acquires a signal to be classified by responding to an emotion recognition instruction input by a user. The voice signal, image signal and text signal in the classified signal are converted into classified data. The emotion features of the voice data, image data and text data in the classified data are extracted respectively to obtain audio emotion features, image emotion features and text emotion features. The emotion features are fused, and finally the emotion fusion features are predicted by an emotion classification model to obtain an emotion classification result. The application can identify emotion data from multiple emotion dimensions by fusing the emotion features of audio, image and text, improve the accuracy of emotion recognition, speed up the emotion classification efficiency and improve the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of emotion classification methods, and in particular to an electronic device and a multi-modal emotion classification method based on multi-path fusion. BACKGROUND

[0002] Emotion recognition is to infer the emotional state by observing the facial expression, behavior and the premise environment of emotional generation. Because the emotional state is internal and contains physiological and psychological changes, only some observable things such as facial expression, behavior, etc. can be obtained. In order to more accurately recognize human emotions, physiological methods are usually used, the first is to detect the physiological signals of the tester such as breathing, heart rate and body temperature, and the second is to detect the emotional behavior of the tester such as facial feature expression recognition, speech emotion recognition and posture recognition.

[0003] Because the tester's facial image and speech are easy to collect, in the field of emotion recognition, the tester's facial image, speech and text content of speech are usually collected to perform emotion recognition. In the field of emotion recognition, one of the above is usually used as a basis for detection to distinguish the tester's emotion. However, the result of such single-modal emotion recognition is often low in accuracy. Because humans tend to control their tone or expression to hide their negative emotions when expressing emotions such as anger, disgust and other negative emotional states. At the same time, compared with multi-modal signals, single-modal signals contain less information and are not sufficient to correctly analyze the emotional state, resulting in errors in emotion classification results. SUMMARY

[0004] Some embodiments of the present application provide a multi-modal emotion classification method based on multi-path fusion to solve the problem that when only single-modal emotion data is used as the basis for emotion recognition, the emotional information contained is less, which cannot accurately identify the emotion category, resulting in errors in emotion classification results, and reducing the poor user experience.

[0005] In a first aspect, some embodiments of the present application provide an electronic device, the electronic device comprising a user interface, a memory and a controller, wherein the user interface is configured to respectively acquire a voice signal, an image signal and a text signal input by a user, the memory stores an emotion classification model, and the controller is configured to:

[0006] In response to an emotion recognition instruction input by a user, acquire a signal to be classified, the signal to be classified comprising a voice signal, an image signal and a text signal input by a user;

[0007] Convert the signal to be classified into classification data, the classification data comprising voice data, image data and text data;

[0008] extracting emotion features from the voice data, the image data, and the text data to obtain audio emotion features, image emotion features, and text emotion features;

[0009] performing feature fusion on the audio emotion features, the image emotion features, and the text emotion features respectively by a feature fusion module to obtain emotion fusion features;

[0010] performing emotion prediction on the emotion fusion features by the emotion classification model to obtain an emotion classification result.

[0011] In a second aspect, some embodiments of the present application provide a multi-modal emotion classification method based on multi-channel fusion, which is applied to an electronic device, the electronic device comprising a user interface, a memory, and a controller, wherein the user interface is configured to acquire voice signals, image signals, and text signals input by a user respectively; the memory stores an emotion classification model; and the method comprises:

[0012] In response to an emotion recognition instruction input by a user, acquiring a signal to be classified, the signal to be classified comprising voice signals, image signals, and text signals input by the user;

[0013] converting the signal to be classified into classification data, the classification data comprising voice data, image data, and text data;

[0014] extracting emotion features from the voice data, the image data, and the text data to obtain audio emotion features, image emotion features, and text emotion features;

[0015] performing feature fusion on the audio emotion features, the image emotion features, and the text emotion features respectively by a feature fusion module to obtain emotion fusion features;

[0016] performing emotion prediction on the emotion fusion features by the emotion classification model to obtain an emotion classification result.

[0017] From the above scheme, the electronic device and the multi-modal emotion classification method based on multi-path fusion are provided. The electronic device acquires a classification signal by responding to an emotion recognition instruction input by a user. The voice signal, image signal and text signal in the classification signal are converted into classification data. The emotion features of the voice data, image data and text data in the classification data are extracted to obtain audio emotion features, image emotion features and text emotion features. The emotion features are fused, and the emotion fusion features are predicted by an emotion classification model to obtain an emotion classification result. The emotion features of audio, image and text are fused, and the emotion fusion features are predicted, so that the emotion data can be recognized from multiple emotion dimensions, the accuracy of emotion recognition is improved, the emotion classification efficiency is improved, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 The use scenario of the electronic device in the embodiment of the present application;

[0020] Figure 2 The hardware configuration diagram of the electronic device in the embodiment of the present application;

[0021] Figure 3 The flowchart of the electronic device performing the multi-modal emotion classification method based on multi-path fusion in the embodiment of the present application;

[0022] Figure 4 The flowchart of the electronic device acquiring audio emotion features in the embodiment of the present application;

[0023] Figure 5 The flowchart of the electronic device acquiring image emotion features in the embodiment of the present application;

[0024] Figure 6 The structural diagram of the residual neural network in the embodiment of the present application;

[0025] Figure 7 The flowchart of the electronic device acquiring text emotion features in the embodiment of the present application;

[0026] Figure 8 The flowchart of the electronic device acquiring emotion fusion features in the embodiment of the present application;

[0027] Figure 9A flowchart of a process of obtaining a sentiment fusion feature by constructing an attention matrix by the electronic device in the embodiments of the present application is shown in FIG. 1.

[0028] Figure 10 A flowchart of a process of calculating a training loss of a sentiment classification model in the embodiments of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0029] For the purpose of making the objects and embodiments of the present application clearer, the following will describe the exemplary embodiments of the present application with reference to the accompanying drawings. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all of them.

[0030] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, but is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0031] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0032] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not have to be limited to all the components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0033] The electronic device provided by the embodiments of the present application can have various implementation forms, for example, can be a television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc.

[0034] Figure 1 A schematic diagram of an operating scenario between the electronic device and the control device according to the embodiments is shown in FIG. 2. As shown in FIG. 2, a user can operate the electronic device 200 through the control device 300 or the control device 100. Figure 1

[0035] In some embodiments, the control device 100 can be a remote controller, and the communication between the remote controller and the electronic device 200 includes infrared protocol communication or Bluetooth protocol communication, and other short-distance communication modes, to control the electronic device 200 through wireless or wired mode. The user can input user instructions through the keys on the remote controller, voice input, control panel input, etc., to control the electronic device 200.​

[0036] In some embodiments, a control device 300 (such as a mobile phone, tablet computer, computer, laptop computer, etc.) may also be used to control the electronic device 200. For example, an application running on the control device 300 may be used to control the electronic device 200.

[0037] In some embodiments, the electronic device 200 may receive instructions not by using the control device 300 or control unit 100 described above, but by receiving user control through touch or gestures.

[0038] In some embodiments, the electronic device 200 may also be controlled in ways other than the control device 100 and the control device 300. For example, it may be controlled by directly receiving the user's voice commands through a module configured inside the electronic device 200 for acquiring voice commands, or it may be controlled by receiving the user's voice commands through a voice control device set outside the electronic device 200.

[0039] In some embodiments, the electronic device 200 also communicates with the server 400. The electronic device 200 may be allowed to communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the electronic device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0040] like Figure 2 As shown, the electronic device 200 may include at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280.

[0041] In some embodiments, controller 250 may include a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0042] The display 260 may include the following components: a display screen component for displaying images; a driving component for driving image display; a component for receiving image signals output from the controller 250 and displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0043] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0044] The communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The electronic device 200 can establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0045] The user interface can be used to receive control signals of the control device 100 (e.g., an infrared remote controller).

[0046] The detector 230 is used to collect signals of the external environment or interaction with the outside. For example, the detector 230 includes a light receiver for collecting ambient light intensity, or an image collector such as a camera for collecting external environmental scenes, user attributes or user interaction gestures, or a sound collector such as a microphone for receiving external sounds.

[0047] The external device interface 240 can include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It can also be a composite input / output interface formed by the above multiple interfaces.

[0048] The tuner demodulator 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals and EPG data signals from multiple wireless or wired broadcast television signals. In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, i.e., the tuner demodulator 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0049] The controller 250 controls the operation of the electronic device and responds to user operations through various software control programs stored on the memory. The controller 250 controls the overall operation of the electronic device 200. For example, in response to receiving a user command for selecting a UI object displayed on the display 260, the controller 250 can perform an operation related to the object selected by the user command.

[0050] In some embodiments, the controller 250 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM (random access memory), a ROM (read-only memory), a first interface to an n-th interface for input / output, a communication bus, and the like.

[0051] Emotion is the attitude of a person to an external event or a conversation activity. The emotions of a person are generally divided into: happy, angry, sad, frightened, and surprised, etc. A machine analyzes the collected human signals to obtain the emotional state of a person, and this process is emotion recognition. Generally, the signals that can be used for emotion recognition include two aspects, one is physiological signals such as breathing, heart rate, and body temperature, and the other is behavioral manifestations including facial expressions, speech, and gestures, etc. Face and speech are often used to recognize the emotion of an object due to the simple collection method. Emotion recognition can help a system to understand the emotional state of an object and its attitude to a certain topic or event.

[0052] Emotion recognition is a very important technical means and can be applied to multiple fields. For example, in the process of user interaction with an artificial intelligence product, the emotion of a user can be recognized to respond to the emotional state of the user. In the process of sales, a sales strategy can be further developed according to the emotion of a user to a product being sold. In the medical industry, the emotion of a patient can be recognized to determine the emotional state of the patient, etc.

[0053] In the field of emotion recognition, one of the above is usually used as a basis for detection to distinguish the emotion of a tester. In some embodiments, the electronic device 200 can obtain the facial feature expression, the spoken speech, or the gesture of a user as data for emotion recognition, and recognize the emotion of the user through one of the above data. However, the result of such single-modal emotion recognition is often low in accuracy. Because the user tends to control his or her tone or expression to hide his or her negative emotion when expressing negative emotions such as anger and disgust. At the same time, compared with multi-modal signals, single-modal signals contain less information and are not sufficient to correctly analyze the emotional state, resulting in errors in emotion classification.

[0054] To improve the accuracy of emotion recognition, some embodiments of the present application provide an electronic device 200, which should at least include a user interface 280, a memory, and a controller 250, wherein the controller 250 is configured to perform a multi-modal emotion classification method based on multi-channel fusion, such as Figure 3As shown, it includes the following:

[0055] S100: In response to a user input emotion recognition instruction, obtain a signal to be classified.

[0056] The emotion recognition instruction is used for emotion recognition of a signal to be classified input by a user through a user interface 280. The signal to be classified includes a voice signal, an image signal, and a text signal input by the user. In the signal to be classified, the subject can input a test audio to an audio input interface in the user interface. After inputting the test audio, the controller 250 can control the audio processor to process the test audio emitted by the subject into a voice signal.

[0057] The voice signal is a signal generated according to the voice emitted by the subject. The voice signal can be a sentence or a word spoken by the subject, or a sound or a shout emitted by the subject. For example, the sound emitted by the subject when frightened or surprised.

[0058] The image signal is a signal generated according to an image capturing the facial expression of the subject. The facial expression in the image signal can be an expression of happiness, anger, sadness, fear, surprise, helplessness, or fright. The user can input the facial expression image of the subject into the electronic device 200 through the user interface 280. The controller 250 can control the graphics processor to process the facial expression image into an image signal.

[0059] In different scenarios, the facial expression image of the subject taken in different scenarios will have different brightness. For example, the facial expression image taken in a night environment is darker, and the facial expression image taken in an environment with more sufficient light is clearer. In order to ensure the accurate conversion of the image signal, in some embodiments, the controller 250 can also adjust the brightness of the facial expression image. By identifying the background brightness of the facial expression image, the background brightness is adjusted to a standard brightness. The controller 250 can also adjust the saturation, brightness value, and hue of the facial expression image according to the standard image value, so that the graphics processor can convert the facial expression image in a non-special scenario to obtain a more accurate image signal.

[0060] The text signal is a voice signal generated according to the text content corresponding to the test audio input by the user. After the controller 250 receives the test audio, the controller 250 can also call a text conversion program and identify the test audio based on the text conversion program to obtain the text content corresponding to the test audio, and generate a text signal according to the text content.

[0061] In some embodiments, the controller 250 can also obtain the text content corresponding to the test audio input by the user through the control device 100 or input the corresponding text content through the virtual keyboard of the electronic device 200 with touch screen function. For test audio such as calling and shouting, the controller 250 can determine the phonetic words of the test audio according to the pitch and frequency band of the test audio, such as "ah" and "wow" as the text content of the test audio.

[0062] In order to ensure the accuracy of the emotion classification result, the corresponding content of the voice signal, image signal and text signal should be the same person at the same time. For example, the person says "I am in pain" while expressing a painful expression. At this time, the corresponding voice signal needs to be generated according to the audio of the person saying "I am in pain", the image signal needs to be generated according to the image with the painful expression of the person, and the text signal needs to be generated according to the text of "I am in pain", so as to ensure that all the signals to be classified are generated based on the voice, facial image and speaking text of the person at the same time.

[0063] S200: converting the signal to be classified into classification data.

[0064] Since the voice signal, image signal and text signal are different modal signals, the controller 250 is difficult to perform the same processing on the voice signal, image signal and text signal.

[0065] In order to facilitate the controller 250 to process different modal signals, in some embodiments, the controller 250 can first convert the voice signal, image signal and text signal into classification data of the same format, wherein the classification data includes voice data, image data and text data.

[0066] The voice data is converted from the voice signal in the signal to be classified, the image data is converted from the image signal in the signal to be classified, and the text data is converted from the text signal in the signal to be classified. In some embodiments, the controller 250 can convert the signal to be classified into classification data by sampling, quantization or encoding.

[0067] S300: extracting emotion features from the voice data, image data and text data.

[0068] In the voice data, image data and text data, the corresponding emotion features are respectively contained, and the controller 250 needs to extract the emotion features from the voice data, image data and text data, so as to facilitate the subsequent emotion prediction of the emotion features in the classification data.

[0069] For three different modal classification data, different emotion feature extraction methods can be used. In some embodiments, for example, the voice data can be extracted by using the Mel-frequency cepstral coefficient (MFCC) method, the image data can be extracted by using the convolutional neural network (CNN) method, and the text data can be extracted by using the long short-term memory (LSTM) method.Figure 4 As shown, for the emotion feature extraction of the voice data, the controller 250 can also perform the following method:

[0070] S3011: input the voice data into a Gaussian filter to compensate the voice data, and obtain a compensated spectrum.

[0071] Since the sound production system of the electronic device 200 can suppress the high frequency part of the voice data, it is difficult to identify the spectrum of the high frequency domain, resulting in that the voice emotion features of the voice data cannot be accurately identified. After the controller 250 inputs the voice data into the Gaussian filter, the compensated spectrum obtained can be more flat, and at the same time, the voice data can be compensated by the Gaussian filter to make up for the high frequency part suppressed by the sound production system, so that the identification of the sound range is more accurate.

[0072] S3012: sequentially perform a framing operation and a windowing operation on the compensated spectrum to obtain a windowed spectrum.

[0073] The controller 250 will take the set of N sampling points in the compensated spectrum as a frame of observation units in time sequence to perform the framing operation. Usually, the value of N can be 256 or 512, covering a time of about 20-30 ms, and the larger the value, the longer the time covered by the frame.

[0074] In some embodiments, in order to avoid a large difference in sound range between adjacent two frames, the controller 250 can set an overlapping area between adjacent two frames, and the overlapping area contains M sampling points, and the value of M is between 1 / 2 and 1 / 3 of the value of N, so that the adjacent two frames can be smoothly transitioned.

[0075] After the framing operation on the compensated spectrum, the controller 250 will bring each frame of spectrum into a window function to eliminate the discontinuity of the voice data between the beginning and the end of each frame, which can cause spectrum leakage. After the controller 250 brings the compensated spectrum after the framing operation into the window function, the windowed spectrum is obtained. In this embodiment, the window functions that can be used include square window function, Hamming window function and Hanning window function, etc. According to the frequency domain characteristics of the window function, Hamming window is usually used.

[0076] S3013: convert the windowed spectrum into a voice energy spectrum by Fourier transform.

[0077] The transform in time domain is difficult to express the characteristics of the voice data, so in the embodiment, the controller 250 performs Fourier transform on each frame of the windowed spectrum to obtain the energy distribution in the frequency spectrum, converts the voice data into the energy distribution in the frequency domain to process the characteristics. After the controller 250 performs Fourier transform on the windowed spectrum, the controller 250 also needs to take the modulus square of the transformed spectrum to obtain the voice energy spectrum of the voice data, so as to better process the characteristics of the voice data.

[0078] S3014: inputting the voice energy spectrum into a mel filter to eliminate the harmonics of the voice energy spectrum to obtain mel filter data.

[0079] The controller 250 inputs the voice energy spectrum into the mel filter, smoothes the energy spectrum through the mel filter, and outputs mel filter data. In the embodiment, by eliminating the harmonics of the voice energy spectrum and highlighting the relatively concentrated energy area in the voice energy spectrum, the computational amount of extracting the audio emotion feature is reduced.

[0080] S3015: sequentially performing logarithmic operation and discrete cosine transform on the mel filter data to obtain the audio emotion feature.

[0081] After the mel filter outputs the mel filter data, the controller 250 calculates the logarithmic energy according to the obtained mel filter data to obtain mel filter energy.

[0082] In some embodiments, if there are multiple mel filters, the controller 250 calculates the logarithmic energy according to the mel filter data output by each mel filter, and sums each logarithmic energy to obtain the mel filter energy.

[0083] After calculating the mel filter energy, the controller 250 inputs the mel filter energy into the discrete cosine transform to obtain the coefficient parameter of the mel filter energy, and determines the audio emotion feature according to the coefficient parameter.

[0084] In some embodiments, as shown in FIG. 8, for image emotion feature extraction of image data, the controller 250 can also perform the following method: Figure 5

[0085] S3021: constructing a residual neural network.

[0086] In order to extract deeper features in the image data, when constructing the residual neural network, the controller 250 usually constructs a deep neural network with more layers. In the residual neural network, each layer can use one-dimensional convolution or three-dimensional convolution to extract features in the image data layer by layer to achieve the purpose of extracting deep features.

[0087] ​When the constructed deep neural network has a large number of layers, problems such as gradient vanishing or gradient explosion may occur in the process of extracting features in the image data, causing the feature extraction of the image data to be inaccurate. Therefore, in this embodiment, the controller 250 can eliminate problems such as gradient vanishing and gradient explosion that occur in the training process through the residual neural network.

[0088] S3022: inputting the image data into the residual neural network to obtain the image sentiment features.

[0089] Figure 6 The structure of the residual neural network is shown in the schematic diagram. The residual neural network is composed of residual neural units and activation functions. The neuron input to the residual neural unit is x, and the expected output is F(x). Before input, the neuron x is copied into two parts, one of which is transmitted to the residual neural unit for inter-layer operation and outputs F(x). The other part is used as a branch and is superimposed with the output F(x). If the deep neural network has reached an optimal solution in the training process at a certain layer, the output of F(x) is 0 at this time, indicating that there is no residual, and the output is the copied other part of the neuron x. At this time, it is indicated that x is the optimal output. In this way, the problems of gradient vanishing and gradient explosion of the deep neural network are solved.

[0090] After the deep neural network is trained through the residual neural network, the controller 250 inputs the image data into the deep neural network to extract the image sentiment features through the deep neural network.

[0091] In some embodiments, as shown in FIG. 6, for text sentiment feature extraction of text data, the controller 250 can also perform the following method: Figure 7

[0092] In some embodiments, the controller 250 can also perform prosody prediction on the text data according to the pronunciation of the text data to obtain prosody prediction data.

[0093] In this embodiment, the controller 250 can perform prosody prediction on a large piece of text. For example, the text data is “I am not hating you, I am jealous of you.” The prosody prediction data obtained by the controller 250 after prosody prediction according to the pinyin coda of the text data is “wobushitaoyanni1woshijiduni2”, where “1” and “2” respectively represent the comma and the period in the text. In some embodiments, other symbols that are different from letters can also be used to represent punctuation marks in the text, such as “@”, “￥”, “#”, etc. This embodiment does not make specific limitations.

[0094] S3031: performing word segmentation on the text data to obtain word segmentation data.

[0095] ​In the embodiment, the text data can include multiple languages. Taking Chinese as an example, there is no word boundary in a Chinese sentence. Therefore, the neural network needs to perform a word segmentation operation on the text data before processing the Chinese text data, so as to better perform sentiment recognition on the Chinese text data.

[0096] In the embodiment, the controller 250 can also identify valid words and invalid words in the text data according to the prosody prediction data. The invalid words are, for example, mood words such as “ah” and “oh”, or words expressing a connection relationship, such as “of” and “and”. The valid words are, for example, words representing a person, such as “you”, “I”, “he”, or “she”, words representing an action, such as “hit”, “sit”, “walk”, and words representing a place, such as “at home” and “at school”. The controller 250 performs a word segmentation operation on the text data according to the valid words and the invalid words, so as to accurately divide the word features of the text data and facilitate subsequent extraction of text sentiment features.

[0097] In some embodiments, the controller 250 can also exclude the invalid words in the text data according to the prosody prediction data. The controller 250 only retains the valid words in the text data. The controller 250 can perform a word segmentation operation on the text data only by distinguishing the valid words in the text data, so as to obtain segmented data.

[0098] S3032: constructing a text tensor according to the tensor construction rule, to obtain a text sentiment feature;

[0099] The tensor construction rule is a rule for assigning a corresponding tensor value to a text language. After obtaining the segmented data, the controller 250 needs to construct a tensor for each word or character in the segmented data according to the tensor construction rule. For example, the segmented data is “I / actually / am not / hate you, I am jealous of you.” After constructing a text tensor according to the tensor construction rule for the above segmented data, the obtained tensor code is:

[0100] “101,2769,1071,2141,679,3221,6374,1328,872,8024,2769,3221,2065,1971,872.” The above code is the text sentiment feature.

[0101] In some embodiments, the controller 250 can obtain the text sentiment feature by using a text recognition model such as a Roberta Chinese pre-training model.

[0102] After the controller 250 obtains the audio sentiment feature, the image sentiment feature, and the text sentiment feature, the controller 250 also needs to perform modal fusion on the audio sentiment feature, the image sentiment feature, and the text sentiment feature, that is:

[0103] S400: Perform feature fusion on the audio emotion feature, the image emotion feature and the text emotion feature respectively through a feature fusion module to obtain an emotion fusion feature.

[0104] At this time, the controller 250 first calls the feature fusion module from the emotion classification model, and inputs the audio emotion feature, the image emotion feature and the text emotion feature into the feature fusion module respectively. The feature fusion module interacts and performs feature fusion on the emotion features of different modalities to obtain an emotion fusion feature. The emotion fusion feature contains emotion features of different modalities, so that the emotion classification model can more accurately identify and detect the emotions of the subject.

[0105] Specifically, in some embodiments, after the audio emotion feature, the image emotion feature and the text emotion feature are input into the feature fusion module, the controller 250 encodes the audio emotion feature, the image emotion feature and the text emotion feature in the feature fusion module through the encoder module of the emotion classification model. The encoder module includes an audio encoder, an image encoder and a text encoder. The audio encoder encodes the audio emotion feature to obtain an audio emotion encoding. The image encoder encodes the image emotion feature to obtain an image emotion encoding. The text encoder encodes the text emotion feature to obtain a text emotion encoding.

[0106] As shown in Figure 8 After the above three modal emotion features are encoded respectively, the controller 250 also extracts modal hidden vectors from the audio emotion encoding, the image emotion encoding and the text emotion encoding through the emotion classification model. The modal hidden vectors include an audio emotion hidden vector, an image emotion hidden vector and a text emotion hidden vector. The modal hidden vectors can respectively reflect the emotion features of the corresponding modalities.

[0107] After the controller 250 obtains the hidden vectors of the three modalities respectively, the audio emotion hidden vector, the image emotion hidden vector and the text emotion hidden vector can be fused through the attention mechanism of the emotion classification model, so that the features of the above three modal hidden vectors are fully interacted to obtain an emotion fusion feature. In this embodiment, the emotion fusion feature combines all the features in the three modal hidden vectors, achieving a full state of the emotion features of the three modalities.

[0108] In some embodiments, the controller 250 can fuse the audio emotion hidden vector, the image emotion hidden vector and the text emotion hidden vector by constructing an attention matrix. In this embodiment, the controller 250 establishes a modal matrix according to the attention mechanism of the emotion classification model. Corresponding to the three different modalities of audio, image and text, the modal matrix includes an image attention matrix, an audio attention matrix and a text attention matrix.

[0109] The controller 250 can fuse the modality hidden vectors by splicing the image attention matrix, the audio attention matrix and the text attention matrix. In the process of matrix splicing, the controller 250 can first splice the attention matrices of any two modalities, and then splice the obtained double-modality matrix with the attention matrix of the other modality. There can be various permutations and combinations of the above-mentioned matrix splicing methods, and the present embodiment takes the example of splicing the image attention matrix and the audio attention matrix first, and then splicing the text attention matrix.

[0110] As shown in Figure 9 , the controller 250 first splices the image attention matrix and the audio attention matrix to obtain an audio-visual attention matrix, which contains the emotional features of the image emotional hidden vector and the audio emotional hidden vector. In order to improve the splicing effect of the matrix and the interaction effect of the image emotional features and the audio emotional features, the controller 250 needs to perform convolution operation on the audio-visual attention matrix through the emotion classification model to deeply extract the audio-visual emotional features and obtain a double-modality matrix.

[0111] After obtaining the double-modality matrix, the controller 250 needs to splice the double-modality matrix with the text attention matrix to fuse the emotional features of the third modality and obtain an audio-visual-text attention matrix. After fusion, the controller 250 performs the same operation on the audio-visual-text attention matrix through the emotion classification model to extract deep audio-visual-text emotional features and obtain a triple-modality matrix. Finally, the controller 250 extracts emotional fusion features according to the triple-modality matrix.

[0112] In the above-mentioned matrix splicing process, the controller 250 can also calculate the linear mapping data of the hidden vectors corresponding to the attention matrices. For example, the controller 250 calculates the audio linear mapping data of the audio emotional hidden vector and the image linear mapping data according to the image emotional hidden vector to more intuitively reflect the emotional features of the audio emotional hidden vector and the image emotional hidden vector, facilitating the feature interaction of the attention matrices of the two modalities.

[0113] Then, the controller 250 calculates the interaction features of the audio linear mapping data and the image linear mapping data, and splices the image attention matrix and the audio attention matrix according to the interaction features to obtain an audio-visual attention matrix. Similarly, after the controller 250 performs convolution operation on the audio-visual attention matrix to obtain a double-modality matrix, the same method can also be used to perform the same operation on the double-modality matrix and the text attention matrix to obtain a triple-modality matrix. This embodiment will not be repeated here.

[0114] S500: Perform emotion prediction on the emotional fusion features through the emotion classification model to obtain an emotion classification result.

[0115] After the emotion fusion feature is output by the emotion classification model, the controller 250 inputs the emotion fusion feature as an input to the full connection layer of the emotion classification model again, and performs emotion prediction on the emotion fusion feature through the full connection layer. The full connection layer maps the emotion fusion feature into an emotion prediction vector, which can be in the form of one-hot encoding. One-hot encoding is a representation of a categorical variable as a binary vector, which requires mapping the classification value of each emotion in the emotion prediction vector to an integer value, and then each integer value is represented as a binary vector.

[0116] For example, after one-hot encoding of "joy" and "sadness" as emotion features, the encoding of "joy" is "10", with the first bit being the valid bit, and the encoding of "sadness" is "01", with the second bit being the valid bit. The controller 250 can map the emotion prediction vector to the one-hot encoding of the corresponding emotion feature to perform emotion feature matching to obtain the emotion prediction result.

[0117] In some embodiments, the controller 250 needs to train the emotion classification model before performing emotion recognition and classification. Specifically, the controller 250 first acquires sample modal data for training the emotion classification model. The sample modal data is audio sample data, image sample data, and text sample data in the same emotion state label.

[0118] After obtaining the sample modal data, the controller 250 inputs the audio sample data, the image sample data, and the text sample data into the encoder module of the training model. The audio encoder in the encoder module extracts audio sample features of the audio sample data, the image encoder in the encoder module extracts image sample features of the image sample data, and the text encoder in the encoder module extracts text sample features of the text sample data.

[0119] After encoding, the controller 250 fuses the audio sample features, the image sample features, and the text sample features to obtain fusion sample features. At this time, the controller 250 inputs the fusion sample features into the full connection layer of the to-be-trained model, performs emotion classification on the extracted fusion sample features through the full connection layer, and outputs an emotion prediction result through the to-be-trained model. Since the to-be-trained model is not trained to convergence, the emotion prediction result output in the training process will have a certain feature loss, which is generally calculated by the prediction result label contained in the emotion prediction result.

[0120] Specifically, the controller 250 calculates the cross-entropy loss between the predicted result label and the emotion state label by using a cross-entropy function, so as to obtain the loss between the emotion result recognized by the predicted result label and the emotion result represented by the emotion state label. After the cross-entropy loss is calculated, the controller 250 calculates the sum of the cross-entropy loss and the feature fusion loss, so as to obtain the training loss of the to-be-trained model.

[0121] In some embodiments, the controller 250 can also determine the convergence degree of the to-be-trained model by setting a loss threshold. If the training loss is less than the loss threshold, it indicates that the emotion classification model has reached the convergence state, and the controller 250 outputs the training parameters of the to-be-trained model to obtain the emotion classification model. If the training loss is greater than the loss threshold, it indicates that the emotion classification model has not converged, and the controller 250 continues to iteratively train the emotion classification model.

[0122] In the above embodiments, the feature fusion loss is the loss that occurs in the process of fusing the audio sample feature, the image sample feature and the text sample feature to obtain the fused sample feature. In order to accurately obtain the feature fusion loss, as shown in FIG. 6, in some embodiments, the controller 250 can also input the fused sample feature into the decoder module of the to-be-trained model, decode the fused sample feature by using the decoder module to obtain the reconstructed feature. The decoder module includes an audio decoder, an image decoder and a text decoder corresponding to the encoder module. The reconstructed feature includes an image reconstructed feature, an audio reconstructed feature and a text reconstructed feature. Figure 10

[0123] In the above process, the controller 250 inputs the fused sample feature into the audio decoder, the image decoder and the text decoder respectively to obtain the corresponding image reconstructed feature, the audio reconstructed feature and the text reconstructed feature. Then, the controller 250 needs to compare the reconstructed feature with the fused sample feature, and calculate the Manhattan distance between the reconstructed feature and the fused sample feature to obtain the feature loss before and after fusion. The controller 250 calculates the image reconstruction loss of the image reconstructed feature and the image sample feature, the audio reconstruction loss of the audio reconstructed feature and the audio sample feature, and the text reconstruction loss of the text reconstructed feature and the text sample feature respectively, and obtains the feature fusion loss by using the image reconstruction loss, the audio reconstruction loss and the text reconstruction loss. The training loss is calculated according to the feature fusion loss. The to-be-trained model is trained by using the feature fusion loss, so as to further improve the training accuracy of the feature model and improve the emotion classification efficiency.

[0124] In some embodiments of the present application, a multi-modal emotion classification method based on multi-channel fusion is also provided, which is applied to any one of the electronic devices 200 described above. The method comprises the following steps:

[0125] ​S100: in response to a user input emotion recognition instruction, obtaining a signal to be classified, the signal to be classified including a voice signal, an image signal and a text signal input by the user;

[0126] S200: converting the signal to be classified into classification data, the classification data including voice data, image data and text data;

[0127] S300: performing emotion feature extraction on the voice data, the image data and the text data to obtain audio emotion features, image emotion features and text emotion features;

[0128] S400: performing feature fusion on the audio emotion features, the image emotion features and the text emotion features respectively by a fusion feature module to obtain emotion fusion features;

[0129] S500: performing emotion prediction on the emotion fusion features by the emotion classification model to obtain an emotion classification result.

[0130] As can be seen from the above solutions, the present application provides an electronic device and a multi-modal emotion classification method based on multi-channel fusion. The electronic device obtains a signal to be classified in response to a user input emotion recognition instruction. The voice signal, the image signal and the text signal in the classification signal are converted into classification data. Emotion features of voice data, image data and text data in the classification data are extracted respectively to obtain audio emotion features, image emotion features and text emotion features. The above emotion features are fused, and finally emotion prediction is performed on the emotion fusion features by an emotion classification model to obtain an emotion classification result. The present application can identify emotion data from multiple emotion dimensions by fusing emotion features of audio, image and text three modalities and performing emotion prediction on the emotion fusion features, thereby improving the accuracy of emotion recognition, accelerating the emotion classification efficiency and improving the user experience.

[0131] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, which can be stored in a computer readable storage medium.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0133] The above description has been made in conjunction with specific embodiments for the convenience of explanation. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed. Various modifications and variations can be derived from the above teachings. The selection and description of the above embodiments are for better explanation of the present disclosure, so that those skilled in the art can better use the embodiments.

Claims

1. An electronic device, comprising: The method comprises the following steps: a user interface configured to obtain a voice signal, an image signal and a text signal input by a user respectively; a memory in which an emotion classification model is stored; a controller configured to: in response to an emotion recognition instruction input by the user, obtain a signal to be classified, the signal to be classified comprising the voice signal, the image signal and the text signal input by the user; convert the signal to be classified into classification data, the classification data comprising voice data, image data and text data; extract emotion features from the voice data, the image data and the text data to obtain audio emotion features, image emotion features and text emotion features; encode the audio emotion features, the image emotion features and the text emotion features through an encoder module of the emotion classification model to obtain audio emotion encoding, image emotion encoding and text emotion encoding respectively; extract a modal hidden vector from the audio emotion encoding, the image emotion encoding and the text emotion encoding, the modal hidden vector comprising an audio emotion hidden vector, an image emotion hidden vector and a text emotion hidden vector; establish a modal matrix through an attention mechanism and the audio emotion hidden vector, the image emotion hidden vector and the text emotion hidden vector, the modal matrix comprising an image attention matrix, an audio attention matrix and a text attention matrix; calculate audio linear mapping data according to the audio emotion hidden vector, and calculate image linear mapping data according to the image emotion hidden vector; calculate an interaction feature of the audio linear mapping data and the image linear mapping data; splice the image attention matrix and the audio attention matrix according to the interaction feature to obtain an image-audio attention matrix; perform a convolution operation on the image-audio attention matrix to obtain a dual-modal matrix; splice the dual-modal matrix and the text attention matrix to obtain an image-audio-text attention matrix; perform a convolution operation on the image-audio-text attention matrix to obtain a triple-modal matrix; extract an emotion fusion feature according to the triple-modal matrix; perform emotion prediction on the emotion fusion feature through the emotion classification model to obtain an emotion classification result.

2. The electronic device of claim 1, wherein, The controller performing emotion feature extraction on the voice data, the image data and the text data is further configured to: input the voice data into a Gaussian filter to compensate the voice data to obtain a compensated spectrum; perform a framing operation and a windowing operation on the compensated spectrum in sequence to obtain a windowed spectrum; convert the windowed spectrum into a voice energy spectrum through Fourier transform; input the voice energy spectrum into a Mel filter to eliminate harmonics of the voice energy spectrum to obtain Mel filter data; perform a logarithmic operation and a discrete cosine transform on the Mel filter data in sequence to obtain the audio emotion features.

3. The electronic device of claim 2, wherein, The controller performing emotion feature extraction on the voice data, the image data and the text data is further configured to: build a residual neural network; input the image data into the residual neural network to obtain the image emotion features.

4. The electronic device of claim 3, wherein, The controller performs emotion feature extraction on the voice data, the image data, and text data, and is further configured to: perform a word segmentation operation on the text data to obtain word segmentation data; construct a text tensor from the word segmentation data according to a tensor construction rule to obtain text emotion features; the tensor construction rule is a rule of assigning a corresponding tensor value to a text language.

5. The electronic device of claim 1, wherein, The controller is further configured to: obtain sample modal data; the sample modal data is audio sample data, image sample data, and text sample data in the same emotion state label; input the audio sample data, the image sample data, and the text sample data into an encoder module of a to-be-trained model to extract audio sample features, image sample features, and text sample features; fuse the audio sample features, the image sample features, and the text sample features to obtain fused sample features; input the fused sample features into a fully connected layer of the to-be-trained model to obtain an emotion prediction result output by the to-be-trained model; the emotion prediction result includes a prediction result label; calculate a cross-entropy loss between the prediction result label and an emotion state label; sum the cross-entropy loss and a feature fusion loss to obtain a training loss; the feature fusion loss is a loss in the process of fusing the audio sample features, the image sample features, and the text sample features; if the training loss is less than the loss threshold, output training parameters of the to-be-trained model to obtain an emotion classification model.

6. The electronic device of claim 5, wherein, The controller is further configured to: input the fused sample features into a decoder module of the to-be-trained model to obtain reconstruction features; the reconstruction features include image reconstruction features, audio reconstruction features, and text reconstruction features; respectively calculate an image reconstruction loss between the image reconstruction features and the image sample features, an audio reconstruction loss between the audio reconstruction features and the audio sample features, and a text reconstruction loss between the text reconstruction features and the text sample features; superimpose the image reconstruction loss, the audio reconstruction loss, and the text reconstruction loss to obtain the feature fusion loss.

7. A multi-modal sentiment classification method based on multi-pass fusion, applied to an electronic device, the electronic device comprising a user interface, a memory and a controller, wherein, The user interface is configured to respectively acquire voice signals, image signals, and text signals input by a user; the memory stores an emotion classification model; the method comprises: in response to a user input emotion recognition instruction, acquiring to-be-classified signals, the to-be-classified signals including voice signals, image signals, and text signals input by a user; convert the to-be-classified signals into classification data, the classification data including voice data, image data, and text data; perform emotion feature extraction on the voice data, the image data, and text data to obtain audio emotion features, image emotion features, and text emotion features; encode the audio emotion features, the image emotion features, and the text emotion features through an encoder module of the emotion classification model to respectively obtain audio emotion encoding, image emotion encoding, and text emotion encoding; extract a modal hidden vector in the audio emotion encoding, the image emotion encoding, and the text emotion encoding; the modal hidden vector includes an audio emotion hidden vector, an image emotion hidden vector, and a text emotion hidden vector; establish a modal matrix through an attention mechanism and the audio emotion hidden vector, the image emotion hidden vector, and the text emotion hidden vector; the modal matrix includes an image attention matrix, an audio attention matrix, and a text attention matrix; calculate audio linear mapping data according to the audio emotion hidden vector, and calculate image linear mapping data according to the image emotion hidden vector; calculate an interaction feature of the audio linear mapping data and the image linear mapping data; splice the image attention matrix and the audio attention matrix according to the interaction feature to obtain an audio-image attention matrix; perform a convolution operation on the audio-image attention matrix to obtain a dual-modal matrix; splice the dual-modal matrix and the text attention matrix to obtain an audio-image-text attention matrix; perform a convolution operation on the audio-image-text attention matrix to obtain a triple-modal matrix; extract an emotion fusion feature according to the triple-modal matrix; perform emotion prediction on the emotion fusion feature through the emotion classification model to obtain an emotion classification result.

Images