Emotion recognition method

Abstract

The invention relates to an emotion recognition method, and the method comprises the steps: obtaining a speaking audio of a target speaker from a first moment to a second moment after the first moment, and the features of each frame formed by dividing the speaking audio; extracting audio emotion features from the features of each frame; fusing the audio emotion features and the facial expression difference features to obtain fused emotion features; wherein the facial expression difference feature is determined according to the difference between a first facial expression feature extracted from a first facial picture of the target speaker and a second facial expression feature extracted from a second facial picture of the target speaker; the first face picture and the second face picture are acquired at the first moment and the second moment respectively; and obtaining an emotion category of the target speaker according to the fused emotion features. By using the method, the accuracy of emotion recognition can be improved.

Description

Technical Field

[0001] The present disclosure relates to an emotion recognition method, an electronic device, a storage medium, a program product, and the like. Background Art

[0002] At present, smart glasses are applicable to more and more occasions, and they basically have audio collection functions. Some smart glasses have camera functions, which can take pictures of the faces of the people they are communicating with. Therefore, it is expected that the emotions of the other party can be recognized in real time during the communication, which can provide the possibility for more interesting interactions, such as choosing different conversation topics according to different emotions, and empathizing with the interlocutor more accurately. Summary of the invention

[0003] Embodiments of the present disclosure provide emotion recognition methods and corresponding electronic devices, non-transitory machine-readable storage media, and computer program products for executing these methods.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an emotion recognition method, comprising: obtaining speech audio of a target speaker from a first moment to a second moment after the first moment, and features of each frame into which the speech audio is divided; extracting audio emotion features from the features of each frame; fusing the audio emotion features with facial expression difference features to obtain fused emotion features, wherein the facial expression difference features are determined based on the difference between a first facial expression feature extracted from a first facial image of the target speaker and a second facial expression feature extracted from a second facial image of the target speaker, and the first facial image and the second facial image are collected at the first moment and the second moment, respectively; and obtaining the emotion category of the target speaker according to the fused emotion features.

[0005] Optionally, the facial expression difference feature is determined based on the difference between the first facial expression feature and the second facial expression feature, including: subtracting the second facial expression feature from the first facial expression feature bitwise to obtain a difference feature; when the maximum value among the absolute values ​​of all elements of the difference feature is less than a preset threshold value, taking a feature with a preset fixed value as the facial expression difference feature; when the maximum value among the absolute values ​​of all elements of the difference feature is not less than the threshold value, taking the difference feature as the facial expression difference feature.

[0006] Optionally, extracting a first facial expression feature from a first facial image of the target speaker and extracting a second facial expression feature from a second facial image of the target speaker comprises: using a facial target detection model to respectively determine facial regions in the first facial image and the second facial image; and using a facial emotion recognition neural network model to respectively obtain the first facial expression feature and the second facial expression feature based on image information in the facial regions in the first facial image and the second facial image; or, using a facial feature extraction model based on facial feature points to respectively extract the first facial expression feature and the second facial expression feature from image information in the facial regions in the first facial image and the second facial image.

[0007] Optionally, the audio emotion feature and the facial expression difference feature are fused to obtain a fused emotion feature, comprising: when at least one of the audio emotion feature and the facial expression difference feature is a non-one-dimensional vector, flattening the non-one-dimensional vector into a one-dimensional vector; and splicing the audio emotion feature and the facial expression difference feature after flattening into the fused emotion feature.

[0008] Optionally, extracting audio emotion features from the features of each frame includes: utilizing a first neural network module to extract audio emotion features from the features of each frame, wherein the first neural network module includes one or a plurality of neural network units connected in series that have a function of preserving historical information.

[0009] Optionally, obtaining the emotion category of the target speaker according to the fused emotion feature includes: inputting the fused emotion feature into a second neural network module, wherein the second neural network module includes one or more processing units connected in series and an output layer connected after the processing unit, the processing unit includes a convolution unit and / or a fully connected unit, the convolution unit includes a convolution layer, a normalization layer and an activation layer, and the fully connected unit includes a fully connected layer, a normalization layer and an activation layer; and outputting confidence scores of all emotion categories from the output layer of the second neural network module, and determining the emotion category of the target speaker according to the confidence scores.

[0010] Optionally, the first neural network module and the second neural network module are trained together as modules in the same neural network model.

[0011] Optionally, during training, a discard layer is connected after one of the processing units in the second neural network module.

[0012] Optionally, the method is performed in a streaming reasoning manner, and the method further includes: post-processing the obtained emotion category of the target speaker, wherein the post-processing includes: low-pass filtering a predetermined plurality of sequentially obtained emotion categories; or when the emotion category of the target speaker is determined by selecting the emotion category corresponding to the highest value among the confidence scores of all emotion categories, calculating the average of multiple confidence scores obtained within a predetermined time for each emotion category in all emotion categories, and selecting the emotion category with the highest average as the emotion category of the target speaker.

[0013] According to a second aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a microphone configured to collect speech audio of a target speaker; a camera configured to collect a facial image of the target speaker; a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in any one of the schemes in the first aspect above.

[0014] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in any one of the schemes in the above-mentioned first aspect.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary machine-readable storage medium is provided, on which an executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes a method as described in any of the schemes in the first aspect above.

[0016] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute a method as described in any one of the schemes in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0018] Figure 1 A schematic structural diagram of smart glasses as an example of an electronic device according to at least one embodiment of the present disclosure is exemplarily shown.

[0019] Figure 2A and Figure 2B Typical emotion recognition schemes combining facial expressions and speech audio are shown respectively.

[0020] Figure 3 A schematic flow chart of an emotion recognition method according to at least one embodiment of the present disclosure is exemplarily shown.

[0021] Figure 4 A schematic flowchart of obtaining audio features according to at least one embodiment of the present disclosure is exemplarily shown.

[0022] Figure 5 A schematic flow chart of an emotion recognition method according to at least one embodiment of the present disclosure is exemplarily shown.

[0023] Figure 6 A structural schematic diagram of an electronic device according to at least one embodiment of the present disclosure is exemplarily shown. DETAILED DESCRIPTION

[0024] The specific embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the specific embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] First, some explanations of terms involved in this disclosure are given:

[0026] CPU: Central Processing Unit, also known as general-purpose processor, is the "brain" of smart devices;

[0027] ASR: Automatic Speech Recognition, automatic speech recognition;

[0028] Speech: human voice;

[0029] Noise: non-human voice, or noise;

[0030] YOLO: “You Only Look Once” is an algorithm for object detection using convolutional neural networks;

[0031] VAD: Voice Activity Detection, which can be used to determine the start and end time points of voice activity in a voice signal;

[0032] MFCC: Mel Frequency Cepstral Coefficient, which is the cepstral parameter extracted in the Mel scale frequency domain. It is a feature widely used in automatic speech and speaker recognition.

[0033] FBANK: filter bank, an audio feature;

[0034] Dense: fully connected layer, for feature extraction and dimension mapping;

[0035] Dropout: Random dropout layer randomly discards some information, so that the trained neural network is more robust;

[0036] Softmax: Activation layer, the sum of all output values ​​is 1;

[0037] Conv1D: 1D convolution;

[0038] Conv2D: 2D convolution;

[0039] Relu: Activation layer, providing nonlinear features, usually following the convolutional layer;

[0040] LSTM: LongStort-Term Memory, long short-term memory network;

[0041] CNN: Convolutional Neural Networks, convolutional neural network.

[0042] Smart glasses are a type of wearable smart product, which can include XR (Extended Reality) glasses, etc. XR glasses can include AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, MR (Mixed Reality) glasses, etc.

[0043] like Figure 1 As shown, in some examples, smart glasses can embed some hardware modules in their frames (including temples), including microphones, cameras, processors, and memories. The microphone and camera can be used to collect the audio and facial images of the communicator, respectively, and then the processor can execute the relevant programs stored in the memory to process the audio and / or facial images, thereby identifying the emotions of the communicator. Those skilled in the art should understand that Figure 1This is merely illustrative and not intended to limit the structure of the smart glasses of the present invention; the smart glasses may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently; or the components in the smart glasses may be deployed in other different locations.

[0044] For example, the processor may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU) (for example, for running the neural network model and / or neural network module described in any embodiment of the present invention), etc., wherein different processing units may be independent devices or integrated into one or more processors. The memory may be used to store computer executable program code, and the executable program code includes instructions. The memory may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an emotion recognition function, etc.), etc. The data storage area may store data created during the use of the smart glasses (such as audio data, picture data, etc.).

[0045] In one example, the processor and memory described above may be integrated into a microcontroller unit. In one example, hardware such as a microphone, a camera, a processor, and / or a memory may also be attached to the glasses instead of being embedded therein, and so on.

[0046] As mentioned above, in some scenarios where smart glasses are used for communication, it is expected to be able to recognize the emotions of the interlocutor in order to better communicate with him or her.

[0047] Current emotion recognition schemes include those based on facial expressions or the voice and semantics of a speech, but their recognition accuracy is relatively low.

[0048] For example, there are two neural network models for recognizing emotions based on facial expressions, one based on Mini-Xception and the other based on CNN. The input is a segmented picture containing a face. After a series of convolution, pooling, and full connection processes in the neural network model, 6 or 7 floating point values ​​are output, representing the confidence of 6 or 7 emotions. However, the recognition accuracy of these models is not high enough.

[0049] In addition, the tone of voice and speech content in a person's speech can also identify emotions, especially the characteristics of voice tone, such as speech rate, amplitude, fundamental frequency and other audio features are related to emotion categories. Therefore, there are also audio-based emotion recognition neural network models, such as CNN-based models and LSTM-based models, whose inputs can be audio feature data, such as FBANK, MFCC, fundamental frequency, energy, etc., and their combinations. Since audio is a time series, it is difficult to infer the speaker's emotions based on short (such as 1 or a few frames) audio, and audio of a sufficiently long time is required to infer emotions. Therefore, most of the neural networks that infer emotions based on audio input are audio features of a period of time, such as 2 seconds or more of audio. However, similarly, the accuracy of audio-based emotion recognition models is not high enough, and there is still a lot of room for improvement.

[0050] Therefore, we can consider combining the above two features to identify emotions, and propose an emotion recognition scheme that combines facial expressions and speech audio.

[0051] Figure 2A A combination scheme is shown, in which the expression-based emotion recognition model and the speech-based emotion recognition model are trained and inferred independently, and then post-processing is used to further process the results of the two models. For example, the result is considered correct only when the expression-based reasoning result and the speech-based reasoning result are consistent, otherwise the reasoning result is abandoned. However, since expressions change quickly, the reasoning is based on a photo of a moment, while the audio needs to be over a period of time, so the two are not easy to synchronize.

[0052] Figure 2B Another combination scheme is shown, which simply trains the expression-based emotion recognition network and the speech-based emotion recognition network together, and the loss function Loss comprehensively considers the errors of the two networks. However, with this scheme, not only are the two networks out of sync in time, but the joint training is difficult to converge and the training data is difficult to obtain.

[0053] Therefore, some embodiments of the present disclosure propose a new multimodal emotion recognition scheme that combines facial expressions and spoken audio, which can improve the accuracy of emotion recognition by fusing facial expression difference features with audio emotion features for final emotion recognition. In some examples, since the changes in expressions are more obvious and faster, they can better express emotional changes. Therefore, by fusing facial expression difference features into the audio network, it is more conducive to network convergence. In some examples of applying the method of the present disclosure to smart glasses, even if the resources on the smart glasses are limited, the accuracy of emotion recognition can be improved as much as possible.

[0054] It should be understood that the emotion recognition method provided by the embodiments of the present disclosure can be applied not only to the aforementioned smart glasses, but also to various other types of handheld devices (such as mobile phones, personal digital assistants (PDAs), etc.), various types of computers (such as tablets, laptops, ultra-mobile personal computers (UMPCs), netbooks, laptops, etc.), wearable devices, vehicle-mounted devices, AR / VR devices and other electronic devices. The embodiments of the present disclosure do not impose any restrictions on the specific types of application devices such as electronic devices, as long as the application device is equipped with a microphone and a camera and has a demand for emotion recognition.

[0055] As an example but not limitation, when the electronic device is a wearable device, the wearable device can also be a general term for wearable devices that are developed by applying wearable technology to intelligently design daily wearables, such as gloves, watches, AR head-mounted display devices, VR head-mounted display devices, or MR head-mounted display devices equipped with far-field communication modules and / or near-field communication modules.

[0056] The following will be combined Figures 3 to 5 Some embodiments of the present disclosure are described in detail. In the present disclosure, "plurality" or similar expressions refer to two or more.

[0057] Figure 3 The schematic flowchart of the emotion recognition method according to at least one embodiment of the present disclosure is exemplarily shown. Figure 4 Given Figure 3 A specific implementation example of the operation of obtaining audio features in Figure 5 Given Figure 3 A specific implementation example of the method.

[0058] like Figure 3 As shown, in step S310, speech audio of the target speaker from a first moment to a second moment after the first moment, and features of each frame divided from the speech audio are obtained.

[0059] For example, in the aforementioned scenario where the user wears smart glasses to communicate, the target speaker is the person who is talking to the user, and the microphone of the smart glasses can be used to obtain the speech audio. In order to obtain the emotion of the target speaker, a speech audio that lasts for a period of time is usually required, such as 2 seconds of speech audio, that is, the time interval from the first moment to the second moment is 2 seconds.

[0060] In some implementations, the speech audio is a segment of continuously collected audio, and the audio sampling rate is, for example, 16 kHz, the bit width is 16 bits, and it is a single channel.

[0061] In order to process in real time, the speech audio can be cut into segments (i.e., divided into multiple frames), usually 10ms as a frame. Generally speaking, a 10ms segment of audio can extract meaningful features and results. In the subsequent processing, in order to prevent spectrum leakage, the method of partially overlapping the previous and next frames is adopted during processing. Each frame length is 25ms, of which 15ms is historical information, that is, 15ms overlap, and the actual movement is only 10ms.

[0062] For example, the acquired speech audio is 160ms long and is divided into 16 frames. The feature of each frame is 40-dimensional data, and the features of the 16 frames are 16*40 matrices. In some embodiments, the features of each frame are frequency domain features, because the frequency domain features of sound are more recognizable than time domain features. Frequency domain features can include MFCC features, FBANK features, fundamental frequency, energy, etc. The FBANK feature is obtained by summing the squares of the amplitude of the power spectrum of the Mel filter and then taking the logarithm. Compared with MFCC, the calculation of the discrete cosine transform step is omitted.

[0063] Figure 4 An exemplary process for obtaining FBANK features for each frame of spoken audio is given: generally, the audio file is pre-emphasized first, in order to enhance the high-frequency signal; then it is framed and windowed. Framing is to separate the audio signal into frames of 10ms. Windowing is to prevent spectrum leakage. Each time, 25ms of signal is used to calculate the features, that is, each time it moves 10ms, and 25ms of signal is actually used, with 15ms of historical overlapping information. Then, the time domain signal is used to obtain the frequency domain signal, that is, the spectrum, through Fourier transform. The frequency domain of a certain time is accumulated to obtain the speech spectrum (the power spectrum is taken and then the amplitude is squared), and then the frequency is mapped to the Mel frequency scale through the Mel filter bank, and finally the logarithm is taken to obtain the FBANK feature. In the application, the Mel filter bank is generally 40 or 80, that is, each frame of audio corresponds to 40 or 80 outputs.

[0064] Back to Figure 3 After obtaining the features of each frame, in step S320, audio emotion features are extracted from the features of each frame.

[0065] In some embodiments, the first neural network module can be used to extract audio emotion features from the features of each frame, wherein the first neural network module includes one or more neural network units in series that have the function of saving historical information. For example, the neural network units that have the function of saving historical information include: LSTM, GRU (Gated Recurrent Unit), FSMN (Feedforward Sequential Memory Network) and / or various attention mechanism neural network units, etc. Any number of neural network units can be selected as needed, and various types of neural network units can be arbitrarily combined and connected in series.

[0066] As mentioned earlier, audio that is too short cannot determine emotions. Only a continuous audio (such as 2 seconds of audio) can determine people's emotions. However, the time of each input frame is usually not long enough, so the use of a neural network unit with the function of saving historical information can save previous historical audio data, so that the audio emotional features can be better extracted. This is particularly beneficial for streaming reasoning. For example, when using 2 seconds of audio for emotion recognition, the neural network does not need to wait 2 seconds to make an inference, but usually can be one frame (such as 10ms, 20ms or 25ms long) or multiple frames for reasoning. The previous historical audio data can be saved in a neural network with a memory function. When calculating the current frame, it is equivalent to considering the historical audio data.

[0067] For example, the first neural network module may include two LSTMs connected in series, and the audio feature data inputted therein may be MFCC features, FBANK features, fundamental frequency features, energy, or a combination thereof, etc. After the first neural network module calculates one or more frames of audio feature data, a one-dimensional vector or a two-dimensional matrix is ​​outputted as the audio emotion feature. Alternatively, one LSTM layer or more than two LSTM layers may be used as required.

[0068] In some examples, the first neural network module can be part of an audio network, which is a neural network model that mainly processes audio features for emotion recognition, and includes a first neural network module, a fusion layer described later, and a second neural network module, wherein the facial expression difference feature fused at the fusion layer described later can be regarded as an externally introduced feature to assist the audio network in inferring emotions. The audio networks are trained and inferred together, so the audio emotion feature extracted in step S320 can be regarded as an intermediate feature of an audio neural network model for emotion recognition.

[0069] like Figure 3As shown, after the audio emotion feature is extracted, in step S330, the audio emotion feature is fused with the facial expression difference feature to obtain a fused emotion feature, wherein the facial expression difference feature is determined based on the difference between a first facial expression feature extracted from a first facial image of the target speaker and a second facial expression feature extracted from a second facial image of the target speaker, and the first facial image and the second facial image are collected at the first moment and the second moment, respectively.

[0070] Since people's facial features vary greatly, it is difficult to judge emotions from static images. For example, some people are born with a cold face, and a slight upward movement of the corners of their mouth may mean that they are very happy, while others are born with a smiling face, and their eyebrows are still raised even when they are particularly sad. A better way to judge a person's emotional changes is the changes before and after, such as the changes in the degree of smiling eyebrows and the upward movement of the corners of the mouth, which represent the changes in emotions. Therefore, the solution disclosed in this disclosure uses the difference in the changes in facial expressions before and after in time, integrates the facial expression difference features into the neural network, and infers the current user's emotions together with the audio features.

[0071] Facial expression difference features can be obtained in various ways. In some embodiments, the method according to the present disclosure can directly receive the facial expression difference features from the outside and integrate them into the audio network to assist emotion recognition. In other words, the scheme of the present disclosure does not limit the part of specifically calculating the facial expression difference features. In other embodiments, the part of collecting facial images and specifically calculating the facial expression difference features based on the facial images can also be used as a part of the method according to the present disclosure, which is performed independently of the operations of the aforementioned steps S310 and S320, that is, the order of operations on both sides is not limited.

[0072] Various methods can be used to extract facial expression features from the target speaker's facial image. In some embodiments, a two-stage solution can be used, that is, first using a face target detection model to determine the face region in the face image, and then using a face emotion recognition neural network model or a face feature extraction model based on face feature points to extract facial expression features from the image information in the face region.

[0073] For example, various face target detection models can be used, such as models using various neural networks (such as YOLO) or various statistical feature methods (such as face detection in OpenCV), to find the face area (such as a rectangular area containing the face) in the face image, segment the face area, and provide it to the next stage for facial expression feature extraction.

[0074] For example, a facial emotion recognition neural network model can be used to obtain facial expression features based on the image information in the face area of ​​the facial image. Exemplarily, the facial emotion recognition neural network model can be obtained by removing the last one or more layers of the trained neural network model for obtaining emotion categories based on face images. That is, the facial emotion recognition neural network model can be a part of various neural network models (such as the aforementioned Mini-Xception-based or CNN-based neural network models) that are trained to obtain emotion categories based on face images. For example, a classification model that uses a neural network to obtain emotion categories based on face images can be trained first, and its input is the image information in the face area, and the output is the confidence value of a predetermined plurality of emotion categories (such as 7 emotion categories); after the training is completed, the last layer (output layer) or the last multiple layers of the classification model can be removed as needed to obtain the facial emotion recognition neural network model of the present application, so the facial expression features obtained in the present application are actually the output of a certain intermediate layer of the aforementioned classification model, which usually has more dimensions than the emotion category, and the facial expression information has been extracted from the input image.

[0075] Alternatively, various facial feature extraction models based on facial feature points, such as a model based on statistics (such as the common 68 feature points), can be used to extract facial expression features from image information in the face area of ​​a facial image.

[0076] The same method (eg, using the same neural network model) may be used to extract the first facial expression feature and the second facial expression feature from the first facial picture and the second facial picture of the target speaker, respectively.

[0077] In some embodiments, the second facial expression feature and the first facial expression feature may be bitwise subtracted to obtain a difference feature as the facial expression difference feature. In other embodiments, after extracting two facial expression features from two facial pictures before and after the target speaker, the facial expression difference feature may be determined according to a preset threshold value. Specifically, the second facial expression feature and the first facial expression feature may be bitwise subtracted to obtain a difference feature; when the maximum value among the absolute values ​​of all elements of the difference feature is less than a preset threshold value, a feature with a preset fixed value is used as the facial expression difference feature; when the maximum value among the absolute values ​​of all elements of the difference feature is not less than the threshold value, the difference feature is used as the facial expression difference feature. For example, the feature with a preset fixed value may be an all-zero feature having the same dimension as the difference feature.

[0078] When the difference between the facial expressions collected before and after is less than the threshold value, it can be considered that the expression has not changed. At this time, the difference feature is set to a fixed value (for example, an all-0 feature) and the all-0 feature is fused into the audio network. Because the all-0 feature is relatively simple, this facilitates the training convergence of the neural network and also helps to improve accuracy.

[0079] In some embodiments, two features may be fused by concatenation. For example, when at least one of the audio emotion feature and the facial expression difference feature is a non-one-dimensional vector, the non-one-dimensional vector is flattened into a one-dimensional vector; and after flattening, the audio emotion feature and the facial expression difference feature are concatenated into the fused emotion feature. The fused emotion feature obtained after concatenation is a one-dimensional vector.

[0080] In some examples, the camera of the smart glasses can be used to collect the facial image (first facial image) of the target speaker at the first moment, such as a black-and-white or color image with a resolution of 480*480, and the first facial expression feature is extracted from the first facial image by any of the above methods, and then the first facial expression feature is saved in the memory, so as to compare the difference between the changes before and after with the facial expression feature (second facial expression feature) after that. At a later moment (second moment), such as 2s later, the camera collects the facial image (second facial image) of the target speaker again, and extracts the second facial expression feature from the second facial image in the same way, and then saves the newly extracted facial expression feature (second facial expression feature) in the memory (it can be a newly saved feature or the previously saved feature is overwritten and updated after the previously saved feature is read out), and is bitwise subtracted from the previously saved first facial expression feature, and each difference obtained by the subtraction is compared with the threshold value. When the maximum difference (absolute value) is less than the threshold value, the facial expression difference feature is directly set to the all-0 feature; the significance of this is that the picture with a small difference is considered to have no change in emotion, and the all-0 feature facilitates the learning and convergence of the neural network. When the maximum difference (absolute value) is not less than the threshold value, the difference feature obtained by the previous subtraction is used as the facial expression difference feature.

[0081] Then, the facial expression difference feature is fused into the audio network, which is equivalent to using the expression difference as a fused feature to assist the audio network in inferring emotions. For example, the facial expression difference feature and the audio emotion feature are spliced ​​together to expand the dimension. Assuming that the facial expression difference feature is 100-dimensional and the audio feature is 1582-dimensional, the dimension of the audio emotion feature after LSTM×2 is 32*1582, and it is flattened to 50624 dimensions. The fused emotion feature obtained after fusion is 50624+100=50724 dimensions.

[0082] like Figure 3 As shown, after the fused emotion feature is obtained, in step S340, the emotion category of the target speaker is obtained according to the fused emotion feature.

[0083] For example, various neural networks may be used to obtain the confidence scores of each emotion category from the fused emotion features (similar to a classification model), and then the final emotion category of the target speaker may be determined based on the confidence scores.

[0084] In some embodiments, the fused emotion feature is input into a second neural network module, wherein the second neural network module includes one or more processing units connected in series and an output layer connected after the processing unit, the processing unit includes a convolution unit and / or a fully connected unit, the convolution unit includes a convolution layer, a normalization layer and an activation layer, and the fully connected unit includes a fully connected layer, a normalization layer and an activation layer; and the confidence scores of all emotion categories are output from the output layer of the second neural network module, and the emotion category of the target speaker is determined according to the confidence scores. For example, the second neural network module can include one or more convolution units connected in series as needed, and optionally a fully connected unit is connected in series thereafter.

[0085] In the convolutional unit and the fully connected unit, the convolutional layer and the fully connected layer are used to further extract the fused emotion features. The fully connected layer can also project the output of the previous layer onto the feature dimension. The normalization layer is usually placed after the convolutional layer / fully connected layer and before the activation layer to constrain the data to a mean of 0 and a variance of 1. In this way, the activation function in the activation layer will not be saturated and truncated, which reduces the error and is more conducive to the convergence of neural network training. Batch normalization is usually used for the convolutional layer, and batch normalization or layer normalization can be used for the fully connected layer. The activation function is used in the activation layer to bring nonlinear characteristics to the neural network. The convolutional, fully connected and other layers themselves cannot bring nonlinear characteristics. The neural network is like a function that transforms the input data into the desired output data. A network without nonlinear characteristics is not enough to complete this transformation. You can use activation functions such as Relu. The activation function does not change the data dimension, that is, the dimension of the output feature is the same as the dimension of the input feature.

[0086] The convolution layer / fully connected layer, normalization layer, and activation layer constitute a processing unit. Multiple processing units can be set as needed to fit a stronger network model. During deployment, a processing unit can be optimized into a fusion operator to speed up the calculation.

[0087] For example, the output layer of the second neural network module can also be a fully connected layer, and its output is the final confidence score of each emotion category, so the output dimension is the number of categories. For example, there are 7 emotion categories: angry, happy, afraid, sad, surprised, disgusted, and natural, then the output dimension is 7, and the output features are 7 floating point numbers, which respectively represent the confidence scores of these 7 emotion categories.

[0088] For example, the second neural network module and the first neural network module are trained together as modules in the same neural network model, as shown below Figure 5 That is, the second neural network module and the first neural network module are respectively part of the same neural network model, and the neural network model is trained and reasoned in an integrated manner. In some cases, the solution of the present disclosure is only used to design and train the neural network model, such as the following Figure 5 The illustrated audio network, and the facial expression difference features are obtained using an existing trained model. Therefore, the multimodal emotion recognition solution combining facial expressions and spoken audio disclosed in the present invention can simply and efficiently improve the accuracy of emotion recognition.

[0089] In addition, a dropout layer may be connected after one of the processing units in the second neural network module during training. The dropout layer is only available during the training phase, not during the inference phase. In order to prevent overfitting during training, some convolutions are randomly invalidated, for example, some convolution weights are invalidated, so that the trained weights have better generalization performance and avoid overfitting.

[0090] In some examples, the second neural network module may include a processing unit and a fully connected layer as an output layer, and also include a dropout layer for random dropout processing during training, wherein the processing unit includes a convolution layer / fully connected layer, a normalization layer, and an activation layer. The convolution layer / fully connected layer performs convolution / fully connected processing on the input fused emotion features, which is then normalized by the normalization layer, which is then activated by the activation layer, which is then randomly dropped during training by the dropout layer, and finally fully connected by the fully connected layer as the output layer, outputting confidence scores for each emotion category, and then determining the final emotion category based on the confidence score.

[0091] In addition, in some embodiments, the emotion recognition method disclosed in the present invention is performed in a streaming reasoning manner. For example, assuming that an inference is to be made on data every 2 seconds, the previous 2 seconds of audio and photo data can be collected every 300ms. In this way, 1700ms of the inference data before and after are overlapped, and only 300ms are different data, and a series of inference results are obtained. For example, streaming reasoning produces a result every 0.3s. Therefore, these inference results must be post-processed to determine the final emotion category. In other words, the interval between the first moment and the second moment is 2 seconds, and the difference between the first moment and the second moment in the two inferences is 300ms, and the difference between the second moments is also 300ms.

[0092] Therefore, in some embodiments, the method according to the present disclosure may further include: post-processing the obtained emotion category of the target speaker. The post-processing may be performed in two ways: low-pass filtering to remove burrs, or similar moving average technology. For example, the post-processing may include: low-pass filtering a predetermined plurality of sequentially obtained emotion categories; or, when the emotion category of the target speaker is determined by selecting the emotion category corresponding to the highest value among the confidence scores of all emotion categories, calculating the average of multiple confidence scores obtained within a predetermined time for each emotion category in all emotion categories, and selecting the emotion category with the highest average as the emotion category of the target speaker.

[0093] For example, the following 10 inference results are obtained in 3 seconds: 1. Happy; 2. Happy; 3. Happy; 4. Happy; 5. Natural; 6. Happy; 7. Natural; 8. Happy; 9. Happy; 10. Happy. Low-pass filtering can be used to filter out the 5th and 7th abnormal points. Alternatively, the confidence scores of various emotion categories obtained from these 10 inferences can be averaged, and then the emotion category with the highest average value can be used as the final determined emotion category of the target speaker.

[0094] Combine the following Figure 5 An exemplary implementation of an emotion recognition method according to at least one embodiment of the present disclosure is described.

[0095] like Figure 5 As shown, the target speaker's speech audio, such as 2s of speech audio, can be collected using, for example, a microphone provided in the smart glasses.

[0096] Then, in step S1, audio features are extracted from the speech audio to obtain audio features of each frame (for example, using Figure 4 The FBANK feature of each frame is obtained by the method and input into the subsequent audio network.

[0097] Then, in step S2, the audio emotion feature is extracted from the input audio feature using the first neural network module. The first neural network module used includes two LSTMs connected in series, and the audio feature data inputted can be MFCC features, FBANK features, fundamental frequency features, energy, or a combination thereof, etc. After the first neural network module calculates one or more frames of audio feature data, a one-dimensional vector or a two-dimensional matrix is ​​output as the audio emotion feature. Alternatively, one LSTM layer or more than two LSTM layers can be used as needed.

[0098] like Figure 5 As shown, the first neural network module is part of an audio network, and the audio network is a neural network model that mainly processes audio features for emotion recognition, which includes a first neural network module, a fusion layer, and a second neural network module, wherein the facial expression difference feature fused at the fusion layer can be regarded as an externally introduced feature to assist the audio network in inferring emotions. The audio networks are trained and inferred together, so the audio emotion feature extracted by the first neural network module can also be regarded as an intermediate feature of an audio neural network model for emotion recognition.

[0099] In addition, if Figure 5 As shown, it can also be performed independently from the aforementioned collection of speech audio and steps S1 and S2. Figure 5 The operations shown in the left branch are steps S3 to S5 of collecting facial images and then specifically calculating facial expression difference features based on the facial images.

[0100] For example, the camera of the smart glasses can be used to collect the facial image (first facial image) of the target speaker at the current moment (first moment), such as a black and white or color image with a resolution of 480*480, which is input into step S3, and the facial expression features are extracted from the facial image in any of the above-mentioned ways, and then the facial expression features (first facial expression features) extracted from the currently collected image are stored in the memory in step S4, so as to compare the difference between the changes before and after with the facial expression features (second facial expression features) thereafter. At a later moment (second moment), such as 2s later, the camera collects the facial image (second facial image) of the target speaker again, and also inputs it into step S3 to extract the facial expression features (second facial expression features), and then in step S4, the newly extracted facial expression features (second facial expression features) are stored in the memory (can be a newly added saved feature or after reading out the previously saved features, the previously saved features are covered and updated), and are input into step S5 for bitwise subtraction with the previously saved first facial expression features, and each difference obtained by subtraction is compared with the threshold value. When the maximum difference (absolute value) is less than the threshold value, the facial expression difference feature is directly set to all 0 features; the significance of this is that pictures with small differences are considered to have no change in emotion, and all 0 features facilitate neural network learning and convergence. When the maximum difference (absolute value) is not less than the threshold value, the difference feature obtained by previous subtraction is used as the facial expression difference feature.

[0101] Then, in step S6, the facial expression difference features are fused into the audio network, which is equivalent to using the expression difference as a fused feature to assist the audio network in inferring emotions.

[0102] For example, in step S6, the facial expression difference feature and the audio emotion feature are spliced ​​together to expand the dimension. Assuming that the facial expression difference feature is 100-dimensional and the audio feature is 1582-dimensional, the dimension of the audio emotion feature after LSTM×2 is 32*1582, and it is flattened to 50624 dimensions. The fused emotion feature obtained after fusion is 50624+100=50724 dimensions.

[0103] Then, in steps S7 to S11, the emotion category of the target speaker is obtained according to the fused emotion feature, wherein each step is performed by each layer in the second neural network module. Figure 5 The second neural network module in the embodiment includes a processing unit and a fully connected layer as an output layer, and during training also includes a dropout layer for random dropout processing as shown in a dotted box, wherein the processing unit includes a convolutional layer / fully connected layer, a normalization layer, and an activation layer.

[0104] Specifically, step S7 is the convolution / full connection processing of the input fusion emotion features by the convolution layer / full connection layer, S8 is the normalization processing of the subsequent normalization layer, S9 is the activation processing of the subsequent activation layer, S10 is the random discard processing of the discard layer during training, and S11 is the final full connection processing of the fully connected layer as the output layer. S11 outputs the confidence score of each emotion category, and then determines the final emotion category based on the confidence score.

[0105] Figure 6 A schematic diagram of the structure of an electronic device that can be used to implement the above-mentioned emotion recognition method according to at least one embodiment of the present disclosure is shown.

[0106] See also Figure 6 , the electronic device 600 includes a memory 610 and a processor 620 .

[0107] The processor 620 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 620 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 620 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0108] The memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 620 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 610 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0109] The memory 610 stores executable codes, and when the executable codes are processed by the processor 620 , the processor 620 can execute the emotion recognition method mentioned above.

[0110] In some embodiments, the electronic device may further include a microphone and a camera, wherein the microphone is configured to collect speech audio of the target speaker, and the camera is configured to collect facial images of the target speaker. Thus, the processor and memory of the electronic device may utilize the audio and image data collected by the microphone and camera of the electronic device to implement the emotion recognition method described in the above embodiments of the present disclosure.

[0111] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0112] Alternatively, the present disclosure may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present disclosure.

[0113] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0114] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An emotion recognition method, comprising: Obtaining speech audio of a target speaker from a first moment to a second moment after the first moment, and features of each frame divided from the speech audio; Extracting audio emotion features from the features of each frame; fusing the audio emotion feature with the facial expression difference feature to obtain a fused emotion feature, wherein the facial expression difference feature is determined based on a difference between a first facial expression feature extracted from a first facial picture of the target speaker and a second facial expression feature extracted from a second facial picture of the target speaker, and the first facial picture and the second facial picture are collected at the first moment and the second moment, respectively; as well as The emotion category of the target speaker is obtained according to the fused emotion feature.

2. The method according to claim 1, wherein: The facial expression difference feature is determined based on the difference between the first facial expression feature and the second facial expression feature, and includes: Subtracting the second facial expression feature from the first facial expression feature bit by bit to obtain a difference feature; When the maximum value among the absolute values ​​of all elements of the difference feature is less than a preset threshold value, taking a feature of a preset fixed value as the facial expression difference feature; When the maximum value among the absolute values ​​of all elements of the difference feature is not less than the threshold value, the difference feature is used as the facial expression difference feature.

3. The method according to claim 1, wherein: Extracting a first facial expression feature from a first facial image of the target speaker and extracting a second facial expression feature from a second facial image of the target speaker include: Using a face object detection model, respectively determining face regions in the first face image and the second face image; and Using a facial emotion recognition neural network model, obtaining the first facial expression feature and the second facial expression feature according to image information in the face regions of the first facial image and the second facial image respectively; Alternatively, the first facial expression feature and the second facial expression feature are extracted from the image information in the face area of ​​the first facial image and the second facial image respectively using a facial feature extraction model based on facial feature points.

4. The method according to claim 1, wherein: The audio emotion feature and the facial expression difference feature are fused to obtain a fused emotion feature, including: In the case where at least one of the audio emotion feature and the facial expression difference feature is a non-one-dimensional vector, flattening the non-one-dimensional vector into a one-dimensional vector; and After flattening, the audio emotion feature and the facial expression difference feature are concatenated into the fused emotion feature.

5. The method according to claim 1, wherein: Extracting audio emotion features from the features of each frame includes: The audio emotion features are extracted from the features of each frame using a first neural network module, wherein the first neural network module includes one or a plurality of neural network units connected in series and having a function of preserving historical information.

6. The method according to claim 5, wherein: Obtaining the emotion category of the target speaker according to the fused emotion feature includes: Inputting the fused emotion feature into a second neural network module, wherein the second neural network module comprises one or a plurality of processing units connected in series and an output layer connected after the processing unit, the processing unit comprises a convolution unit and / or a fully connected unit, the convolution unit comprises a convolution layer, a normalization layer and an activation layer, and the fully connected unit comprises a fully connected layer, a normalization layer and an activation layer; and The confidence scores of all emotion categories are output from the output layer of the second neural network module, and the emotion category of the target speaker is determined according to the confidence scores.

7. The method according to claim 6, wherein: The first neural network module and the second neural network module are trained together as modules in the same neural network model, and / or a dropout layer is connected after one of the processing units in the second neural network module during training.

8. The method according to claim 1, wherein: The method is performed in a streaming reasoning manner, and the method further includes: Post-processing the obtained emotion category of the target speaker, The post-processing includes: Low-pass filtering is performed on a predetermined plurality of sequentially obtained emotion categories; or In a case where the emotion category of the target speaker is determined by selecting the emotion category corresponding to the highest value among the confidence scores of all emotion categories, an average value of multiple confidence scores obtained within a predetermined time is calculated for each emotion category among all emotion categories, and the emotion category with the highest average value is selected as the emotion category of the target speaker.

9. An electronic device, comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 8.

10. A computer program product, comprising an executable code, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 8.

Images