Emotion recognition method and apparatus, storage medium, and processor

Abstract

The application discloses an emotion recognition method and device, a storage medium, and a processor. It relates to the field of artificial intelligence. The method comprises the following steps: obtaining a to-be-recognized face image and to-be-recognized voice information of a target object; determining a face feature in the to-be-recognized face image and a voice feature in the to-be-recognized voice information; fusing the face feature and the voice feature to obtain a fused feature; analyzing the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, each set of fused classification data in the plurality of sets of fused classification data comprises the fused feature and a calibrated fused classification value; and determining an emotion recognition result of the target object based on the fused classification value. The application solves the problem of inaccurate emotion recognition results in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular, to a method and device for emotion recognition, a storage medium, and a processor. BACKGROUND

[0002] With the rapid development of computer vision technology, human-computer interaction has gradually been valued. Emotion recognition, as an important part of human-computer interaction, has also begun to attract widespread attention. How to use a computer to quickly recognize human emotions has become a research hotspot in the field of computer vision.

[0003] Currently, emotion recognition methods at home and abroad are mainly divided into single-modal-based recognition and multi-modal-based recognition. Single-modal-based emotion recognition refers to collecting and analyzing information of a single channel, and finally obtaining the emotional state of a tester through certain technical means. This method has some limitations. Multi-modal-based methods fuse information of multiple channels to make emotional judgments. The mainstream fusion method includes decision-level fusion.

[0004] The decision-level fusion method uses different classifiers to model different emotional expressions, and finally fuses the recognition results of each classifier. The implementation is to use different modules to process audio and video features respectively, to obtain the corresponding recognition rates, and finally to obtain the final recognition result according to the proportional scoring mechanism and the corresponding weight distribution strategy.

[0005] However, the existing decision fusion method has the following shortcomings: 1. The proportion and weight of the results of different classifiers have no strict basis and unified standard. Different researchers of the same project often get different results according to different proportions and weights. 2. The decision fusion method focuses on the fusion of recognition results, but ignores the internal relationship between different channel features.

[0006] Therefore, based on the above shortcomings, the existing emotion recognition method has the problem of inaccurate recognition results.

[0007] In view of the problem of inaccurate emotion recognition results in the related art, no effective solution has been proposed so far. SUMMARY

[0008] The main purpose of the present application is to provide an emotion recognition method, device, storage medium, and processor to solve the problem of inaccurate emotion recognition results in the related art.

[0009] To achieve the above object, according to one aspect of the present application, a method for emotion recognition is provided. The method comprises: obtaining a to-be-recognized face image and to-be-recognized voice information of a target object; determining a face feature in the to-be-recognized face image and a voice feature in the to-be-recognized voice information; fusing the face feature and the voice feature to obtain a fused feature; analyzing the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, each set of the fused classification data comprising a fused feature and a calibrated fused classification value; and determining an emotion recognition result of the target object based on the fused classification value.

[0010] Optionally, determining the emotion recognition result of the target object based on the fused classification value comprises: judging whether the fused classification value is not less than a predetermined threshold; in a case where the fused classification value is not less than the predetermined threshold, determining the emotion recognition result of the target object based on the fused classification value; in a case where the fused classification value is less than the predetermined threshold, obtaining a face classification value by analyzing the face feature based on a face classification model and obtaining a voice classification value by analyzing the voice feature based on a voice classification model; and determining the emotion recognition result of the target object based on the fused classification value, the face classification value and the voice classification value, wherein the face classification model is trained by a plurality of sets of face classification data through machine learning, each set of the face classification data comprising a face feature and a calibrated face classification value, and the voice classification model is trained by a plurality of sets of voice classification data through machine learning, each set of the voice classification data comprising a voice feature and a calibrated voice classification value.

[0011] Optionally, obtaining the to-be-recognized face image and the to-be-recognized voice information of the target object comprises: obtaining a to-be-recognized video of the target object; decomposing the to-be-recognized video into an image frame sequence and a voice signal; pre-processing the image frame sequence by using a face detection algorithm to obtain the to-be-recognized face image; and pre-processing the voice signal by using a mel-frequency cepstral coefficient to obtain the to-be-recognized voice information.

[0012] Optionally, pre-processing the image frame sequence by using the face detection algorithm to obtain the to-be-recognized face image comprises: inputting the image frame sequence into a preset face recognition model to recognize a face region of each image in the image frame sequence, wherein the preset face recognition model is trained by a plurality of sets of data through machine learning, each set of the data comprising a face image and a calibrated face region in the face image; and cropping the face region from each image in the image frame sequence as the to-be-recognized face image.

[0013] Optionally, the pre-processing of the speech signal by using the mel frequency cepstral coefficient to obtain the to-be-identified speech information comprises: pre-emphasizing, framing and windowing the speech signal to obtain at least one analysis window; obtaining the frequency spectrum of each analysis window through fast Fourier transform; performing mel filtering on the frequency spectrum to obtain a mel spectrum; performing cepstrum analysis on the mel spectrum to obtain a mel frequency spectrum coefficient; and determining the mel frequency spectrum coefficient as the to-be-identified speech information.

[0014] Optionally, the determining of the face feature in the to-be-identified face image and the speech feature in the to-be-identified speech information comprises: extracting the face feature of the to-be-identified face image by using a face feature extraction model, wherein a loss function for training the face feature extraction model is determined based on a soft maximum loss function softmax loss and a center loss function centerloss, and the proportion of the soft maximum loss function softmax loss and the center loss function centerloss is controlled by a preset hyperparameter; and extracting the speech feature of the to-be-identified speech information by using a speech feature extraction model, wherein a loss function for training the speech feature extraction model is a cross-entropy loss function.

[0015] Optionally, the fusing of the face feature and the speech feature to obtain the fusion feature comprises: obtaining a first output feature vector of a last full connection layer of the face feature extraction model; obtaining a second output feature vector of a last full connection layer of the speech feature extraction model; performing dimension reduction on the first output feature vector and the second output feature vector; and calculating the first output feature vector and the second output feature vector after dimension reduction by using a convolution operation to determine the fusion feature.

[0016] To achieve the above object, according to another aspect of the present application, a kind of emotion recognition device is provided.The device comprises: obtaining unit, for obtaining the to-be-identified face image of target object and to-be-identified speech information;First determining unit, for determining the face feature in the to-be-identified face image and the speech feature in the to-be-identified speech information;Fusion unit, for fusing the face feature and the speech feature, obtains fusion feature;Analysis unit, for based on feature fusion classification model analysis the fusion feature, obtains fusion classification numerical value, wherein the feature fusion classification model is by machine learning training from multiple groups of fusion classification data, each group of fusion classification data in the multiple groups of fusion classification data includes: fusion feature and calibrated fusion classification numerical value;Second determining unit, for based on the fusion classification numerical value determines the emotion recognition result of the target object.

[0017] To achieve the above object, according to an aspect of the present application, a processor is provided. The processor is used to run a program, wherein the program performs the above-mentioned emotion recognition method when running.

[0018] To achieve the above object, according to an aspect of the present application, an electronic device is provided. The electronic device comprises one or more processors and a memory, the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned emotion recognition method.

[0019] Through the present application, the following steps are adopted: obtaining a to-be-recognized face image and to-be-recognized voice information of a target object; determining a face feature in the to-be-recognized face image and a voice feature in the to-be-recognized voice information; fusing the face feature and the voice feature to obtain a fused feature; analyzing the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, and each set of fused classification data in the plurality of sets of fused classification data comprises the fused feature and a calibrated fused classification value; determining an emotion recognition result of the target object based on the fused classification value; solve the problem that the emotion recognition result is not accurate in the related art, and then determine the emotion recognition result based on the fused classification value of the fused feature, realize the determination of the emotion recognition result based on the quantized value after the feature fusion, and achieve the effect of improving the accuracy of the emotion recognition result. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein in their entirety, and the illustrative embodiments of the application and their description serve to explain the application. In the drawings:

[0021] Figure 1 is a flowchart of an emotion recognition method provided according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a voice and facial expression bimodal emotion fusion recognition method based on a neural network provided according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an emotion recognition device according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other in the case of no conflict. The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0026] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, not necessarily to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] The present application will be described below in combination with preferred implementation steps, Figure 1 is a flowchart of an emotion recognition method provided according to an embodiment of the present application, as shown in the figure, the method comprises the following steps: Figure 1

[0029] Step S102, obtaining a to-be-recognized face image and to-be-recognized voice information of a target object;

[0030] Step S104, determining a face feature in the to-be-recognized face image and a voice feature in the to-be-recognized voice information;

[0031] Step S106, fusing the face feature and the voice feature to obtain a fused feature;

[0032] Step S108, analyzing the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, and each set of fused classification data in the plurality of sets of fused classification data comprises a fused feature and a calibrated fused classification value;

[0033] Step S110, determining an emotion recognition result of the target object based on the fused classification value.

[0034] ​The emotion recognition method provided in the embodiments of the present application comprises the following steps: obtaining a to-be-recognized face image and to-be-recognized voice information of a target object; determining a face feature in the to-be-recognized face image and a voice feature in the to-be-recognized voice information; fusing the face feature and the voice feature to obtain a fused feature; analyzing the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, and each set of fused classification data in the plurality of sets of fused classification data comprises the fused feature and a calibrated fused classification value; and determining an emotion recognition result of the target object based on the fused classification value. The emotion recognition result is inaccurate in the related art, and the emotion recognition result is determined based on the fused classification value of the fused feature, the emotion recognition result is determined based on the quantized value after feature fusion, and the accuracy of the emotion recognition result is improved.

[0035] In the step S102, the target object can be an object to be recognized, the to-be-recognized face image is a face image of the target object, and the to-be-recognized voice information is voice information of the target object.

[0036] In the step S106, the fused classification value is used to represent a probability that the fused feature belongs to a certain emotion classification, wherein the fused classification value has a corresponding emotion classification, for example, the fused classification value can represent a probability that the target object belongs to a positive emotion.

[0037] For example, the emotion classification includes a positive emotion and a negative emotion, the fused classification value represents a probability that the target object belongs to the positive emotion, and then the probability that the target object belongs to the positive emotion is determined to be 60% based on the analysis of the fused feature based on the feature fusion classification model, and 60% is the fused classification value of the target object.

[0038] Optionally, the higher the fused classification value is, the greater the probability that the emotion of the target object belongs to the emotion classification corresponding to the fused classification value is, and the more accurate the emotion recognition result is.

[0039] As an optional embodiment, the determining the sentiment recognition result of the target object based on the fusion classification value comprises: determining whether the fusion classification value is not less than a predetermined threshold; in a case where the fusion classification value is not less than the predetermined threshold, determining the sentiment recognition result of the target object based on the fusion classification value; in a case where the fusion classification value is less than the predetermined threshold, obtaining a face classification value based on the face feature by using the face classification model, and obtaining a speech classification value based on the speech feature by using the speech classification model; and determining the sentiment recognition result of the target object based on the fusion classification value, the face classification value and the speech classification value, wherein the face classification model is trained by using a plurality of sets of face classification data through machine learning, each set of the face classification data comprises the face feature and a labeled face classification value, and the speech classification model is trained by using a plurality of sets of speech classification data through machine learning, each set of the speech classification data comprises the speech feature and a labeled speech classification value.

[0040] Optionally, in a case where the fusion classification value is not less than the predetermined threshold, the sentiment of the target object is closer to the sentiment classification corresponding to the fusion classification value, and thus the sentiment classification can be used as the sentiment recognition result of the target object, so as to obtain an accurate sentiment recognition result.

[0041] Optionally, in a case where the fusion classification value is less than the predetermined threshold, it is indicated that the sentiment of the target object has a low possibility to belong to the sentiment classification corresponding to the fusion classification value, and thus in a case where it is necessary to determine the sentiment recognition result of the target object, a decision fusion manner is used to determine the sentiment recognition result of the target object based on the face classification value of the face recognition result, the speech classification value of the speech recognition result and the fusion classification value, so as to obtain an accurate sentiment recognition result.

[0042] Optionally, the face classification model is used to determine the face classification value corresponding to the face feature.

[0043] Optionally, the face classification value is used to represent a probability that the face feature belongs to a certain sentiment classification, wherein the face classification value has a corresponding sentiment classification, for example, the face classification value can represent a probability that the target object belongs to a positive sentiment.

[0044] Optionally, the speech classification model is used to determine the speech classification value corresponding to the speech feature.

[0045] Optionally, the speech classification value is used to represent a probability that the speech feature belongs to a certain sentiment classification, wherein the speech classification value has a corresponding sentiment classification, for example, the speech classification value can represent a probability that the target object belongs to a positive sentiment.

[0046] As an optional embodiment, the determining the emotion recognition result of the target object based on the fusion classification value, the face classification value and the speech classification value comprises: determining the emotion recognition result according to the product of the fusion classification value, the face classification value and the speech classification value.

[0047] Optionally, the expression for determining the emotion recognition result of the target object based on the fusion classification value, the face classification value and the speech classification value is: p = p f *p v *p m Wherein, P is the emotion recognition result, used to represent the probability that the target object belongs to the target emotion classification; p f is the face classification value, used to represent the probability that the face feature belongs to the target emotion classification; p v is the speech classification value, used to represent the probability that the speech feature belongs to the target emotion classification, and p m is the fusion classification value, used to represent the probability that the fusion feature belongs to the target emotion classification.

[0048] As an optional embodiment, the obtaining the to-be-recognized face image and the to-be-recognized speech information of the target object comprises: obtaining a to-be-recognized video of the target object; decomposing the to-be-recognized video into an image frame sequence and a speech signal; preprocessing the image frame sequence by using a face detection algorithm to obtain the to-be-recognized face image; and preprocessing the speech signal by using a mel-frequency cepstral coefficient to obtain the to-be-recognized speech information.

[0049] In the above embodiments of the present application, the to-be-recognized video of the target object comprises: a face image of the target object and a speech signal of the target object, the to-be-recognized video can be decomposed into an image frame sequence and a speech signal by decomposing the to-be-recognized video, the to-be-recognized face image is obtained from the image frame sequence by preprocessing the image frame sequence by using a face detection algorithm, and the to-be-recognized speech information is obtained by preprocessing the speech signal by using a mel-frequency cepstral coefficient, so that the to-be-recognized face image and the to-be-recognized speech information of the target object are obtained.

[0050] Optionally, the speech signal can be kept in an MP3 format.

[0051] As an optional embodiment, the preprocessing the image frame sequence by using a face detection algorithm to obtain the to-be-recognized face image comprises: inputting the image frame sequence into a preset face recognition model to identify the face regions of each image in the image frame sequence, wherein the preset face recognition model is trained by machine learning using a plurality of sets of data, each set of data in the plurality of sets of data comprises: a face image and a marked face region in the face image; and the face region is cropped from each image in the image frame sequence as the to-be-recognized face image.

[0052] It should be noted that the face image is an image including a face, and the face region is used to represent the size of the face part in the image, highlighting the face features. For example, the face region can be a rectangular region in the image where the face is located.

[0053] Optionally, the image frame sequence is preprocessed using a face detection algorithm to obtain the face image to be recognized, including: using a face detection algorithm to locate the face of the image frame sequence, and uniformly processing the face image into a size of M x M pixels through cropping, scaling and the like.

[0054] As an optional embodiment, the speech signal is preprocessed using the Mel frequency cepstrum coefficient to obtain the speech information to be recognized, including: pre-emphasizing, framing and windowing the speech signal to obtain at least one analysis window; obtaining the frequency spectrum of each analysis window through fast Fourier transform; performing Mel filtering on the frequency spectrum to obtain the Mel spectrum; performing cepstrum analysis on the Mel spectrum to obtain the Mel spectrum coefficient; and determining the Mel spectrum coefficient as the speech information to be recognized.

[0055] In the above embodiments of the present application, the speech features in the speech signal are extracted using the Mel frequency cepstrum coefficient MFCC, including: pre-emphasizing, framing and windowing the speech signal; obtaining the frequency spectrum of each analysis window through fast Fourier transform FFT; obtaining the Mel spectrum by Mel filtering on the above frequency spectrum; and performing cepstrum analysis on the Mel spectrum to obtain the Mel spectrum coefficient, which can be used as the MFCC feature, i.e., the speech information to be recognized.

[0056] As an optional embodiment, the face features in the face image to be recognized and the speech features in the speech information to be recognized are determined, including: extracting the face features of the face image to be recognized using a face feature extraction model, wherein the loss function for training the face feature extraction model is determined based on a soft maximum loss function softmax loss and a center loss function center loss, and the proportion of the soft maximum loss function softmax loss and the center loss function center loss is controlled by a preset hyperparameter; and extracting the speech features of the speech information to be recognized using a speech feature extraction model, wherein the loss function for training the speech feature extraction model is a cross-entropy loss function.

[0057] Optionally, the loss function for training the face feature extraction model is determined based on a soft maximum loss function softmax loss and a center loss function center loss, and the expression is:

[0058] Center loss function center loss

[0059]

[0060] Soft maximum loss function softmax loss

[0061]

[0062] L = L C + λL s ;

[0063] where λ is a hyper-parameter, used to control the proportion of the soft-max loss function softmax loss and the center loss function center loss, x i is the feature extracted from the i-th sample, c yi represents the average feature of all sample features corresponding to the class of sample i, or the center point of the y i -th class sample feature, and m represents the number of samples.

[0064] It should be noted that in the context of machine learning, a hyper-parameter is a parameter whose value is set before the learning process begins, rather than a parameter data obtained by training. Generally, the hyper-parameters need to be optimized to select a set of optimal hyper-parameters for the learning machine, so as to improve the performance and effect of learning.

[0065] Optionally, the loss function for training the speech feature extraction model is a cross-entropy loss function, and the expression is:

[0066]

[0067] where m is the number of classes, y ic ∈ [0, 1], and if the real class of sample i is equal to c, it takes 1, otherwise it takes 0, p ic is the predicted probability that the observed sample i belongs to the class c.

[0068] As an optional embodiment, the face feature and the speech feature are fused to obtain the fusion feature, including: obtaining a first output feature vector of a last full connection layer of the face feature extraction model; obtaining a second output feature vector of a last full connection layer of the speech feature extraction model; dimensionally reducing the first output feature vector and the second output feature vector; and calculating the dimensionally reduced first output feature vector and the dimensionally reduced second output feature vector by using a convolution operation to determine the fusion feature.

[0069] Optionally, the face feature and the speech feature are fused to obtain the fusion feature, which is expressed by the following formula:

[0070]

[0071] wherein, represents a sketching / flow algorithm count sketch dimensionality reduction operation, h and s are parameters of the operation, h is used to randomly select elements in a feature vector, s is used to randomly select whether to make the element negative, * represents a convolution operation.

[0072] The emotion recognition method provided in the embodiments of the present application comprises the following steps: obtaining a to-be-recognized face image and to-be-recognized voice information of a target object; determining a face feature in the to-be-recognized face image and a voice feature in the to-be-recognized voice information; fusing the face feature and the voice feature to obtain a fused feature; analyzing the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, each set of fused classification data in the plurality of sets of fused classification data comprises the fused feature and a calibrated fused classification value; and determining an emotion recognition result of the target object based on the fused classification value. The emotion recognition result is inaccurate in the related art, and the emotion recognition result is determined based on the fused classification value of the fused feature, so that the emotion recognition result is determined based on the quantized value after feature fusion, and the accuracy of the emotion recognition result is improved.

[0073] The present application also provides a preferred embodiment, which provides a neural network-based speech and facial expression bimodal emotion fusion recognition method.

[0074] Figure 2 is a schematic diagram of a neural network-based speech and facial expression bimodal emotion fusion recognition method according to the embodiments of the present application, as shown in Figure 2 The steps are as follows:

[0075] Step S1, extracting audio and image frames from the to-be-recognized video:

[0076] Optionally, the to-be-recognized video is decomposed into an image frame sequence, and the voice signal in the video is extracted and kept in MP3 format.

[0077] Step S2, face image preprocessing.

[0078] Optionally, the image frame sequence obtained in step S1 is positioned by using a face detection algorithm Viola&Jones, and the face image is uniformly processed to have a size of M x M pixels through cropping, scaling and the like.

[0079] Step S3, voice signal preprocessing.

[0080] Optionally, the speech features of the speech signal in step S1 are extracted by using Mel frequency cepstral coefficients (MFCCs), including: pre-emphasizing, framing and windowing the speech signal; obtaining the spectrum of each analysis window by fast Fourier transform (FFT); obtaining Mel spectrum by Mel filtering the spectrum; and performing cepstrum analysis on the Mel spectrum to obtain Mel spectrum coefficients which can be used as the MFCC features, i.e., the to-be-recognized speech information.

[0081] Step S4: feature extraction of the face image.

[0082] Optionally, the face features of the face image obtained in step S2 are extracted by using a residual network (ResNet).

[0083] Optionally, the loss function used by the residual network (ResNet) is a soft-max loss function (softmax loss) combined with a center loss function (center loss), and the expression is as follows:

[0084] L = L c + λL s , where L c is the center loss function (center loss), L s is the soft-max loss function (softmax loss), and λ is a hyperparameter used to control the proportion of the two.

[0085]

[0086] where x i is the feature extracted from the i-th sample, c yi represents the average feature of all sample features corresponding to the class of the sample i, or the center point of the y i -th class sample feature, and m represents the number of samples.

[0087] Step S5: feature extraction of the speech signal.

[0088] Optionally, the MFCC features (i.e., the to-be-recognized speech information) obtained in step S3 are further extracted by using a general one-dimensional convolutional neural network.

[0089] Optionally, the loss function used in the model training is a cross-entropy loss function (Cross Entropy Loss Function), and the expression is as follows:

[0090]

[0091] where m is the number of classes, y ic ∈ [0, 1], and if the real class of the sample i is equal to c, it takes 1, otherwise it takes 0, and p ic is the predicted probability of the observation sample i belonging to the class c.

[0092] Step S6, feature fusion.

[0093] Optionally, the output feature vector x of the last fully connected layer (before the activation function) of the face feature extraction model in step S4 and the output feature vector y of the last fully connected layer (before the activation function) of the speech feature extraction model in step S5 are reduced in dimension using a sketching algorithm, and then the two feature vectors are calculated using convolution operation to obtain the fused feature, which is expressed as follows:

[0094]

[0095] wherein, represents the count sketch dimension reduction operation, h and s are parameters of the operation, h is used to randomly select elements in the feature vector, s is used to randomly select whether to change the element to negative, and * represents the convolution operation.

[0096] Optionally, the fused feature is input into a long short term memory network (LSTM) for further learning, and then the classification result of the feature fusion classification model is obtained.

[0097] Step S6, the fusion of the face classification model, the speech classification model and the feature fusion classification model in decision making obtains the final emotion recognition result of the object to be recognized.

[0098] Optionally, in the case where the classification result (i.e. the fusion classification value) obtained by the feature fusion classification model in step S6 is greater than or equal to a predetermined threshold a, the classification result of the feature fusion classification model is directly used as the final recognition result.

[0099] Step S41, using the face classification model to analyze the face feature in step S4, obtaining the result p f (i.e. face classification value) of the face classification model.

[0100] Step S51, using the speech classification model to analyze the speech feature in step S5, obtaining the result p v (i.e. speech classification value) of the speech classification model.

[0101] Optionally, in the case where the classification result (i.e. the fusion classification value) obtained in step S6 is less than a predetermined threshold a, the results p f (i.e. face classification value) of the face classification model in step S41, the result p v (i.e. speech classification value) of the speech classification model in step S51 and the classification result p m(fusion classification value), and finally, the recognition result is obtained, and the expression of fusion is as follows: p=p f *p v *p m .

[0102] Optionally, the final recognition expression of the model is as follows:

[0103] The application uses the convolutional neural network (CNN) which is very successful in the field of image application to extract voice and facial expression features, so that the influence of subjective factors in manual feature selection can be avoided; the model fusion is performed by using the feature fusion and decision fusion, so that the internal relationship between the face features and the voice features can be effectively utilized, and the accuracy and reliability of emotion recognition can be improved.

[0104] The application overcomes the shortcomings of the existing dual-modal emotion recognition technology, and provides a method which is simple to realize, has strong migration, and can effectively utilize the internal relationship between different channel features.

[0105] The application provides a neural network-based voice and facial expression dual-modal emotion fusion recognition method, which uses the neural network to extract voice and facial expression features, so that the manual feature selection method can be improved; finally, the model fusion is performed by using the combination of the feature fusion and the decision fusion.

[0106] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0107] The application embodiment further provides an emotion recognition device, and it should be noted that the emotion recognition device of the application embodiment can be used to execute the emotion recognition method provided by the application embodiment. The emotion recognition device provided by the application embodiment is introduced as follows.

[0108] Figure 3 is a schematic diagram of an emotion recognition device according to the application embodiment. As shown in Figure 3As shown, the apparatus comprises: an acquisition unit 31 configured to acquire a to-be-identified face image and to-be-identified voice information of a target object; a first determination unit 33 configured to determine a face feature in the to-be-identified face image and a voice feature in the to-be-identified voice information; a fusion unit 35 configured to fuse the face feature and the voice feature to obtain a fused feature; an analysis unit 37 configured to analyze the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, and each set of the fused classification data comprises a fused feature and a labeled fused classification value; and a second determination unit 39 configured to determine a sentiment recognition result of the target object based on the fused classification value.

[0109] It should be noted that the acquisition unit 31 in this embodiment can be configured to perform step S102 in the embodiments of the present application, the first determination unit 33 in this embodiment can be configured to perform step S104 in the embodiments of the present application, the fusion unit 35 in this embodiment can be configured to perform step S106 in the embodiments of the present application, the analysis unit 37 in this embodiment can be configured to perform step S108 in the embodiments of the present application, and the second determination unit 39 in this embodiment can be configured to perform step S110 in the embodiments of the present application. The above units and the examples and application scenarios realized by the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0110] As an optional embodiment, the second determination unit comprises: a judgment module configured to judge whether the fused classification value is not less than a predetermined threshold; a first determination module configured to, in a case where the fused classification value is not less than the predetermined threshold, determine the sentiment recognition result of the target object based on the fused classification value; and a second determination module configured to, in a case where the fused classification value is less than the predetermined threshold, analyze the face feature based on a face classification model to obtain a face classification value, analyze the voice feature based on a voice classification model to obtain a voice classification value, and determine the sentiment recognition result of the target object based on the fused classification value, the face classification value and the voice classification value, wherein the face classification model is trained by a plurality of sets of face classification data through machine learning, and each set of the face classification data comprises a face feature and a labeled face classification value, and the voice classification model is trained by a plurality of sets of voice classification data through machine learning, and each set of the voice classification data comprises a voice feature and a labeled voice classification value.

[0111] As an optional embodiment, the acquisition unit comprises: an acquisition module, configured to acquire a to-be-identified video of a target object; a decomposition module, configured to decompose the to-be-identified video into an image frame sequence and a speech signal; a first preprocessing module, configured to preprocess the image frame sequence by using a face detection algorithm to obtain a to-be-identified face image; and a second preprocessing module, configured to preprocess the speech signal by using a mel-frequency cepstral coefficient to obtain to-be-identified speech information.

[0112] As an optional embodiment, the first preprocessing module comprises: an identification module, configured to input the image frame sequence into a preset face recognition model to identify a face region of each image in the image frame sequence, wherein the preset face recognition model is trained by machine learning using a plurality of sets of data, and each set of data in the plurality of sets of data comprises a face image and a labeled face region in the face image; and a clipping module, configured to clip the face region in each image of the image frame sequence as the to-be-identified face image.

[0113] As an optional embodiment, the second preprocessing module comprises: a first processing module, configured to pre-emphasize, frame and window the speech signal to obtain at least one analysis window; a second processing module, configured to obtain a frequency spectrum of each analysis window by using a fast Fourier transform; a third processing module, configured to perform mel filtering on the frequency spectrum to obtain a mel spectrum; an analysis module, configured to perform cepstrum analysis on the mel spectrum to obtain a mel spectrum coefficient; and a first determination module, configured to determine the mel spectrum coefficient as the to-be-identified speech information.

[0114] As an optional embodiment, the first determination unit comprises: a first extraction module, configured to extract a face feature of the to-be-identified face image by using a face feature extraction model, wherein a loss function for training the face feature extraction model is determined based on a soft-max loss function softmax loss and a center loss function center loss, and a proportion of the soft-max loss function softmax loss and the center loss function center loss is controlled by a preset hyperparameter; and a second extraction module, configured to extract a speech feature of the to-be-identified speech information by using a speech feature extraction model, wherein a loss function for training the speech feature extraction model is a cross-entropy loss function.

[0115] As an optional embodiment, the fusion unit comprises: a first acquisition module, configured to acquire a first output feature vector of a last full connection layer of the face feature extraction model; a second acquisition module, configured to acquire a second output feature vector of a last full connection layer of the speech feature extraction model; a dimension reduction module, configured to reduce dimensions of the first output feature vector and the second output feature vector; and a second determination module, configured to calculate the first output feature vector and the second output feature vector after dimension reduction by using a convolution operation to determine a fusion feature.

[0116] The emotional recognition device provided in the embodiment of the present application obtains a to-be-recognized face image and to-be-recognized voice information of a target object, determines a face feature in the to-be-recognized face image and a voice feature in the to-be-recognized voice information, fuses the face feature and the voice feature to obtain a fused feature, analyzes the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, each set of fused classification data in the plurality of sets of fused classification data includes the fused feature and a calibrated fused classification value, and determines an emotional recognition result of the target object based on the fused classification value, thereby solving the problem of inaccurate emotional recognition result in the related art, determining the emotional recognition result based on the fused classification value of the fused feature, realizing the determination of the emotional recognition result based on the quantized value after feature fusion, and achieving the effect of improving the accuracy of the emotional recognition result.

[0117] The emotional recognition device includes a processor and a memory, each of the units and modules is stored in the memory as a program unit, and the corresponding function is realized by the processor executing the program unit stored in the memory.

[0118] The processor includes a core, and the core retrieves the corresponding program unit from the memory. The core can be set to one or more, and the emotional recognition result based on the fused classification value of the fused feature is determined by adjusting the core parameters.

[0119] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0120] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the emotional recognition method.

[0121] The embodiment of the present application provides a processor, which is used to run a program, wherein the program is executed to perform the emotional recognition method.

[0122] Figure 4 FIG. 1 is a schematic diagram of an electronic device according to the embodiment of the present application. As shown in FIG. 1, the electronic device includes a processor 101, a memory 102, a communication interface 103, a display 104, and a microphone 105. Figure 4As shown, the embodiment of the present application provides an electronic device 40, the device comprising a processor 403, a memory 401, and a program stored on the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining a to-be-identified face image and to-be-identified voice information of a target object; determining a face feature in the to-be-identified face image and a voice feature in the to-be-identified voice information; fusing the face feature and the voice feature to obtain a fused feature; analyzing the fused feature based on a feature fusion classification model to obtain a fused classification value, wherein the feature fusion classification model is trained by a plurality of sets of fused classification data through machine learning, and each set of fused classification data in the plurality of sets of fused classification data comprises the fused feature and a calibrated fused classification value; and determining a sentiment recognition result of the target object based on the fused classification value.

[0123] Optionally, the processor implements the following steps when executing the program: determining whether the fused classification value is not less than a predetermined threshold value; in a case where the fused classification value is not less than the predetermined threshold value, determining the sentiment recognition result of the target object based on the fused classification value; in a case where the fused classification value is less than the predetermined threshold value, obtaining a face classification value based on the face feature by analyzing the face feature based on a face classification model, and obtaining a voice classification value based on the voice feature by analyzing the voice feature based on a voice classification model; and determining the sentiment recognition result of the target object based on the fused classification value, the face classification value, and the voice classification value, wherein the face classification model is trained by a plurality of sets of face classification data through machine learning, and each set of face classification data in the plurality of sets of face classification data comprises the face feature and a calibrated face classification value, and the voice classification model is trained by a plurality of sets of voice classification data through machine learning, and each set of voice classification data in the plurality of sets of voice classification data comprises the voice feature and a calibrated voice classification value.

[0124] Optionally, the processor implements the following steps when executing the program: obtaining a to-be-identified video of the target object; decomposing the to-be-identified video into an image frame sequence and a voice signal; preprocessing the image frame sequence by using a face detection algorithm to obtain the to-be-identified face image; and preprocessing the voice signal by using a mel-frequency cepstral coefficient to obtain the to-be-identified voice information.

[0125] Optionally, the processor implements the following steps when executing the program: inputting the image frame sequence into a preset face recognition model to identify a face region of each image in the image frame sequence, wherein the preset face recognition model is trained by a plurality of sets of data through machine learning, and each set of data in the plurality of sets of data comprises a face image and a calibrated face region in the face image; and cropping the face region in each image of the image frame sequence as the to-be-identified face image.

[0126] Optionally, the processor implements the following steps when executing the program: pre-emphasizing, framing and windowing the speech signal to obtain at least one analysis window; obtaining the spectrum of each analysis window through fast Fourier transform; performing Mel filtering on the spectrum to obtain a Mel spectrum; performing cepstrum analysis on the Mel spectrum to obtain a Mel spectrum coefficient; and determining the Mel spectrum coefficient as the speech information to be recognized.

[0127] Optionally, the processor implements the following steps when executing the program: extracting the facial features of the face image to be recognized by using a facial feature extraction model, wherein the loss function for training the facial feature extraction model is determined based on a soft-max loss function softmax loss and a center loss function center loss, and the proportion of the soft-max loss function softmax loss and the center loss function center loss is controlled by a preset hyperparameter; and extracting the speech features of the speech information to be recognized by using a speech feature extraction model, wherein the loss function for training the speech feature extraction model is a cross-entropy loss function.

[0128] Optionally, the processor implements the following steps when executing the program: obtaining a first output feature vector of a last full connection layer of the facial feature extraction model; obtaining a second output feature vector of a last full connection layer of the speech feature extraction model; performing dimension reduction on the first output feature vector and the second output feature vector; and determining the fusion features by performing convolution operation on the first output feature vector and the second output feature vector after dimension reduction.

[0129] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0130] The application further provides a computer program product adapted to execute the program of the following method steps when executed on a data processing device: obtaining a face image to be recognized and speech information to be recognized of a target object; determining facial features in the face image to be recognized and speech features in the speech information to be recognized; fusing the facial features and the speech features to obtain fusion features; analyzing the fusion features based on a feature fusion classification model to obtain a fusion classification value, wherein the feature fusion classification model is trained by machine learning from a plurality of sets of fusion classification data, each set of fusion classification data in the plurality of sets of fusion classification data comprising: fusion features and a calibrated fusion classification value; and determining a sentiment recognition result of the target object based on the fusion classification value.

[0131] Optionally, when executed on the data processing device, the program is adapted to execute the following method steps: determining whether the fusion classification value is not less than a predetermined threshold; in a case where the fusion classification value is not less than the predetermined threshold, determining the emotional recognition result of the target object based on the fusion classification value; in a case where the fusion classification value is less than the predetermined threshold, obtaining a face classification value based on analysis of the face feature by a face classification model, and obtaining a speech classification value based on analysis of the speech feature by a speech classification model; and determining the emotional recognition result of the target object based on the fusion classification value, the face classification value, and the speech classification value, wherein the face classification model is trained by a plurality of sets of face classification data through machine learning, each set of the face classification data including the face feature and a labeled face classification value, and the speech classification model is trained by a plurality of sets of speech classification data through machine learning, each set of the speech classification data including the speech feature and a labeled speech classification value.

[0132] Optionally, when executed on the data processing device, the program is adapted to execute the following method steps: obtaining a to-be-recognized video of a target object; decomposing the to-be-recognized video into an image frame sequence and a speech signal; pre-processing the image frame sequence by a face detection algorithm to obtain a to-be-recognized face image; and pre-processing the speech signal by a mel-frequency cepstral coefficient to obtain to-be-recognized speech information.

[0133] Optionally, when executed on the data processing device, the program is adapted to execute the following method steps: inputting the image frame sequence into a preset face recognition model to recognize a face region of each image in the image frame sequence, wherein the preset face recognition model is trained by a plurality of sets of data through machine learning, each set of the data including a face image and a labeled face region in the face image; and cropping the face region from each image in the image frame sequence as the to-be-recognized face image.

[0134] Optionally, when executed on the data processing device, the program is adapted to execute the following method steps: pre-emphasizing, framing, and windowing the speech signal to obtain at least one analysis window; obtaining a frequency spectrum of each analysis window through fast Fourier transform; performing mel filtering on the frequency spectrum to obtain a mel frequency spectrum; performing cepstral analysis on the mel frequency spectrum to obtain a mel frequency spectrum coefficient; and determining the mel frequency spectrum coefficient as the to-be-recognized speech information.

[0135] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: extracting, by using a face feature extraction model, face features of the to-be-identified face image, wherein a loss function for training the face feature extraction model is determined based on a soft-max loss function and a center loss function, and a proportion of the soft-max loss function and the center loss function is controlled by a preset hyperparameter; and extracting, by using a speech feature extraction model, speech features of the to-be-identified speech information, wherein a loss function for training the speech feature extraction model is a cross-entropy loss function.

[0136] Optionally, when executed on a data processing device, the program is adapted to perform the following method steps: obtaining a first output feature vector of a last full connection layer of the face feature extraction model; obtaining a second output feature vector of a last full connection layer of the speech feature extraction model; performing dimension reduction on the first output feature vector and the second output feature vector; and determining the fusion features by performing convolution operation on the dimension-reduced first output feature vector and the dimension-reduced second output feature vector.

[0137] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0138] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0139] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or blocks Figure 1 the function(s) specified in the block or blocks.

[0140] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more processes and / or blocks Figure 1 the function(s) specified in the block or blocks.

[0141] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0142] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information such as computer program instructions. Memory is an example of computer readable media.

[0143] Computer readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memories (CD-ROMs), digital versatile discs (DVDs), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store information that can be accessed by a computing device. According to the definition provided herein, computer readable media excludes transitory computer readable media, such as modulated data signals and carrier waves.

[0144] It is also important to note that the terms "comprises", "comprising", or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without further constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0145] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code thereon for use by a computer or computer system.

[0146] The foregoing is merely illustrative of the embodiments of the present application and is not used to limit the present application. The present application can have various modifications and alternative forms. Any addition, substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. An emotion recognition method, characterized in that, include: Acquire the facial image and speech information of the target object to be identified; Determine the facial features in the face image to be identified and the speech features in the speech information to be identified; The facial features and the voice features are fused to obtain fused features; The fusion features are analyzed based on the feature fusion classification model to obtain fusion classification values. The feature fusion classification model is trained by machine learning from multiple sets of fusion classification data. Each set of fusion classification data includes: fusion features and labeled fusion classification values. The emotion recognition result of the target object is determined based on the fusion classification value; The emotion recognition result of the target object determined based on the fusion classification value includes: Determine whether the fused classification value is not less than a predetermined threshold; If the fusion classification value is not less than a predetermined threshold, the emotion recognition result of the target object is determined based on the fusion classification value; If the fused classification value is less than a predetermined threshold, a face classification value is obtained by analyzing the face features based on a face classification model, and a voice classification value is obtained by analyzing the voice features based on a voice classification model. Based on the fused classification value, the face classification value, and the voice classification value, the emotion recognition result of the target object is determined. The face classification model is trained by machine learning from multiple sets of face classification data, and each set of face classification data includes face features and a labeled face classification value. Similarly, the voice classification model is trained by machine learning from multiple sets of voice classification data, and each set of voice classification data includes voice features and a labeled voice classification value.

2. The method according to claim 1, characterized in that, Obtaining the target object's facial image and speech information includes: Obtain the video of the target object to be identified; The video to be identified is decomposed into a sequence of image frames and an audio signal; The image frame sequence is preprocessed using a face detection algorithm to obtain the face image to be identified; The speech signal is preprocessed using Mel-frequency cepstral coefficients to obtain the speech information to be recognized.

3. The method according to claim 2, characterized in that, The image frame sequence is preprocessed using a face detection algorithm to obtain the face image to be identified, including: The image frame sequence is input into a preset face recognition model to identify the face region of each image in the image frame sequence. The preset face recognition model is trained by machine learning using multiple sets of data. Each set of data includes a face image and a labeled face region in the face image. In each image of the image frame sequence, the face region is cropped out as the face image to be identified.

4. The method according to claim 2, characterized in that, The speech signal is preprocessed using Mel-frequency cepstral coefficients to obtain the speech information to be recognized, including: The speech signal is pre-emphasized, framed, and windowed to obtain at least one analysis window; The spectrum of each analysis window is obtained by fast Fourier transform; Mel filtering is performed on the spectrum to obtain the Mel spectrum; The Mel frequency cepstral coefficients are obtained by performing cepstral analysis on the Mel spectrum. The Mel frequency cepstral coefficients are determined to be the speech information to be recognized.

5. The method according to claim 1, characterized in that, Determining the facial features in the face image to be identified and the speech features in the speech information to be identified includes: The facial features of the face image to be identified are extracted using a facial feature extraction model. The loss function for training the facial feature extraction model is determined based on the softmax loss and the center loss. The weight of the softmax loss and the center loss is controlled by a preset hyperparameter. The speech features of the speech information to be identified are extracted using a speech feature extraction model, wherein the loss function for training the speech feature extraction model is the cross-entropy loss function.

6. The method according to claim 5, characterized in that, The facial features and the voice features are fused to obtain fused features, including: Obtain the first output feature vector of the last fully connected layer of the face feature extraction model; Obtain the second output feature vector of the last fully connected layer of the speech feature extraction model; Dimensionality reduction is performed on the first output feature vector and the second output feature vector; The fused features are determined by calculating the first and second output feature vectors after dimensionality reduction using convolution operations.

7. An emotion recognition device, characterized in that, include: The acquisition unit is used to acquire the face image and voice information of the target object to be recognized. The first determining unit is used to determine the facial features in the face image to be identified and the speech features in the speech information to be identified; The fusion unit is used to fuse the facial features and the voice features to obtain fused features; The analysis unit is used to analyze the fusion features based on the feature fusion classification model to obtain fusion classification values. The feature fusion classification model is trained by machine learning from multiple sets of fusion classification data. Each set of fusion classification data includes: fusion features and labeled fusion classification values. The second determining unit is used to determine the emotion recognition result of the target object based on the fusion classification value; The second determining unit includes: The judgment module is used to determine whether the fused classification value is not less than a predetermined threshold. If the fusion classification value is not less than a predetermined threshold, the emotion recognition result of the target object is determined based on the fusion classification value; The first determining module is configured to, when the fused classification value is less than a predetermined threshold, analyze the facial features based on a facial classification model to obtain a facial classification value, analyze the speech features based on a speech classification model to obtain a speech classification value, and determine the emotion recognition result of the target object based on the fused classification value, the facial classification value, and the speech classification value. The facial classification model is trained from multiple sets of facial classification data using machine learning, and each set of facial classification data includes facial features and a labeled facial classification value. The speech classification model is trained from multiple sets of speech classification data using machine learning, and each set of speech classification data includes speech features and a labeled speech classification value.

8. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the emotion recognition method according to any one of claims 1 to 6.

Images