Emotion recognition method and device and storage medium

Abstract

The invention relates to an emotion recognition method and device and a storage medium. The emotion recognition method comprises the steps of obtaining target data, wherein the total number of the target data is at least two; determining a plurality of first emotion probability results based on the target data; fusing the plurality of first emotion probability results to obtain a second emotion probability result; and based on the second emotion probability result, determining an identified emotion. High computing power of the terminal is not needed, power consumption is saved, and user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of emotion recognition, and in particular to an emotion recognition method, device and storage medium. Background Art

[0002] Currently, the terminal recognizes the current emotion of the terminal user by taking images or collecting audio. However, due to the influence of image quality and audio quality, the result of emotion recognition is not accurate. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides an emotion recognition method, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an emotion recognition method, comprising: acquiring target data, wherein the total number of the target data is at least two; determining a plurality of first emotion probability results based on the target data; fusing the plurality of first emotion probability results to obtain a second emotion probability result; and determining a recognized emotion based on the second emotion probability result.

[0005] In one embodiment, determining a plurality of first emotion probability results based on the target data includes: inputting the target data into a neural network model to obtain a plurality of first emotion probability results.

[0006] In one embodiment, the target data includes an image, and the step of inputting the target data into a neural network model to obtain a plurality of first emotion probability results includes: in response to the image including facial features, inputting the image into a first neural network model to obtain a plurality of first emotion probability results.

[0007] In one embodiment, the target data includes an image and audio, and inputting the target data into a neural network model to obtain a plurality of first emotion probability results includes: in response to the image not including facial features, inputting the audio into a second neural network model to obtain a plurality of first emotion probability results.

[0008] In one embodiment, the target data includes images and audios, and the target data is input into a neural network model to obtain multiple first emotion probability results, including: in response to the image including facial features, the image is input into the first neural network model to obtain M numerical values ​​output by the intermediate layer of the first neural network model corresponding to each image; the audio is input into the second neural network model to obtain N numerical values ​​output by the intermediate layer of the second neural network model corresponding to each audio; the M numerical values ​​corresponding to the first image and the N numerical values ​​corresponding to the first audio are input into a third neural network model to obtain a first emotion probability result, the first audio represents any one of the audios, the first image represents any one of the images, and the position of the first audio in the audio after sorting by acquisition time is the same as the position of the first image in the image after sorting by acquisition time.

[0009] In one embodiment, the target data includes audio, the first emotion probability result includes the probability of a first emotion and the probability of a second emotion, and determining the recognized emotion based on the second emotion probability result includes: in response to the sum of the probability of the first emotion and the probability of the second emotion being greater than or equal to a first threshold, and the probability of the first emotion and the probability of the second emotion being less than or equal to the first threshold, extracting audio features of the audio; inputting the audio features into a fourth neural network model to obtain an output result, the output result including the emotion recognized as the first emotion or the emotion recognized as the second emotion.

[0010] In one implementation, the audio feature includes at least one of the following: a feature related to the time domain; a feature related to the frequency domain; a feature related to Mel-frequency cepstral coefficients (MFCC).

[0011] In one embodiment, after determining the recognized emotion, the method further includes: in response to the recognized emotion being different from the most recently recorded emotion, recording the recognized emotion and time information and location information associated with the recognized emotion.

[0012] In one embodiment, the first neural network model is trained in the following manner: a first image data set and a second image data set are collected, the image quality in the second image data set is lower than the image quality in the first image data set; the first image data set is used to train a model to be trained; and the second image data set is used to adjust the parameters of the model to be trained to obtain the first neural network model.

[0013] In one embodiment, the second neural network model is trained in the following manner: a first audio data set and a second audio data set are collected, the audio quality in the second audio data set is lower than the audio quality in the first audio data set; the first audio data set is used to train the model to be trained; and the second audio data set is used to adjust the parameters of the model to be trained to obtain the second neural network model.

[0014] According to a second aspect of an embodiment of the present disclosure, an emotion recognition device is provided, including: an acquisition unit, used to acquire target data, the total number of the target data being at least two; a determination unit, used to determine a plurality of first emotion probability results based on the target data; a processing unit, used to fuse the plurality of first emotion probability results to obtain a second emotion probability result; the determination unit is further used to determine the recognized emotion based on the second emotion probability result.

[0015] In one embodiment, the determination unit determines a plurality of first emotion probability results based on the target data in the following manner: inputting the target data into a neural network model to obtain a plurality of first emotion probability results.

[0016] In one embodiment, the target data includes an image, and the determination unit inputs the target data into a neural network model in the following manner to obtain multiple first emotion probability results: in response to the image including facial features, the image is input into a first neural network model to obtain multiple first emotion probability results.

[0017] In one embodiment, the target data includes images and audio, and the determination unit inputs the target data into a neural network model in the following manner to obtain multiple first emotion probability results: in response to the image not including facial features, the audio is input into a second neural network model to obtain multiple first emotion probability results.

[0018] In one embodiment, the target data includes images and audios, and the determination unit inputs the target data into a neural network model in the following manner to obtain multiple first emotion probability results: in response to the image including facial features, the image is input into the first neural network model to obtain M numerical values ​​output by the intermediate layer of the first neural network model corresponding to each image; the audio is input into the second neural network model to obtain N numerical values ​​output by the intermediate layer of the second neural network model corresponding to each audio; the M numerical values ​​corresponding to the first image and the N numerical values ​​corresponding to the first audio are input into a third neural network model to obtain a first emotion probability result, the first audio represents any one of the audios, the first image represents any one of the images, and the position of the first audio in the audio after sorting by acquisition time is the same as the position of the first image in the image after sorting by acquisition time.

[0019] In one embodiment, the target data includes audio, the first emotion probability result includes the probability of a first emotion and the probability of a second emotion, and the determination unit determines the recognized emotion based on the second emotion probability result in the following manner: in response to the sum of the probability of the first emotion and the probability of the second emotion being greater than or equal to a first threshold, and the probability of the first emotion and the probability of the second emotion being less than or equal to the first threshold, extracting audio features of the audio; inputting the audio features into a fourth neural network model to obtain an output result, the output result including the emotion recognized as the first emotion or the emotion recognized as the second emotion.

[0020] In one implementation, the audio feature includes at least one of the following: a feature related to the time domain; a feature related to the frequency domain; a feature related to Mel-frequency cepstral coefficients (MFCC).

[0021] In one embodiment, after determining the recognized emotion, the method further includes: in response to the recognized emotion being different from the most recently recorded emotion, recording the recognized emotion and time information and location information associated with the recognized emotion.

[0022] In one embodiment, the processing unit trains the first neural network model in the following manner: collecting a first image data set and a second image data set, the image quality in the second image data set being lower than the image quality in the first image data set; using the first image data set to train a model to be trained; and using the second image data set to adjust the parameters of the model to be trained to obtain the first neural network model.

[0023] In one embodiment, the processing unit trains the second neural network model in the following manner: collecting a first audio data set and a second audio data set, the audio quality in the second audio data set being lower than the audio quality in the first audio data set; using the first audio data set to train the model to be trained; and using the second audio data set to adjust the parameters of the model to be trained to obtain the second neural network model.

[0024] According to the third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory for storing instructions; and a processor for calling the instructions stored in the memory to execute the emotion recognition method in the first aspect and any one of the embodiments of the first aspect or the second aspect and any one of the embodiments of the second aspect.

[0025] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions are executed by a processor, the emotion recognition method in the first aspect or any one of the implementations of the first aspect is executed.

[0026] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by fusing multiple recognition results according to a preset ratio, the fused recognition result can effectively improve the accuracy of emotion recognition, and does not require higher computing power of the terminal, thus saving power consumption and improving user experience.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] Figure 1 The figure is a flow chart of an emotion recognition method according to an exemplary embodiment.

[0030] Figure 2 is a probability distribution diagram of a geometric distribution according to an exemplary embodiment.

[0031] Figure 3 The figure is a flow chart of an emotion recognition method according to an exemplary embodiment.

[0032] Figure 4 The figure is a flow chart of an emotion recognition method according to an exemplary embodiment.

[0033] Figure 5 It is a flow chart of a first neural network model training method according to an exemplary embodiment.

[0034] Figure 6 is a flow chart of a second neural network model training method according to an exemplary embodiment.

[0035] Figure 7 The figure is a block diagram of an emotion recognition device according to an exemplary embodiment.

[0036] Figure 8 The figure is a block diagram of an emotion recognition device according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0038] Currently, the terminal recognizes the current emotion of the terminal user by taking images or collecting audio. For example, the terminal uses an always on camera (AON), which can continuously acquire images in a low-power manner. Such images have low quality. For example, grayscale images, low-pixel images, or infrared images. For another example, the terminal acquires audio through a low-power earpiece, and the quality of such audio is also poor. Affected by the image quality and audio quality, the result of emotion recognition is not accurate.

[0039] In one implementation, the accuracy of emotion recognition can be improved by repairing the image quality or the audio quality. However, repairing the image quality and the audio quality has high requirements on the computing power of the terminal and often results in high power consumption.

[0040] Therefore, the present disclosure provides a method for emotion recognition, which can effectively improve the accuracy of emotion recognition by fusing multiple recognition results according to a preset ratio, and does not require higher computing power of the terminal, saving power consumption and improving user experience.

[0041] Figure 1 is a flow chart of an emotion recognition method according to an exemplary embodiment. Figure 1 As shown, the emotion recognition method includes the following steps.

[0042] In step S11, target data is acquired, and the total number of target data is at least two.

[0043] In one embodiment, target data can be obtained, and different target data can represent different emotions. For example, the target data can be an image. When the image includes a human face, the emotion of the person can be identified based on the facial expressions, such as staring, frowning, etc. For another example, the target data can be audio. When the loudness of the audio is large, or the frequency of the audio is high, or the frequency, loudness, etc. of the audio fluctuate greatly, the corresponding emotion of the person can be identified. However, the present disclosure is only an example, and the target data can also be other data that can represent emotions. Among them, the total number of target data is at least two, which is used to obtain multiple first emotion probability results later. For example, the target data can be at least two images, or at least two audios, or at least one image and at least one audio, etc.

[0044] In step S12, a plurality of first emotion probability results are determined based on the target data.

[0045] In one embodiment, multiple first emotion probability results may be determined based on the target data. A first emotion probability result indicates the probability of different emotions obtained by the terminal predicting the user's emotion. Multiple predictions may be performed to obtain multiple first emotion probability results. For example, emotions may include anger, disgust, fear, happiness, calmness, sadness, surprise, and no emotion, etc., but the present disclosure does not limit this.

[0046] In step S13, multiple first emotion probability results are fused to obtain a second emotion probability result.

[0047] In one embodiment, the probabilities of the same emotion in different first emotion probability results can be fused according to a preset ratio to obtain a second emotion probability result. The preset ratio refers to the ratio of each first emotion probability result to multiple first emotion probability results during the fusion process. For example, each first emotion probability result includes the probabilities of emotion 1, emotion 2, and emotion 3. Assume that the first emotion probability result A includes the probability X1 of emotion 1, the probability Y1 of emotion 2, and the probability Z1 of emotion 3; the second emotion probability result B includes the probability X2 of emotion 1, the probability Y2 of emotion 2, and the probability Z2 of emotion 3; the second emotion probability result C includes the probability X3 of emotion 1, the probability Y3 of emotion 2, and the probability Z3 of emotion 3. Assume that the proportion of the first emotion probability result A is P, the proportion of the first emotion probability result B is Q, and the proportion of the first emotion probability result C is W, P+Q+W=1. The multiple first emotion probability results can be fused according to a preset ratio: X1*P+X2*Q+X3*W=X4, Y1*P+Y2*Q+Y3*W=Y4, Z1*P+Z2*Q+Z3*W=Z4, and the obtained X4 represents the probability of emotion 1 after fusion, Y4 represents the probability of emotion 2 after fusion, and Z4 represents the probability of emotion 3 after fusion. X4, Y4, and Z4 constitute the fused second emotion probability result.

[0048] In one embodiment, the preset ratio can be determined according to the time when the first emotion probability result is determined. For example, the closer the time when the first emotion probability result is determined is to the current time, the greater the ratio of the first emotion probability result to the multiple first emotion probability results. The ratio of each first emotion probability result to the multiple first emotion probability results can be determined according to the probability distribution function. The probability distribution function can be a geometric distribution, a Poisson distribution, etc., which is not limited in the present disclosure. Figure 2 As shown, it is a probability distribution diagram of geometric distribution. Figure 2 The horizontal axis represents the identifier of the first emotion probability result, and the vertical axis represents the proportion of different first emotion probability results to multiple first emotion probability results. For example, the first emotion probability result 0 may be the first emotion probability result obtained in the most recent prediction, which accounts for the largest proportion of multiple first emotion probability results. Figure 2 The number of horizontal axes in the probability distribution diagram shown is only exemplary. The proportion corresponding to any number of first emotion probability results can be determined according to geometric distribution, and the present disclosure does not limit this.

[0049] In step S14, the recognized emotion is determined based on the second emotion probability result.

[0050] In one embodiment, the recognized emotion can be determined according to the maximum probability in the second emotion probability result. Taking the second emotion recognition result including X4, Y4, and Z4 as an example, if X4 is greater than Y4 and Z4, the recognized emotion is emotion 1, and if Y4 is greater than X4 and Z4, the recognized emotion is emotion 2. If Z4 is greater than X4 and Y4, the recognized emotion is emotion 3.

[0051] Of course, A, B, C, X, Y, Z, P, Q, W, etc. involved in this embodiment are only for explanation and do not represent specific values ​​or specific objects.

[0052] The present invention obtains a second emotion probability result by fusing multiple first emotion probability results according to a preset ratio, which can effectively improve the accuracy of emotion recognition, and does not require higher computing power of the terminal, thus saving power consumption and improving user experience.

[0053] In some embodiments, the present disclosure determines the first emotion probability result in the following manner: inputting target data into a neural network model to obtain multiple first emotion probability results.

[0054] In one embodiment, the neural network model may include a first neural network model. When the target data is an image, multiple images are input into the first neural network model respectively to obtain multiple first emotion probability results. The neural network model may include a second neural network model. When the target data is audio, multiple audios can be input into the first neural network model respectively to obtain multiple first emotion probability results. For another example, when the target data includes images and audio, the images can be input into the first neural network model respectively, and the audio can be input into the second neural network model respectively to obtain the first emotion probability result. The present disclosure obtains the first emotion probability result by inputting images, or audio, or images and audio into the neural network model under different conditions, so as to adapt to different situations and obtain the first emotion probability result for fusion to improve the accuracy of emotion recognition.

[0055] In some embodiments, if the target data includes an image, the present disclosure may determine the first emotion probability result in the following manner: in response to the image including facial features, the image is input into a first neural network model to obtain the first emotion probability result.

[0056] In one embodiment, if the target data is an image, the image can be input into the first neural network model when the image includes facial features. It can be understood that when the target data is an image, the number of target data is at least two, which means that the number of images is at least two. At least two are respectively input into the first neural network model to obtain the first emotion probability result corresponding to each image. For example, face recognition can be performed on the image to determine whether the image includes facial features. The method of face recognition can be, for example, detecting feature points in the image, or inputting a neural network model for face recognition, etc. Other existing methods can also be used, and the present disclosure does not limit this.

[0057] Of course, in this embodiment, the image may be directly input into the first neural network model regardless of whether the image includes facial features. For images that do not contain facial features, after inputting into the first neural network model, the output result usually has a large error. Therefore, in response to the image including facial features, inputting the image into the first neural network model can avoid the situation of large errors and improve efficiency. Moreover, in the present disclosure, for low-quality images acquired by low-power cameras, the first emotion probability results corresponding to each low-quality image are fused, which can improve recognition accuracy.

[0058] In some embodiments, if the target data includes images and audio, the present disclosure may determine the first emotion probability result in the following manner: in response to the image not including facial features, the audio is input into a second neural network model to obtain the first emotion probability result.

[0059] In one embodiment, when the image does not include facial features, for example, the user's face is not within the shooting range of the terminal camera, and the multiple images collected do not include facial features, the audio can be input into the second neural network model to obtain a first probability recognition result. Or when only a few of the multiple images collected contain facial features, multiple audios can also be input into the second neural network model to obtain a first emotion probability result. Among them, the number of audios input into the second neural network model is at least two, so as to obtain multiple first emotion probability results, that is, each audio corresponds to a first emotion probability result. It can be understood that in this embodiment, the target data can include only audio, that is, the audio can be directly input into the second neural network model without responding to the fact that the image does not include facial features.

[0060] In the present disclosure, in response to the fact that the image does not include facial features, the audio is input into the second neural network model to obtain a first emotion recognition result. When it is impossible to obtain an image including facial features, multiple first emotion recognition results can be obtained based on multiple audios, thereby improving the accuracy of emotion recognition.

[0061] In some embodiments, Figure 3is a flow chart of an emotion recognition method according to an exemplary embodiment. Figure 3 As shown, the following steps are included.

[0062] In step S21, in response to the fact that the image does not include facial features, the image is input into the first neural network model to obtain M numerical values ​​output by the middle layer of the first neural network model corresponding to each image.

[0063] In one embodiment, an image can be input into the first neural network model, and M numerical values ​​output by the intermediate layer of the first neural network model after each image is input into the first neural network model are obtained. The intermediate layer of the first neural network model can be, for example, a fully connected (FC) layer or a global average pooling layer after the convolution layer of the first neural network model. For example, the FC layer of the residual neural network (resnet) model outputs 1000 numerical values. It is understandable that the M values ​​of the intermediate layer can continue to be input into other FC layers, linear layers, etc. of the first neural network model to output the first emotion probability result.

[0064] In step S22, the audio is input into the second neural network model, and N numerical values ​​output by the middle layer of the second neural network model corresponding to each audio are obtained.

[0065] In one implementation, the audios may be input into the second neural network model respectively, and N values ​​output by the middle layer of the second neural network model after each audio is input into the second neural network model are obtained.

[0066] In step S23, the M numerical values ​​corresponding to the first image and the N numerical values ​​corresponding to the first audio are input into the third neural network model to obtain a second emotion probability result.

[0067] In one embodiment, the fusion of multiple first emotion probability results can be to input M numerical values ​​corresponding to the first image and N numerical values ​​corresponding to the first audio into the third neural network model to obtain the second emotion probability result. For example, the number of target data is at least two, and the target data can be an image and an audio. Input an image into the first neural network model to obtain M numerical values, and input an audio into the second neural network model to obtain N numerical values. The M numerical values ​​and the N numerical values ​​are spliced ​​and input into the third neural network model to obtain the output second emotion probability result. For another example, the number of target data is at least two, and the target data can be multiple images and multiple audios. The multiple images can be sorted according to the time when the images are acquired, and the multiple audios can be sorted according to the time when the audios are acquired. The N numerical values ​​corresponding to the first image and the M numerical values ​​corresponding to the first audio can be spliced ​​and input into the third neural network model to obtain the third emotion probability result corresponding to each image, and the multiple third emotion probability results can be fused according to a preset ratio to obtain the second emotion probability result. Among them, the first image represents any one of the multiple images, and the first audio represents any one of the multiple audios. The position of the first image in the sorted multiple images is the same as the position of the first audio in the sorted multiple audios. The third neural network model may be, for example, a multilayer perceptron (MLP). The first emotion probability results corresponding to the images and audios whose acquisition time is closer to the current time will account for a higher proportion in the subsequent fusion.

[0068] The present invention inputs images and audio into a neural network model and obtains features of the middle layer of the neural network model, and then inputs them into a third neural network model to obtain a first emotion probability result. The obtained first emotion probability result is more accurate, and the second emotion probability result obtained after subsequent fusion is also more accurate, thereby effectively improving the accuracy of emotion recognition.

[0069] In some embodiments, Figure 4 is a flow chart of an emotion recognition method according to an exemplary embodiment. Figure 4 As shown, the emotion recognition method includes the following steps.

[0070] In step S31 , in response to the sum of the probability of the first emotion and the probability of the second emotion being greater than or equal to a first threshold and the sum of the probability of the first emotion and the probability of the second emotion being less than or equal to the first threshold, audio features of the audio are extracted.

[0071] In one embodiment, the first emotion recognition result includes the first emotion and the second emotion. If the sum of the probability of the first emotion and the probability of the second emotion is greater than the first threshold, it can be considered that the emotion identified based on the first emotion probability result is the first emotion or the second emotion. On this basis, if the probability of the first emotion and the probability of the second emotion are less than the second threshold, the first emotion and the second emotion can be two emotions that are not easy to distinguish by the neural network model. For example, the first emotion is anger and the second emotion is happiness. Of course, the present disclosure is just an example. The first emotion and the second emotion can be any emotion that is not easy to distinguish in actual situations, and the present disclosure does not limit it. If the first emotion and the second emotion meet the condition that the sum is greater than the first threshold and the sum is less than the second threshold, the features of multiple audios can be extracted. Among them, the size of the first threshold and the second threshold can be set according to the actual situation. For example, the first threshold is 0.8 and the second threshold is 0.2, which is not limited by the present disclosure.

[0072] In step S32, the audio feature is input into the fourth neural network model to obtain an output result, where the output result includes the emotion recognition as the first emotion or the emotion recognition as the second emotion.

[0073] In one embodiment, the extracted features are input into a fourth neural network model to obtain an output result. The fourth neural network model is used to distinguish the first emotion from the second emotion. The output result of the fourth neural network model is that the emotion is recognized as the first emotion or the emotion is recognized as the second emotion.

[0074] The present disclosure can further use a fourth neural network model to distinguish the first emotion from the second emotion in response to the probability of the first emotion and the second emotion satisfying a condition, so as to avoid the phenomenon that the first emotion and the second emotion are often identified as each other, resulting in recognition errors.

[0075] In some embodiments, in the emotion recognition method provided by the present disclosure, extracting audio features includes at least one of the following:

[0076] Time-domain related features;

[0077] Frequency domain related features;

[0078] Features related to Mel-frequency cepstral coefficients (MFCC).

[0079] In one embodiment, the time domain related features may be, for example, fluctuation, time domain energy, time domain variance, zero crossing rate, short-time average energy, short-time average zero crossing rate, short-time zero crossing rate cumulative value, short-time energy cumulative value, short-time zero crossing rate root mean square value, short-time energy root mean square value, fundamental frequency, peak amplitude, envelope, linear prediction coefficient, short-time autocorrelation function, time domain cross-correlation function, etc. Among them, fluctuation can be considered as amplitude. The frequency domain related features may be, for example, frequency spectrum density, power spectrum density, frequency band energy ratio, spectrum entropy, spectrum centroid, spectrum average amplitude, spectrum skewness, spectrum kurtosis, frequency peak, spectrum envelope, frequency offset, modulation frequency, and spectrum dimension reduction. The features related to Mel-scale frequency cepstral coefficients (MFCC) may be, for example, Mel filter energy distribution on the frequency axis; energy distribution characteristics on the frequency axis; distribution characteristics of cepstral coefficients; correlation between cepstral coefficients; energy distribution characteristics on the time axis; change characteristics on the time axis. Some or all of the above features may be input into the fourth neural network model to achieve the distinction between the first emotion and the second emotion.

[0080] The present disclosure extracts multiple types of features of audio and inputs one or more of the different types of features into a fourth neural network model to improve the prediction accuracy of the fourth neural network model.

[0081] In some embodiments, the emotion recognition method provided by the present disclosure includes: in response to the recognized emotion being different from the most recently recorded emotion, recording the recognized emotion and time information and location information associated with the recognized emotion.

[0082] In one embodiment, after the recognized emotion is determined, it can be compared with the emotion last recorded by the terminal. If the emotion last recorded by the user is different, it can be considered that the user's emotion fluctuates. The recognized emotion can be recorded, and the corresponding time information and location information can be recorded.

[0083] In one embodiment, the change of emotions can be counted based on the recorded emotions and the location information and time information corresponding to the emotions. For example, in response to the number of recorded emotions reaching a threshold, a chart can be generated based on the recorded emotions and the time information and location information corresponding to the emotions, and the chart can be displayed to the user. For another example, a chart can be generated based on the recorded emotions and the time information and location information corresponding to the emotions at intervals, and the chart can be displayed to the user.

[0084] In one embodiment, the recorded emotions and the time information and location information corresponding to the emotions can be transmitted to a third-party application so that the third-party application can detect the degree of correlation between the disease and the emotions, recommend information to the user, etc. For example, if the user's emotions are often angry during a certain period of time, a song list that soothes the emotions can be recommended to the user.

[0085] In one implementation, the location and time at which the user's emotion is happy can be predicted based on the recorded emotion and the time information and location information corresponding to the emotion.

[0086] The present disclosure records the recognized emotions and corresponding location information and time information after determining that the emotions will change, and can generate icons, recommendation information, etc. to improve the utilization rate of emotion recognition results and enhance user experience.

[0087] In some embodiments, Figure 5 FIG. 1 is a flow chart of a first neural network model training method according to an exemplary embodiment. Figure 5 , the first neural network model can be trained as follows:

[0088] In step S41 , a first image data set and a second image data set are acquired.

[0089] In step S42, the model to be trained is trained using the first image data set.

[0090] In step S43, the parameters of the model to be trained are adjusted using the second image data set to obtain a first neural network model.

[0091] In one embodiment, a model to be trained with a model complexity of a first complexity can be constructed. For example, the model to be trained with a first complexity can be an emoticon (EmotiCon) network. The model to be trained with a first complexity can be trained using a first image data set. Among them, the first image data set can be, for example, FER2013, The Extended Cohn-Kanade Dataset. A second image data set can be collected. The image quality of the second image data set is lower than that of the first image data set. For example, the images in the second image data set are images taken by a low-power camera. The model to be trained that has been initially trained can be fine-tuned using the second image data set. Fine-tuning refers to further training on a specific task based on an already trained model, so that the model adapts to the specific data and requirements of the task. That is, training is performed according to the second image data, so as to adjust the parameters to make it more suitable for the second image data set. In the present disclosure, the trained model to be trained with the first complexity is referred to as a third neural network model, and the third neural network model can be used as the first neural network model. Multiple images can be input into a neural network model of the first complexity respectively. Since the neural network model is obtained after fine-tuning using the second image data set, it is more suitable for images captured by low-power cameras. It can reduce terminal power consumption while improving the accuracy of emotion recognition.

[0092] In one embodiment, a model to be trained with a model complexity of the second complexity can be constructed. For example, the model to be trained with the second complexity can be a computer vision (MobileNet) network. Among them, the second complexity is lower than the first complexity. The output result of the model to be trained with the second complexity can be made to learn the probability of the output of the third neural network model. For example, if the output result of the third neural network model is the probability corresponding to each emotion, the output result of the model to be trained with the second complexity can be made to learn the output result of the third neural network model. If the output result of the third neural network model is the predicted emotion, the output result of the model to be trained with the second complexity can be made to learn the probability of the output of the middle layer of the third neural network model. In the present disclosure, the trained model to be trained with the second complexity is referred to as the fourth neural network model. The fourth neural network model can be used as the first neural network model. For example, multiple probabilities output by the trained model to be trained with the first complexity are used as soft targets of the model to be trained with the second complexity. The trained model to be trained with the second complexity includes multiple probabilities in the soft target, that is, the probability of positive samples and the probability of negative samples. Therefore, the expression ability of the trained model of the second complexity on both positive and negative samples is similar to that of the first neural model of the first complexity. The prediction accuracy of the first neural network model of the second complexity can be close to the prediction accuracy of the first neural network model of the first complexity, and the storage space occupied is smaller. The positive sample refers to the actual emotion of the user, and the negative sample refers to the emotion predicted incorrectly by the neural network. For example, if the actual emotion of the user is anger, and the probability of happiness predicted by the first neural network model is almost the same as the probability of anger, or the probability of happiness is higher than the probability of anger, then the positive sample refers to anger, and the negative sample is happiness.

[0093] In some embodiments, if the output result of the trained model of the first complexity to be trained is the probability corresponding to each emotion, the trained model of the first complexity to be trained can be used as the first neural network model. Or, the output result of the trained model of the second complexity to be trained is made to learn the output result of the trained model of the first complexity to be trained. If the output result of the trained model of the first complexity to be trained is an emotion, the output result of the trained model of the second complexity to be trained can be made to learn the probability vector of the middle layer of the trained model of the first complexity to be trained. That is, the model of the second complexity to be trained is used to learn the probability of the emotion output by the trained model of the first complexity to be trained.

[0094] In some embodiments, Figure 6 FIG. 1 is a flow chart of a second neural network model training method according to an exemplary embodiment. Figure 6 , the second neural network model can be trained as follows:

[0095] In step S51 , a first audio data set and a second audio data set are collected.

[0096] In step S52, the model to be trained is trained using the first audio data set.

[0097] In step S53, the parameters of the model to be trained are adjusted using the second audio data set to obtain a second neural network model.

[0098] The specific implementation of step S51-step S53 can refer to the implementation of step S51-step S53. For example, the first audio data set can be used to train the second neural network model to be trained of the first complexity. Correspondingly, the second audio data can be collected. The audio quality of the second audio data set is lower than the audio quality of the first audio data set. For example, the audio in the second audio data set is audio collected by a low-power microphone. Correspondingly, the second audio data set can be used to fine-tun the second neural network model. This disclosure will not go into details.

[0099] The present disclosure obtains a neural network model of the first complexity after fine-tuning the second image data set or the second audio data set, which can be more suitable for images captured by a low-power camera or audio captured by a low-power microphone. The first emotion probability result is obtained based on the neural network model of the first complexity, which can improve the accuracy of emotion recognition while reducing the power consumption of the terminal. On the basis of the neural network model of the first complexity, multiple probabilities output by the neural network model of the first complexity are used as soft targets of the neural network model of the second complexity to achieve a prediction accuracy of the neural network model of the second complexity that is close to the prediction accuracy of the neural network model of the first complexity. The first emotion probability result is obtained based on the neural network model of the second complexity, which can further save storage space.

[0100] Based on the same concept, the embodiment of the present disclosure also provides an emotion recognition device.

[0101] It is understandable that in order to achieve the above functions, the emotion recognition device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0102] It should be noted that those skilled in the art can understand that the various implementation methods / embodiments involved in the embodiments of the present disclosure can be used in conjunction with the aforementioned embodiments or can be used independently. Whether used alone or in conjunction with the aforementioned embodiments, the implementation principle is similar. In the implementation of the present disclosure, some embodiments are described in terms of implementation methods used together. Of course, those skilled in the art can understand that such examples are not limitations of the embodiments of the present disclosure.

[0103] Figure 7 is a block diagram of an emotion recognition device 100 according to an exemplary embodiment. Figure 7 The device 100 includes: an acquisition unit 101, a determination unit 102 and a processing unit 103.

[0104] The acquisition unit 101 is used to acquire target data, and the total number of target data is at least two. The determination unit 102 is used to determine multiple first emotion probability results based on the target data. The processing unit 103 is used to merge the multiple first emotion probability results to obtain a second emotion probability result. The determination unit 102 is also used to determine the recognized emotion based on the second emotion probability result.

[0105] In one implementation, the determination unit 102 determines a plurality of first emotion probability results based on the target data in the following manner: inputting the target data into a neural network model to obtain a plurality of first emotion probability results.

[0106] In one embodiment, the target data includes an image, and the determination unit 102 inputs the target data into a neural network model in the following manner to obtain multiple first emotion probability results: in response to the image including facial features, the image is input into a first neural network model to obtain multiple first emotion probability results.

[0107] In one embodiment, the target data includes images and audio, and the determination unit 102 inputs the target data into a neural network model in the following manner to obtain multiple first emotion probability results: in response to the image not including facial features, the audio is input into a second neural network model to obtain multiple first emotion probability results.

[0108] In one embodiment, the target data includes images and audios, and the determination unit 102 inputs the target data into a neural network model in the following manner to obtain a plurality of first emotion probability results: in response to the image including facial features, the image is input into the first neural network model to obtain M numerical values ​​output by the middle layer of the first neural network model corresponding to each image. The audio is input into the second neural network model to obtain N numerical values ​​output by the middle layer of the second neural network model corresponding to each audio. The M numerical values ​​corresponding to the first image and the N numerical values ​​corresponding to the first audio are input into the third neural network model to obtain the first emotion probability result, the first audio represents any audio in the audio, the first image represents any image in the image, and the position of the first audio in the audio after being sorted by the acquisition time is the same as the position of the first image in the image after being sorted by the acquisition time.

[0109] In one embodiment, the target data includes audio, the first emotion probability result includes the probability of the first emotion and the probability of the second emotion, and the determination unit 102 determines the recognized emotion based on the second emotion probability result in the following manner: in response to the probability of the first emotion and the probability of the second emotion being greater than or equal to the first threshold, and the probability of the first emotion and the probability of the second emotion being less than or equal to the first threshold, extract the audio features of the audio. The audio features are input into the fourth neural network model to obtain an output result, and the output result includes the emotion being recognized as the first emotion or the emotion being recognized as the second emotion.

[0110] In one embodiment, the audio feature includes at least one of the following: a feature related to the time domain, a feature related to the frequency domain, and a feature related to Mel-frequency cepstral coefficients (MFCC).

[0111] In one embodiment, after determining the recognized emotion, the method further includes: in response to the recognized emotion being different from the most recently recorded emotion, recording the recognized emotion and time information and location information associated with the recognized emotion.

[0112] In one embodiment, the processing unit 103 trains the first neural network model in the following manner: collect the first image data set and the second image data set, the image quality in the second image data set is lower than the image quality in the first image data set. Use the first image data set to train the model to be trained. Use the second image data set to adjust the parameters of the model to be trained to obtain the first neural network model.

[0113] In one embodiment, the processing unit 103 trains the second neural network model in the following manner: collect the first audio data set and the second audio data set, the audio quality in the second audio data set is lower than the audio quality in the first audio data set. Use the first audio data set to train the model to be trained. Use the second audio data set to adjust the parameters of the model to be trained to obtain the second neural network model.

[0114] Figure 8 is a block diagram of an emotion recognition device 200 according to an exemplary embodiment.

[0115] like Figure 8 As shown, apparatus 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0116] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0117] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0118] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0119] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0120] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0121] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0122] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0123] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0124] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0125] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0126] The present invention fuses multiple recognition results according to a preset ratio to obtain a fused recognition result that can effectively improve the accuracy of emotion recognition, and does not require a higher computing power of the terminal, thus saving power consumption and improving user experience.

[0127] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0128] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0129] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0130] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0131] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

Claims

1. An emotion recognition method, characterized in that: include: Acquire target data, where the total number of the target data is at least two; Based on the target data, determining a plurality of first emotion probability results; Merging the multiple first emotion probability results to obtain a second emotion probability result; Based on the second emotion probability result, a recognized emotion is determined.

2. The method according to claim 1, characterized in that The determining of a plurality of first emotion probability results based on the target data comprises: The target data is input into a neural network model to obtain a plurality of first emotion probability results.

3. The method according to claim 2, characterized in that The target data includes an image, and the target data is input into a neural network model to obtain a plurality of first emotion probability results, including: In response to the image including facial features, the image is input into a first neural network model to obtain a plurality of first emotion probability results.

4. The method according to claim 2, characterized in that: The target data includes images and audio, and inputting the target data into the neural network model includes: In response to the image not including facial features, the audio is input into a second neural network model to obtain a plurality of first emotion probability results.

5. The method according to claim 2, characterized in that: The target data includes images and audio, and inputting the target data into the neural network model includes: In response to the image including a facial feature, the image is input into a first neural network model, and M numerical values ​​output by an intermediate layer of the first neural network model corresponding to each image are obtained; Input the audio into the second neural network model, and obtain N values ​​output by the middle layer of the second neural network model corresponding to each audio; The step of fusing the plurality of first emotion probability results to obtain a second emotion probability result includes: The M numerical values ​​corresponding to the first image and the N numerical values ​​corresponding to the first audio are input into the third neural network model to obtain a second emotion probability result, the first audio represents any one of the audios, the first image represents any one of the images, and the position of the first audio in the audio after sorting by acquisition time is the same as the position of the first image in the image after sorting by acquisition time.

6. The method according to claim 2, characterized in that The target data includes audio, the first emotion probability result includes a probability of a first emotion and a probability of a second emotion, and determining the recognized emotion based on the second emotion probability result includes: In response to the sum of the probability of the first emotion and the probability of the second emotion being greater than or equal to a first threshold, and the sum of the probability of the first emotion and the probability of the second emotion being less than or equal to the first threshold, extracting audio features of the audio; The audio feature is input into a fourth neural network model to obtain an output result, wherein the output result includes emotion recognition as a first emotion or emotion recognition as a second emotion.

7. The method according to claim 6, characterized in that The audio feature includes at least one of the following: Time-domain related features; Frequency domain related features; Features related to Mel-frequency cepstral coefficients (MFCC).

8. The method according to claim 1, characterized in that After determining the recognized emotion, the method further includes: In response to the recognized emotion being different from the most recently recorded emotion, the recognized emotion and time information and location information associated with the recognized emotion are recorded.

9. The method according to claim 3 or 5, characterized in that: The first neural network model is trained in the following way: Acquiring a first image data set and a second image data set, wherein image quality in the second image data set is lower than image quality in the first image data set; Using the first image data set to train the model to be trained; The parameters of the model to be trained are adjusted using the second image data set to obtain the first neural network model.

10. The method according to claim 4 or 5, characterized in that: The second neural network model is trained in the following way: collecting a first audio data set and a second audio data set, wherein audio quality in the second audio data set is lower than audio quality in the first audio data set; Using the first audio data set to train the model to be trained; The parameters of the model to be trained are adjusted using the second audio data set to obtain the second neural network model.

11. An emotion recognition device, characterized in that: include: An acquisition unit, used for acquiring target data, wherein the total number of the target data is at least two; A determination unit, configured to determine a plurality of first emotion probability results based on the target data; A processing unit, configured to merge the plurality of first emotion probability results to obtain a second emotion probability result; The determining unit is further configured to determine the recognized emotion based on the second emotion probability result.

12. An electronic device, characterized in that: include: A memory for storing instructions; as well as A processor, configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 10.

13. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed by the processor, the method according to any one of claims 1 to 10 is executed.

Images