Facial emotion recognition method, device, equipment and storage medium

Abstract

The present application discloses a facial emotion recognition method, apparatus, device and storage medium, relating to the field of emotion recognition technology, including: obtaining facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period; performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information; performing feature analysis on the audio acquisition information to obtain sound features contained in the audio acquisition information; performing feature fusion on the facial features and the sound features to obtain an emotion vector; inputting the emotion vector into an emotion decision model to perform emotion recognition to obtain an emotion recognition result, which can combine the two dimensions of facial feature recognition and sound feature recognition to perform emotion recognition, and can improve the accuracy of emotion recognition.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition technology, and in particular to a method, apparatus, device and storage medium for facial emotion recognition. Background Art

[0002] Facial emotion recognition is widely used in many fields, such as autonomous driving, human-computer interaction, and health monitoring. In current facial emotion recognition schemes, the facial features of the person being identified are usually recognized. However, human emotions are not only reflected on the face, but also on the voice. That is to say, in current emotion recognition schemes, only the facial emotions of the person being identified are recognized. When the person being identified disguises his emotions, this emotion recognition method cannot accurately recognize the emotions of the person being identified. Therefore, a more accurate emotion recognition scheme is needed for emotion recognition.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a facial emotion recognition method, device, equipment and storage medium, aiming to solve the technical problem in the existing technology of low accuracy of emotion recognition results caused by single facial emotion recognition.

[0005] To achieve the above objectives, the present application proposes a method for facial emotion recognition, which includes:

[0006] Acquire facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period;

[0007] Performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information;

[0008] Performing feature analysis on the audio collection information to obtain sound features contained in the audio collection information;

[0009] Performing feature fusion on the facial features and the voice features to obtain an emotion vector;

[0010] The emotion vector is input into an emotion decision model to perform emotion recognition and obtain an emotion recognition result.

[0011] In one embodiment, the step of performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information includes:

[0012] Normalizing and gray-scaling the facial image acquisition information to obtain facial information to be recognized;

[0013] Determining facial feature points of the facial information to be recognized based on a facial recognition model;

[0014] determining a facial region according to the facial feature points;

[0015] Performing feature recognition in the facial area to determine key feature points in the facial area;

[0016] extracting a plurality of local features of the facial region according to the key feature points;

[0017] The multiple local features are aggregated to obtain the facial features.

[0018] In one embodiment, the step of determining facial feature points of the facial information to be recognized based on a facial recognition model further includes:

[0019] Detecting the facial information to be identified based on a detection window of the facial recognition model, extracting key points to be selected from the facial information to be identified, and moving the detection window along a preset trajectory on the facial information to be identified;

[0020] Fitting the candidate key points to obtain a feature area;

[0021] The candidate key points are screened according to the feature area, and the candidate key points in the feature area are determined as facial feature points.

[0022] In one embodiment, the step of performing feature analysis on the audio collection information to obtain sound features contained in the audio collection information further includes:

[0023] Performing noise suppression on the audio collection information to obtain audio preprocessing information;

[0024] Converting the audio preprocessing information into digital audio information;

[0025] Performing audio enhancement on the digital audio information to obtain enhanced audio information;

[0026] Performing spectrum analysis on the enhanced audio information to separate human voice audio information from the enhanced audio information;

[0027] Feature extraction is performed on the human voice audio information to obtain the sound feature.

[0028] In one embodiment, the step of converting the audio preprocessing information into digital audio information further includes:

[0029] Sampling the audio preprocessing information at a preset interval to obtain an audio sampling sequence;

[0030] Extracting sample values ​​from the audio sampling series, mapping the sample values ​​to obtain quantized values ​​corresponding to the sample values, and obtaining a quantized sequence;

[0031] The quantized sequence is binary-encoded to obtain the digital audio information.

[0032] In one embodiment, the step of performing spectrum analysis on the enhanced audio information and separating the enhanced audio information to obtain human voice audio information further comprises:

[0033] performing spectrum analysis on the enhanced audio information to determine harmonics in the enhanced audio information;

[0034] Performing spectrum subtraction on the enhanced audio information and the harmonics to obtain the human voice audio information.

[0035] In one embodiment, the step of inputting the emotion vector into an emotion decision model for emotion recognition to obtain an emotion recognition result further comprises:

[0036] Inputting the emotion vector into an emotion decision model to obtain facial feature weights and voice feature weights in the emotion decision model;

[0037] parsing the emotion vector to determine facial-voice feature pairs in the emotion vector;

[0038] Determining an emotion set of the facial feature and an emotion set of the voice feature in the face-feature pair, respectively, wherein the emotion set includes an emotion name and a predicted probability of each emotion, and the emotion name with the highest predicted probability is a target predicted emotion in the emotion set;

[0039] The emotion recognition result is obtained based on the predicted probability of the target predicted emotion in the emotion set of the facial features and the emotion set of the voice features, the facial feature weights and the voice feature weights.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a facial emotion recognition device, which includes:

[0041] An information collection module, configured to obtain facial image collection information and audio collection information corresponding to the facial image collection information within a preset time period;

[0042] A facial analysis module, configured to perform feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information;

[0043] A sound analysis module, configured to perform feature analysis on the audio collection information to obtain sound features contained in the audio collection information;

[0044] A feature fusion module, configured to fuse the facial features with the voice features to obtain an emotion vector;

[0045] The emotion decision module is used to input the emotion vector into the emotion decision model to perform emotion recognition and obtain an emotion recognition result.

[0046] In addition, to achieve the above-mentioned purpose, the present application also proposes a facial emotion recognition device, which includes: a memory, a processor, and a computer program stored in the memory and runnable on the processor, and the computer program is configured to implement the steps of the facial emotion recognition method described above.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the facial emotion recognition method described above are implemented.

[0048] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the facial emotion recognition method as described above.

[0049] One or more technical solutions proposed in this application have at least the following technical effects: obtaining facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period; performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information; performing feature analysis on the audio acquisition information to obtain sound features contained in the audio acquisition information; performing feature fusion on the facial features and the sound features to obtain an emotion vector; inputting the emotion vector into an emotion decision model to perform emotion recognition to obtain an emotion recognition result, which can combine the two dimensions of facial feature recognition and sound feature recognition to perform emotion recognition, and can improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 A flowchart of the first embodiment of the facial emotion recognition method provided by the applicant;

[0053] Figure 2 This is a schematic diagram of information collection according to an embodiment of the facial emotion recognition method of the present application;

[0054] Figure 3 This is a schematic diagram of facial feature recognition according to an embodiment of the present application's facial emotion recognition method;

[0055] Figure 4 This is a schematic diagram of audio collection information according to an embodiment of the facial emotion recognition method of the present application;

[0056] Figure 5 This is a fusion diagram of an embodiment of the facial emotion recognition method of the present application;

[0057] Figure 6 This is a schematic diagram of the module structure of the facial emotion recognition device according to an embodiment of the present application;

[0058] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the facial emotion recognition method in the embodiment of the present application.

[0059] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0061] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0062] The main solution of the embodiment of the present application is: obtaining facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period; performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information; performing feature analysis on the audio acquisition information to obtain sound features contained in the audio acquisition information; performing feature fusion of the facial features and the sound features to obtain an emotion vector; inputting the emotion vector into an emotion decision model to perform emotion recognition to obtain an emotion recognition result, which can combine the two dimensions of facial feature recognition and sound feature recognition to perform emotion recognition, and can improve the accuracy of emotion recognition.

[0063] In this embodiment, for ease of description, the following description is made with the facial emotion recognition device as the execution subject.

[0064] In the existing technology, when performing emotion recognition, the method usually adopted is to identify the facial features of the person to be identified, determine the expression features of the facial features, and analyze the facial expression features to determine the emotions of the person to be identified. However, determining human emotions with a single feature will be interfered with by the subjective consciousness of the person to be identified. Therefore, determining emotions from only one perspective of facial features is more unstable, resulting in a lower recognition accuracy.

[0065] This application provides a solution that can combine facial information and sound information during the current facial information collection time, and jointly determine the emotional characteristics of the person being identified by combining facial expression features and sound features, thereby improving the accuracy of emotion recognition.

[0066] It can be seen from the above embodiments that the present application obtains facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period; performs feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information; performs feature analysis on the audio acquisition information to obtain sound features contained in the audio acquisition information; performs feature fusion on the facial features and the sound features to obtain an emotion vector; inputs the emotion vector into an emotion decision model to perform emotion recognition to obtain an emotion recognition result, which can combine the two dimensions of facial feature recognition and sound feature recognition to perform emotion recognition, and can improve the accuracy of emotion recognition.

[0067] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions, a facial emotion recognition device, etc. The following uses a facial emotion recognition device as an example to illustrate this embodiment and the following embodiments.

[0068] Based on this, the embodiment of the present application provides a method for facial emotion recognition, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the facial emotion recognition method of the present application.

[0069] In this embodiment, the facial emotion recognition method includes steps S10 to S50:

[0070] Step S10: acquiring facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period.

[0071] It should be noted that facial image acquisition information is image information collected by a camera, wherein the facial image acquisition information includes the front facial features of the target person; audio acquisition information refers to the audio signal obtained through a sound acquisition device, such as audio information collected by a microphone. The audio acquisition signal obtained through the sound acquisition device is an analog signal.

[0072] It should be understood that in the process of emotion recognition of the target person, the facial features of the target person and the sound information in the current environment will be collected at the same time. The facial features include the target person's eyebrows, eyes, nose, mouth and other feature information, and the audio collection information is the audio information of a period of time. This period of time is determined according to the collection time of the face image collection information. Since the image collection is an instantaneous action, if the audio information corresponding to the instantaneous moment is a single-point audio, it is impossible to identify the spectrum information and emotional information in the sound. Therefore, the audio collection device continuously collects volume, that is, the audio collection device will generate an audio sequence. When the image collection device collects the face image collection information, it records the current collection time and intercepts the corresponding audio collection information in the audio sequence.

[0073] In the specific implementation, refer to Figure 2 , Figure 2 The figure is a schematic diagram of information collection. When performing emotion recognition, the audio acquisition device can start audio acquisition, continuously collect audio information in the current environment, and constantly monitor the acquisition instructions of the image acquisition device. When the acquisition instruction of the image acquisition device is detected, the current acquisition time of the facial image acquisition information is recorded and the facial image acquisition information is output. After obtaining the acquisition time of the facial image acquisition information, the audio information can be cropped according to the preset template of the audio sequence to determine the starting point of the audio sequence. The time interval between the time corresponding to the starting point of the audio sequence and the image acquisition time is half of the duration of the audio sequence, and the other half ends at half of the duration of the audio sequence after the image acquisition time. The audio sequence corresponding to this time interval is then output as the audio acquisition information.

[0074] Step S20: performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information.

[0075] It should be noted that facial features refer to the facial features of a person in the face image collection information, usually including eyes, eyebrows, nose, mouth and other facial features that can reflect emotions.

[0076] In a specific implementation, feature analysis is performed on the face image acquisition information to determine the facial information in the face image acquisition information, the facial contour features of the face are determined, and the facial features are highlighted. Specifically, the face image acquisition information can be preprocessed, and facial feature recognition is performed on the preprocessed face image acquisition information to determine the shape contour of each facial feature and use curves to outline the features. If some facial features are blocked, the blocked facial features will be left vacant and feature extraction will not be performed. The facial features that can be extracted will be output to determine the position of the facial features and the facial features in the face image acquisition information.

[0077] Exemplarily, the step of performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information includes:

[0078] Normalizing and gray-scaling the facial image acquisition information to obtain facial information to be recognized;

[0079] Determining facial feature points of the facial information to be recognized based on a facial recognition model;

[0080] determining a facial region according to the facial feature points;

[0081] Performing feature recognition in the facial area to determine key feature points in the facial area;

[0082] extracting a plurality of local features of the facial region according to the key feature points;

[0083] The multiple local features are aggregated to obtain the facial features.

[0084] It should be noted that the facial recognition model is trained by the convolutional neural network CNN. When performing feature analysis on facial image collection information, the facial recognition model can perform feature analysis based on the facial image collection information, determine facial feature points by collecting facial image information, and convolve the facial image collection information through the convolution kernel in the convolutional neural network to obtain facial features.

[0085] In the specific implementation, the facial image acquisition information is normalized and grayscaled to obtain the facial information to be identified. Due to the large differences between the image acquisition environments, there are also large differences between the quality of the acquired images. Therefore, when performing feature analysis on the facial image acquisition information, the facial image acquisition information needs to be preprocessed. The preprocessing process is to scale the image and adjust the image to a fixed size. After adjusting the size of the image, the first step of preprocessing is performed by grayscale. After obtaining the grayscale image, in order to reduce the generalization error of the model training and the model size required for fitting the training data, local contrast normalization can be used. Before model training, the image is quickly localized and the brightness value at the image (i, j) is set to m(i, j). The brightness value after local contrast normalization is set to m′(i, j). The calculation formula is:

[0086]

[0087] Where i∈{1,2,…,M}, j∈{1,2,…,N}, M and N refer to the dimension values, μ is the mean brightness of the image, σ is the standard deviation of the brightness, and c is the brightness correction value.

[0088] After normalizing and gray-scaling the facial image information, the facial information to be identified can be obtained. The facial information to be identified is then identified based on the facial recognition model, and facial feature points are determined. Multiple facial feature points are combined to determine the facial area. Feature recognition is performed within the facial area, and key feature points in the facial area are determined. Key feature points are indispensable key feature points that constitute facial features, such as dividing points and curve change points. At the same time, feature points outside the feature area formed by the key feature points are removed to eliminate interference from other factors. The feature area formed by the key feature points is the facial feature. Multiple local features of the facial area are then extracted based on the key feature points, such as the left eye feature area, the mouth feature area, and other local feature areas. The multiple local feature areas obtained are aggregated to obtain the entire facial feature.

[0089] Exemplarily, the step of determining facial feature points of the facial information to be recognized based on the facial recognition model further includes:

[0090] Detecting the facial information to be identified based on a detection window of the facial recognition model, extracting key points to be selected from the facial information to be identified, and moving the detection window along a preset trajectory on the facial information to be identified;

[0091] Fitting the candidate key points to obtain a feature area;

[0092] The candidate key points are screened according to the feature area, and the candidate key points in the feature area are determined as facial feature points.

[0093] In a specific implementation, when determining the facial feature points of the face information to be identified, the face information to be identified can be detected based on the detection window of the face recognition model, wherein the detection window is a sliding window with a certain pixel size, which slides on the face information to be identified according to a preset step size. In the detection window, the pixel value of each pixel in the window is extracted, and the pixel value of each pixel in the window is referenced. Figure 3 , Figure 3 This is a schematic diagram of facial feature recognition. In each detection window, each pixel value can be convolved. Specifically, the maximum value convolution can be selected to determine the convolution value of the current detection window, and the convolution values ​​of the detection window are summarized. After determining the convolution value of the detection window, the candidate key points in the current facial information to be identified can be obtained, and the detection window is continuously moved on the facial information to be identified according to a preset trajectory, and the candidate key points are further fitted to obtain a feature area. The candidate key points are screened according to the feature area, and the candidate key points in the feature area are determined as facial feature points.

[0094] Step S30: performing feature analysis on the audio collection information to obtain sound features contained in the audio collection information.

[0095] It should be noted that the audio collection information includes the noise in the environment and the voice information of the target person. The audio information of the time before and after the image collection is determined based on the time of image collection to reflect the emotional information in the current environment.

[0096] In the specific implementation, the characteristics of the audio collection information are analyzed, referring to Figure 4 , Figure 4 This is a diagram of audio capture information. When analyzing audio capture information, it's necessary to separate the human voice from the background noise. Because human voices typically have a harmonic structure, while background noise typically does not, feature analysis can extract the harmonic spectrum from the audio capture information. By subtracting the audio capture information from the harmonic spectrum, we can separate the background noise and identify the human voice. Feature analysis of the human voice then determines the vocal characteristics of pitch, speaking rate, and volume.

[0097] Exemplarily, the step of performing feature analysis on the audio collection information to obtain sound features contained in the audio collection information further includes:

[0098] Performing noise suppression on the audio collection information to obtain audio preprocessing information;

[0099] Converting the audio preprocessing information into digital audio information;

[0100] Performing audio enhancement on the digital audio information to obtain enhanced audio information;

[0101] Performing spectrum analysis on the enhanced audio information to separate human voice audio information from the enhanced audio information;

[0102] Feature extraction is performed on the human voice audio information to obtain the sound feature.

[0103] In a specific implementation, noise suppression is performed on the audio acquisition information to obtain audio preprocessing information. The audio acquisition information can then be filtered using a filter. The noise can be filtered using a low-pass filter, removing most of the noise while retaining clearer human voice information. The audio preprocessing information is then converted into digital audio information. The analog signal is sampled at a preset sampling interval, and the amplitude of the analog signal is recorded at a fixed frequency to discretize the analog signal. According to the sampling theorem, the sampling frequency must be higher than twice the highest frequency in the signal. Therefore, before sampling, the frequency of the audio sampling information needs to be detected to determine the highest frequency. After sampling, the analog signal becomes a series of equally spaced sample values, which are stored in a sample data sequence. The digital audio information is then audio enhanced to obtain enhanced audio information. Spectral analysis is performed on the enhanced audio information, and human voice audio information is separated from the enhanced audio information.

[0104] Exemplarily, the step of converting the audio preprocessing information into digital audio information further includes:

[0105] Sampling the audio preprocessing information at a preset interval to obtain an audio sampling sequence;

[0106] Extracting sample values ​​from the audio sampling series, mapping the sample values ​​to obtain quantized values ​​corresponding to the sample values, and obtaining a quantized sequence;

[0107] The quantized sequence is binary-encoded to obtain the digital audio information.

[0108] In a specific implementation, after sampling, the sample values ​​need to be quantized. Quantization is the process of mapping each sample value into a discrete numerical representation. During the sampling process, each sample value is a continuous analog quantity. The quantization operation determines the representation of each sample value within the digital range, but this process will introduce quantization errors because the analog signal is continuous, while the digital quantization representation is finitely discrete. The quantization sequence is then binary-encoded to obtain the digital audio information. Encoding is the process of representing the quantized sample values ​​as binary codes. Although the signal has become a digital signal after quantization, there are many ways to represent binary words. Therefore, it is necessary to select the required binary encoding method based on multiple angles such as accuracy, dynamic range, and implementation cost.

[0109] Exemplarily, the step of performing spectrum analysis on the enhanced audio information and separating the human voice audio information from the enhanced audio information further includes:

[0110] performing spectrum analysis on the enhanced audio information to determine harmonics in the enhanced audio information;

[0111] Performing spectrum subtraction on the enhanced audio information and the harmonics to obtain the human voice audio information.

[0112] In a specific implementation, spectrum analysis is performed on the enhanced audio information to determine harmonics in the enhanced audio information, and spectrum subtraction is performed between the enhanced audio information and the harmonics to obtain the human voice audio information.

[0113] Step S40: Fusing the facial features with the voice features to obtain an emotion vector.

[0114] It should be noted that the emotion vector refers to the initial recognition used to represent the current facial features and voice features, and can determine the emotion probability list corresponding to the facial features and voice features separately.

[0115] In the specific implementation, facial features and voice features are fused, referring to Figure 5 , Figure 5 It is a fusion diagram. It can combine the facial features and voice features at the same time, and the current facial features P i With sound characteristics S i The corresponding emotion recognition sequence:

[0116] {<P1,v1> ,<P2,v2> ,… <P n ,v n >,<S1,v1> ,<S2,v2> ,… n ,v n >}.

[0117] ​Step S50: input the emotion vector into an emotion decision model to perform emotion recognition and obtain an emotion recognition result.

[0118] It should be noted that the emotion decision model is a neural network model built based on the convolutional neural network (CNN), which can be used to analyze the input facial features and voice features to determine the final emotion recognition results.

[0119] Exemplarily, the step of inputting the emotion vector into an emotion decision model for emotion recognition to obtain an emotion recognition result further includes:

[0120] Inputting the emotion vector into an emotion decision model to obtain facial feature weights and voice feature weights in the emotion decision model;

[0121] parsing the emotion vector to determine facial-voice feature pairs in the emotion vector;

[0122] Determining an emotion set of the facial feature and an emotion set of the voice feature in the face-feature pair, respectively, wherein the emotion set includes an emotion name and a predicted probability of each emotion, and the emotion name with the highest predicted probability is a target predicted emotion in the emotion set;

[0123] The emotion recognition result is obtained based on the predicted probability of the target predicted emotion in the emotion set of the facial features and the emotion set of the voice features, the facial feature weights and the voice feature weights.

[0124] In a specific implementation, after the emotion decision model is trained, facial feature weights c1 and voice feature weights c2 are preset in the emotion decision model; the emotion vector is parsed to determine the facial-voice feature pair in the emotion vector; the emotion set of the facial features and the emotion set of the voice features in the face-feature pair are determined respectively, the emotion set including the emotion name and predicted probability of each emotion, and the emotion name with the largest predicted probability is the target predicted emotion in the emotion set; the emotion recognition result is obtained based on the predicted probability of the target predicted emotion in the emotion set of the facial features and the emotion set of the voice features, the facial feature weights, and the voice feature weights. Specifically, it can be expressed as establishing a time series matrix for the facial features and voice features in the image, expressed as:

[0125] F t →CNNmodel→[P(F t ) k ] t

[0126] Where: F t is the t-th image input to the CNN expression recognition model, CNNmodel is the emotion decision model, [P(Ft ) k ] t is the time series matrix of expression recognition results, P(F t ) k is the probability that F is recognized as the k-type expression, and t is the temporal order of the recognition result.

[0127] F t The time series matrix of the expression recognition results obtained by inputting the emotion decision model is:

[0128] [P(F t ) k ] t =[P(F t )0,…P(F t )6] t

[0129] Among them, emotions can be divided into 7 categories, corresponding to 0-anger, 1-sadness, 2-fear, 3-disgust, 4-surprise, 5-happy, and 6-neutral.

[0130] Therefore, an emotional decision set can be established. For details, refer to Table 1, which is an emotional decision table:

[0131]

[0132] The probability value with the highest probability determined from Table 1 is the target predicted emotion of the current facial features and voice features.

[0133] This embodiment provides a facial emotion recognition method, which obtains facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period; performs feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information; performs feature analysis on the audio acquisition information to obtain sound features contained in the audio acquisition information; performs feature fusion on the facial features and the sound features to obtain an emotion vector; inputs the emotion vector into an emotion decision model to perform emotion recognition to obtain an emotion recognition result. The method can combine the two dimensions of facial feature recognition and sound feature recognition to perform emotion recognition, and can improve the accuracy of emotion recognition.

[0134] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the facial emotion recognition method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0135] This application also provides a facial emotion recognition device, please refer to Figure 6 , the facial emotion recognition device comprises:

[0136] The information collection module 10 is used to obtain facial image collection information and audio collection information corresponding to the facial image collection information within a preset time period.

[0137] The facial analysis module 20 is configured to perform feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information.

[0138] The sound analysis module 30 is configured to perform feature analysis on the audio collection information to obtain sound features contained in the audio collection information.

[0139] The feature fusion module 40 is used to fuse the facial features with the voice features to obtain an emotion vector.

[0140] The emotion decision module 50 is used to input the emotion vector into the emotion decision model to perform emotion recognition and obtain an emotion recognition result.

[0141] The facial emotion recognition device provided in this application, utilizing the facial emotion recognition method described in the aforementioned embodiments, can address the technical issue in the prior art of low accuracy in emotion recognition results due to the singular focus on facial emotion recognition. Compared to the prior art, the beneficial effects of the facial emotion recognition device provided in this application are the same as those of the facial emotion recognition method described in the aforementioned embodiments. Other technical features of the facial emotion recognition device are the same as those disclosed in the aforementioned embodiments and are not further elaborated upon here.

[0142] The present application provides a facial emotion recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the facial emotion recognition method in the above-mentioned embodiment one.

[0143] Reference below Figure 7 , which shows a schematic diagram of the structure of a facial emotion recognition device suitable for implementing the embodiments of the present application. The facial emotion recognition device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7The facial emotion recognition device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0144] like Figure 7 As shown, the facial emotion recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the device. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the facial emotion recognition device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a facial emotion recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0145] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0146] The facial emotion recognition device provided in this application, utilizing the facial emotion recognition method described in the aforementioned embodiment, can resolve the technical issue in the prior art of low accuracy in emotion recognition results due to the singular focus on facial emotion recognition. Compared to the prior art, the beneficial effects of the facial emotion recognition device provided in this application are the same as those of the facial emotion recognition method described in the aforementioned embodiment. Other technical features of the facial emotion recognition device are the same as those disclosed in the aforementioned embodiment and are not further elaborated upon here.

[0147] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0148] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0149] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the facial emotion recognition method in the above-mentioned embodiment.

[0150] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0151] The computer-readable storage medium may be included in the facial emotion recognition device; or it may exist independently without being assembled into the facial emotion recognition device.

[0152] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the facial emotion recognition device, the facial emotion recognition device:

[0153] Acquire facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period;

[0154] Performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information;

[0155] Performing feature analysis on the audio collection information to obtain sound features contained in the audio collection information;

[0156] Performing feature fusion on the facial features and the voice features to obtain an emotion vector;

[0157] The emotion vector is input into an emotion decision model to perform emotion recognition and obtain an emotion recognition result.

[0158] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0159] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0160] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0161] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described facial emotion recognition method. This computer-readable storage medium can address the technical problem in the prior art of low accuracy in emotion recognition results due to the single-target facial emotion recognition method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the facial emotion recognition method provided in the above-described embodiment, and are not further elaborated here.

[0162] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned facial emotion recognition method when executed by a processor.

[0163] The computer program product provided in this application can address the technical problem in the prior art of low accuracy in emotion recognition results due to the singular focus on facial emotion recognition. Compared to the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the facial emotion recognition method provided in the aforementioned embodiments, and are not further elaborated here.

[0164] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A facial emotion recognition method, characterized in that: The method includes: Acquire facial image acquisition information and audio acquisition information corresponding to the facial image acquisition information within a preset time period; Performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information; Performing feature analysis on the audio collection information to obtain sound features contained in the audio collection information; Performing feature fusion on the facial features and the voice features to obtain an emotion vector; Inputting the emotion vector into an emotion decision model to perform emotion recognition and obtain an emotion recognition result; The step of performing feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information includes: Normalizing and gray-scaling the facial image acquisition information to obtain facial information to be recognized; Determining facial feature points of the facial information to be recognized based on a facial recognition model; determining a facial region according to the facial feature points; Performing feature recognition in the facial area to determine key feature points in the facial area; extracting a plurality of local features of the facial region according to the key feature points; Aggregating the multiple local features to obtain the facial features; The step of determining facial feature points of the facial information to be recognized based on the facial recognition model further comprises: Detecting the facial information to be identified based on a detection window of the facial recognition model, extracting key points to be selected from the facial information to be identified, and moving the detection window along a preset trajectory on the facial information to be identified; Fitting the candidate key points to obtain a feature area; Screening the candidate key points according to the feature area, and determining the candidate key points in the feature area as facial feature points; The step of inputting the emotion vector into the emotion decision model for emotion recognition to obtain the emotion recognition result further comprises: The emotion vector is input into the emotion decision model for identification, and a time series feature matrix is ​​established based on the facial features and voice features. t →CNN model→[P(F t ) k ] t , where: F t is the t-th image input to the CNN expression recognition model, CNN model is the emotion decision model, [P(F t ) k ] t is the time series matrix of expression recognition results, P(F t ) k is the probability that F is recognized as type k expression, and t is the temporal order of the recognition result; According to the expression recognition result time series matrix [P(F t ) k ] t =[P(F t )0,…P(F t )6] t Generate an emotion decision set, which can classify emotions into 7 categories, corresponding to 0-anger, 1-sadness, 2-fear, 3-disgust, 4-surprise, 5-happiness, and 6-neutral; Determining the maximum probability value in the emotion decision set as the target predicted emotion, and obtaining an emotion recognition result based on the target predicted emotion; The step of inputting the emotion vector into the emotion decision model for emotion recognition to obtain the emotion recognition result further comprises: Inputting the emotion vector into an emotion decision model to obtain facial feature weights and voice feature weights in the emotion decision model; parsing the emotion vector to determine facial-voice feature pairs in the emotion vector; Determining an emotion set of the facial feature and an emotion set of the voice feature in the facial-voice feature pair, respectively, wherein the emotion set includes an emotion name and a predicted probability of each emotion, and the emotion name with the highest predicted probability is a target predicted emotion in the emotion set; The emotion recognition result is obtained based on the predicted probability of the target predicted emotion in the emotion set of the facial features and the emotion set of the voice features, the facial feature weights and the voice feature weights.

2. The method according to claim 1, wherein The step of performing feature analysis on the audio collection information to obtain sound features contained in the audio collection information further includes: Performing noise suppression on the audio collection information to obtain audio preprocessing information; Converting the audio preprocessing information into digital audio information; Performing audio enhancement on the digital audio information to obtain enhanced audio information; Performing spectrum analysis on the enhanced audio information to separate human voice audio information from the enhanced audio information; Feature extraction is performed on the human voice audio information to obtain the sound feature.

3. The method according to claim 2, wherein The step of converting the audio preprocessing information into digital audio information also includes: Sampling the audio preprocessing information at a preset interval to obtain an audio sampling sequence; Extracting sample values ​​from the audio sampling series, mapping the sample values ​​to obtain quantized values ​​corresponding to the sample values, and obtaining a quantized sequence; The quantized sequence is binary-encoded to obtain the digital audio information.

4. The method according to claim 2, wherein The step of performing spectrum analysis on the enhanced audio information to separate the human voice audio information from the enhanced audio information further comprises: performing spectrum analysis on the enhanced audio information to determine harmonics in the enhanced audio information; Performing spectrum subtraction on the enhanced audio information and the harmonics to obtain the human voice audio information.

5. A facial emotion recognition device, characterized in that: The device comprises: An information collection module, configured to obtain facial image collection information and audio collection information corresponding to the facial image collection information within a preset time period; A facial analysis module, configured to perform feature analysis on the facial image acquisition information to obtain facial features contained in the facial image acquisition information; A sound analysis module, configured to perform feature analysis on the audio collection information to obtain sound features contained in the audio collection information; A feature fusion module, configured to fuse the facial features with the voice features to obtain an emotion vector; An emotion decision module, configured to input the emotion vector into an emotion decision model for emotion recognition to obtain an emotion recognition result; The facial features included in the facial image acquisition information obtained by performing feature analysis on the facial image acquisition information include: Normalizing and gray-scaling the facial image acquisition information to obtain facial information to be recognized; Determining facial feature points of the facial information to be recognized based on a facial recognition model; determining a facial region according to the facial feature points; Performing feature recognition in the facial area to determine key feature points in the facial area; extracting a plurality of local features of the facial region according to the key feature points; Aggregating the multiple local features to obtain the facial features; The determining of facial feature points of the facial information to be recognized based on the facial recognition model further comprises: Detecting the facial information to be identified based on a detection window of the facial recognition model, extracting key points to be selected from the facial information to be identified, and moving the detection window along a preset trajectory on the facial information to be identified; Fitting the candidate key points to obtain a feature area; Screening the candidate key points according to the feature area, and determining the candidate key points in the feature area as facial feature points; The step of inputting the emotion vector into the emotion decision model for emotion recognition to obtain the emotion recognition result further comprises: The emotion vector is input into the emotion decision model for identification, and a time series feature matrix is ​​established based on the facial features and voice features. t →CNN model→[P(F t ) k ] t , where: F t is the t-th image input to the CNN expression recognition model, CNN model is the emotion decision model, [P(F t ) k ] t is the time series matrix of expression recognition results, P(F t ) k is the probability that F is recognized as type k expression, and t is the temporal order of the recognition result; According to the expression recognition result time series matrix [P(F t ) k ] t =[P(F t )0,…P(F t )6] t Generate an emotion decision set, which can classify emotions into 7 categories, corresponding to 0-anger, 1-sadness, 2-fear, 3-disgust, 4-surprise, 5-happiness, and 6-neutral; Determining the maximum probability value in the emotion decision set as the target predicted emotion, and obtaining an emotion recognition result based on the target predicted emotion; The step of inputting the emotion vector into the emotion decision model for emotion recognition to obtain the emotion recognition result further comprises: Inputting the emotion vector into an emotion decision model to obtain facial feature weights and voice feature weights in the emotion decision model; parsing the emotion vector to determine facial-voice feature pairs in the emotion vector; Determining an emotion set of the facial feature and an emotion set of the voice feature in the facial-voice feature pair, respectively, wherein the emotion set includes an emotion name and a predicted probability of each emotion, and the emotion name with the highest predicted probability is a target predicted emotion in the emotion set; The emotion recognition result is obtained based on the predicted probability of the target predicted emotion in the emotion set of the facial features and the emotion set of the voice features, the facial feature weights and the voice feature weights.

6. A facial emotion recognition device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the facial emotion recognition method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the facial emotion recognition method according to any one of claims 1 to 4 are implemented.

Images