Facial emotion recognition method and device and storage medium

Through the comprehensive analysis of audio and video data, facial expressions are accurately identified, which solves the problem of inaccurate emotional judgment by advertising designers, improves the accuracy of emotional recognition, and helps advertising designers understand the viewer's true feelings.

CN118968602AInactive Publication Date: 2024-11-15ZHE JIANG YUAN YI ZHI NENG KE JI YOU XIAN GONG SI

Patent Information

Application Number
CN202411447884.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-17
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When the advertising designer analyzes the viewer's facial expressions, he fails to accurately judge emotions, resulting in inaccurate judgment of emotions.

Method used

By obtaining audio data and video data, facial images are extracted and divided into different regions, scaling and grayscale processing are performed, facial feature points are extracted, local emotional state is calculated, and the audio data is analyzed in a comparison and credibility verification to ensure accuracy in facial emotions recognition.

Benefits of technology

Improves the accuracy of facial emotions recognition and ensures that ad designers can understand the viewer’s true feelings, thereby improving ad design.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968602A_ABST
    Figure CN118968602A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of emotion analysis, and particularly discloses a facial emotion recognition method and device and a storage medium, and the method comprises the steps: obtaining audio data and video data, and extracting a facial image from the video data; dividing the facial image into sub-images of different areas, calculating local emotional states of the sub-images, and integrating the local emotional states of all the sub-images to obtain a facial emotional state of the facial image; performing semantic analysis and intonation analysis on the audio data to obtain a voice emotional state; when the facial emotion state is the same as the voice emotion state, the facial emotion state is a result. According to the method, the facial image is divided into different sub-images for recognition, then the recognition results of the sub-images are summarized, the obtained facial emotion is more accurate, then semantic analysis, speech speed analysis and intonation analysis are performed on the audio data to obtain the voice emotion, and then the facial emotion and the voice emotion are compared, so that the accuracy of the facial emotion is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of emotion analysis, and in particular to a facial emotion recognition method, device and storage medium. Background Art

[0002] In people's daily communication, 7% of the information is transmitted through language, 38% through sound, and 55% through facial expressions. This shows that facial expressions are an important carrier of human communication and an important way of non-verbal communication. The various emotions revealed in facial expressions are a universal and effective way to express human intentions. Analyzing people's intentions through the emotions revealed by their faces is called facial emotion recognition technology, which is applied to many fields such as security detection.

[0003] Emotion is a stress response of human beings to the external environment, and it is also a comprehensive manifestation of multiple sensory organs. With the popularization of advertising videos, advertising designers also need to know how people feel about the advertisements. Currently, facial expressions are one of the most powerful, natural and common signals for humans to convey emotional states and intentions. Therefore, advertising designers will analyze the facial expressions of viewers to know how people feel about the advertisements. However, the results obtained by only analyzing facial expressions are not accurate, and there is no step to verify the results, which leads to inaccurate emotional judgments of advertising designers on viewers. Summary of the invention

[0004] The purpose of the present invention is to provide a facial emotion recognition method and system to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for recognizing facial emotions, the method comprising: Acquire audio data and video data corresponding to the audio data, and extract a facial image from the video data; Divide the facial image into sub-images of different regions, scale and grayscale each sub-image, extract facial feature points of each sub-image, calculate the local emotional state of each sub-image based on all facial feature points of each sub-image, comprehensively calculate the local emotional states of all sub-images in the same facial image, and obtain the facial emotional state of the facial image; Performing semantic analysis, speech rate analysis and intonation analysis on the audio data corresponding to the facial image to obtain the speech emotional state corresponding to the audio data; Compare the facial emotional state and the voice emotional state. When the facial emotional state and the voice emotional state are the same, the facial emotional state is the result. When the facial emotional state and the voice emotional state are different, confirm the credibility of the facial emotional state and the voice emotional state. When the credibility of the voice emotional state is greater than the first credibility threshold and the credibility of the facial emotional state is greater than the second credibility threshold, the facial emotion recognition result is the sum of the credibility of the voice emotional state multiplied by the voice weight coefficient plus the credibility of the facial emotional state multiplied by the facial weight coefficient.

[0006] As a further solution of the present invention: the steps of dividing the facial image into sub-images of different regions, scaling and grayscale processing each sub-image, extracting facial feature points of each sub-image, calculating the local emotional state of the sub-image based on all facial feature points of each sub-image, and comprehensively calculating the local emotional states of all sub-images in the same facial image to obtain the facial emotional state of the facial image include: Divide the facial image into sub-images according to the cheek area, eye area, nose area, mouth area and chin area, and mark the facial feature points in each sub-image; Each sub-image is scaled to obtain a normalized sub-image, and then the normalized sub-image is gray-processed using a maximum value method to obtain a processed sub-image; Extract facial feature points in the processed sub-image, divide the coordinates of the facial feature points by the height and width of the normalized facial region in the sub-image to obtain the facial feature vector of the sub-image, establish an expression discrimination feature model, substitute all facial feature vectors of the sub-image into the expression discrimination feature model, and obtain the local emotional state score of the sub-image; The local emotional state score of the sub-image is multiplied by the weight parameter of the sub-image to obtain the emotional state score of the facial image, thereby obtaining the facial emotional state of the facial image.

[0007] As a further solution of the present invention: the step of establishing the expression discrimination feature model includes: Acquire a first image of a face without expression, and acquire a second image showing changes in facial features during a process in which the face is expressionless to an expression in the video data; Determine facial feature points, and obtain change parameters of the facial feature points based on the first image and the second image; An expression discrimination feature model is established based on the change parameters of facial feature points.

[0008] As a further solution of the present invention: the step of performing semantic analysis, speech rate analysis and intonation analysis on the audio data corresponding to the facial image to obtain the speech emotional state corresponding to the audio data comprises: Performing semantic analysis on the audio data corresponding to the facial image to obtain a semantic score; Perform speech speed analysis on the audio data corresponding to the facial image to obtain a speech speed score; Performing intonation analysis on the audio data corresponding to the facial image to obtain an intonation score; The semantic score, speaking speed score and intonation score are multiplied by their respective weights to obtain a speech emotion score, and the speech emotion state is determined based on the speech emotion score.

[0009] As a further solution of the present invention: the step of confirming the credibility of the facial emotional state and the voice emotional state includes: Identify subtle movements of facial muscles and confirm the credibility of facial emotional states; Identify the prosodic and spectral features in the audio data and substitute them into the I-Vector model to confirm the credibility of the emotional state of the speech.

[0010] The technical solution of the present invention also provides a facial emotion recognition device, the device comprising: Video equipment, used to obtain video data of users; Voice equipment, used to obtain audio data from users; processor; The memory is used to store the facial emotion recognition program of the processor, and the facial emotion recognition program, when executed by the processor, implements the steps of the facial emotion recognition method as described above.

[0011] A computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of the facial emotion recognition method as described above.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention divides the facial image into different sub-images for recognition, and then summarizes the recognition results of the sub-images, so that the obtained facial emotions are more accurate, and then performs semantic analysis, speech rate analysis and intonation analysis on the audio data to obtain the voice emotions, and then compares the facial emotions and voice emotions, so as to ensure the accuracy of the facial emotions and then know the user's true feelings about the advertisement. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0014] Figure 1 A flowchart of a facial emotion recognition method.

[0015] Figure 2 This is a second sub-flow chart of the facial emotion recognition method.

[0016] Figure 3 This is a block diagram of the third sub-process of the facial emotion recognition method.

[0017] Figure 4 This is a partial fourth sub-process flowchart of the facial emotion recognition method.

[0018] Figure 5 The figure is a structural block diagram of the facial emotion recognition device. DETAILED DESCRIPTION

[0019] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] Embodiment 1: Figure 1 The flowchart of the facial emotion recognition method is as follows. In an embodiment of the present invention, a facial emotion recognition method includes: Acquire audio data and video data corresponding to the audio data, and extract a facial image from the video data; Each piece of video data has a corresponding piece of real-time audio data. The combination of the two can more conveniently, accurately and quickly analyze the user's real-time emotions; audio data is used to more accurately judge the emotions expressed in the language, and video data needs to be converted into facial images in order to better analyze facial expressions. By processing the video data in frames, facial images can be obtained, which is convenient for subsequent analysis of facial expressions, and then knowing the user's true feelings about the advertisement.

[0021] Divide the facial image into sub-images of different regions, scale and grayscale each sub-image, extract facial feature points of each sub-image, calculate the local emotional state of each sub-image based on all facial feature points of each sub-image, comprehensively calculate the local emotional states of all sub-images in the same facial image, and obtain the facial emotional state of the facial image; In order to better identify the emotions expressed in facial images, it is necessary to process the facial images so that the facial images can be recognized more accurately. First, the facial images are divided into different sub-images according to different regions. When people express emotions on their faces, they will form different expressions. Facial feature points can be determined based on facial expressions. Since the faces included in each sub-image are from different regions, the facial feature points included in each sub-image are different (for example, the positions of the eyes and mouth are different when happy and angry). By identifying and analyzing the facial feature points in each sub-image, the emotions expressed by the sub-image can be determined, which is called the local emotional state of the sub-image. By combining the local emotional states of all sub-images in the same facial image, the emotional state expressed by the facial image can be obtained.

[0022] The audio data corresponding to the facial image is subjected to semantic analysis, speech rate analysis and intonation analysis to obtain the speech emotional state corresponding to the audio data.

[0023] Identifying the emotions of audio data is to verify facial emotions. By analyzing the emotions expressed by the audio data from different aspects, we can more accurately derive the speech emotion state corresponding to the audio data. Semantics refers to the meaning expressed by the speech, speaking speed refers to the speed of speaking, and intonation refers to the high and low pitch of the speech. By comprehensively analyzing the audio data from three aspects: semantics, speaking speed and intonation, the speech emotion state identified will be more accurate.

[0024] Compare the facial emotional state and the voice emotional state. When the facial emotional state and the voice emotional state are the same, the facial emotional state is the result. When the facial emotional state and the voice emotional state are different, confirm the credibility of the facial emotional state and the voice emotional state. When the credibility of the voice emotional state is greater than the first credibility threshold and the credibility of the facial emotional state is greater than the second credibility threshold, the facial emotion recognition result is the sum of the credibility of the voice emotional state multiplied by the voice weight coefficient plus the credibility of the facial emotional state multiplied by the facial weight coefficient.

[0025] After the facial emotional state and the voice emotional state are identified, the facial emotional state and the voice emotional state need to be mutually verified to further ensure the accuracy of the facial emotional state. When the identified facial emotional state and the voice emotional state are the same, the facial emotional state can be directly used as the result. When the emotions identified by the two methods are inconsistent, the credibility of the facial emotional state and the voice emotional state are confirmed respectively. When the credibility of the voice emotional state recognition result is to be obtained, multiple measurement values ​​of the variable are first obtained through multiple voice emotion recognition processes, and then the mean of the multiple measurement values ​​is used as an estimated value. Then, the probability that the error range between the estimated value and the true value of the variable is within a certain range is estimated by the interval estimation method. The larger the probability value, the more accurate the estimated value, that is, the higher the credibility of the current voice emotional state recognition result. Only when the credibility of the voice emotional state recognition result is greater than the first credibility threshold and the credibility of the facial emotional state recognition result is greater than the second credibility threshold, can the voice emotional state recognition result and the facial emotional state recognition result be confirmed to be valid. At this time, the voice weight coefficient and the face weight coefficient can be set to allocate the influence weight of the voice emotional state recognition result and the facial emotional state recognition result on the final emotional state judgment. For example, if the credibility of the voice emotion state recognition result is X1, the voice weight coefficient is a, the credibility of the facial emotion state recognition result is X2, and the facial weight coefficient is b, then y=aX1+bX2 can be obtained, where a and b are values ​​ranging from 0 to 1. At this time, the final facial emotion recognition result is judged according to the data of y. When the value of y is greater than or equal to the third credibility threshold, the facial emotion state recognition result is taken as the result. When the value of y is less than the third credibility threshold, the voice emotion state recognition result is taken as the result. Based on the results, the user's true feelings about the advertisement can be known, which is convenient for advertising designers to conduct analysis and subsequent improvements.

[0026] Figure 2 This is a second sub-flow diagram of the facial emotion recognition method. In the embodiment of the present invention, the steps of dividing the facial image into sub-images of different regions, scaling and grayscale processing each sub-image, extracting facial feature points of each sub-image, calculating the local emotional state of the sub-image based on all facial feature points of each sub-image, and comprehensively calculating the local emotional states of all sub-images in the same facial image to obtain the facial emotional state of the facial image include: Divide the facial image into sub-images according to the cheek area, eye area, nose area, mouth area and chin area, and mark the facial feature points in each sub-image; The facial image is divided into cheek area, eye area, nose area, mouth area and chin area. Each area has its specific facial feature points, such as the distance between the eyes and the mouth, the shape of the mouth, etc. The positions of these facial feature points will be different under different expressions. The images of these areas can fully form the user's expression. Emotional expression can be analyzed from the expression. Each expression is formed by different facial features, and facial features are composed of different facial feature points. Therefore, marking the facial feature points can confirm the facial features and then determine the facial emotions.

[0027] Each sub-image is scaled to obtain a normalized sub-image, and then the normalized sub-image is gray-processed using a maximum value method to obtain a processed sub-image; In order to facilitate analysis, each sub-image needs to be transformed into an image with the same height and width, so each sub-image needs to be enlarged or reduced, that is, a standardized sub-image. In order to clearly identify the facial feature points in the sub-image, the sub-image is grayed out. Graying is to eliminate the color of the image. Most color images now use the RGB color mode. When processing images, the three RGB components must be processed separately. Using the maximum value method to grayscale the sub-image is to use the maximum value of the brightness of the three RGB components in the color image as the grayscale value of the grayscale image. This method not only avoids visible strip distortion, but also reduces the size of the sub-image file, speeds up recognition, and significantly reduces the network's memory usage on the hardware computing platform, thereby reducing subsequent processing pressure.

[0028] The facial feature points in the processed sub-image are extracted, the coordinates of the facial feature points are divided by the height and width of the facial area in the normalized sub-image to obtain the facial feature vector of the sub-image, an expression discriminant feature model is established, all the facial feature vectors of the sub-image are substituted into the expression discriminant feature model to obtain the local emotional state score of the sub-image.

[0029] By determining all the facial feature points in the sub-image, the coordinates of the facial feature points can be known. Since all the sub-images have unified height and width at this time, the vector composed of the facial feature points in all the sub-images can be known, which is the facial feature vector. The facial feature vector represents the shape and position of the facial features at this time. By substituting the facial feature vector into the expression discrimination feature model for iterative training, the facial emotion score represented by the sub-image can be obtained, which is the local emotional state score of the sub-image.

[0030] The local emotional state score of the sub-image is multiplied by the weight parameter of the sub-image to obtain the emotional state score of the facial image, thereby obtaining the facial emotional state of the facial image.

[0031] Different sub-images represent different areas of the face. Different areas contain different numbers and positions of facial feature points. Therefore, the proportion of the local emotional state score of each sub-image to the emotional score expressed in the facial image is different. This proportion is the weight parameter of the sub-image. The local emotional state scores of all sub-images are multiplied by the weight parameter of the sub-image and the sum of the scores is the emotional state score of the facial image. By comparing the preset rules with the emotional state scores, it can be confirmed which type of facial emotion the facial image belongs to, positive emotion, neutral emotion or negative emotion. Negative emotions include anger, disgust, fear, anxiety, etc., neutral emotions include calmness, surprise, stress, etc., positive emotions include happiness, excitement, etc., and by judging the category of facial emotions, we can continue to determine which specific emotion it is.

[0032] The step of establishing the expression discrimination feature model comprises: Acquire a first image of a face without expression, and acquire a second image showing changes in facial features during a process in which the face is expressionless to an expression in the video data; When the face is expressionless, the position of each facial feature point is fixed. When the face has expression, the position of the facial feature points will change. For example, the distance between the eyebrows will change (brows bent when smiling), the shape of the mouth will change (pursed lips), the distance between the eyes and the mouth will also change (upturned lips), etc. By obtaining an image of the face with expression, we can know the position of the facial feature points at this time, and also know the amount of change of the facial feature points. These expressions can indicate the emotions expressed, so the change of facial feature points can indicate the change of emotions.

[0033] Facial feature points are determined, and change parameters of the facial feature points are obtained based on the first image and the second image.

[0034] The change in the position of the facial feature points forms different facial expressions, and the user's emotions can be judged based on the facial expressions. The changes in the facial feature points in the first image and the second image can be used to judge the changes in the user's expressions, and further the changes in the user's emotions can be judged.

[0035] An expression discrimination feature model is established based on the change parameters of facial feature points.

[0036] The position change of different facial feature points relative to the initial facial feature points can form a one-to-one correspondence with the expression, and then achieve a one-to-one correspondence with the emotion. Based on this principle, a large number of images corresponding to different emotions can be selected to establish an expression discrimination feature model, that is, the one-to-one correspondence between facial feature points and emotions. Here, the facial expressions can be divided into training sets and test sets, and the ratio can be 4:1. The expression discrimination feature model is established using the training set and the binary neural network, and then the trained expression discrimination feature model is used to test the test set. The recognition effect of the expression discrimination feature model is evaluated based on the test results. The binary neural network refers to a neural network that only uses 1 and -1 values ​​to represent weights and excitations. The binary neural network will significantly reduce the network's memory usage on the hardware computing platform and quickly obtain results. When the evaluation result shows that the expression discrimination feature model is effective, the expression discrimination feature model can be put into use.

[0037] Figure 3 The third sub-flow chart of the facial emotion recognition method, wherein the steps of performing semantic analysis, speech rate analysis and intonation analysis on the audio data corresponding to the facial image to obtain the speech emotion state corresponding to the audio data include: Performing semantic analysis on the audio data corresponding to the facial image to obtain a semantic score; The audio data is denoised to reduce the impact of noise on the analysis, and then input into a pre-trained semantic recognition model for recognition and extraction of semantic feature parameters. The semantic feature parameters are feature parameters that can reflect the essence of semantics and can output the emotional tendency of the current text, that is, the semantic emotion scores corresponding to the positive, negative, and neutral emotional tendencies contained in the text. The semantic emotion scores that can intuitively reflect each emotional tendency and ensure the accuracy of semantic emotion recognition can be used to derive the true ideas that the audio data wants to express and evaluate the semantic score.

[0038] Perform speech speed analysis on the audio data corresponding to the facial image to obtain a speech speed score; The user's speaking speed is different under different emotions. The speaking speed will increase when the user is excited, and will slow down when the user is depressed. The audio data is substituted into the pre-trained speech speed recognition model to obtain the speech speed score of the audio data.

[0039] Performing intonation analysis on the audio data corresponding to the facial image to obtain an intonation score; The same words but different intonations express different emotions. The emotions expressed by saying a word in a low and slow voice are different from those expressed by saying the same word in a high and fast voice. By substituting the audio data into a pre-trained intonation recognition model, the emotional tendency of the audio data can be obtained, which can intuitively reflect the intonation emotion scores of each emotion type, thereby obtaining the intonation score.

[0040] The semantic score, speaking speed score and intonation score are multiplied by their respective weights to obtain a speech emotion score, and the speech emotion state is determined based on the speech emotion score.

[0041] Judging the user's emotions from audio data requires a comprehensive judgment from three aspects: semantics, speaking speed and intonation, so that the user's emotional judgment is more accurate. The proportion of semantics to the importance of speech is the first weight, the proportion of speaking speed to the importance of speech is the second weight, and the proportion of intonation to the importance of speech is the third weight. The sum of the semantic score multiplied by the first weight, the speaking speed score multiplied by the second weight, and the intonation score multiplied by the third weight is the speech emotion score, so as to accurately judge the emotional state expressed by the audio data.

[0042] Figure 4 This is a partial fourth sub-flow diagram of the facial emotion recognition method, wherein the step of confirming the credibility of the facial emotion state and the voice emotion state includes: Identify subtle movements of facial muscles and confirm the credibility of facial emotional states; Because facial expressions are formed by the movement of facial muscles, muscle movement will form various features on the face, such as wrinkles. If the user only moves the zygomatic major muscle when smiling, the user is faking a smile, because a sincere smile requires the use of both the zygomatic major muscle and the orbicularis oculi muscle. Therefore, paying attention to the subtle movements of the facial muscles can confirm whether the facial emotional state is real.

[0043] Identify the prosodic and spectral features in the audio data and substitute them into the I-Vector model to confirm the credibility of the emotional state of the speech.

[0044] Prosodic features are also known as "super-linguistic features". Their ability to distinguish emotions has been confirmed by researchers in the field of speech emotions. Common prosodic features include duration, fundamental frequency, energy, etc. Spectral features are a reflection of the correlation between changes in vocal tract shape and vocal movements, and have been successfully applied in the field of speech signal processing. For example, speech expressing happy emotions shows high energy in the high frequency band, and speech expressing sad emotions shows significantly different low energy in the same frequency band. Common spectral features include LPC, LFPC, etc. Sound quality features are used to measure whether the speech is pure, clear, and easy to recognize, and are closely related to the expression of speech emotions. Common sound quality features include resonance peak frequency and its bandwidth, frequency perturbation, amplitude perturbation, etc. The I-Vector model models global differences and models prosodic features, spectral features, etc. as a whole, relaxing restrictions on audio data, making calculations simple and obtaining accurate results, thereby obtaining the credibility of the emotional state of speech.

[0045] Embodiment 2: Figure 5The structure block diagram of a facial emotion recognition device is shown in FIG. 1 . In an embodiment of the present invention, a facial emotion recognition device includes: Video equipment, used to obtain video data of users; Voice equipment, used to obtain audio data from users; processor; The memory is used to store the facial emotion recognition program of the processor, and the facial emotion recognition program, when executed by the processor, implements the steps of the facial emotion recognition method as described above.

[0046] A computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of the facial emotion recognition method as described above.

[0047] The functions that can be achieved by the facial emotion recognition method are all completed by a computer device, which includes one or more processors and one or more memories, and the one or more memories store at least one program code. The program code is loaded and executed by the one or more processors to achieve the functions of the user behavior prediction method based on big data.

[0048] The processor takes out instructions from the memory one by one, analyzes the instructions, and then completes the corresponding operations according to the instruction requirements, generating a series of control commands, so that the various parts of the computer can automatically, continuously and coordinately move to become an organic whole, realize the input of programs, the input of data, and the calculation and output of results. The arithmetic operations or logical operations generated in this process are all completed by the operator; the memory includes a read-only memory (ROM), which is used to store computer programs, and a protection device is provided outside the memory.

[0049] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0050] Those skilled in the art will understand that the description of the above service equipment is merely an example and does not constitute a limitation on the terminal equipment. It may include more or fewer components than described above, or a combination of certain components, or different components, for example, it may include input and output devices, network access equipment, buses, etc.

[0051] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire user terminal.

[0052] The above-mentioned memory can be used to store computer programs and / or modules. The above-mentioned processor realizes various functions of the above-mentioned terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an information collection template display function, a product information release function, etc.), etc.; the data storage area can store data created according to the use of the berth status display system (such as product information collection templates corresponding to different product types, product information that different product providers need to release, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0053] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the modules / units in the above-mentioned embodiment system, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can realize the functions of the above-mentioned various system embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media can include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0054] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0055] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A facial emotion recognition method, characterized in that: The method comprises: Acquire audio data and video data corresponding to the audio data, and extract a facial image from the video data; Divide the facial image into sub-images of different regions, scale and grayscale each sub-image, extract facial feature points of each sub-image, calculate the local emotional state of each sub-image based on all facial feature points of each sub-image, comprehensively calculate the local emotional states of all sub-images in the same facial image, and obtain the facial emotional state of the facial image; Performing semantic analysis, speech rate analysis and intonation analysis on the audio data corresponding to the facial image to obtain the speech emotional state corresponding to the audio data; Compare the facial emotional state and the voice emotional state. When the facial emotional state and the voice emotional state are the same, the facial emotional state is the result. When the facial emotional state and the voice emotional state are different, confirm the credibility of the facial emotional state and the voice emotional state. When the credibility of the voice emotional state is greater than the first credibility threshold and the credibility of the facial emotional state is greater than the second credibility threshold, the facial emotion recognition result is the sum of the credibility of the voice emotional state multiplied by the voice weight coefficient plus the credibility of the facial emotional state multiplied by the facial weight coefficient.

2. The facial emotion recognition method according to claim 1, characterized in that: The steps of dividing the facial image into sub-images of different regions, scaling and grayscale processing each sub-image, extracting facial feature points of each sub-image, calculating the local emotional state of the sub-image based on all facial feature points of each sub-image, and comprehensively calculating the local emotional states of all sub-images in the same facial image to obtain the facial emotional state of the facial image include: Divide the facial image into sub-images according to the cheek area, eye area, nose area, mouth area and chin area, and mark the facial feature points in each sub-image; Each sub-image is scaled to obtain a normalized sub-image, and then the normalized sub-image is gray-processed using a maximum value method to obtain a processed sub-image; Extract facial feature points in the processed sub-image, divide the coordinates of the facial feature points by the height and width of the normalized facial region in the sub-image to obtain the facial feature vector of the sub-image, establish an expression discrimination feature model, substitute all facial feature vectors of the sub-image into the expression discrimination feature model, and obtain the local emotional state score of the sub-image; The local emotional state score of the sub-image is multiplied by the weight parameter of the sub-image to obtain the emotional state score of the facial image, thereby obtaining the facial emotional state of the facial image.

3. The facial emotion recognition method according to claim 2, characterized in that: The step of establishing the expression discrimination feature model comprises: Acquire a first image of a face without expression, and acquire a second image showing changes in facial features during a process in which the face is expressionless to an expression in the video data; Determine facial feature points, and obtain change parameters of the facial feature points based on the first image and the second image; An expression discrimination feature model is established based on the change parameters of facial feature points.

4. The facial emotion recognition method according to claim 1, characterized in that: The step of performing semantic analysis, speech rate analysis and intonation analysis on the audio data corresponding to the facial image to obtain the voice emotional state corresponding to the audio data comprises: Performing semantic analysis on the audio data corresponding to the facial image to obtain a semantic score; Perform speech speed analysis on the audio data corresponding to the facial image to obtain a speech speed score; Performing intonation analysis on the audio data corresponding to the facial image to obtain an intonation score; The semantic score, speaking speed score and intonation score are multiplied by their respective weights to obtain a speech emotion score, and the speech emotion state is determined based on the speech emotion score.

5. The facial emotion recognition method according to claim 1 or 4, characterized in that: The step of confirming the credibility of the facial emotional state and the voice emotional state comprises: Identify subtle movements of facial muscles and confirm the credibility of facial emotional states; Identify the prosodic and spectral features in the audio data and substitute them into the I-Vector model to confirm the credibility of the emotional state of the speech.

6. A facial emotion recognition device, characterized in that: The device comprises: Video equipment, used to obtain video data of users; Voice equipment, used to obtain audio data from users; processor; The memory is used to store a facial emotion recognition program of the processor, and when the facial emotion recognition program is executed by the processor, the steps of the facial emotion recognition method as described in any one of claims 1 to 5 are implemented.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the facial emotion recognition method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Emotion recognition method and device, electronic equipment and storage medium

    CN114120425A

  • Emotion recognition method and device based on facial image and computer storage medium

    CN115331292A

  • Multi-modal emotion recognition method and system

    CN116189669A

  • Server, display device and voice interaction method

    CN117809616A

  • Face emotion recognition method and device, equipment and storage medium

    CN118609187A

Cited By

  • Voice interaction method and device, equipment and storage medium

    CN119673160A

  • Intelligent transaction risk prevention and control method and device based on emotion recognition

    CN120598569A

  • Character emotion recognition method and device

    CN121483308A