Emotion recognition system and method based on micro expressions and macro expressions, equipment and medium

Abstract

The invention provides an emotion recognition system and method based on micro expressions and macro expressions, equipment and a medium. The method comprises the following steps: detecting and positioning a human face from input data; performing facial feature extraction on the detected face; performing micro-expression recognition and macro-expression recognition on the extracted facial features; and analyzing the emotional state according to the micro-expression recognition result and the macro-expression recognition result. According to the method, the micro-expressions of the figures in the video or the real-time image can be efficiently and accurately recognized, the corresponding emotional state is analyzed according to the micro-expressions, powerful technical support is provided for the fields of psychological counseling, man-machine interaction, safety monitoring and the like, the intelligent level of related applications can be improved, the emotion analysis result is visually displayed, and the user experience is improved. A user can conveniently and intuitively know emotion changes of a character, and user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and emotional computing technology, and in particular to an emotion recognition system, method, device, and medium based on micro-expressions and macro-expressions. Background Art

[0002] Traditional emotion recognition methods rely solely on macro-level changes in facial expressions, failing to capture the fleeting and subtle changes in micro-expressions. Micro-expressions, as a true manifestation of human emotion, often unconsciously reveal inner feelings. Therefore, developing a system that can efficiently and accurately recognize micro-expressions and use them as a supplementary tool for analyzing macro-expressions is crucial. Summary of the Invention

[0003] In order to achieve the above-mentioned purpose and other advantages of the present invention, a first object of the present invention is to provide an emotion recognition method based on micro-expressions and macro-expressions, comprising the following steps: Detect and locate faces from input data; Extract facial features from detected faces; Perform micro-expression recognition and macro-expression recognition on the extracted facial features; Analyze emotional states based on micro-expression recognition results and macro-expression recognition results.

[0004] Furthermore, the input data includes real-time video stream or static image.

[0005] Furthermore, the step of detecting and locating a face from the input data includes: The face in the real-time video stream or the static image is located using a face detection algorithm, and the face coordinates are returned.

[0006] Furthermore, the face detection algorithm adopts an Adaboost cascade face detection classifier based on Haar features.

[0007] Furthermore, the step of extracting facial features from the detected face includes: Preprocess the detected faces using a pre-trained face detection model; Use facial feature extraction algorithm to extract key facial feature points; Estimate the key points of the face through the shape predictor; Multiple standard facial expression units are formed through the key points of the face.

[0008] Furthermore, the step of performing micro-expression recognition on the extracted facial features includes: For each image containing a face, the activity levels of multiple standard expression units are calculated based on the positional relationship of key points.

[0009] Furthermore, macro expression recognition of the extracted facial features includes: A macro-expression classification model is used to classify people's macro-expressions according to the combination of expression units.

[0010] Furthermore, the step of analyzing the emotional state based on the micro-expression recognition results and the macro-expression recognition results includes: The macro-expression and micro-expression features are fused and input into a fusion recognition model for emotional state analysis.

[0011] Furthermore, the method further comprises the steps of: The emotional state analysis results are displayed in a visual form on a video or image for users to view and analyze.

[0012] The second object of the present invention is to provide an emotion recognition system based on micro-expressions and macro-expressions, which applies the above method and includes a face detection module, a facial feature extraction module, an expression recognition module, and an emotion analysis module; wherein, The face detection module is used to detect and locate faces from input data; The facial feature extraction module is used to extract facial features from the detected face; The expression recognition module is used to perform micro-expression recognition and macro-expression recognition on the extracted facial features; The emotion analysis module is used to analyze the emotional state based on the micro-expression recognition results and the macro-expression recognition results.

[0013] A third object of the present invention is to provide a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0014] A fourth object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides an emotion recognition system, method, device, and medium based on micro-expressions and macro-expressions. The method comprises the following steps: detecting and locating a human face from input data; extracting facial features from the detected face; performing micro-expression recognition and macro-expression recognition on the extracted facial features; and analyzing the emotional state based on the micro-expression and macro-expression recognition results. The present invention can efficiently and accurately identify a person's micro-expressions in video or real-time images and analyze the corresponding emotional state accordingly. This provides strong technical support for fields such as psychological counseling, human-computer interaction, and security monitoring, helping to enhance the intelligence level of related applications. By visually displaying the emotion analysis results, users can intuitively understand the person's emotional changes and improve the user experience.

[0016] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and to implement it according to the contents of the description, the following preferred embodiments of the present invention are described in detail with reference to the accompanying drawings. The specific implementation methods of the present invention are given in detail by the following embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 This is a flow chart of the emotion recognition method based on micro-expressions and macro-expressions of Example 1; Figure 2 This is a flow chart of facial feature extraction for a detected face in Example 1; Figure 3 This is a schematic diagram of the calculation results of the activity level of the standard expression unit in Example 1; Figure 4 Schematic diagram of macro-expression and micro-expression feature fusion recognition in Example 1; Figure 5 Schematic diagram of the emotion recognition system based on micro-expressions and macro-expressions of Example 2; Figure 6 This is a schematic diagram of the computer equipment of Example 3; Figure 7 Schematic diagram of a computer-readable storage medium of Example 4. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. It should be noted that, without conflict, the embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0019] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0020] The figure numbers in this application are only used to distinguish the various steps in the scheme and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0022] Example 1 An emotion recognition method based on micro-expressions and macro-expressions, such as Figure 1 As shown, the method includes the following steps: S1. Detect and locate faces from input data; specifically, detect and locate faces from videos or real-time images.

[0023] In some embodiments, the input data includes a real-time video stream or a still image. For example, the system receives a real-time video stream or a still image from a camera or other video source.

[0024] In some embodiments, the step of detecting and locating a face from input data includes: The face in the real-time video stream or the static image is quickly located using a face detection algorithm, and the face coordinates are returned. Specifically, the face detection algorithm uses an Adaboost cascade face detection classifier based on Haar features.

[0025] S2, extract facial features of the detected face, such as accurate extraction of eyes, nose, mouth, etc.; In some embodiments, as Figure 2 As shown, the step of extracting facial features from the detected face includes: S21. Preprocess the detected face using a pre-trained facial detection model. The pre-trained facial detection model may be a dlib library of the Python machine learning open source library. The dlib library provides a pre-trained facial detection model.

[0026] S22, using facial feature extraction algorithms (such as HOG features, LBP features, etc.) to extract facial key feature points; S23. Estimate the locations of facial key points using a shape predictor. The shape predictor can be provided by the dlib library. The shape predictor is further used to estimate the locations of facial key points, such as the outline points of the eyes, nose, and mouth. These key points not only provide a benchmark for subsequent feature extraction but also help calibrate the facial image, reducing errors caused by factors such as angle and size.

[0027] S24. Form multiple standard facial expression units using key points of the face. Specifically, the facial feature points returned by the dlib library constitute multiple facial expression units (AUs), which are the basic units for facial expression analysis.

[0028] Expression units (AUs) describe the degree of activity in different parts of a person's face during an expression. These units are considered to be independently analyzable and recognizable and are often used as the building blocks of facial expression recognition systems. The meaning of each AU is as follows: AU1: Inner Brow Lift, describes the degree to which the eyebrows move upward.

[0029] AU2: Outer Brow Lift, describes the degree to which the eyebrows move upward.

[0030] AU4: Eyebrow ptosis, describes the degree to which the eyebrows move downward.

[0031] AU5: Upper eyelid lift, describes the degree to which the upper eyelid moves upward.

[0032] AU6: Zygomatic lift, describes the degree to which the zygomatic bones move upward.

[0033] AU7: Eyelid tightening, describes the degree of eyelid tightening.

[0034] AU9: Nose wrinkles, describing the degree of wrinkling of the nose.

[0035] AU10: Upper lip lift, describes the degree to which the upper lip moves upward.

[0036] AU12: Mouth corner pull, describes the degree to which the corners of the mouth move upward.

[0037] AU14: Cheekbone Recession, describes the degree of cheekbone depression.

[0038] AU15: Mouth corner droop, describes the degree to which the corners of the mouth move downward.

[0039] AU17: Chin lift, describes the degree to which the chin moves upward.

[0040] AU20: Lip extension, describing the lips being pulled laterally and becoming flattened.

[0041] AU23: Lips pursed, describes the degree of lip tightening.

[0042] AU25: Lips parted, describe the degree to which the lips are parted.

[0043] AU26: Chin drop, describes the degree to which the chin moves downward.

[0044] AU45: Blinking, describe the number of times you blink.

[0045] Multiple expression units can be combined to express various micro-expressions. For example, AU12, AU06, and AU25 can be combined to represent a smile; AU15 and AU17 can be combined to represent a pout. For example: Double smile = (au_regs["AU12"] + au_regs["AU06"]+ au_regs["AU25"]) / 13.0; Double frown = (au_regs["AU15"] + au_regs["AU17"]) / 12.0; Double brow_up = (au_regs["AU01"] + au_regs["AU02"]) / 10.0; Double brow_down = au_regs["AU04"] / 5.0; Double eye_widen = au_regs["AU05"] / 3.0; Double nose_wrinkle = au_regs["AU09"] / 4.0.

[0046] S3. Perform micro-expression recognition and macro-expression recognition on the extracted facial features; specifically, perform micro-expression recognition on the extracted facial features based on a deep learning algorithm; and perform macro-expression recognition on the extracted facial features based on a deep learning algorithm.

[0047] In some embodiments, the step of performing micro-expression recognition on the extracted facial features includes: For each image containing a face, the activity levels of multiple standard expression units are calculated based on the position relationship of key points. For example, the activity levels of 45 AUs such as AU1-AU45 are calculated. The calculation results are as follows: Figure 3 shown.

[0048] In combination with the above embodiment, performing macro expression recognition on the extracted facial features includes: A macro-expression classification model is used to classify a person's macro-expressions based on the combination of expression units, such as six common macro-expressions: sadness, disgust, anger, calmness, fear, happiness, and surprise. The macro-expression classification model can be constructed based on a deep learning algorithm or leverage an existing macro-expression classification model.

[0049] S4. Analyze emotional state based on micro-expression recognition results and macro-expression recognition results.

[0050] In some embodiments, as Figure 4 As shown, the step of analyzing the emotional state based on the micro-expression recognition results and the macro-expression recognition results includes: The macro-expression and micro-expression features are fused and input into a fusion recognition model for emotional state analysis.

[0051] This embodiment fuses micro-expression and macro-expression features to improve the accuracy and granularity of emotion recognition. Emotional state analysis is performed using a fusion recognition model that takes into account the interactions and relationships between micro-expressions and macro-expressions.

[0052] In some embodiments, the steps further include: The emotional state analysis results are displayed on the video or image in a visual form (such as emotional labels, timeline markers, etc.) for users to view and analyze.

[0053] This embodiment provides an emotion recognition method based on micro-expressions and macro-expressions. The method includes the following steps: detecting and locating human faces from input data; extracting facial features from the detected faces; performing micro-expression recognition and macro-expression recognition on the extracted facial features; and analyzing emotional states based on the micro-expression and macro-expression recognition results. This embodiment uses deep learning algorithms and image processing techniques to accurately recognize micro-expressions in videos or real-time images, combined with macro-expression analysis, to derive corresponding emotional states.

[0054] Example 2 An emotion recognition system based on micro-expressions and macro-expressions, using the above method. For a detailed description of the method, please refer to the corresponding description in the above method embodiment, which will not be repeated here. Figure 5 As shown, the system includes a face detection module 51, a facial feature extraction module 52, an expression recognition module 53, and an emotion analysis module 54; wherein, The face detection module is used to detect and locate faces from input data; The facial feature extraction module is used to extract facial features from the detected face; The expression recognition module is used to perform micro-expression recognition and macro-expression recognition on the extracted facial features; The emotion analysis module is used to analyze the emotional state based on the micro-expression recognition results and the macro-expression recognition results.

[0055] Based on the technical solution of the above embodiment, optionally, the input data includes a real-time video stream or a static image.

[0056] Based on the technical solution of the above embodiment, optionally, the step of detecting and locating a face from the input data includes: The face in the real-time video stream or the static image is located using a face detection algorithm, and the face coordinates are returned.

[0057] Based on the technical solution of the above embodiment, optionally, the face detection algorithm adopts an Adaboost cascade face detection classifier based on Haar features.

[0058] Based on the technical solution of the above embodiment, optionally, the step of extracting facial features from the detected face includes: Preprocess the detected faces using a pre-trained face detection model; Use facial feature extraction algorithm to extract key facial feature points; Estimate the key points of the face through the shape predictor; Multiple standard facial expression units are formed through the key points of the face.

[0059] Based on the technical solution of the above embodiment, optionally, the step of performing micro-expression recognition on the extracted facial features includes: For each image containing a face, the activity levels of multiple standard expression units are calculated based on the positional relationship of key points.

[0060] Based on the technical solution of the above embodiment, optionally, performing macro expression recognition on the extracted facial features includes: A macro-expression classification model is used to classify people's macro-expressions according to the combination of expression units.

[0061] Based on the technical solution of the above embodiment, optionally, the step of analyzing the emotional state according to the micro-expression recognition results and the macro-expression recognition results includes: The macro-expression and micro-expression features are fused and input into a fusion recognition model for emotional state analysis.

[0062] Based on the technical solution of the above embodiment, optionally, the method further includes the following steps: The emotional state analysis results are displayed in a visual form on a video or image for users to view and analyze.

[0063] This embodiment provides an emotion recognition system based on micro- and macro-expressions. The system includes a face detection module, a facial feature extraction module, an expression recognition module, and an emotion analysis module. This embodiment automatically captures, recognizes, and analyzes micro-expressions in videos or real-time images, providing powerful support for fields such as psychological counseling, human-computer interaction, and security monitoring.

[0064] Example 3 A computer device 600, such as Figure 6 As shown, the present invention includes a memory 610, a processor 620, and a computer program 630 stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for emotion recognition based on micro-expressions and macro-expressions are implemented. For a detailed description of the method, please refer to the corresponding description in the above method embodiment and will not be repeated here.

[0065] Example 4 A computer-readable storage medium such as Figure 7 As shown, a computer program is stored thereon, and when the computer program is executed by the processor, the steps of a method for emotion recognition based on micro-expressions and macro-expressions are implemented. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiment, and no further details will be given here.

[0066] Example 5 A computer program product, comprising a computer program that, when executed by a processor, implements the steps of a method for emotion recognition based on micro-expressions and macro-expressions. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiment and will not be repeated here.

[0067] The number of devices and processing scales described herein are intended to simplify the description of the present invention. Applications, modifications, and variations of the present invention will be readily apparent to those skilled in the art.

[0068] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0069] The apparatus, computer device, non-volatile computer storage medium, and method provided in the embodiments of this specification correspond to each other. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0070] Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, it is entirely possible to implement the same functionality by programming the method steps logically, such as through logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered structures within the hardware component. Alternatively, the devices for implementing various functions can be considered both software units implementing the method and structures within the hardware component.

[0071] The systems, devices, or units described in the above embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function, with each unit described separately. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware components.

[0072] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, the embodiments of this specification may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0076] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0077] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0078] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0079] The foregoing is merely an example of the present invention and is not intended to limit the present invention to one or more embodiments. It will be apparent to those skilled in the art that various modifications and variations may be made to the present invention to one or more embodiments. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention to one or more embodiments shall be included within the scope of the claims of the present invention to one or more embodiments.

Claims

1. An emotion recognition method based on micro-expressions and macro-expressions, characterized in that: The following steps are involved: Detect and locate faces from input data; Extract facial features from detected faces; Perform micro-expression recognition and macro-expression recognition on the extracted facial features; Analyze emotional state based on micro-expression recognition results and macro-expression recognition results.

2. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 1, characterized in that: The input data includes real-time video stream or static images.

3. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 2, characterized in that: The step of detecting and locating a face from input data comprises: The face in the real-time video stream or the static image is located by a face detection algorithm, and the face coordinates are returned.

4. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 3, characterized in that: The face detection algorithm adopts Adaboost cascade face detection classifier based on Haar features.

5. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 1, characterized in that: The step of extracting facial features from the detected face comprises: Preprocess the detected faces using a pre-trained face detection model; Extract key facial feature points using facial feature extraction algorithm; Estimate the key point locations of the face through the shape predictor; Multiple standard facial expression units are formed through the key points of the face.

6. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 5, characterized in that: The step of performing micro-expression recognition on the extracted facial features comprises: For each image containing a human face, the activity levels of multiple standard expression units are calculated based on the positional relationship of key points.

7. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 6, characterized in that: Macro expression recognition of extracted facial features includes: The macro-expression classification model is used to classify people's macro-expressions according to the combination of expression units.

8. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 7, characterized in that: The step of analyzing the emotional state according to the micro-expression recognition results and the macro-expression recognition results comprises: The macro-expression and micro-expression features are fused and input into a fusion recognition model for emotional state analysis.

9. The method for emotion recognition based on micro-expressions and macro-expressions as claimed in claim 1, characterized in that: Also includes the steps: The emotional state analysis results are displayed in a visual form on a video or image for users to view and analyze.

10. An emotion recognition system based on micro-expressions and macro-expressions, using the method according to any one of claims 1 to 9, characterized in that: It includes face detection module, facial feature extraction module, expression recognition module and emotion analysis module; among them, The face detection module is used to detect and locate faces from input data; The facial feature extraction module is used to extract facial features from the detected face; The expression recognition module is used to perform micro-expression recognition and macro-expression recognition on the extracted facial features; The emotion analysis module is used to analyze the emotional state according to the micro-expression recognition results and the macro-expression recognition results.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images