Emotion recognition method and device, electronic equipment and storage medium

Abstract

The invention relates to an emotion recognition method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining image data of a target object; wherein the image data comprises a face image and a hand image; determining a hand posture of the target object according to the hand image; determining a facial expression of the target object according to the facial image; recognizing a preliminary emotion of the target object according to the hand posture; and correcting the initial emotion according to the facial expression to obtain a final recognition emotion of the target object. According to the method, the hand posture can be determined through the hand image, the initial emotion of the target object is recognized according to the hand posture, the facial expression of the target object can be determined according to the facial image, the initial emotion is corrected according to the facial expression, and the final recognized emotion is obtained. The final recognition emotion is the emotion obtained by performing multi-dimensional recognition in combination with the hand posture and the facial expression, so that the accuracy of emotion recognition is improved.

Description

Technical Field

[0001] The present application relates to the technical field of emotion recognition, and in particular to an emotion recognition method, device, electronic device and storage medium. Background Art

[0002] In existing emotion recognition technologies, a single signal source is usually used to analyze an individual's emotional state, such as through facial expression recognition or voice recognition. However, analyzing emotional states with a single signal source has certain limitations. For example, since facial expressions may be artificially controlled or concealed, analysis based solely on facial expressions may not accurately reflect the user's true emotions. In social situations, people often smile, even though they may feel anxious or uneasy inside. Therefore, relying solely on facial expressions may lead to deviations in emotion recognition results. Voice emotion recognition relies on acoustic features, which are easily affected by the external environment (such as noise), the user's physiological state (such as fatigue), and the content of the speech. Different languages ​​and dialects also increase the difficulty of voice emotion recognition, limiting its application in cross-cultural and multilingual environments. Therefore, how to accurately identify the emotions of the target object has become a difficult problem. Summary of the invention

[0003] The present application provides an emotion recognition method, device, electronic device and storage medium to solve the technical problem of how to accurately judge the emotional state of a target object.

[0004] In a first aspect, the present application provides an emotion recognition method, the method comprising:

[0005] Acquire image data of the target object; wherein the image data includes a facial image and a hand image;

[0006] determining a hand posture of the target object according to the hand image;

[0007] determining the facial expression of the target object according to the facial image;

[0008] identifying a preliminary emotion of the target object according to the hand gesture;

[0009] The preliminary emotion is modified according to the facial expression to obtain a final recognized emotion of the target object.

[0010] Optionally, the hand image includes a hand color image and a hand depth image;

[0011] The step of determining the hand posture of the target object according to the hand image comprises:

[0012] Aligning pixels of the hand color image and the hand depth image;

[0013] Performing hand key point detection on the hand color image to obtain two-dimensional coordinate information of the hand key points;

[0014] Extracting depth information from the hand depth image according to the two-dimensional coordinate information of the key points of the hand to obtain the three-dimensional coordinate information of the key points of the hand;

[0015] The hand posture of the target object is determined according to the three-dimensional coordinate information of the hand key points.

[0016] Optionally, the step of performing hand key point detection on the hand color image to obtain two-dimensional coordinate information of the hand key points includes:

[0017] Performing palm detection on the hand color image using a palm detection model to obtain a hand positioning frame for locating the palm position;

[0018] Inputting the target image corresponding to the hand positioning frame into a hand key point detection model to predict the hand key points in the target image;

[0019] Rendering the hand key points to the hand color image;

[0020] The two-dimensional coordinate information of the corresponding positions of the key points of the hand is extracted from the hand color image.

[0021] Optionally, the step of extracting depth information from the hand depth image according to the two-dimensional coordinate information of the hand key points to obtain the three-dimensional coordinate information of the hand key points includes:

[0022] Extracting depth information corresponding to the two-dimensional coordinate information from the hand depth image aligned with pixels of the hand color image according to the two-dimensional coordinate information of the hand key points;

[0023] Convert the two-dimensional coordinate information and the depth information into three-dimensional point coordinates in a camera coordinate system;

[0024] The three-dimensional point coordinates are used as the three-dimensional coordinate information of the hand key points.

[0025] Optionally, the step of determining the hand posture of the target object according to the three-dimensional coordinate information of the hand key points includes:

[0026] Determine left hand key points and right hand key points from the hand key points;

[0027] Determine the coordinates of the left hand center point based on the left hand key points, and determine the coordinates of the right hand center point based on the right hand key points;

[0028] Determine the distance between the two hands according to the coordinates of the left hand center point and the coordinates of the right hand center point;

[0029] Determining a hand motion according to the left hand key points and the right hand key points;

[0030] Using the hand movement and the distance between the two hands as the hand posture of the target object;

[0031] Accordingly, the step of identifying the preliminary emotion of the target object according to the hand gesture comprises:

[0032] Obtaining a preset mapping relationship; wherein the mapping relationship is a mapping relationship between hand gestures and emotions;

[0033] The preliminary emotion of the target object is estimated according to the hand motion, the distance between the two hands and the mapping relationship.

[0034] Optionally, the step of determining the facial expression of the target object according to the facial image comprises:

[0035] Performing facial detection on the facial image and extracting facial key points;

[0036] Extracting facial coordinates of the facial key points;

[0037] Determining expression features according to the facial coordinates of the facial key points;

[0038] A facial expression is determined based on the expression features and a pre-trained machine learning model.

[0039] Optionally, the method further comprises:

[0040] generating emotion guidance information according to the final recognized emotion of the target object;

[0041] The emotion guidance information is fed back to the target object.

[0042] In a second aspect, the present application provides an emotion recognition device, the device comprising:

[0043] An acquisition module, used to acquire image data of a target object; wherein the image data includes a facial image and a hand image;

[0044] A first determining module, configured to determine a hand posture of the target object according to the hand image;

[0045] A second determination module, configured to determine the facial expression of the target object according to the facial image;

[0046] A first recognition module, configured to recognize a preliminary emotion of the target object according to the hand gesture;

[0047] The second recognition module is used to modify the preliminary emotion according to the facial expression to obtain the final recognized emotion of the target object.

[0048] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0049] Memory, used to store computer programs;

[0050] The processor is used to implement the emotion recognition method described in any embodiment of the first aspect when executing the program stored in the memory.

[0051] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the emotion recognition method as described in any one of the embodiments of the first aspect is implemented.

[0052] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application obtains the image data of the target object; wherein the image data includes a facial image and a hand image; the hand posture of the target object is determined according to the hand image; the facial expression of the target object is determined according to the facial image; the preliminary emotion of the target object is identified according to the hand posture; the preliminary emotion is corrected according to the facial expression to obtain the final recognized emotion of the target object. This method can determine the hand posture through the hand image, and then identify the preliminary emotion of the target object according to the hand posture, and can determine the facial expression of the target object according to the facial image, and then correct the preliminary emotion according to the facial expression to obtain the final recognized emotion. Since the final recognized emotion is the emotion obtained by multi-dimensional recognition combining the hand posture and the facial expression, the accuracy of emotion recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0056] Figure 1 A system architecture diagram of an emotion recognition method provided for one embodiment of the present application;

[0057] Figure 2 A flowchart of an emotion recognition method provided by an embodiment of the present application;

[0058] Figure 3 A schematic diagram of an emotion recognition system provided by an embodiment of the present application;

[0059] Figure 4 A schematic diagram of an emotion recognition method provided by an embodiment of the present application;

[0060] Figure 5 A schematic diagram of the structure of an emotion recognition device provided by one embodiment of the present application;

[0061] Figure 6 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0063] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0064] In order to solve the technical problem of how to accurately judge the emotional state of a target object in the prior art, the present application provides an emotion recognition method, device, electronic device and storage medium, which can perform multi-dimensional recognition of the target object's emotions in combination with hand gestures and facial expressions, thereby improving the accuracy of emotion recognition.

[0065] The first embodiment of the present application provides an emotion recognition method, which can be applied to Figure 1 The system architecture shown includes at least a data acquisition module 101 and a data processing module 102 , and the data acquisition module 101 and the data processing module 102 establish a communication connection.

[0066] Next, the emotion recognition method is described in detail based on the system architecture. Figure 2 , the emotion recognition method comprises:

[0067] Step 201, obtaining image data of a target object; wherein the image data includes a facial image and a hand image.

[0068] The image data of the target object can be collected by a data collection module such as a camera. The image data can be a single frame image or a continuous multi-frame image. The image data includes a facial image and a hand image of the target object. The target object can be a user for emotion recognition.

[0069] Step 202: determine the hand posture of the target object according to the hand image.

[0070] In one embodiment, the hand image includes a hand color image and a hand depth image.

[0071] The steps of determining the hand posture of the target object according to the hand image include: aligning the pixels of the hand color image and the hand depth image; detecting the hand key points on the hand color image to obtain the two-dimensional coordinate information of the hand key points; extracting the depth information from the hand depth image according to the two-dimensional coordinate information of the hand key points to obtain the three-dimensional coordinate information of the hand key points; and determining the hand posture of the target object according to the three-dimensional coordinate information of the hand key points.

[0072] In this embodiment, the camera for image acquisition may be a depth camera, such as a RealSense depth camera, which can capture color images (also referred to as RGB images) and depth images simultaneously, that is, the hand image includes a hand color image and a hand depth image, and the face image includes a face color image and a face depth image. Before image acquisition, the RealSense depth camera may be initialized first, and the camera parameters may be configured to capture color images and depth images simultaneously.

[0073] In the process of determining the hand posture of the target object based on the hand image, the hand color image and the hand depth image are first aligned in pixels to ensure that the pixels of the hand color image and the hand depth image correspond one to one. Next, the hand key point detection is performed on the hand color image. When detecting the hand key point, the Mediapipe model can be used for detection to obtain the two-dimensional coordinate information of the hand key point. For example, the Mediapipe model can be used to detect the 21 hand key points of the hand from the hand color image collected by the RealSense depth camera, and the two-dimensional coordinate information of the hand key points in the hand color image can be obtained. Since the pixel alignment of the hand color image and the hand depth image has been performed before, the depth information of the corresponding position can be extracted from the hand depth image, so that the three-dimensional coordinate information of each hand key point can be determined. According to the three-dimensional coordinate information of each hand key point, the hand posture of the target object can be determined. The hand gesture may include hand movements and the distance between the two hands, wherein the hand movement may be subdivided into finger opening degree, palm rotation angle, etc. For example, the finger opening degree may be opening, slightly clenched, clenched into a fist, etc., and the palm rotation angle may be upward, outward, downward, inward, etc. It should be understood that the finger opening degree may include the opening degree of the left hand fingers and the opening degree of the right hand fingers, and the palm rotation angle may include the rotation angle of the left hand palm and the rotation angle of the right hand palm.

[0074] Although body posture can provide clues about emotions, it is usually not directly related to the expression of emotions. For example, a person may maintain an upright posture in a meeting, but this does not necessarily mean that he feels confident. It may be just out of politeness or professional requirements. Hand gestures are usually subconscious reactions, so the user's potential emotional state can be assessed through hand gestures to improve the accuracy of emotion estimation.

[0075] In one embodiment, the steps of performing hand key point detection on a hand color image to obtain two-dimensional coordinate information of the hand key points include: performing palm detection on the hand color image through a palm detection model to obtain a hand positioning frame for locating the palm position; inputting a target image corresponding to the hand positioning frame into the hand key point detection model to predict the hand key points in the target image; rendering the hand key points to the hand color image; and extracting the two-dimensional coordinate information of the corresponding positions of the hand key points from the hand color image.

[0076] In this embodiment, the Mediapipe model includes a palm detection model and a hand key point detection model. The palm detection model can be used to perform palm detection on the hand color image to detect whether there is a hand in the image. If there is, the hand positioning frame of the palm position is obtained. The hand positioning frame can be a square boundary frame. For example, by scanning the entire hand color image, a directional hand positioning frame is used to locate the position of the palm. The palm detection model uses a codec extractor similar to a feature pyramid, which can perceive the context in a larger scene and improve the detection ability of large and small objects. Next, the square boundary frame of the palm part is cropped out and input into the hand key point detection model. The hand key point detection can use a regression model to predict multiple key point coordinates of the hand from the cropped image, such as predicting 21 key point coordinates of the hand. If the prediction result is higher than the preset threshold of the hand, the two-dimensional coordinates of the 21 hand key points are output. The key points can be rendered to the input frame image (a single frame of the hand color image), and finally the rendered image with the hand key points is displayed.

[0077] In one embodiment, the step of extracting depth information from a hand depth image based on the two-dimensional coordinate information of the hand key points to obtain the three-dimensional coordinate information of the hand key points includes: extracting depth information corresponding to the two-dimensional coordinate information from a hand depth image aligned with the hand color image pixels based on the two-dimensional coordinate information of the hand key points; converting the two-dimensional coordinate information and the depth information into three-dimensional point coordinates in a camera coordinate system; and using the three-dimensional point coordinates as the three-dimensional coordinate information of the hand key points.

[0078] In this embodiment, the two-dimensional coordinate information (x, y) of the key point of the hand is the pixel coordinate of the key point of the hand in the hand color image. Based on the pixel coordinate, the depth information corresponding to the pixel coordinate, such as the depth value z, can be extracted from the hand depth image after pixel alignment with the hand color image, so as to convert the two-dimensional coordinate information (x, y) and the depth value z into the three-dimensional point coordinates (X, Y, Z) in the camera coordinate system to complete the spatial positioning of the hand position. The conversion formula can be as follows:

[0079]

[0080] Among them, X, Y, Z represent the three-dimensional point coordinates, x, y represent the pixel coordinates, z is the depth value, Cx and Cy are the camera optical center coordinates, and fx and fy are the camera focal lengths.

[0081] In one embodiment, the step of determining the hand posture of the target object based on the three-dimensional coordinate information of the hand key points includes: determining the left hand key points and the right hand key points from the hand key points; determining the coordinates of the left hand center point based on the left hand key points, and determining the coordinates of the right hand center point based on the right hand key points; determining the distance between the two hands based on the coordinates of the left hand center point and the coordinates of the right hand center point; determining the hand movement based on the left hand key points and the right hand key points; and using the hand movement and the distance between the two hands as the hand posture of the target object.

[0082] In this embodiment, after determining the three-dimensional coordinate information of each hand key point, the hand posture of the target object can be determined according to the three-dimensional coordinate information of the hand key point. Specifically, the left hand key point and the right hand key point can be determined from all the hand key points. For example, the left hand key point can be the left wrist, the left index finger metacarpophalangeal joint and the left little finger metacarpophalangeal joint, and the right hand key point can be the right wrist, the right index finger metacarpophalangeal joint and the right little finger metacarpophalangeal joint. The coordinates of the center point of the left hand can be determined according to the left wrist, the left index finger metacarpophalangeal joint and the left little finger metacarpophalangeal joint, the coordinates of the center point of the right hand can be determined according to the right wrist, the right index finger metacarpophalangeal joint and the right little finger metacarpophalangeal joint, and the distance between the two hands can be determined according to the coordinates of the center point of the left hand and the coordinates of the center point of the right hand.

[0083] Specifically, the distance between the two hands can be represented by the Euclidean distance between the left hand center point coordinate and the right hand center point coordinate. Assume that the left hand center point coordinate is L = (L x , L y , L z ), the coordinates of the right hand center point are R=(R x , R y , R z ), then the distance d between the two hands is as follows:

[0084]

[0085] Further, the hand motion may be determined according to the left hand key points and the right hand key points. The hand motion may include the finger opening degree and the palm rotation angle.

[0086] For example, the finger opening degree can be calculated as follows:

[0087] Get the coordinates of the key points of the hand. For example, the key points of the hand can be the center point of the palm, the fingertips of each finger, and the base points of each finger (close to the joints of the palm).

[0088] Calculate vectors. For each finger, calculate the vector from the center of the palm to the fingertip, and calculate the vectors between adjacent fingers.

[0089] Calculate the angle. Use the dot product formula to calculate the angle between adjacent finger vectors. The angle θ is calculated as follows:

[0090] θ = arccos((v1·v2) / (|v1|*|v2|)), where v1 and v2 are the vectors of adjacent fingers.

[0091] Assess the degree of opening. The greater the angle, the greater the degree of opening of the fingers.

[0092] For example, the palm rotation angle can be calculated as follows:

[0093] Calculate the vector from the wrist to the center of the palm. Create a vector using the coordinates of the wrist point and the center of the palm.

[0094] Calculate the angle between the vector and the horizontal line. Use the inverse tangent function to calculate the angle between the vector formed by the palm center and the wrist and the horizontal line to obtain the palm rotation angle.

[0095] Interpret the palm rotation angle. Angle of 0°: Palm facing up (or parallel to the x-axis). Angle of 90°: Palm rotated to the right (clockwise). Angle of -90°: Palm rotated to the left (counterclockwise). Angle of 180° or -180°: Palm facing down.

[0096] Step 203: determining the facial expression of the target object according to the facial image.

[0097] In one embodiment, the step of determining the facial expression of the target object based on the facial image includes: performing facial detection on the facial image and extracting facial key points; extracting facial coordinates of the facial key points; determining expression features based on the facial coordinates of the facial key points; and determining the facial expression based on the expression features and a pre-trained machine learning model.

[0098] In this embodiment, facial expressions can be recognized by the Mediapipe model, which includes a facial detection model and a facial key point detection model. The facial detection model can be used to perform facial detection to determine that a facial image includes a human face, and then the facial key point detection model can be used to detect facial key points in the facial image and extract the two-dimensional coordinate information of the facial key points. Mediapipe provides 468 facial key points, each of which contains three coordinates: x, y, and z. The facial key points detected can be used to calculate expression features, and the pre-trained machine learning model can be used to classify expression features and determine facial expressions. For example, emotions such as smile, sadness, surprise, anger, disgust, and fear can be determined based on the relative positions of facial key points.

[0099] Step 204: Identify the preliminary emotion of the target object based on the hand gesture.

[0100] In one embodiment, the step of identifying the preliminary emotion of the target object based on hand gestures includes: obtaining a preset mapping relationship; wherein the mapping relationship is a mapping relationship between hand gestures and emotions; estimating the preliminary emotion of the target object based on hand movements, the distance between the two hands and the mapping relationship.

[0101] In this embodiment, a mapping relationship between a preset hand gesture and an emotion can be obtained. Specifically, the mapping relationship may include a mapping relationship between hand movements (such as finger opening degree and palm rotation angle) and emotions, and a mapping relationship between the distance between the two hands and emotions. According to the hand gesture determined in step 202 and the mapping relationship, the preliminary emotion of the target object can be estimated. The hand gestures corresponding to the six main emotions can be shown in Table 1.

[0102] Table 1

[0103]

[0104]

[0105] Step 205, correcting the preliminary emotion according to the facial expression to obtain the final recognized emotion of the target object.

[0106] This method can determine the hand posture through the hand image, and then identify the preliminary emotion of the target object based on the hand posture. In addition, it can determine the facial expression of the target object based on the facial image, and then correct the preliminary emotion based on the facial expression to obtain the final recognized emotion. Since the final recognized emotion is the emotion obtained by multi-dimensional recognition combining hand posture and facial expression, the accuracy of emotion recognition is improved.

[0107] In the above embodiments, by combining Mediapipe's hand key point detection technology with the RealSense depth camera, the three-dimensional position information of the hand can be accurately obtained, thereby accurately measuring the distance between the hands, making the emotion recognition method perform better in complex three-dimensional space scenes, overcoming the limitations of relying solely on two-dimensional images. And by comprehensively analyzing facial expressions and hand postures (such as the distance between the hands, etc.), multimodal emotion recognition is achieved. This multi-dimensional information fusion makes emotional judgment more comprehensive and accurate, and reduces misjudgments caused by a single signal source. In addition, the embodiments of the present application can accurately measure hand distance and posture without contacting the user by using depth cameras and computer vision technology, providing a more natural and convenient means of emotion analysis.

[0108] In one embodiment, the method further includes: generating emotion guidance information according to the final recognized emotion of the target object; and feeding back the emotion guidance information to the target object.

[0109] In this embodiment, emotion guidance information can be generated based on the final recognized emotion of the target object, and providing feedback or guidance to the user through the emotion guidance information can help the user better understand their emotional state and take appropriate measures. The detailed steps of providing feedback or guidance to the user based on the final recognized emotion can be as follows:

[0110] Define emotions and corresponding feedback or guidance. Define each identified emotional state and provide corresponding feedback or guidance for each emotion, such as setting up simple text messages for these feedback or guidance through large models, or more complex suggestions or action steps.

[0111] Display feedback or guidance to the user. For example, feedback can be displayed on an image, recorded in a terminal or log, or converted into speech using text-to-speech (TTS) technology to provide users with more intuitive feedback.

[0112] In a specific embodiment, Figure 3 This is a schematic diagram of an emotion recognition system provided in this embodiment. The emotion recognition system may be composed of the following main modules:

[0113] Initialization and Configuration Module: Used to initialize and configure the Intel RealSense depth camera to capture both color and depth images.

[0114] Image capture and alignment module: The RealSense depth camera can be used to collect the user's facial and hand image data, and the color image and depth image of the image data can be pixel-aligned.

[0115] Hand key point detection module: Use Mediapipe technology to detect 21 key points of the hand from RealSense RGB images.

[0116] Depth information extraction and distance calculation module: Use the RealSense depth camera to obtain the depth image of the hand, and calculate the distance between the two hands based on the key point information.

[0117] Face recognition and expression recognition module: Detect facial key points from facial images obtained from RealSense's RGB camera and use a pre-trained expression recognition model to analyze the user's emotional state.

[0118] Emotion analysis module: integrates hand gestures and facial expressions to obtain the final emotion analysis results.

[0119] Result output module: displays the analysis results to the user or passes them to other systems for further processing.

[0120] Figure 4 A schematic diagram of an emotion recognition method provided in this embodiment, the method includes:

[0121] Step 1. Real-time video. Initialize and configure the Intel RealSense depth camera to capture color images and depth images simultaneously, and extract single-frame images from real-time video, each of which includes a captured color image and a depth image.

[0122] Step 2. RealSense: Image capture and alignment are performed through RealSense. The color image and depth image captured by RealSense are aligned through the image alignment module to ensure that the pixels of the two correspond one to one.

[0123] Step 3. Mediapipe. Input the aligned image to Mediapipe.

[0124] Step 4. Face recognition and expression recognition: Use Mediapipe to detect facial key points and then calculate the expression based on the key points.

[0125] Step 5. Hand key point detection and hand motion recognition. Use the Mediapipe model to detect the hand key points and obtain the two-dimensional coordinate information of the hand key points. Based on the obtained two-dimensional coordinate information, extract the corresponding depth information from the depth image and calculate the three-dimensional coordinates of the hand key points. Further convert these three-dimensional coordinates into three-dimensional points in the camera coordinate system to complete the spatial positioning of the hand position, calculate the distance between the two hands and the hand motion, so that the potential emotions can be judged based on the hand motion and the distance between the two hands.

[0126] Step 6. Emotion recognition: Comprehensively analyze facial expressions, hand movements, and the distance between the hands, and use algorithms to determine the user's emotional state.

[0127] Step 7. Result output: Output the final recognized emotional state and provide appropriate feedback or guidance to the user based on the recognition result.

[0128] Specifically, step 4 may specifically include: importing the Mediapipe library, and initializing the facial detection model and facial key point detection model in Mediapipe. Use a camera to capture real-time images or load existing facial images, and use the Mediapipe model to perform facial detection and key point extraction. Extract the two-dimensional coordinate information of facial key points. Mediapipe provides 468 facial key points, each of which contains three coordinates: x, y, and z. Use the detected facial key points to calculate the expression features, and use the pre-trained machine learning model to classify the expressions. According to the relative positions of the facial key points, emotions such as smile, sadness, surprise, anger, disgust, and fear can be calculated.

[0129] In step 5, the hand key point detection may specifically include: inputting a single frame image into a palm detection model, detecting whether there is a hand in the image, and if so, obtaining a square bounding box of the palm position. The palm detector locates the palm position by scanning the entire image with a directional hand positioning frame. The palm detection model uses a codec extractor similar to a feature pyramid, which can perceive the context in a larger scene and improve the detection ability of large and small objects. The bounding box of the palm part is cropped out and input into the hand key point detection model. The hand key point detection can use a regression model to predict the coordinates of 21 key points of the hand from the cropped image. If it is higher than the threshold of the hand, the coordinates of the 21 key points are output, and the key points are rendered to the input frame, and finally the rendered image is displayed. When calculating the distance between the two hands, the number of key points output is used to determine whether it is one hand or two hands. If it is one hand, no operation is performed. If it is two hands, the coordinates are output separately for the left and right hands. Select three key points of each hand: the wrist, the metacarpophalangeal joint of the index finger, and the metacarpophalangeal joint of the little finger. Obtain the coordinates of the center points of the left and right hands respectively through the average value of these three key points.

[0130] The hand action recognition in step 5 may specifically include: converting the normalized coordinates of the hand center into pixel coordinates to obtain depth information; checking whether the pixel coordinates are within the valid range, and if so, obtaining the depth value of the corresponding pixel coordinates; converting the pixel coordinates and the depth value into three-dimensional coordinates, and converting the pixel coordinates (x, y) and the depth value z in the image into three-dimensional points (X, Y, Z) in the camera coordinate system, and the formula is as follows:

[0131]

[0132] Among them, uppercase X, Y, and Z represent the coordinates of three-dimensional points, lowercase x and y represent pixel coordinates, z is the depth value, Cx and Cy are the coordinates of the camera optical center, and fx and fy are the focal lengths of the camera.

[0133] Calculate the three-dimensional distance between the two hands, assuming that the coordinates of the center point of the left hand are L = (L x , L y , L z ), the coordinates of the right hand center point are R=(R x , R y , R z ), then the distance d between the two hands is as follows:

[0134]

[0135] Further, the hand motion may be determined according to the left hand key points and the right hand key points. The hand motion may include the finger opening degree and the palm rotation angle.

[0136] For example, the finger opening degree can be calculated as follows:

[0137] Get the coordinates of the key points of the hand. The key points of the hand can be the center point of the palm, the fingertips of each finger, and the base points of each finger (close to the joints of the palm).

[0138] Calculate vectors. For each finger, calculate the vector from the center of the palm to the fingertip, and calculate the vectors between adjacent fingers.

[0139] Calculate the angle. Use the dot product formula to calculate the angle between adjacent finger vectors. The angle θ is calculated as follows:

[0140] θ = arccos((v1·v2) / (|v1|*|v2|)), where v1 and v2 are the vectors of adjacent fingers.

[0141] Assess the degree of opening. The greater the angle, the greater the degree of opening of the fingers.

[0142] The palm rotation angle can be calculated as follows:

[0143] Calculate the vector from the wrist to the center of the palm. Create a vector using the coordinates of the wrist point and the center of the palm.

[0144] Calculate the angle between the vector and the horizontal line. Use the inverse tangent function to calculate the angle between the vector formed by the palm center and the wrist and the horizontal line to obtain the palm rotation angle.

[0145] Interpret the palm rotation angle. Angle of 0°: Palm facing up (or parallel to the x-axis). Angle of 90°: Palm rotated to the right (clockwise). Angle of -90°: Palm rotated to the left (counterclockwise). Angle of 180° or -180°: Palm facing down.

[0146] Assess the user's underlying emotional state through hand movements.

[0147] In this embodiment, the use of the Mediapipe module can analyze the input image in real time and accurately identify the coordinates of the key points of the hand. Through the Mediapipe algorithm, stable and accurate recognition of multiple gestures is achieved. Compared with other models and algorithms, the resource usage is extremely low, and it can be deployed on platforms with low computing power such as development boards, mobile devices, and AR devices. Mediapipe uses outdoor data sets, outdoor data sets, and synthetic data sets with different geometric shapes, different lighting conditions, and different hand appearances. It has extremely strong robustness and is less affected by lighting and background interference. The model for detecting key point coordinates predicts the 2.5D key point coordinates of the hand through regression. The key point coordinate regression model can learn the inherent consistency of hand posture representation during use, and therefore exhibits very good robustness when the hand is partially missing or self-occluded.

[0148] The RealSense camera can directly output color and depth information without further processing, so speed and real-time performance can be guaranteed. In addition, its depth sensing module can be used to obtain depth information of key points of the hand, and the coordinates of key points of the hand can be fused with the depth information to obtain the precise position of the user's hand in the real scene.

[0149] Traditional gesture recognition technology mainly relies on two-dimensional images, which leads to limitations in obtaining depth information and measuring hand distance. This embodiment combines Mediapipe's hand key point detection technology with the RealSense depth camera to accurately obtain the three-dimensional position information of the hand, thereby accurately measuring the distance between the hands. This improvement makes the system perform better in complex three-dimensional space scenes, overcoming the limitations of relying only on two-dimensional images.

[0150] This embodiment performs multimodal data fusion. Related technologies usually rely on a single data source (such as facial expressions) for emotion recognition, which can easily lead to biased results. This embodiment achieves multimodal emotion recognition by comprehensively analyzing facial expressions and hand gestures (such as the distance between the two hands and hand movements). This multi-dimensional information fusion makes emotion judgment more comprehensive and accurate, reducing misjudgments caused by a single signal source.

[0151] In addition, some existing emotion and posture analysis systems may require contact sensors to obtain hand or body posture information, which may interfere with natural movement expression. This embodiment uses a depth camera and computer vision technology to accurately measure hand distance and posture without contacting the user, providing a more natural and convenient non-contact measurement emotion analysis method.

[0152] The above embodiments of the present application all have the following effects:

[0153] Improved accuracy: By combining deep information and multimodal analysis, the accuracy of emotion recognition has been significantly improved, and it can accurately judge the user's true emotional state in complex scenarios.

[0154] Wide range of applications: Not only suitable for emotional computing and human-computer interaction, but also can be applied to remote health monitoring, psychological counseling and other fields to provide users with more accurate emotional state feedback.

[0155] Non-contact: Non-contact measurement technology enables the system to efficiently obtain accurate posture and emotion information while maintaining user comfort, suitable for various application scenarios.

[0156] Based on the same technical concept, the second embodiment of the present application provides an emotion recognition device, such as Figure 5, the device comprises:

[0157] The acquisition module 501 is used to acquire image data of the target object; wherein the image data includes a facial image and a hand image;

[0158] A first determining module 502, configured to determine the hand posture of the target object according to the hand image;

[0159] A second determination module 503, configured to determine the facial expression of the target object according to the facial image;

[0160] A first recognition module 504, configured to recognize the preliminary emotion of the target object according to the hand gesture;

[0161] The second recognition module 505 is used to modify the preliminary emotion according to the facial expression to obtain the final recognized emotion of the target object.

[0162] The device can determine the hand posture through the hand image, and then recognize the preliminary emotion of the target object based on the hand posture, and can determine the facial expression of the target object based on the facial image, and then correct the preliminary emotion based on the facial expression to obtain the final recognized emotion. Since the final recognized emotion is the emotion obtained by multi-dimensional recognition combining hand posture and facial expression, the accuracy of emotion recognition is improved.

[0163] like Figure 6 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0164] A memory 113, for storing computer programs;

[0165] In one embodiment of the present application, the processor 111 is used to implement the emotion recognition method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113.

[0166] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0167] The communication interface is used for communication between the above terminal and other devices.

[0168] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0169] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0170] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the emotion recognition method provided by any of the aforementioned method embodiments is implemented.

[0171] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0173] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0174] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. In the description, the suffixes such as "module", "component" or "unit" used to represent the elements are only used to facilitate the description of the present application and have no specific meaning in themselves. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0175] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. An emotion recognition method, characterized in that: The method comprises: Acquire image data of the target object; wherein the image data includes a facial image and a hand image; determining a hand posture of the target object according to the hand image; determining the facial expression of the target object according to the facial image; identifying a preliminary emotion of the target object according to the hand gesture; The preliminary emotion is modified according to the facial expression to obtain a final recognized emotion of the target object.

2. The method according to claim 1, characterized in that: The hand image includes a hand color image and a hand depth image; The step of determining the hand posture of the target object according to the hand image comprises: Aligning pixels of the hand color image and the hand depth image; Performing hand key point detection on the hand color image to obtain two-dimensional coordinate information of the hand key points; Extracting depth information from the hand depth image according to the two-dimensional coordinate information of the key points of the hand to obtain the three-dimensional coordinate information of the key points of the hand; The hand posture of the target object is determined according to the three-dimensional coordinate information of the hand key points.

3. The method according to claim 2, characterized in that The step of detecting the key points of the hand on the hand color image to obtain the two-dimensional coordinate information of the key points of the hand includes: Performing palm detection on the hand color image using a palm detection model to obtain a hand positioning frame for locating the palm position; Inputting the target image corresponding to the hand positioning frame into the hand key point detection model to predict the hand key points in the target image; Rendering the hand key points to the hand color image; The two-dimensional coordinate information of the corresponding positions of the key points of the hand is extracted from the hand color image.

4. The method according to claim 2, characterized in that The step of extracting depth information from the hand depth image according to the two-dimensional coordinate information of the hand key points to obtain the three-dimensional coordinate information of the hand key points comprises: Extracting depth information corresponding to the two-dimensional coordinate information from the hand depth image aligned with pixels of the hand color image according to the two-dimensional coordinate information of the hand key points; Convert the two-dimensional coordinate information and the depth information into three-dimensional point coordinates in a camera coordinate system; The three-dimensional point coordinates are used as the three-dimensional coordinate information of the hand key points.

5. The method according to claim 2, characterized in that: The step of determining the hand posture of the target object according to the three-dimensional coordinate information of the hand key points comprises: Determine left hand key points and right hand key points from the hand key points; Determine the coordinates of the left hand center point based on the left hand key points, and determine the coordinates of the right hand center point based on the right hand key points; Determine the distance between the two hands according to the coordinates of the left hand center point and the coordinates of the right hand center point; Determining a hand motion according to the left hand key points and the right hand key points; Using the hand movement and the distance between the two hands as the hand posture of the target object; Accordingly, the step of identifying the preliminary emotion of the target object according to the hand gesture comprises: Obtaining a preset mapping relationship; wherein the mapping relationship is a mapping relationship between hand gestures and emotions; The preliminary emotion of the target object is estimated according to the hand motion, the distance between the two hands and the mapping relationship.

6. The method according to claim 1, characterized in that The step of determining the facial expression of the target object according to the facial image comprises: Performing facial detection on the facial image and extracting facial key points; Extracting facial coordinates of the facial key points; Determining expression features according to the facial coordinates of the facial key points; A facial expression is determined based on the expression features and a pre-trained machine learning model.

7. The method according to claim 1, characterized in that The method further comprises: generating emotion guidance information according to the final recognized emotion of the target object; The emotion guidance information is fed back to the target object.

8. An emotion recognition device, characterized in that: The device comprises: An acquisition module, used to acquire image data of a target object; wherein the image data includes a facial image and a hand image; A first determining module, configured to determine a hand posture of the target object according to the hand image; A second determination module, configured to determine the facial expression of the target object according to the facial image; A first recognition module, configured to recognize a preliminary emotion of the target object according to the hand gesture; The second recognition module is used to modify the preliminary emotion according to the facial expression to obtain the final recognized emotion of the target object.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the emotion recognition method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the emotion recognition method according to any one of claims 1 to 7 is implemented.

Images