Facial expression recognition method based on intelligent companion robot

Abstract

The present invention discloses a facial expression recognition method based on an intelligent companion robot, which relates to the field of intelligent recognition technology. The present invention can identify masked expressions (i.e., expressions inconsistent with real emotions) by analyzing the degree of emotional contradiction Qmd, effectively avoiding the problem of emotional misjudgment. In particular, when facing complex or disguised emotions, the robot can identify the true state of the emotion and adjust its feedback intensity based on this information to ensure that the emotional needs of the companion object can be accurately perceived and appropriately responded to. Dynamic feedback adjustment mechanism: By constructing a feedback response intensity value Fqdz and dynamically adjusting the feedback intensity according to the degree of emotional contradiction, the present invention can accurately adjust the robot's interaction intensity according to the emotional state in different situations. By combining initial settings and historical data, the present invention enables the intelligent companion robot to optimize its own behavior pattern based on long-term interactive feedback.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to a facial expression recognition method based on an intelligent companion robot. Background Art

[0002] Intelligent companion robots are an important application in the field of artificial intelligence (AI) and human-computer interaction (HCI), which are designed to provide emotional support and interaction for users. With the advancement of artificial intelligence technology, intelligent companion robots have gradually entered people's daily lives and are used in many scenarios such as family companionship, elderly care, and psychotherapy. Such robots are usually equipped with perception modules such as image acquisition devices and voice acquisition devices. Through the collection and analysis of multimodal data, they can understand the user's emotional state in real time and provide corresponding feedback. However, in terms of facial expression recognition, the relationship between facial expressions and emotions is not always direct, and facial expressions are not always proportional to emotions. Expression recognition technology often faces the problem of emotional misjudgment, especially when the expressions of emotions show similar characteristics.

[0003] In current intelligent companion robot applications, the correlation between facial expressions and emotional labels is still relatively weak. Expression recognition technology often relies on static facial images or video streams, and most models are classified through some common emotion categories (such as happiness, sadness, anger, surprise, etc.). However, these emotion categories often show overlap and ambiguity in practical applications. Relying solely on facial expressions for emotion recognition is also difficult to handle emotional disguise caused by "mask-like expressions" (such as deliberately showing a certain expression due to social pressure or personal wishes). Therefore, existing technologies ignore the capture of the true meaning behind emotions, which may lead to inappropriate reactions for companion robots or make it difficult to meet the actual emotional needs of users.

[0004] Misjudgment between facial expressions and emotions, especially interference from mask-like expressions, is often caused by the difficulty of the robot to timely identify or adjust the feedback intensity when the emotional conflict is large, resulting in inappropriate responses. For example, when the companion object tries to hide its true emotions with mask-like expressions, if the robot relies on conventional facial expression recognition models for emotional judgment, it may mistakenly interpret it as a happy emotion, thereby providing excessive feedback in a situation where the intensity of interaction should be reduced. This emotional misjudgment will not only destroy the trust and emotional connection between the robot and the companion object, but may also cause the companion object's emotions to be ignored or misunderstood, reducing the effectiveness of the companion robot's emotional support. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a facial expression recognition method based on an intelligent companion robot, which solves the problems in the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a facial expression recognition method based on an intelligent companion robot comprises the following steps:

[0007] S1: Using the image acquisition device and voice acquisition device of the intelligent companion robot, the facial image, body behavior image and voice data of the companion object are monitored in real time, and facial feature vectors are generated after feature extraction. , limb feature vector and speech feature vector , construct a vector set;

[0008] S2: Based on the FER-2013 dataset, combined with the facial feature vector obtained in S1 , construct an expression recognition model to identify and classify the facial expressions and corresponding emotion labels of the accompanying object, align the relevant vectors in the vector set, and map the vector set after the vector alignment to the emotion space to obtain the processed vector set;

[0009] S3: Analyze the degree of contradiction between the facial expression of the accompanying object and the corresponding emotional label according to the processed vector set to calculate the emotional contradiction degree Qmd, and identify whether the facial expression of the current accompanying object belongs to a mask expression based on the emotional contradiction degree Qmd value;

[0010] S4: If it is a mask-like expression, then the initial settings and historical data are combined to analyze the feedback intensity required by the intelligent companion robot when facing the mask-like expression of the companion object, so as to construct a feedback response intensity value Fqdz, and based on the feedback response intensity value Fqdz, dynamically adjust the feedback degree of the intelligent companion robot.

[0011] Preferably, the specific steps of S1 include:

[0012] S11: The image acquisition device preset in the intelligent companion robot collects video data of the companion object's face and body in real time, and extracts continuous facial images and body behavior images from the video data, and the voice acquisition device monitors the companion object's voice data in real time, the facial image is used to capture the companion object's facial expression, the body image is used to capture body language such as posture and movement, and the voice data is used to analyze information such as intonation, speech speed, and tone, so as to extract emotional characteristics; wherein the image acquisition device includes a camera, and the voice acquisition device includes a microphone;

[0013] S12: Use Haar cascade classifier to identify the facial area in the facial image, use OpenPose tool to extract the 3D coordinates of the joints in the body behavior image, and use Kalman filtering to remove background noise from the speech data;

[0014] S13: Based on S12, facial landmark detection technology is used to extract the key point coordinates and key coordinate changes in the facial area of ​​the facial image to obtain the facial feature vector , the facial feature vector The expression is: ; Based on the 3D coordinates of the joint points in the extracted limb behavior image and the relative relationship of the joint point positions, a limb feature vector is generated , the limb feature vector The expression is: ; And extract the features of the speech data after background noise removal to obtain the speech feature vector , the speech feature vector The expression is: ;

[0015] The vector set includes facial feature vectors , limb feature vector and speech feature vector .

[0016] Preferably, the specific steps of S2 include:

[0017] S21: Using deep learning technology, and based on the expression samples and emotion labels stored in the FER-2013 dataset, we build and train an expression recognition model. Through the trained expression recognition model, we use the facial feature vectors obtained in S1 to identify the facial features of the facial expression. Facial expression classification is performed to identify the facial expression of the current companion, and the facial expression of the current companion is mapped to the corresponding emotion category through the expression samples and emotion labels stored in the FER-2013 dataset to determine the emotion label corresponding to the facial expression of the current companion.

[0018] Preferably, the specific step S2 also includes:

[0019] S22: By using the normalization technology, all related vectors in the vector set are aligned so that the related vectors in the vector set have the same scale and direction;

[0020] S23: Facial feature vector after vector alignment using the emotion space mapping algorithm , limb feature vector and speech feature vector are mapped to the same emotion space coordinate system to form a processed vector set, where the processed vector set includes the processed facial feature vectors , processed limb feature vector And the processed speech feature vector , the processed facial feature vector The expression is: ; Processed limb feature vector The expression is: ; Processed speech feature vector The expression is: .

[0021] Preferably, the specific steps of S3 include:

[0022] S31: Based on the processed facial feature vector And the processed speech feature vector , analyze the difference between the facial expression and voice state of the current companion, and obtain the difference between facial expression and voice features Myc. Specifically, the difference between facial expression and voice emotion information Myc is obtained according to the following formula:

[0023] ;

[0024] In the formula, is the modulus length of the processed facial feature vector, is the modulus length of the processed speech feature vector, is the dot product of the processed facial feature vector and the processed voice feature vector;

[0025] S32: Based on the processed facial feature vector And the processed limb feature vector , analyze the difference between the facial expression and body state of the current companion, and obtain the difference between the facial expression and body characteristics Mtc. Specifically, the difference between the facial expression and body characteristics Mtc is obtained according to the following formula:

[0026] ;

[0027] In the formula, It is expressed as the modulus of the processed limb feature vector.

[0028] Preferably, the specific step S3 also includes:

[0029] S33: According to the processed vector set and in combination with the contents of S31 and S32, the contradiction degree between the facial expression of the companion object and the corresponding emotional label is analyzed to calculate the emotional contradiction degree Qmd. The emotional contradiction degree Qmd is specifically obtained according to the following formula:

[0030] ;

[0031] In the formula, and They are respectively represented as the weight values ​​of the difference between facial expression and voice emotion information Myc and the difference between facial expression and body features Mtc, where 0 < <1,0< <1, and The specific value is set by the user according to the situation.

[0032] Preferably, the specific step S3 also includes:

[0033] S34: presetting a difference threshold K, and comparing and analyzing the difference threshold K with the emotional conflict degree Qmd to identify whether the facial expression of the current companion is a mask-like expression. The specific identification content is as follows:

[0034] If the emotional conflict degree Qmd≤the abnormal threshold value K, it will be recognized that the facial expression of the current companion is a mask-like expression, and a feedback adjustment instruction will be triggered externally;

[0035] If the emotional conflict degree Qmd>the abnormal threshold K, it will be recognized that the facial expression of the current companion does not belong to the mask expression, and the feedback adjustment instruction will not be triggered externally.

[0036] Preferably, the specific steps of S4 include:

[0037] S41: After receiving the feedback adjustment instruction, the feedback strength required by the intelligent companion robot for the mask-like expression currently facing the companion object is analyzed in combination with the initial settings and historical data to construct a feedback response strength value Fqdz, wherein the initial settings include the initial set feedback response strength value The historical data includes the emotional conflict degree at each monitoring time point in the historical period, and the average emotional conflict degree in the historical period is obtained by statistical averaging algorithm. The feedback response strength value Fqdz is obtained by the following formula:

[0038] ;

[0039] In the formula, Indicates the initial set feedback response strength value, Expressed as the feedback adjustment coefficient, Expressed as emotional conflict.

[0040] Preferably, the specific step S4 also includes:

[0041] S42: The feedback adjustment coefficient It reflects the sensitivity of the regulatory system to emotional misjudgment, which can be obtained according to the following formula:

[0042] ;

[0043] In the formula, represents the indicator function, Represents the average degree of emotional conflict during the historical period.

[0044] Preferably, the specific step S4 also includes:

[0045] S43: Based on the feedback adjustment coefficient in S42 , and dynamically adjust the feedback degree of the intelligent companion robot by combining the emotional label corresponding to the facial expression of the current companion object.

[0046] The present invention provides a method for recognizing facial expressions based on an intelligent companion robot, which has the following beneficial effects:

[0047] Emotion recognition and feedback: By real-time monitoring of the facial expressions, body movements and voice data of the companion object, and combining the expression recognition model trained with the FER-2013 data set, the present invention can accurately identify the facial expressions of the companion object and its corresponding emotion labels. Through comprehensive analysis of multimodal data, it can effectively make up for the shortcomings of a single facial expression recognition method, improve the accuracy of emotion recognition, and ensure that the intelligent companion robot can make appropriate feedback according to the emotional state of the companion object. Dealing with the problem of masked expressions: By analyzing the emotional contradiction degree Qmd, the present invention can identify masked expressions (i.e., expressions inconsistent with real emotions), effectively avoiding the problem of emotional misjudgment. In particular, when facing complex or disguised emotions, the robot can identify the true state of emotions and adjust its feedback intensity based on this information to ensure that the emotional needs of the companion object can be accurately perceived and appropriately responded to. Dynamic feedback adjustment mechanism: By constructing a feedback response intensity value Fqdz and dynamically adjusting the feedback intensity according to the emotional contradiction degree, the present invention can accurately adjust the interaction intensity of the robot according to the emotional state in different situations. For example, in the case of masked expressions, the robot can moderately reduce the feedback intensity to avoid emotional conflicts caused by excessive responses, thereby improving the naturalness and comfort of companionship interaction. By combining the initial settings and historical data, the present invention enables the intelligent companion robot to optimize its own behavior patterns based on long-term interactive feedback, and gradually improve the adaptability and personalization of emotional responses. By comprehensively analyzing facial expressions, body language and voice information, the robot can better understand and adapt to the emotional state of the companion object, provide a more emotionally resonant interactive experience, and thus improve the quality of the companionship experience and increase the emotional investment and dependence of the companion object. In summary, the present invention not only effectively solves the problem of misjudgment of facial expressions in the prior art, but also further optimizes the interactive behavior of the intelligent companion robot by introducing emotional conflict analysis and dynamic feedback adjustment mechanism, so that it can adapt to the emotional needs of the companion object more accurately and personally, thereby greatly improving the emotional response ability of the companion robot and the naturalness of human-computer interaction.

[0048] Improved accuracy of multimodal emotion feature analysis: By comprehensively analyzing multiple emotion features such as facial expressions, voice and body behaviors, the emotional state of the companion can be accurately assessed. Specifically, step S31 can reveal the possible emotional contradiction or inconsistency between the facial expression and voice expression of the current companion by calculating the difference between facial expression and voice features Myc. The calculation of this difference can help the robot identify potential inconsistencies in emotional expression, thereby avoiding incorrect emotional interpretation and inappropriate reactions, and improving the interactive quality of the companion robot. Multi-dimensional emotion judgment and emotion coordination: Step S32 further supplements the dimension of emotion analysis by calculating the difference between facial expression and body features Mtc, and can comprehensively consider whether the emotional state of the companion is consistent between visual expression (face) and behavioral expression (body). Through this analysis, the robot can identify whether there is inconsistency or conflict in emotional expression, so as to more accurately understand the true emotional state of the companion. This multi-dimensional emotion judgment method can significantly improve the accuracy of emotion recognition and the robot's response sensitivity. Emotional conflict identification and robot feedback optimization: By analyzing the differences in facial expressions, voice and body behaviors, the present invention can effectively identify mask-like expressions or emotional conflicts and avoid emotional misjudgment. By calculating the difference between face and voice Myc and the difference between face and body Mtc, the intelligent companion robot can adjust its feedback response level when facing contradictory emotional expressions, making its feedback more accurate and natural, thereby enhancing the emotional experience of the companion object.

[0049] Emotional conflict analysis and masked expression recognition: Through the comprehensive analysis of the differences between facial expressions, voice and body features in step S33, the emotional conflict Qmd can be accurately calculated. This calculation is based on multiple features (such as the differences between facial expressions and voice, facial expressions and body behaviors), so as to more comprehensively capture the inconsistency or potential contradictions in the emotional expression of the companion. Compared with single emotion analysis, the conflict analysis based on multiple features can more accurately identify the emotional state, especially for the detection of masked expressions, which can effectively avoid the misjudgment between expression and emotion labels and improve the robot's emotional understanding ability. Dynamic feedback adjustment mechanism: Through the comparison analysis of the difference threshold K and the emotional conflict Qmd in step S34, the intelligent companion robot can judge in real time whether the emotional state of the companion is in line with expectations. This mechanism can ensure that the robot adjusts its feedback strategy when facing emotional conflicts or pretended emotions, avoids giving the companion unreal or inappropriate emotional responses, and improves the comfort and naturalness of human-computer interaction.

[0050] After the relationship between the current emotional label and the mask-like expression is clearly identified, the feedback response intensity will be flexibly adjusted according to the actual emotional conflict and the feedback adjustment coefficient. This dynamic feedback adjustment not only enables the robot to better understand and respond to the emotional needs of the companion, but also provides more natural and emotionally responsive feedback in different emotional scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The present invention is a flow chart of a method for recognizing facial expressions based on an intelligent companion robot. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Example 1: Please refer to Figure 1 The present invention provides a method for recognizing facial expressions based on an intelligent companion robot, comprising the following steps:

[0054] S1: Using the image acquisition device and voice acquisition device of the intelligent companion robot, the facial image, body behavior image and voice data of the companion object are monitored in real time, and facial feature vectors are generated after feature extraction. , limb feature vector and speech feature vector , construct a vector set;

[0055] S2: Based on the FER-2013 dataset, combined with the facial feature vector obtained in S1 , construct an expression recognition model to identify and classify the facial expressions and corresponding emotion labels of the accompanying object, align the relevant vectors in the vector set, and map the vector set after the vector alignment to the emotion space to obtain the processed vector set;

[0056] S3: Analyze the degree of contradiction between the facial expression of the accompanying object and the corresponding emotional label according to the processed vector set to calculate the emotional contradiction degree Qmd, and identify whether the facial expression of the current accompanying object belongs to a mask expression based on the emotional contradiction degree Qmd value;

[0057] S4: If it is a mask-like expression, then the initial settings and historical data are combined to analyze the feedback intensity required by the intelligent companion robot when facing the mask-like expression of the companion object, so as to construct a feedback response intensity value Fqdz, and based on the feedback response intensity value Fqdz, dynamically adjust the feedback degree of the intelligent companion robot.

[0058] In this embodiment, the accuracy of facial expression recognition is improved: by combining the FER-2013 data set and the real-time collected facial feature vectors, the present invention introduces the steps of emotional space mapping and vector alignment in the facial expression recognition model. This step effectively reduces the limitations of the traditional facial expression recognition model through comprehensive analysis of facial, body and voice feature vectors, and improves the robot's recognition ability of complex emotional expressions. Accurately identify masked expressions: by calculating the contradiction degree Qmd between facial expressions and emotional labels, the present invention can accurately identify whether the companion object presents a masked expression (i.e., the situation where the emotion does not match the expression). This feature enables the robot to avoid misjudgment and provide more accurate feedback when facing the user's emotional disguise, thereby improving the user's emotional experience. Dynamically adjust the feedback intensity: according to the recognition results of the masked expression and the value of the emotional contradiction degree, the intelligent companion robot can dynamically adjust its feedback response intensity Fqdz according to historical data and current emotional state. This adaptive adjustment mechanism can ensure that when the robot interacts with the companion object, the feedback level always meets the user's real emotional needs, avoids excessive or insufficient emotional feedback, and effectively improves the user's emotional resonance and interaction quality. Enhance the effectiveness of robot emotional support: By combining facial expressions, body language, and voice data, the intelligent companion robot can fully understand the emotional state of the companion object, thereby providing more personalized and accurate emotional support. This multimodal emotion recognition method not only improves the robot's emotion understanding ability, but also improves the companion robot's adaptability in human-computer interaction, especially for complex situations with user emotional fluctuations and emotional disguise. In summary, this method can effectively improve the emotion recognition and feedback capabilities of the intelligent companion robot, enhance the naturalness and flexibility of human-computer interaction, and thus improve the service quality and user experience of the companion robot in practical applications.

[0059] Example 2: Please refer to Figure 1 , specifically: S1 specific steps include:

[0060] S11: The image acquisition device preset in the intelligent companion robot collects video data of the companion object's face and body in real time, and extracts continuous facial images and body behavior images from the video data, and the voice acquisition device monitors the companion object's voice data in real time, the facial image is used to capture the companion object's facial expression, the body image is used to capture body language such as posture and movement, and the voice data is used to analyze information such as intonation, speech speed, and tone, so as to extract emotional characteristics; wherein the image acquisition device includes a camera, and the voice acquisition device includes a microphone;

[0061] S12: Use Haar cascade classifier to identify the facial area in the facial image, use OpenPose tool to extract the 3D coordinates of the joints in the body behavior image, and use Kalman filtering to remove background noise from the speech data;

[0062] S13: Based on S12, facial landmark detection technology is used to extract the coordinates of key points in the facial area of ​​the facial image (such as the positions of eyes, nose, and mouth) and key coordinate changes to obtain facial feature vectors. , the facial feature vector The expression is: ; Based on the 3D coordinates of the joint points in the extracted limb behavior image and the relative relationship of the joint point positions, a limb feature vector is generated , the limb feature vector The expression is: ; And extract the features of the speech data after background noise removal to obtain the speech feature vector , the speech feature vector The expression is: ;

[0063] The vector set includes facial feature vectors , limb feature vector and speech feature vector ;

[0064] For the relative relationship of the joint positions mentioned above: when generating the limb feature vector When talking about the body, the relative relationship refers to the spatial position and motion state between the joints of the human body. These relative relationships can be used to describe and understand limb movements. The relative relationship can be analyzed from the following perspectives:

[0065] 1. Spatial distance between joints: for example, the distance between the shoulder and the elbow, the distance between the knee and the ankle, etc. These relative distances can be used to reflect the range of motion or changes in posture of the limbs, and can be calculated using the Euclidean distance;

[0066] 2. Joint angle: The angle between joints is also an important feature of limb movements, such as the angles of knees, elbows, shoulders and other joints. For example, the angle between the elbow and the shoulder can be calculated by the dot product formula of the vector;

[0067] 3. Movement trajectory between joints: The relative position changes between joints also reflect the movement of the human body. For example, the movement trajectory of an arm or leg can reflect more complex movements. Calculation method: By calculating the continuous position of key points over a period of time, the movement trajectory of the joint can be drawn;

[0068] 4. Relative speed and acceleration: In the time dimension, the speed and acceleration of relative movement between joints can be analyzed. For example, the movement speed of the hand or the acceleration of the knee;

[0069] By extracting the coordinates of the key points of the human body from the input image, the relative relationships between the joints, such as spatial distance, angle, speed, and acceleration, can be calculated based on these coordinates. These relative relationships constitute the limb feature vector, which can be further used to analyze and identify limb movements, behavior patterns, etc.

[0070] The specific steps of S2 include:

[0071] S21: Using deep learning technology, and based on the expression samples and emotion labels stored in the FER-2013 dataset, we build and train an expression recognition model. Through the trained expression recognition model, we use the facial feature vectors obtained in S1 to identify the facial features of the facial expression. Facial expression classification is performed to identify the facial expression of the current companion, and the facial expression of the current companion is mapped to the corresponding emotion category (such as happiness, anger, surprise, etc.) through the expression samples and emotion labels stored in the FER-2013 dataset to determine the emotion label corresponding to the facial expression of the current companion.

[0072] The specific steps of S2 also include:

[0073] S22: By using the normalization technology, all related vectors in the vector set are aligned so that the related vectors in the vector set have the same scale and direction;

[0074] S23: Use an emotion space mapping algorithm (e.g., an emotion representation method based on self-supervised learning, or a calibration using emotion space) to align the facial feature vectors , limb feature vector and speech feature vector are mapped to the same emotion space coordinate system to form a processed vector set, where the processed vector set includes the processed facial feature vectors , processed limb feature vector And the processed speech feature vector , the processed facial feature vector The expression is: ; Processed limb feature vector The expression is: ; Processed speech feature vector The expression is: .

[0075] In this embodiment, efficient emotion recognition of multimodal data: through the image acquisition device and voice acquisition device in the intelligent companion robot, the facial, body and voice data of the companion object can be obtained in real time, and the facial feature vector, body feature vector and voice feature vector can be extracted respectively. This combination of multimodal data can comprehensively analyze the emotional state of the companion object and avoid the emotional understanding bias that may be caused by a single data source. Facial images are used to capture expressions, body images are used to recognize body language, and voice data helps to analyze emotional features such as tone and intonation, so as to more accurately identify the emotions of the companion object. Accurate facial expression recognition and emotion classification: through deep learning technology, the expression recognition model is trained based on the expression samples and emotion labels in the FER-2013 data set, which can efficiently recognize the facial expressions of the companion object, and accurately map the facial expressions to the corresponding emotion categories (such as happiness, anger, surprise, etc.) according to the emotion labels in the FER-2013 data set. The automation and precision of this process provide the intelligent companion robot with better emotion understanding capabilities, so that the robot can adjust the interaction strategy in time according to the facial expressions. Vector alignment improves recognition consistency: The normalization technology is used to align the facial, body and voice feature vectors in the vector set, and these feature vectors are converted into a unified standard with the same scale and direction. This process ensures the consistency between multimodal features, so that the robot can perform comprehensive analysis more efficiently and accurately when processing complex emotional data. Through this alignment method, the weight differences between different features are eliminated, which improves the accuracy of emotion classification. Emotional space mapping improves comprehensive emotion recognition ability: Through the emotional space mapping algorithm (such as the emotion representation method based on self-supervised learning or emotional space calibration), the facial, body and voice feature vectors are mapped to the same emotional space coordinate system. This operation enables various features to be uniformly processed under the same emotional dimension, thereby improving the depth and dimension of emotional understanding. Through this mapping, the intelligent companion robot can combine multiple emotional clues to achieve more accurate emotion recognition, and make more appropriate feedback and responses based on this. Personalized and accurate emotional feedback: The processed facial feature vectors, body feature vectors and voice feature vectors can work together in the recognition and feedback mechanism of the emotional space, ensuring that the robot can accurately identify and understand the emotional state of the companion object, and further realize personalized and intelligent emotional response. For example, when negative emotions such as sadness or anger are identified, the robot can make corresponding adjustments based on historical data and feedback intensity to ensure harmony and comfort in the interaction process.To sum up, the present invention combines multimodal data analysis of image, voice and body features, adopts deep learning and emotion space mapping algorithms, which not only improves the accuracy and consistency of intelligent companion robots in facial expression recognition and emotion classification, but also optimizes the emotional feedback mechanism, enabling the robot to interact more accurately and naturally based on the emotional state of the companionship object, thereby greatly improving the quality of the companionship experience and the user's emotional satisfaction.

[0076] Example 3: Please refer to Figure 1 , specifically: S3 specific steps include:

[0077] S31: Based on the processed facial feature vector And the processed speech feature vector , analyze the difference between the facial expression and voice state of the current companion, and obtain the difference between facial expression and voice features Myc. Specifically, the difference between facial expression and voice emotion information Myc is obtained according to the following formula:

[0078] ;

[0079] In the formula, is the modulus length of the processed facial feature vector, is the modulus length of the processed speech feature vector, is the dot product of the processed facial feature vector and the processed voice feature vector, which represents the inner product of the two vectors. This formula is used to measure the angle between the two vectors, that is, whether the two vectors point to similar emotional directions.

[0080] S32: Based on the processed facial feature vector And the processed limb feature vector , analyze the difference between the facial expression and body state of the current companion, and obtain the difference between the facial expression and body characteristics Mtc. Specifically, the difference between the facial expression and body characteristics Mtc is obtained according to the following formula:

[0081] ;

[0082] In the formula, Expressed as the modulus of the processed limb feature vector, this formula is used to measure the angle between two vectors, that is, whether the two vectors point to similar emotional directions.

[0083] In this embodiment, accurate emotional difference analysis: by analyzing the difference between facial feature vectors and voice feature vectors, the intelligent companion robot can accurately evaluate the emotional consistency between the facial expression and voice state of the companion object. The calculation of the difference Myc can reveal the potential conflict or inconsistency between facial expressions and voice information, and further optimize the robot's response to emotional expression. For example, when there is a difference in the emotions expressed by facial expressions and voice, the robot can recognize this contradiction and avoid making inappropriate feedback due to emotional misjudgment. Multimodal emotional synchronization recognition: facial expressions and voice information belong to two different ways of expressing emotions, and there may be certain differences between facial expressions and voice information. By calculating the difference Myc, the robot can effectively analyze the relationship between facial expressions and voice information, and then determine whether the companion object is in a contradictory emotional state. This technology can help the robot better understand the true emotions of the companion object and provide a more natural and accurate emotional response. For example, if the emotional direction revealed by facial expressions and voice is consistent, the robot can give more positive feedback; if there is an inconsistency, the robot can recognize this emotional difference and adjust the response strategy. Emotional consistency detection of body posture and facial expression: By analyzing the difference between facial feature vectors and body feature vectors, the intelligent companion robot can not only evaluate the emotional state of facial expressions, but also further verify the emotional state through body behaviors (such as posture and movement). The calculation of the difference Mtc can measure the emotional consistency between facial expressions and body posture. This analysis helps the robot determine whether the companion is showing a mask-like expression or whether there is an unreal expression caused by emotional conflict. This emotional consistency detection can effectively prevent the robot from misunderstanding the companion's true emotions and improve the accuracy of its emotion recognition.

[0084] Example 4: Please refer to Figure 1 , specifically: S3 specific steps also include:

[0085] S33: According to the processed vector set and in combination with the contents of S31 and S32, the contradiction degree between the facial expression of the companion object and the corresponding emotional label is analyzed to calculate the emotional contradiction degree Qmd. The emotional contradiction degree Qmd is specifically obtained according to the following formula:

[0086] ;

[0087] In the formula, and They are respectively represented as the weight values ​​of the difference between facial expression and voice emotion information Myc and the difference between facial expression and body features Mtc, where 0 < <1,0< <1, and The specific value is set by the user according to the situation.

[0088] The specific steps of S3 also include:

[0089] S34: presetting a difference threshold K, and comparing and analyzing the difference threshold K with the emotional conflict degree Qmd to identify whether the facial expression of the current companion is a mask-like expression. The specific identification content is as follows:

[0090] If the emotional conflict degree Qmd≤the abnormal threshold value K, it will be recognized that the facial expression of the current companion is a mask-like expression, and a feedback adjustment instruction will be triggered externally;

[0091] If the emotional conflict degree Qmd>the abnormal threshold K, it will be recognized that the facial expression of the current companion does not belong to the mask expression, and the feedback adjustment instruction will not be triggered externally.

[0092] In this embodiment, accurate calculation of emotional contradiction: through step S33, combined with the difference between facial expression and voice emotional information Myc and facial expression and body features Mtc, the present invention can quantify the emotional contradiction Qmd. The calculation of this emotional contradiction can deeply analyze whether the emotional expression of the companion object is consistent, so as to effectively identify potential emotional disharmony or contradiction. This multi-dimensional emotional analysis enables the intelligent companion robot to accurately capture emotional conflicts, thereby providing a more reliable data basis for subsequent feedback adjustment. Flexibility and accuracy of dynamic feedback adjustment: The introduction of emotional contradiction Qmd enables the robot to make feedback adjustments more flexibly and accurately when facing complex emotional states. Step S34 determines whether there is a mask expression (i.e., the surface emotion is inconsistent with the real emotion) by setting the threshold K of emotional contradiction; this mechanism enables the robot to accurately distinguish the expression of different emotional states in the interaction, avoiding excessive or insufficient emotional responses. Reduce emotional misjudgment and improve human-computer interaction experience: through the analysis of emotional contradiction Qmd, the robot can determine whether there is emotional disguise or mask expression, and avoid making inappropriate responses due to emotional misjudgment. For example, when the person being accompanied shows an expression that is inconsistent with his or her inner emotions due to external reasons (such as pretending to be happy or concealing anger), the robot can recognize and make appropriate response adjustments to avoid causing unnecessary distress or discomfort to the person being accompanied.

[0093] Example 5: Please refer to Figure 1 , specifically: S4 specific steps include:

[0094] S41: After receiving the feedback adjustment instruction, the feedback strength required by the intelligent companion robot for the mask-like expression currently facing the companion object is analyzed in combination with the initial settings and historical data to construct a feedback response strength value Fqdz, wherein the initial settings include the initial set feedback response strength value The historical data includes the emotional conflict degree at each monitoring time point in the historical period, and the average emotional conflict degree in the historical period is obtained by statistical averaging algorithm. The feedback response strength value Fqdz is obtained by the following formula:

[0095] ;

[0096] In the formula, Indicates the initial set feedback response strength value, Expressed as the feedback adjustment coefficient, Expressed as emotional conflict.

[0097] The specific steps of S4 also include:

[0098] S42: The feedback adjustment coefficient It reflects the sensitivity of the regulatory system to emotional misjudgment, which can be obtained according to the following formula:

[0099] ;

[0100] In the formula, represents the indicator function, if If the condition is true, the output is 1. When , the condition is false and the output is 0; Represents the average degree of emotional conflict during the historical period.

[0101] The specific steps of S4 also include:

[0102] S43: Based on the feedback adjustment coefficient in S42 , and dynamically adjust the feedback degree of the intelligent companion robot by combining the emotional label corresponding to the facial expression of the current companion object.

[0103] In this embodiment, accurate feedback response intensity adjustment: through the analysis based on the initial setting and historical data in step S41, the intelligent companion robot can calculate the appropriate feedback response intensity value Fqdz. This intensity value not only combines the feedback response level set initially, but also introduces the average value of emotional contradiction in the historical period, so that the robot can make more accurate adjustments based on historical emotional performance, ensuring that the feedback is highly matched with the current emotional contradiction. This dynamic adjustment mechanism avoids the emotional imbalance problem that may be caused by fixed feedback intensity, thereby improving the flexibility and adaptability of interaction. Optimize the sensitivity of emotional misjudgment response: through the calculation of the feedback adjustment coefficient in step S42, the intelligent companion robot can make fine-grained adjustments according to the response sensitivity of emotional misjudgment. The feedback adjustment coefficient can automatically adjust the response intensity to emotional contradictions through the analysis of historical emotional contradictions, so that the robot can more sensitively identify and make corresponding adjustments when processing mask-like expressions. In this way, the robot can make more personalized feedback according to the specific emotional state, avoiding emotional imbalance caused by excessive or insufficient feedback. Improve the naturalness and comfort of emotional feedback: Combined with the dynamic feedback adjustment based on the feedback adjustment coefficient in step S43, the robot can adjust its feedback strength in real time according to the current facial expression and the corresponding emotional label of the companion object. This dynamic adjustment mechanism ensures that the robot can provide more appropriate and natural emotional responses when facing different emotional states, thereby improving the comfort and affinity of human-computer interaction. Reduce unnatural responses caused by mask-like expressions: Through the dynamic adjustment of the feedback response intensity value Fqdz, the robot can accurately control its response intensity to avoid emotional misjudgment or unnatural reactions caused by inappropriate feedback intensity. For example, when facing mask-like expressions, the robot can ensure that the interaction is more real and natural by adjusting the intensity of the response, and effectively avoid distortion between expression and emotion. In summary, the present invention can effectively solve the problems of mask-like expressions and emotional misjudgment through a dynamic feedback adjustment mechanism, combined with historical data and current emotional state, improve the emotional understanding ability of the intelligent companion robot, optimize the emotional feedback accuracy and naturalness of human-computer interaction, ensure that the robot can provide a more appropriate and comfortable interactive experience, and promote the application of intelligent companion robots in human emotional communication.

[0104] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A facial expression recognition method based on an intelligent companion robot, characterized in that: The following steps are involved: S1: Using the image acquisition device and voice acquisition device of the intelligent companion robot, the facial image, body behavior image and voice data of the companion object are monitored in real time, and facial feature vectors are generated after feature extraction. , limb feature vector and speech feature vector , construct a vector set; S2: Based on the FER-2013 dataset, combined with the facial feature vector obtained in S1 , construct an expression recognition model to identify and classify the facial expressions and corresponding emotion labels of the accompanying object, align the relevant vectors in the vector set, and map the vector set after the vector alignment to the emotion space to obtain the processed vector set; S3: Analyze the degree of contradiction between the facial expression of the accompanying object and the corresponding emotional label according to the processed vector set to calculate the emotional contradiction degree Qmd, and identify whether the facial expression of the current accompanying object belongs to a mask expression based on the emotional contradiction degree Qmd value; S4: If it is a mask-like expression, then the initial settings and historical data are combined to analyze the feedback intensity required by the intelligent companion robot when facing the mask-like expression of the companion object, so as to construct a feedback response intensity value Fqdz, and based on the feedback response intensity value Fqdz, dynamically adjust the feedback degree of the intelligent companion robot.

2. The method for facial expression recognition based on an intelligent companion robot according to claim 1, characterized in that: The specific steps of S1 include: S11: collecting video data of the face and limbs of the companion object in real time through an image acquisition device preset in the intelligent companion robot, extracting continuous facial images and limb behavior images from the video data, and monitoring the voice data of the companion object in real time through a voice acquisition device, wherein the image acquisition device includes a camera and the voice acquisition device includes a microphone; S12: Use Haar cascade classifier to identify the facial area in the facial image, use OpenPose tool to extract the 3D coordinates of the joints in the body behavior image, and use Kalman filtering to remove background noise from the speech data; S13: Based on S12, facial landmark detection technology is used to extract the key point coordinates and key coordinate changes in the facial area of ​​the facial image to obtain the facial feature vector , the facial feature vector The expression is: ; Based on the 3D coordinates of the joint points in the extracted limb behavior image and the relative relationship of the joint point positions, a limb feature vector is generated , the limb feature vector The expression is: ; And extract the features of the speech data after background noise removal to obtain the speech feature vector , the speech feature vector The expression is: ; The vector set includes facial feature vectors , limb feature vector and speech feature vector .

3. The method for facial expression recognition based on an intelligent companion robot according to claim 2, characterized in that: The specific steps of S2 include: S21: Using deep learning technology, and based on the expression samples and emotion labels stored in the FER-2013 dataset, we build and train an expression recognition model. Through the trained expression recognition model, we use the facial feature vectors obtained in S1 to identify the facial features of the facial expression. Facial expression classification is performed to identify the facial expression of the current companion, and the facial expression of the current companion is mapped to the corresponding emotion category through the expression samples and emotion labels stored in the FER-2013 dataset to determine the emotion label corresponding to the facial expression of the current companion.

4. The method for facial expression recognition based on an intelligent companion robot according to claim 3, characterized in that: The specific steps of S2 also include: S22: By using the normalization technology, all related vectors in the vector set are aligned so that the related vectors in the vector set have the same scale and direction; S23: Facial feature vector after vector alignment using the emotion space mapping algorithm , limb feature vector and speech feature vector are mapped to the same emotion space coordinate system to form a processed vector set, where the processed vector set includes the processed facial feature vectors , processed limb feature vector And the processed speech feature vector , the processed facial feature vector The expression is: ; Processed limb feature vector The expression is: ; Processed speech feature vector The expression is: .

5. The method for facial expression recognition based on an intelligent companion robot according to claim 4, characterized in that: The specific steps of S3 include: S31: Based on the processed facial feature vector And the processed speech feature vector , analyze the difference between the facial expression and voice state of the current companion, and obtain the difference between facial expression and voice features Myc. Specifically, the difference between facial expression and voice emotion information Myc is obtained according to the following formula: ； In the formula, is the modulus length of the processed facial feature vector, is the modulus length of the processed speech feature vector, is the dot product of the processed facial feature vector and the processed voice feature vector; S32: Based on the processed facial feature vector And the processed limb feature vector , analyze the difference between the facial expression and body state of the current companion, and obtain the difference between the facial expression and body characteristics Mtc. Specifically, the difference between the facial expression and body characteristics Mtc is obtained according to the following formula: ； In the formula, It is expressed as the modulus of the processed limb feature vector.

6. The method for facial expression recognition based on an intelligent companion robot according to claim 5, characterized in that: The specific steps of S3 also include: S33: According to the processed vector set and in combination with the contents of S31 and S32, the contradiction degree between the facial expression of the companion object and the corresponding emotional label is analyzed to calculate the emotional contradiction degree Qmd. The emotional contradiction degree Qmd is specifically obtained according to the following formula: ； In the formula, and They are respectively represented as the weight values ​​of the difference between facial expression and voice emotion information Myc and the difference between facial expression and body features Mtc, where 0 < <1,0< <1, and The specific value is set by the user according to the situation.

7. The method for facial expression recognition based on an intelligent companion robot according to claim 1, characterized in that: The specific steps of S3 also include: S34: presetting a difference threshold K, and comparing and analyzing the difference threshold K with the emotional conflict degree Qmd to identify whether the facial expression of the current companion is a mask-like expression, and the specific identification content is as follows: If the emotional conflict degree Qmd≤the abnormal threshold value K, it will be recognized that the facial expression of the current companion is a mask-like expression, and a feedback adjustment instruction will be triggered externally; If the emotional conflict degree Qmd>the abnormal threshold K, it will be recognized that the facial expression of the current companion does not belong to the mask expression, and the feedback adjustment instruction will not be triggered externally.

8. The method for facial expression recognition based on an intelligent companion robot according to claim 7, characterized in that: The specific steps of S4 include: S41: After receiving the feedback adjustment instruction, the feedback strength required by the intelligent companion robot for the mask-like expression currently facing the companion object is analyzed in combination with the initial settings and historical data to construct a feedback response strength value Fqdz, wherein the initial settings include the initial set feedback response strength value The historical data includes the emotional conflict degree at each monitoring time point in the historical period, and the average emotional conflict degree in the historical period is obtained by statistical averaging algorithm. The feedback response strength value Fqdz is obtained by the following formula: ； In the formula, Indicates the initial set feedback response strength value, Expressed as the feedback adjustment coefficient, Expressed as emotional conflict.

9. The method for facial expression recognition based on an intelligent companion robot according to claim 8, characterized in that: The specific steps of S4 also include: S42: Feedback adjustment coefficient It reflects the sensitivity of the regulatory system to emotional misjudgment, which can be obtained according to the following formula: ； In the formula, represents the indicator function, Represents the average degree of emotional conflict during the historical period.

10. The method for facial expression recognition based on an intelligent companion robot according to claim 1, characterized in that: The specific steps of S4 also include: S43: Based on the feedback adjustment coefficient in S42 , and dynamically adjust the feedback degree of the intelligent companion robot by combining the emotional label corresponding to the facial expression of the current companion object.

Images