AI digital human interaction system based on emotion recognition

Abstract

The invention discloses an AI digital human interaction system based on emotion recognition, relates to the technical field of emotion calculation, and improves the accuracy of emotion recognition through multi-modal emotion data acquisition and processing of an emotion primary recognition module and a secondary recognition module. The system effectively distinguishes complex emotions, such as fatigue and depression, of the trainee in the virtual education task by calculating the first emotion coefficient and the second emotion coefficient; secondary emotion recognition is carried out, so that misjudgment is reduced, and the emotion state recognition accuracy is ensured; through an emotion feedback module, the system dynamically adjusts the expression, intonation and interaction frequency of the AI digital human according to a second emotion coefficient, so that the emotion feedback is more natural, and the emotion recognition sense of the student is enhanced; besides, the culture adaptation analysis module collects cross-culture emotion data based on the culture background of the trainee, calculates and evaluates a culture adaptation coefficient, and automatically adjusts an emotion recognition and feedback strategy, thereby improving the adaptability of the system in a multi-culture environment and the accuracy of emotion expression.

Description

Technical Field

[0001] The present invention relates to the field of affective computing technology, and in particular to an AI digital human interaction system based on emotion recognition. Background Art

[0002] The origin of AI digital human interaction system can be traced back to the early virtual assistant and human-computer dialogue technology, which was originally used for simple voice recognition and dialogue interaction. With the rapid development of artificial intelligence, emotional computing, computer vision and other technologies, AI digital humans have gradually expanded from single voice interaction to multimodal interaction, and have more anthropomorphic image and emotion recognition capabilities. In recent years, the integration of emotion recognition technology has enabled AI digital humans to perceive the user's emotional changes through multiple signals such as expressions, sounds, and postures, improving the naturalness and personalization of the interaction. This system is widely used in intelligent customer service, virtual education, entertainment, and medical assistance, and has promoted the innovation of intelligent interaction.

[0003] AI digital human interaction systems are widely used in a wide range of fields, including virtual education, but they often have the following technical shortcomings:

[0004] 1. Insufficient accuracy in emotion recognition: Although emotion recognition technology has made great progress, AI digital humans may not be able to accurately distinguish complex emotions because students express a variety of emotions in different scenarios. For example, the expressions of fatigue and frustration may be similar, causing the system to misjudge during recognition and thus fail to provide correct interactive feedback.

[0005] 2. Limitations of emotional expression: AI digital human’s emotional feedback is usually based on preset expression libraries and algorithms, and lacks the ability to dynamically adjust to subtle changes in emotions. For example, when a student shows slight anxiety, the system may not be able to delicately adjust the interaction method, resulting in an emotional response that is not natural or appropriate.

[0006] 3. Differences in cross-cultural understanding of emotions: Emotion recognition models usually rely on data training in a specific cultural context, and there are significant differences in the way emotions are expressed in different cultures. For example, Asian students tend to be more restrained in expressing their emotions, while European and American students are more overt in expressing their emotions. AI systems may not be able to accurately capture these differences, resulting in insufficient adaptability of emotion recognition in cross-cultural environments. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides an AI digital human interaction system based on emotion recognition, which solves the problems of insufficient emotion recognition accuracy, limitations in emotion expression, and cross-cultural differences in emotion understanding in the background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: an AI digital human interaction system based on emotion recognition, including a virtual interaction module, an emotion primary recognition module, an emotion secondary recognition module, an emotion feedback module, a cultural adaptation analysis module and an emotion summary feedback module;

[0009] The virtual interaction module is used to create a digital human interactive environment based on VR and AR for students, in which the AI ​​digital human generates a personalized virtual assistant according to the learning scenario and displays emotions and instructions through 3D virtual characters to help students participate in virtual education tasks;

[0010] The emotion initial recognition module is used to collect multimodal related data, including first facial expression related data, first voice intonation related data and first body movement related data, from the initial to a fixed period when the trainee participates in the virtual education task, and then calculate and evaluate the first emotion coefficient Qems, and start the emotion misjudgment re-recognition instruction;

[0011] The emotion secondary recognition module is used to continue to collect the interactive behavior related data of the students participating in the virtual education task outside the fixed period, including the second facial expression related data, the second voice tone related data and the second body movement related data, and then construct an emotional state set; and identify the emotional pattern characteristics through data clustering analysis, so as to generate and evaluate the second emotional coefficient Xevs;

[0012] The emotional feedback module is used to generate AI digital human feedback instructions for adjusting the content and method of interaction based on the emotional state set and the evaluation content of the second emotional coefficient Xevs, including generating relief prompts and dynamically adjusting the digital human's expression, tone and interaction frequency;

[0013] The cultural adaptation analysis module is used to collect cross-cultural emotion-related data in real time based on the cultural background information of the students; at the same time, it constructs and combines the cross-cultural emotion expression model to correct the cultural deviation of the cross-cultural emotion-related data, and then generates and evaluates the cultural adaptation coefficient Ccx. If an abnormality is detected between the student's emotional expression and the emotion recognition model, the emotion recognition and feedback strategy is automatically adjusted;

[0014] The emotion summary feedback module is used for comprehensive emotion state summary; by collecting the emotion recognition results of the trainees in different time periods, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx, an emotion change curve and an emotion state history set are constructed.

[0015] Preferably, the virtual interaction module builds a virtual interactive environment that matches the learning tasks for students based on preset educational scene data through integrated VR devices and AR devices; by identifying the identity information and learning progress of the students, a personalized virtual assistant suitable for their learning needs is automatically generated, wherein the personalized virtual assistant is presented in the form of a 3D virtual character, and the personalized virtual assistant is used to provide real-time emotional interaction and task guidance to the students.

[0016] Preferably, the emotion initial recognition module includes an emotion initial calculation unit and an emotion initial evaluation unit;

[0017] The emotion initial calculation unit is used to collect multimodal related data in real time through integrated sensors and cameras during the initial stage and fixed period of the student's participation in the virtual education task, and after preprocessing and dimensionless processing, calculate and obtain the first emotion coefficient Qems through the following formula:

[0018]

[0019] Wherein, Qyj represents the eye openness in the multimodal related data, Qzj represents the curvature of the mouth corners in the multimodal related data, Qyd represents the amplitude of intonation fluctuation in the multimodal related data, and Qtd represents the head movement rate in the multimodal related data.

[0020] Preferably, the emotion initial assessment unit performs assessment by presetting the emotion threshold T and the first emotion coefficient Qems, and the specific assessment content is as follows:

[0021] If the first emotion coefficient Qems ≤ emotion threshold T, it means that the current emotion recognition is normal, and the system will directly adopt the current emotion recognition result;

[0022] If the first emotion coefficient Qems>emotion threshold T, the emotion misjudgment recognition program is started, indicating that the current emotion recognition is abnormal.

[0023] Preferably, the emotion sub-recognition module includes an emotion sub-calculation unit and an emotion sub-evaluation unit;

[0024] The emotion sub-calculation unit is used to collect and record the interactive behavior-related data of the trainees in real time through the integrated sensors and cameras during the interaction process, construct an emotional state set after pre-processing and dimensionless processing the interactive behavior-related data, and obtain the second emotion coefficient Xevs by extracting the emotional state set. The specific calculation formula is as follows:

[0025]

[0026] Where Xfy represents the interactive reaction time in the emotional state set, Xyq represents the eye movement frequency in the emotional state set, Xsq represents the body tilt angle in the emotional state set, and Xyb represents the volume change value in the emotional state set.

[0027] Preferably, the emotion sub-evaluation unit is used to compare and evaluate the preset emotion sub-threshold value Y with the second emotion coefficient Xevs, and the specific contents are as follows:

[0028] When the second emotion coefficient Xevs ≤ the emotion sub-threshold Y, it means that the student's emotion is normal; at this time, the result of the first emotion coefficient Qems is a misjudgment, and the current teaching interaction process continues;

[0029] When the second emotion coefficient Xevs>emotional sub-threshold Y, it means that the student's emotions are abnormal. At this time, the result of the first emotion coefficient Qems is correct, and the emotion feedback mechanism is triggered to regulate the student's emotions and adjust the interaction method of the personalized virtual assistant.

[0030] Preferably, the emotional feedback module reflects friendly and relaxed emotional feedback by adjusting the facial expressions of the personalized virtual assistant, including increasing smiles and reducing serious expressions; and displays a peaceful and soothing emotional effect by adjusting the voice and tone of the personalized virtual assistant, including slowing down the speech speed and lowering the pitch of the personalized virtual assistant; and reduces the emotional burden of the trainees by adjusting the interaction frequency of the personalized virtual assistant, including reducing the frequency of instructions for complex tasks.

[0031] Preferably, the cultural adaptation analysis module includes a cultural adaptation quantification unit and a cultural adaptation evaluation unit;

[0032] The cultural adaptation quantification unit collects the third facial expression related data, the third voice intonation related data and the third body movement related data of the trainee again through sensors, camera equipment and voice analysis system, and then combines the cultural background information of the trainee to form cross-cultural emotion related data;

[0033] Finally, based on the cross-cultural emotion-related data, the cultural differences in emotion expression are identified through the cross-cultural emotion expression model to obtain the cultural adaptation coefficient Ccx. The specific calculation formula is as follows:

[0034]

[0035] In the formula, 5 represents the number of lower-level parameters involved in the operation, and i is a subscript index used to represent the sequence number of each time point;

[0036] Cwx represents the frequency of smiles in cross-cultural emotion-related data, Cqd represents the intensity of speech in cross-cultural emotion-related data, Ctx represents the head tilt angle in cross-cultural emotion-related data, Cfy represents the number of cultural background-specific non-verbal behaviors in cross-cultural emotion-related data, and Cys represents the frequency of eye contact in cross-cultural emotion-related data;

[0037] CxD i Indicates the smile frequency Cwx, Cqd at the i-th time point i Represents the sound intensity Cqd, Ctx at the i-th time point i Indicates the head tilt angle Ctx, Cfy at the i-th time point i represents the number of cultural background-specific non-verbal behaviors at the i-th time point Cfy, Cys i Cys represents the eye contact frequency at the i-th time point.

[0038] Preferably, the cultural adaptation evaluation unit is used to compare and evaluate the preset cultural adaptation threshold U with the cultural adaptation coefficient Ccx, and adjust the emotion recognition and feedback strategy. The specific evaluation content is as follows:

[0039] When the cultural adaptation coefficient Ccx ≤ the cultural adaptation threshold U, it means that the student's emotional expression meets the cultural adaptation range of the system model, and the student's emotions are correctly identified in the current cultural context; at this time, the system does not need to be adjusted and continues to use the current emotional feedback strategy;

[0040] When the cultural adaptation coefficient Ccx>cultural adaptation threshold U, it means that the student's emotional expression does not conform to the cultural adaptation range of the system model, and the student's emotions are misidentified in the current cultural context; at this time, the system makes adjustments, including introducing specific emotional expression rules through cultural background data.

[0041] Preferably, the emotion summary feedback module collects in real time the emotion recognition results from the emotion initial recognition module, the emotion secondary recognition module and the cultural adaptation analysis module during different time periods when the students participate in the virtual education task, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx; through the data aggregation algorithm, the emotion coefficients of different time periods are sorted and analyzed to generate a historical set of emotion states, and an emotion change curve is constructed according to the emotion fluctuations of the students, so as to comprehensively display the emotion change trends of the students in the interactive process of the entire virtual education task through the human-computer interaction system.

[0042] The present invention provides an AI digital human interaction system based on emotion recognition. It has the following beneficial effects:

[0043] (1) This AI digital human interaction system based on emotion recognition can significantly improve the accuracy of emotion recognition through multimodal emotion data collection and processing of emotion primary recognition module and emotion secondary recognition module; by calculating the first emotion coefficient Qems and the second emotion coefficient Xevs, the system can effectively distinguish the complex emotional expressions of students in virtual education tasks, such as the subtle difference between fatigue and frustration; when the first emotion coefficient Qems exceeds the preset emotion threshold T, the system triggers the misjudgment recognition program, and corrects the misjudgment by evaluating the second emotion coefficient Xevs and performing secondary recognition, thereby reducing emotion recognition errors, ensuring the accuracy of emotion state recognition, and improving the correctness of interactive feedback and the participation experience of students;

[0044] (2) This AI digital human interaction system based on emotion recognition, through the dynamic adjustment function of the emotional feedback module, can flexibly adjust the expression, tone and interaction frequency based on the student's emotion change speed coefficient Xevs; for example, by adjusting the virtual assistant's expression, speech speed and tone, increasing smiles, reducing serious expressions, slowing down speech speed, etc., it can show more natural emotional feedback; this dynamic feedback mechanism can be adjusted according to the student's anxiety or stress state, so as to make the emotional response more refined, improve the student's emotional identification and interaction quality, and reduce the limitations of emotional expression;

[0045] (3) This AI digital human interaction system based on emotion recognition, through the cultural adaptation analysis module, can collect cross-cultural emotion-related data in real time based on the cultural background information of the students, such as smile frequency Cwx, voice intensity Cqd, head tilt angle Ctx, number of cultural background-specific non-verbal behaviors Cfy and eye contact frequency Cys; by constructing a cross-cultural emotion expression model, the system identifies the impact of cultural differences on emotion expression, calculates the cultural adaptation coefficient Ccx and compares and evaluates it with the preset cultural adaptation threshold U; when the cultural adaptation coefficient Ccx exceeds U, the system automatically adjusts the emotion recognition and feedback strategy to improve the accuracy of cross-cultural emotion understanding; this solves the problem of insufficient adaptability of the emotion recognition model in a cross-cultural environment, enabling the system to more accurately capture the emotional expression of students from different cultural backgrounds and improve the interaction effect of AI digital humans in a multicultural environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the framework structure of an AI digital human interaction system based on emotion recognition in the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Example 1

[0049] See also Figure 1 , the present invention provides an AI digital human interaction system based on emotion recognition, including a virtual interaction module, an emotion primary recognition module, an emotion secondary recognition module, an emotion feedback module, a cultural adaptation analysis module and an emotion summary feedback module;

[0050] The virtual interaction module is used to create a digital human interactive environment based on VR and AR for students, in which the AI ​​digital human generates a personalized virtual assistant according to the learning scenario and displays emotions and instructions through 3D virtual characters to help students participate in virtual education tasks;

[0051] The emotion initial recognition module is used to collect multimodal related data, including first facial expression related data, first voice intonation related data and first body movement related data, from the initial to a fixed period when the trainee participates in the virtual education task, and then calculate and evaluate the first emotion coefficient Qems, and start the emotion misjudgment re-recognition instruction;

[0052] The emotion secondary recognition module is used to continue to collect the interactive behavior related data of the students participating in the virtual education task outside the fixed period, including the second facial expression related data, the second voice tone related data and the second body movement related data, and then construct an emotional state set; and identify the emotional pattern characteristics through data clustering analysis, so as to generate and evaluate the second emotional coefficient Xevs;

[0053] The emotional feedback module is used to generate AI digital human feedback instructions for adjusting the content and method of interaction based on the emotional state set and the evaluation content of the second emotional coefficient Xevs, including generating relief prompts and dynamically adjusting the digital human's expression, tone and interaction frequency;

[0054] The cultural adaptation analysis module is used to collect cross-cultural emotion-related data in real time based on the cultural background information of the students; at the same time, it constructs and combines the cross-cultural emotion expression model to correct the cultural deviation of the cross-cultural emotion-related data, and then generates and evaluates the cultural adaptation coefficient Ccx. If an abnormality is detected between the student's emotional expression and the emotion recognition model, the emotion recognition and feedback strategy is automatically adjusted;

[0055] The emotion summary feedback module is used for comprehensive emotion state summary; by collecting the emotion recognition results of the trainees in different time periods, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx, an emotion change curve and an emotion state history set are constructed.

[0056] In this embodiment, the virtual interaction module can create a digital human interactive environment based on VR and AR for students, and help students participate in virtual education tasks by generating personalized virtual assistants, thereby enhancing the immersive interactive experience; the emotion initial recognition module collects multimodal related data and calculates the first emotion coefficient Qems, thereby realizing the preliminary judgment of the emotional state, and can effectively identify misjudgments and start the re-recognition program to reduce emotion recognition errors; the emotion secondary recognition module further collects emotion-related data of students, constructs an emotion state set and calculates the second emotion coefficient Xevs, thereby identifying the characteristics of emotion patterns and improving the accuracy of emotion recognition; the emotion feedback module dynamically generates an AI number based on the evaluation results of the second emotion coefficient Xevs. The feedback instructions of the characters can adjust facial expressions, intonation and interaction frequency, so as to achieve accurate emotional feedback and enhance the interactive experience of students; the cultural adaptation analysis module collects cross-cultural emotion-related data in real time, combines cultural background information, generates a cultural adaptation coefficient Ccx, and automatically adjusts the emotion recognition and feedback strategy when cultural adaptation deviation is detected, effectively solving the problem of cross-cultural emotion understanding differences; the emotion summary feedback module summarizes the emotion recognition results in different time periods, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx, constructs an emotion change curve and an emotion state history collection, realizes comprehensive tracking and analysis of the students' emotional state, and optimizes subsequent emotional interaction strategies.

[0057] Example 2

[0058] The virtual interaction module builds a virtual interactive environment that matches the learning tasks for students based on preset educational scene data through integrated VR devices and AR devices; by identifying the identity information and learning progress of the students, it automatically generates a personalized virtual assistant that suits their learning needs, where the personalized virtual assistant is presented in the form of a 3D virtual character, and the personalized virtual assistant provides real-time emotional interaction and task guidance to the students.

[0059] In this embodiment, the virtual interaction module integrates VR devices and AR devices to build a virtual interactive environment that is highly matched with the learning tasks for the students based on preset educational scene data; the module automatically generates a virtual assistant that meets the personalized needs of the students by identifying their identity information and learning progress. The virtual assistant is presented in the form of a 3D virtual character and interacts with the students emotionally and guides them in tasks in real time; the setting of this personalized virtual assistant effectively enhances the students' sense of participation and interactive experience, enhances the immersive effect of the virtual learning environment, and makes it easier for students to engage in learning tasks; in the system, this module plays a bridging role, and through personalized adjustments, ensures that the interactive process of virtual educational tasks is more in line with the individual needs of the students, thereby improving learning effects and optimizing emotional interactions; this humanized interactive design enhances the system's emotional understanding and responsiveness to students, thereby improving the overall educational effect.

[0060] Example 3

[0061] The emotion initial recognition module includes an emotion initial calculation unit and an emotion initial evaluation unit;

[0062] The emotion initial calculation unit is used to collect multimodal related data in real time through integrated sensors and cameras during the initial stage and fixed period of the student's participation in the virtual education task, and after preprocessing and dimensionless processing, calculate and obtain the first emotion coefficient Qems through the following formula:

[0063]

[0064] Wherein, Qyj represents the eye openness in the multimodal related data, Qzj represents the curvature of the mouth corners in the multimodal related data, Qyd represents the amplitude of intonation fluctuation in the multimodal related data, and Qtd represents the head movement rate in the multimodal related data.

[0065] The emotion initial assessment unit performs assessment by presetting the emotion threshold T and the first emotion coefficient Qems. The specific assessment contents are as follows:

[0066] If the first emotion coefficient Qems ≤ emotion threshold T, it means that the current emotion recognition is normal, and the system will directly adopt the current emotion recognition result;

[0067] If the first emotion coefficient Qems>emotion threshold T, the emotion misjudgment recognition program is started, indicating that the current emotion recognition is abnormal.

[0068] In this embodiment, through the initial emotion calculation unit and the initial emotion evaluation unit, multimodal related data can be collected and processed in real time during the initial stage and fixed period of the student's participation in the virtual education task, and the first emotion coefficient Qems can be obtained through formula calculation; wherein, the eye openness Qyj reflects the student's attention level and alertness state; the curvature of the mouth corners Qzj is used to identify the changes in the student's expression and distinguish emotions such as pleasure and frustration; the amplitude of voice fluctuation Qyd represents the intensity of emotional changes in the voice, which is used to judge the student's emotional excitement; the head movement rate Qtd reflects the student's body language and emotion-related performance; through the collection of these lower-level parameters, the system can accurately calculate the student's emotional state and obtain the first emotion coefficient Qems to provide a basis for subsequent evaluation; the initial emotion evaluation unit evaluates the first emotion coefficient Qems through a preset emotion threshold T, and starts the misjudgment recognition program based on the evaluation content, indicating that the current emotion recognition may be abnormal, thereby improving the accuracy of emotion recognition, reducing the occurrence of misjudgments, and ensuring that the student's emotions are effectively understood and fed back.

[0069] Example 4

[0070] The emotion sub-recognition module includes an emotion sub-calculation unit and an emotion sub-evaluation unit;

[0071] The emotion sub-calculation unit is used to collect and record the emotion-related data of the trainees in real time through the integrated sensors and cameras during the interaction process, construct an emotion state set after preprocessing and dimensionless processing the emotion-related data, and obtain the second emotion coefficient Xevs by extracting the emotion state set. The specific calculation formula is as follows:

[0072]

[0073] Where Xfy represents the interactive reaction time in the emotional state set, Xyq represents the eye movement frequency in the emotional state set, Xsq represents the body tilt angle in the emotional state set, and Xyb represents the volume change value in the emotional state set.

[0074] The emotion sub-evaluation unit is used to compare and evaluate the preset emotion sub-threshold value Y with the second emotion coefficient Xevs, and the specific contents are as follows:

[0075] When the second emotion coefficient Xevs ≤ the emotion sub-threshold Y, it means that the student's emotion is normal; at this time, the result of the first emotion coefficient Qems is a misjudgment, and the current teaching interaction process continues;

[0076] When the second emotion coefficient Xevs>emotional sub-threshold Y, it means that the student's emotions are abnormal. At this time, the result of the first emotion coefficient Qems is correct, and the emotion feedback mechanism is triggered to regulate the student's emotions and adjust the interaction method of the personalized virtual assistant.

[0077] In this embodiment, through the emotion sub-calculation unit and the emotion sub-evaluation unit, the emotion-related data of the trainee can be collected and recorded in real time during the interaction process, and an emotion state set is constructed after preprocessing and dimensionless processing, and the second emotion coefficient Xevs is calculated by the formula; wherein, the interaction reaction time Xfy reflects the reaction speed of the trainee in the task, and can identify whether the trainee is in an anxious, confused or relaxed state; the eye movement frequency Xyq captures the movement of the trainee's eyes to identify the attention and emotional fluctuations; the body inclination angle Xsq is used to reflect the body language of the trainee and identify the state of relaxation or tension; the volume change value Xyb identifies the degree of emotional excitement by analyzing the change in the volume of the trainee's voice; the collection and calculation of these lower-level parameters enable the system to dynamically evaluate the emotional state of the trainee and generate a second emotion coefficient Xevs for subsequent evaluation; the emotion sub-evaluation unit compares the second emotion coefficient Xevs with the preset emotion sub-threshold Y, and according to the evaluation content of the second emotion coefficient Xevs, the system confirms the emotion and triggers the emotion feedback mechanism, regulates the student's emotion and adjusts the interaction method of the virtual assistant, thereby improving the accuracy of emotion recognition and the student's interactive experience.

[0078] Example 5

[0079] The emotional feedback module reflects friendly and relaxed emotional feedback by adjusting the facial expressions of the virtual assistant, including increasing smiles and reducing serious expressions; and displays a peaceful and soothing emotional effect by adjusting the voice and tone of the personalized virtual assistant, including slowing down the speech speed and lowering the pitch of the personalized virtual assistant; and reduces the emotional burden of students by adjusting the interaction frequency of the personalized virtual assistant, including reducing the frequency of instructions for complex tasks.

[0080] In this embodiment, by dynamically adjusting the virtual assistant's facial expressions, voice intonation, and interaction frequency, a more friendly and emotionally-friendly interactive experience can be provided to students. By increasing smiles and reducing serious expressions, the virtual assistant can show more relaxed and positive emotional feedback, thereby alleviating students' emotional stress. By adjusting speech speed and pitch, the virtual assistant can show a calm and soothing effect, thereby helping students relax when they are anxious or nervous. At the same time, by reducing the frequency of instructions for complex tasks, the system effectively reduces the emotional burden of students, allowing them to maintain a more positive emotional state in learning tasks. In the system, the significance of this module is to enhance the virtual assistant's flexible response capabilities to students' emotional states, and through the emotional feedback mechanism, make the interactive process more personalized and humane, thereby improving students' participation and learning outcomes.

[0081] Example 6

[0082] The cultural adaptation analysis module includes a cultural adaptation quantification unit and a cultural adaptation evaluation unit;

[0083] The cultural adaptation quantification unit collects the third facial expression related data, the third voice intonation related data and the third body movement related data of the trainee again through sensors, camera equipment and voice analysis system, and then combines the cultural background information of the trainee to form cross-cultural emotion related data;

[0084] Finally, based on the cross-cultural emotion-related data, the cultural differences in emotion expression are identified through the cross-cultural emotion expression model to obtain the cultural adaptation coefficient Ccx. The specific calculation formula is as follows:

[0085]

[0086] In the formula, 5 represents the number of lower-level parameters involved in the operation, and i is a subscript index used to represent the sequence number of each time point;

[0087] Cwx represents the frequency of smiles in cross-cultural emotion-related data, Cqd represents the intensity of speech in cross-cultural emotion-related data, Ctx represents the head tilt angle in cross-cultural emotion-related data, Cfy represents the number of cultural background-specific non-verbal behaviors in cross-cultural emotion-related data, and Cys represents the frequency of eye contact in cross-cultural emotion-related data;

[0088] CxD i Indicates the smile frequency Cwx, Cqd at the i-th time point i Represents the sound intensity Cqd, Ctx at the i-th time point i Indicates the head tilt angle Ctx, Cfy at the i-th time point i represents the number of cultural background-specific non-verbal behaviors at the i-th time point Cfy, Cys i represents the eye contact frequency Cys at the i-th time point.

[0089] The cultural adaptation evaluation unit is used to compare and evaluate the preset cultural adaptation threshold U and the cultural adaptation coefficient Ccx, and adjust the emotion recognition and feedback strategy. The specific evaluation content is as follows:

[0090] When the cultural adaptation coefficient Ccx ≤ the cultural adaptation threshold U, it means that the student's emotional expression meets the cultural adaptation range of the system model, and the student's emotions are correctly identified in the current cultural context; at this time, the system does not need to be adjusted and continues to use the current emotional feedback strategy;

[0091] When the cultural adaptation coefficient Ccx> the cultural adaptation threshold U, it means that the student's emotional expression does not conform to the cultural adaptation range of the system model, and the student's emotions are misidentified in the current cultural background; at this time, the system makes adjustments, including introducing specific emotional expression rules through cultural background data;

[0092] In this embodiment, the cross-cultural emotion-related data of the trainees are effectively collected through the cultural adaptation quantification unit and the cultural adaptation evaluation unit, and analyzed in combination with the cultural background information; among them, the smile frequency Cwx is used to identify the intensity of the trainees' emotional expression in different cultures, reflecting the degree of emotional exposure in the cultural background; the voice intensity Cqd is used to judge the changes in the trainees' voice when expressing emotions, and to identify the voice characteristics of emotions in different cultures; the head tilt angle Ctx reflects the changes in the trainees' body posture in different cultures, and is used to judge the body language expression of emotions; the number of non-verbal behaviors specific to the cultural background Cfy is used to identify the unique body language used by the trainees in a specific cultural background, Such as gestures, nodding, etc.; eye contact frequency Cys is used to analyze the eye communication characteristics of students in different cultures, helping to determine the impact of cultural differences on emotional communication; through the collection and calculation of these parameters, the cultural adaptation coefficient Ccx is generated, and compared with the preset cultural adaptation threshold U for evaluation to determine whether the student's emotional expression conforms to the system's cultural adaptation model; when the cultural adaptation coefficient Ccx exceeds the cultural adaptation threshold U, the system automatically adjusts the emotion recognition and feedback strategy, and optimizes the student's emotion recognition and feedback mechanism by introducing specific emotion expression rules, thereby improving the accuracy of cross-cultural emotion understanding and ensuring the system's adaptability and emotional interaction effect in a multicultural context.

[0093] Example 7

[0094] The emotion summary feedback module collects the emotion recognition results from the emotion initial recognition module, the emotion secondary recognition module and the cultural adaptation analysis module in real time during the different time periods when the students participate in the virtual education task, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx; through the data aggregation algorithm, the emotion coefficients of different time periods are sorted and analyzed to generate a historical set of emotion states, and an emotion change curve is constructed according to the emotion fluctuations of the students, and the emotion change trend of the students in the interactive process of the entire virtual education task is fully displayed through the human-computer interaction system.

[0095] In this embodiment, by real-time aggregation of the emotion recognition results of the emotion primary recognition module, the emotion secondary recognition module and the cultural adaptation analysis module, the emotion change data of the students in different time periods can be fully integrated, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx; through the data aggregation algorithm, the module organizes and analyzes the emotion data, generates a historical set of emotion states, and constructs an emotion change curve according to the emotion fluctuations of the students; the advantage of this module is that it can track the emotion changes of the students in the virtual education tasks throughout the process, and provide historical data support for subsequent emotion feedback and emotion recognition strategies, so that the system can adjust the interaction mode more accurately; its significance in the system is that the emotion summary feedback module enables the system to achieve a comprehensive understanding and control of the emotional state of the students, and through the visual display of emotion change trends, ensure that the system can respond to the needs of the students more flexibly and individually in future emotional interactions, and further improve the interaction effect and learning experience.

[0096] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An AI digital human interaction system based on emotion recognition, characterized by: It includes virtual interaction module, emotion initial recognition module, emotion secondary recognition module, emotion feedback module, cultural adaptation analysis module and emotion summary feedback module; The virtual interaction module is used to create a digital human interactive environment based on VR and AR for students, in which the AI ​​digital human generates a personalized virtual assistant according to the learning scenario and displays emotions and instructions through 3D virtual characters to help students participate in virtual education tasks; The emotion initial recognition module is used to collect multimodal related data, including first facial expression related data, first voice intonation related data and first body movement related data, from the initial to a fixed period when the trainee participates in the virtual education task, and then calculate and evaluate the first emotion coefficient Qems, and start the emotion misjudgment re-recognition instruction; The emotion secondary recognition module is used to continue to collect the interactive behavior related data of the students participating in the virtual education task outside the fixed period, including the second facial expression related data, the second voice tone related data and the second body movement related data, and then construct an emotional state set; and identify the emotional pattern characteristics through data clustering analysis, so as to generate and evaluate the second emotional coefficient Xevs; The emotional feedback module is used to generate AI digital human feedback instructions for adjusting the content and method of interaction based on the emotional state set and the evaluation content of the second emotional coefficient Xevs, including generating relief prompts and dynamically adjusting the digital human's expression, tone and interaction frequency; The cultural adaptation analysis module is used to collect cross-cultural emotion-related data in real time based on the cultural background information of the students; at the same time, it constructs and combines the cross-cultural emotion expression model to correct the cultural deviation of the cross-cultural emotion-related data, and then generates and evaluates the cultural adaptation coefficient Ccx. If an abnormality is detected between the student's emotional expression and the emotion recognition model, the emotion recognition and feedback strategy is automatically adjusted; The emotion summary feedback module is used for comprehensive emotion state summary; by collecting the emotion recognition results of the trainees in different time periods, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx, an emotion change curve and an emotion state history set are constructed.

2. The AI ​​digital human interaction system based on emotion recognition according to claim 1, characterized in that: The virtual interaction module builds a virtual interactive environment that matches the learning tasks for students based on preset educational scene data through integrated VR devices and AR devices; by identifying the identity information and learning progress of the students, it automatically generates a personalized virtual assistant that suits their learning needs, where the personalized virtual assistant is presented in the form of a 3D virtual character, and the personalized virtual assistant provides real-time emotional interaction and task guidance to the students.

3. The AI ​​digital human interaction system based on emotion recognition according to claim 1, characterized in that: The emotion initial recognition module includes an emotion initial calculation unit and an emotion initial evaluation unit; The emotion initial calculation unit is used to collect multimodal related data in real time through integrated sensors and cameras during the initial stage and fixed period of the student's participation in the virtual education task, and after preprocessing and dimensionless processing, calculate and obtain the first emotion coefficient Qems through the following formula: Wherein, Qyj represents the eye openness in the multimodal related data, Qzj represents the curvature of the mouth corners in the multimodal related data, Qyd represents the amplitude of intonation fluctuation in the multimodal related data, and Qtd represents the head movement rate in the multimodal related data.

4. The AI ​​digital human interaction system based on emotion recognition according to claim 3, characterized in that: The emotion initial assessment unit performs assessment by presetting the emotion threshold T and the first emotion coefficient Qems. The specific assessment contents are as follows: If the first emotion coefficient Qems ≤ emotion threshold T, it means that the current emotion recognition is normal, and the system will directly adopt the current emotion recognition result; If the first emotion coefficient Qems>emotion threshold T, the emotion misjudgment recognition program is started, indicating that the current emotion recognition is abnormal.

5. The AI ​​digital human interaction system based on emotion recognition according to claim 1, characterized in that: The emotion sub-recognition module includes an emotion sub-calculation unit and an emotion sub-evaluation unit; The emotion sub-calculation unit is used to collect and record the interactive behavior-related data of the trainees in real time through the integrated sensors and cameras during the interaction process, construct an emotional state set after pre-processing and dimensionless processing the interactive behavior-related data, and obtain the second emotion coefficient Xevs by extracting the emotional state set. The specific calculation formula is as follows: Where Xfy represents the interactive reaction time in the emotional state set, Xyq represents the eye movement frequency in the emotional state set, Xsq represents the body tilt angle in the emotional state set, and Xyb represents the volume change value in the emotional state set.

6. The AI ​​digital human interaction system based on emotion recognition according to claim 5, characterized in that: The emotion sub-evaluation unit is used to compare and evaluate the preset emotion sub-threshold value Y with the second emotion coefficient Xevs, and the specific contents are as follows: When the second emotion coefficient Xevs ≤ the emotion sub-threshold Y, it means that the student's emotion is normal; at this time, the result of the first emotion coefficient Qems is a misjudgment, and the current teaching interaction process continues; When the second emotion coefficient Xevs>emotional sub-threshold Y, it means that the student's emotions are abnormal. At this time, the result of the first emotion coefficient Qems is correct, and the emotion feedback mechanism is triggered to regulate the student's emotions and adjust the interaction method of the personalized virtual assistant.

7. The AI ​​digital human interaction system based on emotion recognition according to claim 1, characterized in that: The emotional feedback module adjusts the personalized virtual assistant's facial expressions, including increasing smiles and reducing serious expressions, to reflect friendly and relaxed emotional feedback; and adjusts the personalized virtual assistant's voice tone, including slowing down the personalized virtual assistant's speech speed and lowering the pitch, to reflect a peaceful and soothing emotional effect; Reduce the emotional burden on learners by adjusting the frequency of interactions with personalized virtual assistants, including reducing the frequency of instructions for complex tasks.

8. The AI ​​digital human interaction system based on emotion recognition according to claim 1, characterized in that: The cultural adaptation analysis module includes a cultural adaptation quantification unit and a cultural adaptation evaluation unit; The cultural adaptation quantification unit collects the third facial expression related data, the third voice intonation related data and the third body movement related data of the trainee again through sensors, camera equipment and voice analysis system, and then combines the cultural background information of the trainee to form cross-cultural emotion related data; Finally, based on the cross-cultural emotion-related data, the cultural differences in emotion expression are identified through the cross-cultural emotion expression model to obtain the cultural adaptation coefficient Ccx. The specific calculation formula is as follows: In the formula, 5 represents the number of lower-level parameters involved in the operation, and i is a subscript index used to represent the sequence number of each time point; Cwx represents the frequency of smiles in cross-cultural emotion-related data, Cqd represents the intensity of speech in cross-cultural emotion-related data, Ctx represents the head tilt angle in cross-cultural emotion-related data, Cfy represents the number of cultural background-specific non-verbal behaviors in cross-cultural emotion-related data, and Cys represents the frequency of eye contact in cross-cultural emotion-related data; CxD i Indicates the smile frequency Cwx, Cqd at the i-th time point i Represents the sound intensity Cqd, Ctx at the i-th time point i Indicates the head tilt angle Ctx, Cfy at the i-th time point i represents the number of cultural background-specific non-verbal behaviors at the i-th time point Cfy, Cys i represents the eye contact frequency Cys at the i-th time point.

9. The AI ​​digital human interaction system based on emotion recognition according to claim 8, characterized in that: The cultural adaptation evaluation unit is used to compare and evaluate the preset cultural adaptation threshold U and the cultural adaptation coefficient Ccx, and adjust the emotion recognition and feedback strategy. The specific evaluation content is as follows: When the cultural adaptation coefficient Ccx ≤ the cultural adaptation threshold U, it means that the student's emotional expression meets the cultural adaptation range of the system model, and the student's emotions are correctly identified in the current cultural context; at this time, the system does not need to be adjusted and continues to use the current emotional feedback strategy; When the cultural adaptation coefficient Ccx>cultural adaptation threshold U, it means that the student's emotional expression does not conform to the cultural adaptation range of the system model, and the student's emotions are misidentified in the current cultural context; at this time, the system makes adjustments, including introducing specific emotional expression rules through cultural background data.

10. The AI ​​digital human interaction system based on emotion recognition according to claim 1, characterized in that: The emotion summary feedback module collects the emotion recognition results from the emotion initial recognition module, the emotion secondary recognition module and the cultural adaptation analysis module in real time during the different time periods when the students participate in the virtual education task, including the first emotion coefficient Qems, the second emotion coefficient Xevs and the cultural adaptation coefficient Ccx; through the data aggregation algorithm, the emotion coefficients of different time periods are sorted and analyzed to generate a historical set of emotion states, and an emotion change curve is constructed according to the emotion fluctuations of the students, and the emotion change trend of the students in the interactive process of the entire virtual education task is fully displayed through the human-computer interaction system.

Images