AI Digital Human Interaction Methods and Systems Based on Multimodal Emotion Computing

Abstract

This invention relates to the field of human-computer interaction technology, and more particularly to an AI digital human interaction method and system based on multimodal emotion computing. The method includes: collecting behavioral state indicator parameters; analyzing behavioral state indicator representation values; determining whether the operational stability of the target AI digital human interaction meets a standard based on the difference between the behavioral state indicator representation values ​​and a predetermined behavioral state indicator representation threshold; in response to the target AI digital human interaction operational stability meeting the standard, collecting user experience indicator parameters; analyzing user experience indicator representation values; determining whether the target AI digital human interaction feedback is abnormal based on the comparison between the user experience indicator representation values ​​and a predetermined user experience indicator representation threshold; and in response to the target AI digital human interaction feedback being abnormal, determining a processing strategy based on the difference between the user experience indicator representation values ​​and the predetermined user experience indicator representation threshold. This invention improves the optimization efficiency of AI digital human interaction through adaptive control.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to an AI digital human interaction method and system based on multimodal emotion computing. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI digital humans, as an emerging human-computer interaction interface, have been widely applied in various fields such as customer service, education, and entertainment. Their core objective is to simulate and achieve a natural, fluent, and emotionally resonant human dialogue experience. One key technological path to improving the interactive experience of AI digital humans is the introduction of multimodal affective computing, which aims to identify users' emotional states by analyzing their voice, vision, and text information, and thereby generate more empathetic responses. However, in existing AI digital human interaction systems based on multimodal affective computing, emotional state recognition is often limited to a single or finite modality, resulting in a one-sided evaluation dimension; system responses are lagging, and strategy adjustment mechanisms are static and rigid, making it difficult to adapt to users' dynamic emotional needs in real time; simultaneously, process monitoring and result evaluation are disconnected, lacking closed-loop linkage, leading to low optimization efficiency and difficulty in continuously evolving the experience.

[0003] Chinese Patent Publication No. CN119473003A discloses an AI digital human interaction system based on emotion recognition, relating to the field of affective computing technology. Through multimodal emotion data collection and processing via initial and secondary emotion recognition modules, the system improves the accuracy of emotion recognition. By calculating a first and second emotion coefficient, the system effectively distinguishes complex emotions such as fatigue and frustration experienced by learners in virtual educational tasks. Secondary emotion recognition reduces misjudgments and ensures accurate emotion state identification. Through an emotion feedback module, the system dynamically adjusts the AI ​​digital human's facial expressions, tone of voice, and interaction frequency based on the second emotion coefficient, making emotional feedback more natural and enhancing learners' emotional identification. Furthermore, a cultural adaptation analysis module collects cross-cultural emotion data based on the learner's cultural background, calculates and evaluates the cultural adaptation coefficient, and automatically adjusts emotion recognition and feedback strategies, improving the system's adaptability and accuracy of emotional expression in multicultural environments.

[0004] Chinese Patent Publication No. CN120540557A discloses a multimodal AI digital human intelligent interaction method, system, and device. The method includes: pre-waking up the digital human when a face is detected; further fully waking up the digital human based on recognized preset voice or gesture information; acquiring the user's voice and video information during the interaction and generating keyword extraction results, gesture recognition results, and emotional state tags; retrieving relevant information using a pre-built knowledge base and generating response text using a large language generation model module; inputting the response text into a preset speech synthesis model to generate emotionally expressive speech output; and driving the digital human to output emotionally animated content based on the user's current emotional state tags. This invention can create digital humans that understand user emotions, generate personalized responses, provide emotionally rich speech, and display natural expressions and movements, enabling better interaction with users and providing more humanized and effective services.

[0005] Therefore, it is evident that the existing technology has the following problems: Existing AI digital human interaction technologies suffer from problems such as limited emotional state recognition dimensions, delayed emotional feedback, and static interaction strategy adjustment mechanisms, leading to incomplete and untimely evaluation of user emotional experiences and resulting in low optimization efficiency for AI digital human interaction. Summary of the Invention

[0006] To address this, the present invention provides an AI digital human interaction method and system based on multimodal emotion computing, which overcomes the problems of existing AI digital human interaction technologies, such as single dimension of emotion state recognition, delayed emotion feedback, and static interaction strategy adjustment mechanism leading to incomplete evaluation of user emotional experience and untimely response, resulting in low optimization efficiency of AI digital human interaction.

[0007] To achieve the aforementioned objective, this invention provides an AI digital human interaction method based on multimodal emotion computing, comprising: Collect behavioral status index parameters of the target AI digital human interaction model within a historical period; Analyze the behavioral state index representation values ​​based on the aforementioned behavioral state index parameters; The difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold is used to determine whether the interactive operation stability of the target AI digital human meets the standard. In response to the target AI digital human's interactive operation stability meeting the standards, user experience index parameters of the target AI digital human's interactive model were collected over a historical period. Analyze the user experience indicator representation values ​​based on the aforementioned user experience indicator parameters; The comparison between the user experience index representation value and the predetermined user experience index representation threshold determines whether the target AI digital human's interactive feedback is abnormal. In response to abnormal interaction feedback from the target AI digital human, a processing strategy is determined based on the difference between the user experience index representation value and the predetermined user experience index representation threshold; and the adjustment range of the behavior state index representation threshold is determined. The behavioral status index parameters include speech delay duration, eye contact rate, and speech rate change rate; The user experience metrics include user-initiated interruption rate, average number of sessions, and negative feedback rate.

[0008] Furthermore, the process of analyzing the behavioral state index parameters to determine the behavioral state index representation value includes: Collect data on the speech delay duration, eye contact rate, and speech rate change rate of the target AI digital human interaction model within a historical period; The ratio of the speech delay duration to the predetermined speech delay duration threshold is used as the first row as a limiting characterization parameter; The ratio of the predetermined eye contact rate threshold to the eye contact rate is calculated as the second line of the defined characterization parameter. The ratio of the calculated speech rate change rate to a predetermined speech rate change rate threshold is determined as the third line of the limiting characterization parameter; The summation of the first behavior-limited representation parameter, the second behavior-limited representation parameter, and the third behavior-limited representation parameter is determined as the behavior state index representation value.

[0009] Furthermore, the process of determining whether the target AI digital human's interactive operation stability meets the standard by comparing the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is less than the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to meet the standard.

[0010] Furthermore, the process of determining whether the target AI digital human's interactive operational stability does not meet the standard based on the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is greater than or equal to the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to be non-compliant with the standard.

[0011] Furthermore, the process of analyzing the user experience metric parameters to determine the user experience metric representation value includes: Collect data on the user-initiated interruption rate, average number of sessions, and negative feedback rate of the target AI digital human interaction model within a historical period. The ratio of the user-initiated interruption rate to the predetermined user-initiated interruption rate threshold is determined as the first feedback constraint characterization parameter; The ratio of the predetermined average number of sessions threshold to the average number of sessions is determined as the second feedback constraint characterization parameter; The ratio of the negative feedback rate to the predetermined negative feedback rate threshold is determined as the third feedback constraint characterization parameter; The summation of the first feedback constraint representation parameter, the second feedback constraint representation parameter, and the third feedback constraint representation parameter is determined as the user experience index representation value.

[0012] Furthermore, the process of determining that the target AI digital human's interactive feedback is normal by comparing the user experience indicator value with a predetermined user experience indicator threshold includes: Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the user experience metric value is less than the predetermined user experience metric threshold, then the target AI digital human's interactive feedback is determined to be normal.

[0013] Furthermore, the process of determining abnormal interactive feedback of the target AI digital human by comparing the user experience indicator value with the predetermined user experience indicator threshold includes: Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the user experience metric value is greater than or equal to the predetermined user experience metric threshold, then the target AI digital human's interaction feedback is determined to be abnormal.

[0014] Furthermore, the process of determining the processing strategy based on the difference between the user experience metric representation value and the predetermined user experience metric representation threshold includes: Calculate the difference between the user experience metric value and the predetermined user experience metric threshold; The adjustment range of the behavioral state indicator threshold is determined based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold.

[0015] Furthermore, the process of determining that the target AI digital human's interactive operation stability and feedback both meet the standards includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is less than the predetermined difference threshold, and the user experience indicator value is less than the predetermined user experience indicator threshold, then the target AI digital human's interactive operation stability and feedback both meet the standards.

[0016] Furthermore, the present invention also provides an AI digital human interaction system based on multimodal emotion computing, comprising: The state awareness module is used to collect behavioral state index parameters and user experience index parameters of the target AI digital human's interactive operation stability within a historical period. The sentiment analysis module, which is connected to the state perception module, is used to analyze the behavioral state index representation value and the user experience index representation value. The monitoring and evaluation module, which is connected to the sentiment analysis module, is used to determine whether the stability of the target AI digital human's interactive operation meets the standards. If the difference between the behavior state indicator value and the predetermined behavior state indicator threshold is less than the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to meet the standard. If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is greater than or equal to the predetermined difference threshold, then it is determined that the target AI digital human's interactive operation stability does not meet the standard. The decision-making module, which is connected to the monitoring and evaluation module, is used to determine whether the interaction feedback of the target AI digital human is abnormal in response to the stability of the target AI digital human's interactive operation meeting the standard. If the user experience metric value is less than the predetermined user experience metric threshold, then it is determined that the target AI digital human's interactive feedback is normal. If the user experience metric value is greater than or equal to the predetermined user experience metric threshold, then the target AI digital human's interaction feedback is determined to be abnormal. An interaction management module, which is connected to the decision-making module, is used to respond to abnormal interaction feedback from the target AI digital human and determine a processing strategy based on the difference between the user experience index representation value and the predetermined user experience index representation threshold. The adjustment range of the behavior state indicator threshold is determined based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold. The behavioral status index parameters include speech delay duration, eye contact rate, and speech rate change rate; The user experience metrics include user-initiated interruption rate, average number of sessions, and negative feedback rate.

[0017] Compared with the prior art, the beneficial effect of the present invention is that it provides an AI digital human interaction method and system based on multimodal emotion computing. By constructing a three-tiered linkage system of real-time process monitoring, long-term result evaluation, and closed-loop adaptive adjustment, intelligent, refined, and automated management of AI digital human interaction experience is achieved. Through the fusion and analysis of multimodal emotional behavior parameters such as speech delay duration, eye contact rate, and speech rate change rate, and conditionally triggering correlation analysis with user-initiated interruption rate, average number of conversations, and negative feedback rate, the system overcomes the problems of one-sided evaluation and causal disconnect in traditional methods, achieving precise quantitative attribution from micro-behavior to macro-experience. By calculating the difference between behavioral state representation values ​​and thresholds in real time, the system enables immediate diagnosis and intervention of the interaction process, automatically mapping long-term experience anomalies to processing strategies and dynamic adjustment of behavioral state thresholds. This gives the system the capability for fully automated closed-loop optimization from perception, analysis, decision-making, and execution, significantly improving operational response speed and iteration efficiency. This solution promotes the upgrading of AI digital human interaction experience management from an experience-dependent, static, and lagging manual mode to a data-driven, real-time proactive, and continuously self-evolving intelligent system, fundamentally solving the core pain points of low efficiency, poor effectiveness, and difficulty in sustainability in experience optimization.

[0018] In particular, this invention defines three multimodal behavioral parameters—speech delay duration, eye contact rate, and speech rate change rate—and calculates the ratios of each parameter to their corresponding thresholds before summing them to generate a comprehensive behavioral state index. This enables a quantitative and integrated assessment of user cognitive load, attention, and emotional fluctuations during interaction. Furthermore, by comparing this comprehensive index with a predetermined threshold and introducing the difference threshold as an allowable range, a set of fault-tolerant dynamic judgment logic is constructed. This upgrades traditional single-point rigid judgment to comprehensive tolerance management, reducing false alarms caused by normal fluctuations in a single parameter. It also achieves real-time, automated, and dynamically adjustable accurate assessment of the quality of the interaction process through a unified quantitative scale, laying a core foundation for subsequent adaptive optimization of the system.

[0019] In particular, this invention generates a comprehensive user experience index by summing three key outcome indicators—user-initiated interruption rate, average number of sessions, and negative feedback rate—after comparing each with a predetermined threshold. This enables a quantitative and integrated evaluation of the long-term effectiveness of interactions. Furthermore, by directly comparing this index with the predetermined threshold, it achieves automated and objective diagnosis of the macro-level experience, integrating discrete indicators into a unified metric. This merges different dimensions of data measuring satisfaction, participation, and subjective evaluation into a single, comparable score, resolving the decision-making ambiguity problem in multi-indicator evaluations. It also establishes a clear and efficient automated anomaly detection mechanism, enabling batch and rapid diagnosis of massive historical interaction data, greatly improving operational efficiency.

[0020] In particular, this invention constructs a complete closed loop of perception, analysis, decision-making, and regulation through the collaborative construction of a state perception module, a sentiment analysis module, a monitoring and evaluation module, a decision-making module, and an interaction management module. The state perception module lays the foundation for multi-dimensional quantitative evaluation by integrating the collected process and result data; the sentiment analysis module normalizes and integrates multi-source indicators into clear representation values, solving the problem of evaluation ambiguity; the monitoring and evaluation module realizes real-time diagnosis of the interaction process, transforming lagging monitoring into immediate control; the decision-making module accurately distinguishes between instantaneous problems and system defects through a conditional triggering mechanism, achieving intelligent attribution; finally, the interaction management module executes dynamic strategies based on the difference results and adjusts the threshold in reverse, enabling the system to obtain feedback-based adaptive evolution capabilities. Together, these modules achieve a full-process upgrade from comprehensive perception and accurate evaluation to automatic optimization and continuous evolution, systematically improving the efficiency and quality of interaction experience optimization. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the steps of the AI ​​digital human interaction method based on multimodal emotion computing in an embodiment of the present invention. Figure 2 This invention provides a logic diagram for determining whether a target AI digital human interaction model conforms to a standard. Figure 3 This invention provides a logic diagram for determining whether the interactive feedback of a target AI digital human is abnormal. Figure 4 This is a structural block diagram of an AI digital human interaction system based on multimodal emotion computing, according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0023] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0024] Please see Figure 1 The diagram shown is a flowchart illustrating the steps of an AI digital human interaction method based on multimodal emotion computing according to an embodiment of the present invention. The present invention provides an AI digital human interaction method based on multimodal emotion computing, comprising: Step S1: Collect behavioral state index parameters of the target AI digital human interaction model within the historical period; Step S2: Analyze the behavioral state index representation value based on the behavioral state index parameters; Step S3: Determine whether the stability of the target AI digital human's interactive operation meets the standard based on the difference between the behavior state index representation value and the predetermined behavior state index representation threshold. Step S4: In response to the target AI digital human's interactive operation stability meeting the standard, collect the user experience index parameters of the target AI digital human's interactive model within the historical period; analyze the user experience index representation value based on the user experience index parameters; Step S5: Determine whether the interaction feedback of the target AI digital human is abnormal based on the comparison result between the user experience index representation value and the predetermined user experience index representation threshold. Step S6: In response to abnormal interaction feedback from the target AI digital human, a processing strategy is determined based on the difference between the user experience index representation value and the predetermined user experience index representation threshold; and the adjustment range of the behavior state index representation threshold is determined. The behavioral status index parameters include speech delay duration, eye contact rate, and speech rate change rate; The user experience metrics include user-initiated interruption rate, average number of sessions, and negative feedback rate.

[0025] In this embodiment, a three-layer linkage system of real-time process monitoring, long-term result evaluation, and closed-loop adaptive adjustment is constructed to achieve intelligent, refined, and automated management of the AI ​​digital human interaction experience. By integrating and analyzing multimodal emotional behavior parameters such as speech delay duration, eye contact rate, and speech rate change rate, and performing condition-triggered correlation analysis with user active interruption rate, average number of conversations, and negative feedback rate experience indicators, the system overcomes the problems of one-sided evaluation and causal disconnect of traditional methods, and achieves accurate quantitative attribution from micro-behavior to macro-experience. By calculating the difference between the behavioral state representation value and the threshold in real time, the system achieves immediate diagnosis and intervention of the stability of interactive operation, and automatically maps long-term experience anomalies into processing strategies and dynamic adjustment of behavioral state thresholds. This enables the system to have full-process automated closed-loop optimization capabilities from perception, analysis, decision-making, and execution, significantly improving the operation and maintenance response speed and iteration efficiency.

[0026] Specifically, the process of analyzing the behavioral state index parameters to determine the behavioral state index representation values ​​includes: Collect data on the speech delay duration, eye contact rate, and speech rate change rate of the target AI digital human interaction model within a historical period; The ratio of the speech delay duration to the predetermined speech delay duration threshold is used as the first row as a limiting characterization parameter; The ratio of the predetermined eye contact rate threshold to the eye contact rate is calculated as the second line of the defined characterization parameter. The ratio of the calculated speech rate change rate to a predetermined speech rate change rate threshold is determined as the third line of the limiting characterization parameter; The summation of the first behavior-limited representation parameter, the second behavior-limited representation parameter, and the third behavior-limited representation parameter is determined as the behavior state index representation value.

[0027] In this embodiment, the predetermined speech delay duration threshold, visual contact rate threshold, and tone change rate threshold are all obtained in advance. The speech delay duration, visual contact rate, and speech rate change rate of the target AI digital human interaction model within 3 months of stable operation are collected, and their average values ​​are calculated and determined as the speech delay duration threshold, visual contact rate threshold, and tone change rate threshold.

[0028] In this embodiment, the speech delay duration is calculated by accurately capturing the difference between the timestamps of the user's speech ending and the start of the digital human's response using a speech activity detection algorithm; the eye contact rate relies on camera and computer vision technology, using face detection and eye gaze estimation models to determine the proportion of time the user's gaze overlaps with the preset interaction area; the speech rate variation rate is based on speech signal processing technology, quantifying the degree of fluctuation in speech rhythm by analyzing changes in fundamental frequency trajectory, energy envelope, or syllable boundaries; the technology fusion enables multimodal and quantitative perception of the interaction process, providing a data foundation for real-time evaluation, but its effectiveness is highly dependent on sensor accuracy, algorithm robustness, and on-device processing to ensure user privacy.

[0029] In this embodiment, by defining three multimodal behavioral parameters—speech delay duration, eye contact rate, and speech rate change rate—and calculating the ratios of each parameter to their corresponding thresholds, a comprehensive behavioral state index is generated, thereby achieving a quantitative and integrated assessment of user cognitive load, attention, and emotional fluctuations during the interaction process.

[0030] Please see Figure 2 As shown, this is a logic diagram for determining whether the interactive operation stability of a target AI digital human meets the standard, provided by an embodiment of the present invention. The process for determining whether the interactive operation stability of a target AI digital human meets the standard includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; If the difference between the behavior state indicator value and the predetermined behavior state indicator threshold is less than the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to meet the standard. If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is greater than or equal to the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to be non-compliant with the standard.

[0031] In this embodiment, the predetermined behavioral state index representation threshold is obtained in advance. All behavioral state index representation values ​​of the target AI digital human interaction model are collected within 3 months of stable operation, and their average value is calculated as the behavioral state index representation threshold. The predetermined behavioral state index representation threshold in this embodiment is selected within the range [3.05, 3.35]. Preferably, the predetermined behavioral state index representation threshold in this embodiment is 3.15.

[0032] In this embodiment, the predetermined difference threshold is obtained in advance. The difference between the target AI digital human interaction model behavior status index representation value and the predetermined behavior status index representation threshold is calculated within 3 months of stable operation, and the average value is calculated to determine the difference threshold. In this embodiment, the predetermined difference threshold is selected within the range [0.05, 0.35]. Preferably, the predetermined difference threshold is 0.15.

[0033] In this embodiment, by comparing the difference between the comprehensive characterization value and a predetermined threshold, and introducing the difference threshold as an allowable range, a set of fault-tolerant dynamic judgment logic is constructed. This upgrades the traditional single-point rigid judgment to comprehensive tolerance management, which not only reduces false alarms caused by normal fluctuations of a single parameter, but also achieves real-time, automated, and dynamically adjustable accurate evaluation of the stability quality of interactive operation through a unified quantitative scale, laying a core foundation for subsequent adaptive optimization of the system.

[0034] Specifically, the process of analyzing the user experience metric parameters to determine the user experience metric representation values ​​includes: Collect data on the user-initiated interruption rate, average number of sessions, and negative feedback rate of the target AI digital human interaction model within a historical period. The ratio of the user-initiated interruption rate to the predetermined user-initiated interruption rate threshold is determined as the first feedback constraint characterization parameter; The ratio of the predetermined average number of sessions threshold to the average number of sessions is determined as the second feedback constraint characterization parameter; The ratio of the negative feedback rate to the predetermined negative feedback rate threshold is determined as the third feedback constraint characterization parameter; The summation of the first feedback constraint representation parameter, the second feedback constraint representation parameter, and the third feedback constraint representation parameter is determined as the user experience index representation value.

[0035] It is understood that the number of sessions in the embodiment refers to one session initiated by a user and the other party responding to the completion of the session.

[0036] In this embodiment, the predetermined user-initiated interruption rate threshold, average number of sessions threshold, and negative feedback rate threshold are all obtained in advance. The user-initiated interruption rate, average number of sessions, and negative feedback rate of the target AI digital human interaction model that has been running stably for 3 months are collected, and their average values ​​are calculated as the user-initiated interruption rate threshold, average number of sessions threshold, and negative feedback rate threshold.

[0037] In this embodiment, the formula for calculating the user-initiated interruption rate is as follows: User-initiated interruption rate = Number of interrupted sessions / Total number of sessions In this embodiment, the formula for calculating the negative feedback rate is: Negative feedback rate = Number of negative feedback sessions / Number of feedback sessions Please see Figure 3 As shown, this is a logic diagram for determining whether the interactive feedback of a target AI digital human is abnormal, provided by an embodiment of the present invention. The process for determining whether the interactive feedback of a target AI digital human is abnormal includes: Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the user experience metric value is less than the predetermined user experience metric threshold, then it is determined that the target AI digital human's interactive feedback is normal. If the user experience metric value is greater than or equal to the predetermined user experience metric threshold, then the target AI digital human's interaction feedback is determined to be abnormal.

[0038] In this embodiment, the predetermined user experience index representation threshold is obtained in advance. All user experience index representation values ​​of the target AI digital human interaction model are collected within a stable 3-month period, and their average value is calculated as the predetermined user experience index representation threshold. In this embodiment, the predetermined user experience index representation threshold is selected within the range [3.05, 3.35]. Preferably, the predetermined user experience index representation threshold is 3.15.

[0039] In this embodiment, three key outcome indicators—user-initiated interruption rate, average number of sessions, and negative feedback rate—are used. These indicators are compared with predetermined thresholds and then summed to generate a comprehensive user experience indicator, achieving a quantitative and integrated evaluation of the long-term effectiveness of the interaction. Furthermore, by directly comparing this indicator with the predetermined thresholds, automated and objective diagnosis of the macro-level experience is achieved, completing the integration from discrete indicators to a unified metric. This merges different dimensions of data measuring satisfaction, participation, and subjective evaluation into a single, comparable score, solving the decision ambiguity problem in multi-indicator evaluation. A clear and efficient automated anomaly detection mechanism is established, enabling batch and rapid diagnosis of massive historical interaction data, greatly improving operational efficiency.

[0040] Specifically, the process of determining the processing strategy based on the difference between the user experience metric representation value and the predetermined user experience metric representation threshold includes: Calculate the difference between the user experience metric value and the predetermined user experience metric threshold; The adjustment range of the behavioral state indicator threshold is determined based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold.

[0041] In this embodiment, the predetermined user experience index representation threshold is obtained in advance. All user experience index representation values ​​of the target AI digital human interaction model are collected within a stable 3-month period, and their average value is calculated as the predetermined user experience index representation threshold. In this embodiment, the predetermined user experience index representation threshold is selected within the range [3.05, 3.35]. Preferably, the predetermined user experience index representation threshold is 3.15.

[0042] In this embodiment, the quantitative difference between the user experience index representation value and the preset threshold is accurately calculated, and the specific adjustment value of the underlying behavioral state index representation threshold is dynamically determined based on this difference. When the difference between the user experience index representation value and the predetermined user experience index representation threshold is greater than the predetermined difference threshold, the behavioral state index representation threshold is reduced, thereby constructing an automated decision-making and execution closed loop based on the negative feedback control principle.

[0043] Specifically, the process of determining that the target AI digital human's interactive operation stability and feedback both meet the standards includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is less than the predetermined difference threshold, and the user experience indicator value is less than the predetermined user experience indicator threshold, then the target AI digital human's interactive operation stability and feedback both meet the standards.

[0044] In this embodiment, by requiring that real-time interactive operational stability and long-term user feedback must be met simultaneously, a comprehensive and rigorous quality standard is constructed, realizing a high-confidence, three-dimensional verification of the system's health status. This not only provides a clear quantitative baseline for optimization goals but also guides precise improvement directions through clear attribution. Furthermore, as a key stabilizer in the entire adaptive closed loop, it ensures that the system only temporarily suspends adjustments when it reaches this ideal state, effectively preventing misadjustment and performance oscillations. Thus, experience management is elevated from fuzzy evaluation to a scientific operation and maintenance level that can be precisely controlled and continuously converged.

[0045] Please see Figure 4The diagram shown is a structural block diagram of an AI digital human interaction system based on multimodal emotion computing according to an embodiment of the present invention. The present invention also provides an AI digital human interaction system based on multimodal emotion computing, comprising: The state awareness module is used to collect behavioral state index parameters and user experience index parameters of the target AI digital human interaction model within a historical period. The sentiment analysis module, which is connected to the state perception module, is used to analyze the behavioral state index representation value and the user experience index representation value. The monitoring and evaluation module, which is connected to the sentiment analysis module, is used to determine whether the stability of the target AI digital human's interactive operation meets the standards. If the difference between the behavior state indicator value and the predetermined behavior state indicator threshold is less than the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to meet the standard. If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is greater than or equal to the predetermined difference threshold, then it is determined that the target AI digital human's interactive operation stability does not meet the standard. The decision-making module, which is connected to the monitoring and evaluation module, is used to determine whether the interaction feedback of the target AI digital human is abnormal in response to the stability of the target AI digital human's interactive operation meeting the standard. If the user experience metric value is less than the predetermined user experience metric threshold, then it is determined that the target AI digital human's interactive feedback is normal. If the user experience metric value is greater than or equal to the predetermined user experience metric threshold, then the target AI digital human's interaction feedback is determined to be abnormal. An interaction management module, which is connected to the decision-making module, is used to respond to abnormal interaction feedback from the target AI digital human and determine a processing strategy based on the difference between the user experience index representation value and the predetermined user experience index representation threshold. The adjustment range of the behavior state indicator threshold is determined based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold. The behavioral status index parameters include speech delay duration, eye contact rate, and speech rate change rate; The user experience metrics include user-initiated interruption rate, average number of sessions, and negative feedback rate.

[0046] In this embodiment, a complete closed loop of perception, analysis, decision-making, and adjustment is collaboratively constructed through a state perception module, a sentiment analysis module, a monitoring and evaluation module, a decision-making module, and an interaction management module. The state perception module lays the foundation for multi-dimensional quantitative evaluation by integrating the collected process and result data; the sentiment analysis module normalizes and integrates multi-source indicators into clear representation values, solving the problem of evaluation ambiguity; the monitoring and evaluation module realizes real-time diagnosis of the interaction process, transforming lagging monitoring into immediate control; the decision-making module accurately distinguishes between instantaneous problems and system defects through a conditional triggering mechanism, achieving intelligent attribution; finally, the interaction management module executes dynamic strategies based on the difference results and adjusts the threshold in reverse, enabling the system to obtain feedback-based adaptive evolution capabilities. Together, they realize a full-process upgrade from comprehensive perception and accurate evaluation to automatic optimization and continuous evolution, systematically improving the efficiency and quality of interaction experience optimization.

[0047] The technical solution of the present invention has been described in conjunction with the embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific implementation methods of the embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An AI digital human interaction method based on multimodal emotion computing, characterized in that, include: Collect behavioral status index parameters of the target AI digital human interaction model within a historical period; Analyze the behavioral state index representation values ​​based on the aforementioned behavioral state index parameters; The process of analyzing the behavioral state indicator representation value based on the aforementioned behavioral state indicator parameters includes: Collect data on the speech delay duration, eye contact rate, and speech rate change rate of the target AI digital human interaction model within a historical period; Calculate the ratio of the speaking delay duration to the predetermined speaking delay duration threshold, and determine the first row as a limiting characterization parameter; Calculate the ratio of the predetermined line-of-sight contact rate threshold to the line-of-sight contact rate, and determine the second row as a limiting characterization parameter; Calculate the ratio of the speech rate change rate to a predetermined speech rate change rate threshold to determine the third row as a limiting characterization parameter; The summation of the first behavior-limited representation parameter, the second behavior-limited representation parameter, and the third behavior-limited representation parameter is determined as the behavior state index representation value; The determination of whether the target AI digital human's interactive operational stability meets the standard is based on the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold. The specific process includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; If the difference between the behavior state indicator value and the predetermined behavior state indicator threshold is less than the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to meet the standard. In response to the target AI digital human's interactive operation stability meeting the standards, user experience index parameters of the target AI digital human's interactive model were collected over a historical period. Analyze the user experience indicator representation values ​​based on the aforementioned user experience indicator parameters; The process of analyzing the user experience metric representation value based on the aforementioned user experience metric parameters includes: Collect data on the user-initiated interruption rate, average number of sessions, and negative feedback rate of the target AI digital human interaction model within a historical period. The ratio of the user-initiated interruption rate to the predetermined user-initiated interruption rate threshold is determined as the first feedback constraint characterization parameter; The ratio of the predetermined average number of sessions threshold to the average number of sessions is determined as the second feedback constraint characterization parameter; The ratio of the negative feedback rate to the predetermined negative feedback rate threshold is determined as the third feedback constraint characterization parameter; The summation of the first feedback limiting representation parameter, the second feedback limiting representation parameter, and the third feedback limiting representation parameter is determined as the user experience indicator representation value. The comparison between the user experience index representation value and the predetermined user experience index representation threshold determines whether the target AI digital human's interactive feedback is abnormal. In response to abnormal interaction feedback from the target AI digital human, a processing strategy is determined based on the difference between the user experience index representation value and the predetermined user experience index representation threshold; and the adjustment range of the behavior state index representation threshold is determined. The process of determining the processing strategy includes: Calculate the difference between the user experience metric value and the predetermined user experience metric threshold; The adjustment range of the behavior state indicator threshold is determined based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold. The behavioral status index parameters include speech delay duration, eye contact rate, and speech rate change rate; The user experience metrics include user-initiated interruption rate, average number of sessions, and negative feedback rate.

2. The AI ​​digital human interaction method based on multimodal emotion computing according to claim 1, characterized in that, The process of determining whether the target AI digital human's interactive operation stability does not meet the standard based on the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is greater than or equal to the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to be non-compliant with the standard.

3. The AI ​​digital human interaction method based on multimodal emotion computing according to claim 2, characterized in that, The process of determining that the target AI digital human's interactive feedback is normal based on the comparison between the user experience index representation value and the predetermined user experience index representation threshold includes: Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the user experience metric value is less than the predetermined user experience metric threshold, then the target AI digital human's interactive feedback is determined to be normal.

4. The AI ​​digital human interaction method based on multimodal emotion computing according to claim 3, characterized in that, The process of determining abnormal interactive feedback of the target AI digital human based on the comparison results between the user experience index representation value and the predetermined user experience index representation threshold includes: Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the user experience metric value is greater than or equal to the predetermined user experience metric threshold, then the target AI digital human's interaction feedback is determined to be abnormal.

5. The AI ​​digital human interaction method based on multimodal emotion computing according to claim 4, characterized in that, The process of ensuring that the stability and feedback of the target AI digital human's interactive operation meet the standards includes: Calculate the difference between the behavioral state index representation value and the predetermined behavioral state index representation threshold; Extract the comparison results of user experience metric values ​​with predetermined user experience metric thresholds; If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is less than the predetermined difference threshold, and the user experience indicator value is less than the predetermined user experience indicator threshold, then the target AI digital human's interactive operation stability and feedback both meet the standards.

6. A system using the AI ​​digital human interaction method based on multimodal emotion computing as described in any one of claims 1 to 5, characterized in that, include: The state awareness module is used to collect behavioral state index parameters and user experience index parameters of the target AI digital human interaction model within a historical period. The sentiment analysis module, which is connected to the state perception module, is used to analyze the behavioral state index representation value and the user experience index representation value. The monitoring and evaluation module, which is connected to the sentiment analysis module, is used to determine whether the stability of the target AI digital human's interactive operation meets the standards. If the difference between the behavior state indicator value and the predetermined behavior state indicator threshold is less than the predetermined difference threshold, then the stability of the target AI digital human's interactive operation is determined to meet the standard. If the difference between the behavioral state indicator value and the predetermined behavioral state indicator threshold is greater than or equal to the predetermined difference threshold, then it is determined that the target AI digital human's interactive operation stability does not meet the standard. The decision-making module, which is connected to the monitoring and evaluation module, is used to determine whether the interaction feedback of the target AI digital human is abnormal in response to the stability of the target AI digital human's interactive operation meeting the standard. If the user experience metric value is less than the predetermined user experience metric threshold, then it is determined that the target AI digital human's interactive feedback is normal. If the user experience metric value is greater than or equal to the predetermined user experience metric threshold, then the target AI digital human's interaction feedback is determined to be abnormal. An interaction management module, which is connected to the decision-making module, is used to respond to abnormal interaction feedback from the target AI digital human and determine a processing strategy based on the difference between the user experience index representation value and the predetermined user experience index representation threshold. The adjustment range of the behavior state indicator threshold is determined based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold. The behavioral status index parameters include speech delay duration, eye contact rate, and speech rate change rate; The user experience metrics include user-initiated interruption rate, average number of sessions, and negative feedback rate.

Images