AI digital human interaction method and system based on multi-modal emotion calculation

Abstract

The invention relates to the technical field of human-computer interaction, in particular to an AI digital human interaction method and system based on multi-modal emotion calculation, and the method comprises the steps: collecting behavior state index parameters; analyzing the characterization value of the behavior state index; determining whether the target AI digital human interaction operation stability meets a standard or not based on a difference result between the behavior state index characterization value and a preset behavior state index characterization threshold value; in response to the fact that the target AI digital human interaction operation stability meets the standard, collecting user experience index parameters; analyzing the characterization value of the user experience index; determining whether target AI digital human interaction feedback is abnormal or not based on a comparison result of the user experience index representation value and a preset user experience index representation threshold value; and in response to a target AI digital human interaction feedback abnormity, determining a processing strategy based on a difference result between the user experience index characterization value and a predetermined user experience index characterization threshold value. According to the invention, the AI digital human interaction optimization efficiency is improved through adaptive regulation and control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to an AI digital human interaction method and system based on multi-modal emotion computing. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, AI digital humans, as a new human-computer interaction interface, have been widely used in customer service, education, entertainment and other fields. Its core goal is to simulate and realize natural, smooth and emotionally rich human conversation experience. One of the key technical paths to improve the AI digital human interaction experience is to introduce multi-modal emotion computing, which aims to analyze user's multi-channel information of voice, vision and text, recognize the user's emotional state, and generate more empathetic responses accordingly. However, in the existing AI digital human interaction system based on multi-modal emotion computing, the emotional state recognition is often limited to a single or limited modality, resulting in one-sided evaluation dimensions; the system response has a lag and the strategy adjustment mechanism is static and fixed, making it difficult to adapt to the user's dynamic emotional needs in real time; at the same time, the process monitoring and result evaluation are mutually disjointed, lacking closed-loop linkage, resulting in low optimization efficiency and difficulty in continuous evolution of experience.

[0003] Chinese patent publication CN119473003A discloses an AI digital human interaction system based on emotion recognition, which relates to the technical field of emotion computing. Through multi-modal emotion data acquisition and processing of the primary and secondary emotion recognition modules, the accuracy of emotion recognition is improved. The system effectively distinguishes complex emotions such as fatigue and frustration of students in virtual education tasks by calculating the first and second emotion coefficients. It also performs secondary emotion recognition to reduce misjudgment and ensure the accuracy of emotion state recognition. Through the emotion feedback module, the system dynamically adjusts the AI digital human's expression, tone and interaction frequency based on the second emotion coefficient, making the emotional feedback more natural and enhancing the students' emotional identification. In addition, the cultural adaptation analysis module collects cross-cultural emotion data based on the cultural background of students, calculates the cultural adaptation coefficient and evaluates it, automatically adjusts the emotion recognition and feedback strategy, and improves the adaptability and accuracy of emotional expression of the system in a multicultural environment.

[0004] Chinese patent publication No. CN120540557A discloses an AI digital human intelligent interaction method, system and device based on multi-modal, the method comprising: pre-waking up the digital human when a face is detected, and further waking up the digital human based on the recognized preset voice information or preset gesture information; obtaining the voice and video information of the user in the interaction process and generating keyword extraction results, gesture recognition results and emotion state labels, retrieving related information using a pre-constructed knowledge base and generating answer text using a large language generation model module, inputting the answer text into a preset voice synthesis model to generate emotional voice output, and driving the digital human animation based on the current emotion state label of the user to output emotionally. The present application can create a digital human that understands user emotions, generates personalized answers, provides emotionally rich voice, and displays natural expressions and actions, better interacts with users, and provides more personalized and effective services.

[0005] Therefore, the prior art has the following problems: The prior art AI digital human interaction has the problems of single emotional state recognition dimension, emotional feedback lag, static interaction strategy adjustment mechanism, resulting in incomplete evaluation of user emotional experience and delayed response of the system, thereby reducing the optimization efficiency of AI digital human interaction. SUMMARY

[0006] To this end, the present application provides an AI digital human interaction method and system based on multi-modal emotional computing to overcome the problems of single emotional state recognition dimension, emotional feedback lag, static interaction strategy adjustment mechanism, resulting in incomplete evaluation of user emotional experience and delayed response of the system, thereby reducing the optimization efficiency of AI digital human interaction.

[0007] To achieve the above purpose, the present application provides an AI digital human interaction method based on multi-modal emotional computing, comprising: Collecting behavior state index parameters of a target AI digital human interaction model in a historical period; Analyzing behavior state index representation values based on the behavior state index parameters; Determining whether the target AI digital human interaction running stability meets the standard based on the difference between the behavior state index representation values and the predetermined behavior state index representation threshold value; In response to the target AI digital human interaction running stability meeting the standard, collecting user experience index parameters of the target AI digital human interaction model in the historical period; Analyzing user experience index representation values based on the user experience index parameters; Determining whether the target AI digital human interaction feedback is abnormal based on the comparison result between the user experience index representation values and the predetermined user experience index representation threshold value; In response to the target AI digital human interaction feedback being abnormal, a processing strategy is determined based on a difference result of the user experience index representation value and a predetermined user experience index representation threshold value; and an adjustment range of the behavior state index representation threshold value is determined. The behavior state index parameters include speech delay duration, line of sight contact rate, and speech speed change rate. The user experience index parameters include user active interruption rate, average conversation times, and negative feedback rate.

[0008] Further, the process of analyzing the behavior state index representation value based on the behavior state index parameters includes: The speech delay duration, line of sight contact rate, and speech speed change rate of the target AI digital human interaction model in a historical period are collected; The ratio of the speech delay duration to a predetermined speech delay duration threshold value is calculated as a first behavior limiting representation parameter; The ratio of the line of sight contact rate to a predetermined line of sight contact rate threshold value is calculated as a second behavior limiting representation parameter The ratio of the speech speed change rate to a predetermined speech speed change rate threshold value is calculated as a third behavior limiting representation parameter; The first behavior limiting representation parameter, the second behavior limiting representation parameter, and the third behavior limiting representation parameter are summed to determine the behavior state index representation value.

[0009] Further, the process of determining whether the target AI digital human interaction running stability meets the standard based on the difference result of the behavior state index representation value and the predetermined behavior state index representation threshold value includes: The difference between the behavior state index representation value and the predetermined behavior state index representation threshold value is calculated; If the difference between the behavior state index representation value and the predetermined behavior state index representation threshold value is less than a predetermined difference threshold value, it is determined that the target AI digital human interaction running stability meets the standard.

[0010] Further, the process of determining whether the target AI digital human interaction running stability meets the standard based on the difference result of the behavior state index representation value and the predetermined behavior state index representation threshold value includes: The difference between the behavior state index representation value and the predetermined behavior state index representation threshold value is calculated; If the difference between the behavior state index representation value and the predetermined behavior state index representation threshold value is greater than or equal to a predetermined difference threshold value, it is determined that the target AI digital human interaction running stability does not meet the standard.

[0011] Further, the process of analyzing the user experience index representation value based on the user experience index parameters includes: collecting a user voluntary interruption rate, an average conversation times, and a negative feedback rate of the target AI digital human interaction model in a historical period; calculating a ratio of the user voluntary interruption rate and a predetermined user voluntary interruption rate threshold value to determine a first feedback limited representation parameter; calculating a ratio of the average conversation times and a predetermined average conversation times threshold value to determine a second feedback limited representation parameter; calculating a ratio of the negative feedback rate and a predetermined negative feedback rate threshold value to determine a third feedback limited representation parameter; summing the first feedback limited representation parameter, the second feedback limited representation parameter, and the third feedback limited representation parameter to determine the user experience index representation value.

[0012] Further, the process of determining that the target AI digital human interaction feedback is abnormal based on a comparison result of the user experience index representation value and a predetermined user experience index representation threshold value includes: extracting the comparison result of the user experience index representation value and the predetermined user experience index representation threshold value; if the user experience index representation value is less than the predetermined user experience index representation threshold value, determining that the target AI digital human interaction feedback is abnormal.

[0013] Further, the process of determining that the target AI digital human interaction feedback is abnormal based on a comparison result of the user experience index representation value and a predetermined user experience index representation threshold value includes: extracting the comparison result of the user experience index representation value and the predetermined user experience index representation threshold value; if the user experience index representation value is greater than or equal to the predetermined user experience index representation threshold value, determining that the target AI digital human interaction feedback is abnormal.

[0014] Further, the process of determining the processing strategy based on a difference result of the user experience index representation value and a predetermined user experience index representation threshold value includes: calculating the difference between the user experience index representation value and the predetermined user experience index representation threshold value; determining an adjustment range of the behavior state index representation threshold value based on the difference between the user experience index representation value and the predetermined user experience index representation threshold value.

[0015] Further, the process of determining that the target AI digital human interaction running stability and feedback both meet the standard includes: calculating the difference between the behavior state index representation value and the predetermined behavior state index representation threshold value; extracting the comparison result of the user experience index representation value and the predetermined user experience index representation threshold value; If the difference between the behavior state indicator representation value and the predetermined behavior state indicator representation threshold value is less than the predetermined difference threshold value and the user experience indicator representation value is less than the predetermined user experience indicator representation threshold value, the target AI digital human interaction running stability and feedback both meet the standard.

[0016] Further, the present application also provides an AI digital human interaction system based on multi-modal emotion computing, comprising: a state perception module configured to collect behavior state indicator parameters and user experience indicator parameters of the target AI digital human interaction running stability in a historical period; an emotion analysis module connected to the state perception module and configured to analyze the behavior state indicator representation value and the user experience indicator representation value; a monitoring and evaluation module connected to the emotion analysis module and configured to determine whether the target AI digital human interaction running stability meets the standard; If the difference between the behavior state indicator representation value and the predetermined behavior state indicator representation threshold value is less than the predetermined difference threshold value, it is determined that the target AI digital human interaction running stability meets the standard; If the difference between the behavior state indicator representation value and the predetermined behavior state indicator representation threshold value is greater than or equal to the predetermined difference threshold value, it is determined that the target AI digital human interaction running stability does not meet the standard; a decision determination module connected to the monitoring and evaluation module and configured to determine whether the target AI digital human interaction feedback is abnormal in response to the target AI digital human interaction running stability meeting the standard; If the user experience indicator representation value is less than the predetermined user experience indicator representation threshold value, it is determined that the target AI digital human interaction feedback is not abnormal; If the user experience indicator representation value is greater than or equal to the predetermined user experience indicator representation threshold value, it is determined that the target AI digital human interaction feedback is abnormal; an interaction management module connected to the decision determination module and configured to determine a processing strategy based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold value in response to the target AI digital human interaction feedback being abnormal; determine the adjustment range of the behavior state indicator representation threshold value based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold value; The behavior state indicator parameters include speech delay duration, eye contact rate, and speech speed change rate. The user experience indicator parameters include user active interruption rate, average conversation times, and negative feedback rate.

[0017] Compared with the prior art, the present application has the beneficial effects that the present application provides an AI digital human interaction method and system based on multi-modal emotion calculation. By constructing a three-layer linkage system of real-time process monitoring, long-period result evaluation and closed-loop adaptive adjustment, intelligent, refined and automated management of AI digital human interaction experience is realized. By fusing and analyzing multi-modal emotion behavior parameters such as speech delay time, eye contact rate and speech speed change rate, and performing conditional trigger type correlation analysis with user active interruption rate, average conversation times and negative feedback rate experience indicators, the problems of one-sided evaluation and cause disconnection of traditional methods are overcome, and accurate quantitative attribution from micro behavior to macro experience is realized. By calculating the difference between the behavior state representation value and the threshold value in real time, immediate diagnosis and intervention of the interaction process are realized, and long-term experience abnormalities are automatically mapped to processing strategies and dynamic adjustment of the behavior state threshold, so that the system has the full-process automated closed-loop optimization capability from perception, analysis, decision-making and execution, greatly improving the operation and maintenance response speed and iteration efficiency. The scheme promotes AI digital human interaction experience management from experience-dependent, static lagging manual mode to data-driven, real-time active and continuously self-evolving intelligent system, fundamentally solving the core pain points of low experience optimization efficiency, poor effect and difficulty in continuous improvement.

[0018] Especially, by defining three multi-modal behavior parameters of speech delay time, eye contact rate and speech speed change rate, and summing up the ratios after comparing each parameter with the corresponding threshold value, a comprehensive behavior state indicator representation value is generated, realizing the quantitative fusion evaluation of user cognitive load, attention and emotional fluctuation in the interaction process. Then, by comparing the comprehensive representation value with the predetermined threshold value and introducing the difference threshold value as the tolerance interval, a set of dynamic judgment logic with fault tolerance capability is constructed, upgrading the traditional single-point rigid judgment to comprehensive tolerance management, reducing false positives caused by normal fluctuations of a single parameter, and realizing real-time, automated and dynamically adjustable precise evaluation of the quality of the interaction process through a unified quantitative scale, laying a core foundation for subsequent system adaptive optimization.

[0019] Especially, by defining three multi-modal behavior parameters of speech delay time, eye contact rate and speech speed change rate, and summing up the ratios after comparing each parameter with the corresponding threshold value, a comprehensive behavior state indicator representation value is generated, realizing the quantitative fusion evaluation of user cognitive load, attention and emotional fluctuation in the interaction process. Then, by comparing the comprehensive representation value with the predetermined threshold value and introducing the difference threshold value as the tolerance interval, a set of dynamic judgment logic with fault tolerance capability is constructed, upgrading the traditional single-point rigid judgment to comprehensive tolerance management, reducing false positives caused by normal fluctuations of a single parameter, and realizing real-time, automated and dynamically adjustable precise evaluation of the quality of the interaction process through a unified quantitative scale, laying a core foundation for subsequent system adaptive optimization.

[0020] Especially, the application builds a complete perception, analysis, decision-making, and regulation closed loop through the state perception module, the emotion analysis module, the monitoring and evaluation module, the decision-making module, and the interaction management module. The state perception module lays the foundation for multi-dimensional quantitative evaluation by fusing the collected process and result data. The emotion analysis module solves the evaluation ambiguity problem by integrating multi-source indicators into clear representation values. The monitoring and evaluation module realizes real-time diagnosis of the interaction process, changes the lagging monitoring into immediate control. The decision-making module accurately distinguishes between transient problems and system defects through a conditional trigger mechanism, and realizes intelligent attribution. Finally, the interaction management module executes dynamic strategies and reversely adjusts thresholds based on the difference results, so that the system obtains adaptive evolution ability based on feedback, and realizes the whole-process upgrade from comprehensive perception, accurate evaluation to automatic optimization and continuous evolution, systematically improving the efficiency and quality of interaction experience optimization. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A step flowchart of the AI digital human interaction method based on multi-modal emotion computing of the embodiment of the application is shown in the figure. Figure 2 A determination logic diagram for determining whether the target AI digital human interaction model meets the standard is provided for the embodiment of the application. Figure 3 A determination logic diagram for determining whether the target AI digital human interaction feedback is abnormal is provided for the embodiment of the application. Figure 4 A structural block diagram of the AI digital human interaction system based on multi-modal emotion computing of the embodiment of the application. DETAILED DESCRIPTION

[0022] In order to make the objects and advantages of the application clearer, the application will be further described below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the protection scope of the application.

[0023] The preferred embodiments of the application will be described below with reference to the accompanying drawings. Those skilled in the art should understand that the embodiments are only used to explain the technical principles of the application and do not limit the protection scope of the application.

[0024] Please refer to Figure 1 The figure shows a step flowchart of the AI digital human interaction method based on multi-modal emotion computing of the embodiment of the application. The application provides an AI digital human interaction method based on multi-modal emotion computing, which comprises: Step S1, collecting behavior state indicator parameters of a target AI digital human interaction model in a historical period; Step S2, analyzing behavior state indicator representation values based on the behavior state indicator parameters; Step S3, determining whether the target AI digital human interaction running stability meets the standard based on the difference between the behavior state indicator representation value and the predetermined behavior state indicator representation threshold value; Step S4, in response to the target AI digital human interaction running stability meeting the standard, collecting user experience indicator parameters of the target AI digital human interaction model in the historical period; and analyzing a user experience indicator representation value based on the user experience indicator parameters; Step S5, determining whether the target AI digital human interaction feedback is abnormal based on the comparison result between the user experience indicator representation value and the predetermined user experience indicator representation threshold value; Step S6, in response to the target AI digital human interaction feedback being abnormal, determining a processing strategy based on the difference between the user experience indicator representation value and the predetermined user experience indicator representation threshold value; and determining an adjustment range of the behavior state indicator representation threshold value; The behavior state indicator parameters include speech delay duration, line of sight contact rate, and speech speed change rate. The user experience indicator parameters include user active interruption rate, average conversation times, and negative feedback rate.

[0025] In the embodiment, through the construction of a three-layer linkage system of real-time process monitoring, long-period result evaluation, and closed-loop adaptive adjustment, intelligent, refined, and automated management of AI digital human interaction experience is realized. Through the fusion analysis of multi-modal emotional behavior parameters such as speech delay duration, line of sight contact rate, and speech speed change rate, and the conditional trigger type correlation analysis with user experience indicators such as user active interruption rate, average conversation times, and negative feedback rate, the problems of one-sided evaluation and causal disconnection of traditional methods are overcome, and accurate quantitative attribution from micro-behavior to macro-experience is realized. Through real-time calculation of the difference between the behavior state representation value and the threshold value, immediate diagnosis and intervention of the interaction running stability are realized, and long-term experience abnormalities are automatically mapped to processing strategies and dynamic adjustment of the behavior state threshold value, so that the system has the full-process automated closed-loop optimization capability from perception, analysis, decision-making, and execution, and the operation and maintenance response speed and iteration efficiency are greatly improved.

[0026] Specifically, the process of analyzing the behavior state indicator representation value based on the behavior state indicator parameters includes: Collecting speech delay duration, line of sight contact rate, and speech speed change rate of the target AI digital human interaction model in the historical period; Calculating the ratio of the speech delay duration to the predetermined speech delay duration threshold value as a first behavior limited representation parameter; Calculating the ratio of the predetermined line of sight contact rate threshold value to the line of sight contact rate as a second behavior limited representation parameter Calculating the ratio of the speech speed change rate to the predetermined speech speed change rate threshold value as a third behavior limited representation parameter; Summing up the first behavior limiting characterization parameter, the second behavior limiting characterization parameter and the third behavior limiting characterization parameter to determine the behavior state index characterization value.

[0027] In the embodiment, the predetermined speech delay duration threshold value, the visual contact rate threshold value and the speech rate threshold value are obtained in advance, all speech delay durations, visual contact rates and speech rate changes of the target AI digital human interaction model within 3 months of stable operation are collected, and the average values are calculated to determine the speech delay duration threshold value, the visual contact rate threshold value and the speech rate threshold value.

[0028] In the embodiment, the speech delay duration is accurately captured by a speech activity detection algorithm to calculate the timestamp difference between the end of user speech and the start of digital human response; the visual contact rate depends on the camera and computer vision technology, and the coincidence time proportion of the user's visual line and the preset interaction area is determined through face detection and visual line estimation model; the speech rate change rate is based on speech signal processing technology, and the fluctuation degree of speech rhythm is quantified by analyzing the changes of fundamental frequency trajectory, energy envelope or syllable boundary; the fusion of technologies realizes the multi-modal and quantitative perception of the interaction process, and provides a data basis for real-time evaluation, but its effectiveness is highly dependent on the sensor accuracy, algorithm robustness and end-side processing to protect user privacy.

[0029] In the embodiment, by defining the speech delay duration, the visual contact rate and the speech rate change rate as three multi-modal behavior parameters, and calculating the ratio after comparing them with the corresponding threshold values respectively, the comprehensive behavior state index characterization value is generated, and the quantitative fusion evaluation of the user's cognitive load, attention and emotional fluctuation in the interaction process is realized.

[0030] Please refer to Figure 2 As shown in the figure, the embodiment of the present application provides a determination logic diagram for determining whether the running stability of the target AI digital human interaction meets the standard, and the process provided by the embodiment of the present application for determining whether the running stability of the target AI digital human interaction meets the standard includes: Calculating the difference between the behavior state index characterization value and the predetermined behavior state index characterization threshold value; If the difference between the behavior state index characterization value and the predetermined behavior state index characterization threshold value is less than the predetermined difference threshold value, it is determined that the running stability of the target AI digital human interaction meets the standard; If the difference between the behavior state index characterization value and the predetermined behavior state index characterization threshold value is greater than or equal to the predetermined difference threshold value, it is determined that the running stability of the target AI digital human interaction does not meet the standard.

[0031] In the embodiment, the predetermined behavior state indicator threshold value is obtained in advance, all behavior state indicator values of the target AI digital human interaction model within 3 months of stable operation are collected, and the average value is calculated as the behavior state indicator threshold value. The predetermined behavior state indicator threshold value in the embodiment is selected within the interval [3.05, 3.35], and the predetermined behavior state indicator threshold value in the embodiment is preferably 3.15.

[0032] In the embodiment, the predetermined difference threshold value is obtained in advance, the difference between the behavior state indicator value of the target AI digital human interaction model within 3 months of stable operation and the predetermined behavior state indicator threshold value is calculated, and the average value is calculated to determine the difference threshold value. The predetermined difference threshold value in the embodiment is selected within the interval [0.05, 0.35], and the predetermined difference threshold value in the embodiment is preferably 0.15.

[0033] In the embodiment, by comparing the comprehensive indicator value with the predetermined threshold value and introducing the difference threshold value as the tolerance interval, a set of dynamic judgment logic with fault tolerance capability is constructed, which upgrades the traditional single-point rigid judgment to comprehensive tolerance management, reduces false positives caused by normal fluctuations of a single parameter, and realizes real-time, automatic, and dynamically adjustable precise evaluation of the interaction running stability quality through a unified quantitative scale, thereby laying a core foundation for subsequent system adaptive optimization.

[0034] Specifically, the process of analyzing the user experience indicator value by the user experience indicator parameter includes: collecting the user active interruption rate, average session number, and negative feedback rate of the target AI digital human interaction model within a historical period; calculating the ratio of the user active interruption rate to the predetermined user active interruption rate threshold value to determine a first feedback limiting representation parameter; calculating the ratio of the predetermined average session number threshold value to the average session number to determine a second feedback limiting representation parameter; calculating the ratio of the negative feedback rate to the predetermined negative feedback rate threshold value to determine a third feedback limiting representation parameter; summing the first feedback limiting representation parameter, the second feedback limiting representation parameter, and the third feedback limiting representation parameter to determine the user experience indicator value.

[0035] It can be understood that the session number in the embodiment refers to one user initiation, and one user session response completion is determined as one session number.

[0036] In the embodiment, the predetermined user active interruption rate threshold value, the average conversation frequency threshold value and the negative feedback rate threshold value are obtained in advance, the user active interruption rate, the average conversation frequency and the negative feedback rate of the target AI digital human interaction model in the stable operation of 3 months are collected, and the average value is calculated as the user active interruption rate threshold value, the average conversation frequency threshold value and the negative feedback rate threshold value.

[0037] In the embodiment, the calculation formula of the user active interruption rate is: User active interruption rate = interrupted conversation number / total conversation number In the embodiment, the calculation formula of the negative feedback rate is: Negative feedback rate = negative feedback conversation number / feedback conversation number Please refer to Figure 3 As shown in the figure, it is a determination logic diagram for determining whether the target AI digital human interaction feedback is abnormal, and the process for determining whether the target AI digital human interaction feedback is abnormal comprises: Extracting the comparison result of the user experience index representation value and the predetermined user experience index representation threshold value; If the user experience index representation value is less than the predetermined user experience index representation threshold value, it is determined that the target AI digital human interaction feedback is not abnormal; If the user experience index representation value is greater than or equal to the predetermined user experience index representation threshold value, it is determined that the target AI digital human interaction feedback is abnormal.

[0038] In the embodiment, the predetermined user experience index representation threshold value is obtained in advance, all user experience index representation values of the target AI digital human interaction model in the stable operation of 3 months are collected, and the average value is calculated as the predetermined user experience index representation threshold value. In the embodiment, the predetermined user experience index representation threshold value is selected in the interval [3.05, 3.35], and the predetermined user experience index representation threshold value is preferably 3.15.

[0039] In the embodiment, the user active interruption rate, the average conversation frequency and the negative feedback rate are three key result indicators, and the sum is generated after comparing them with the predetermined threshold value respectively to generate a comprehensive user experience index representation value, which realizes the quantitative fusion evaluation of the long-term effectiveness of interaction. Then, by directly comparing the representation value with the predetermined threshold value, the automatic and objective diagnosis of macro experience is realized, the integration from discrete indicators to unified measurement is completed, the different dimension data of measuring satisfaction, participation and subjective evaluation are fused into a single and comparable score, the decision fuzzy problem in multi-index evaluation is solved, an explicit and efficient automatic abnormal judgment mechanism is established, batch and rapid diagnosis of massive historical interaction data is realized, and the operation and maintenance efficiency is greatly improved.

[0040] Specifically, the process of determining the processing strategy based on the difference between the user experience index representation value and the predetermined user experience index representation threshold value includes: calculating the difference between the user experience index representation value and the predetermined user experience index representation threshold value; determining the adjustment range of the behavior state index representation threshold value based on the difference between the user experience index representation value and the predetermined user experience index representation threshold value.

[0041] In an embodiment, the predetermined user experience index representation threshold value is obtained in advance by collecting all user experience index representation values of the target AI digital human interaction model in a stable running period of 3 months and calculating the average value as the predetermined user experience index representation threshold value. In an embodiment, the predetermined user experience index representation threshold value is selected in the interval [3.05, 3.35], and in an embodiment, the predetermined user experience index representation threshold value is 3.15.

[0042] In an embodiment, the quantitative difference between the user experience index representation value and the preset threshold value is accurately calculated, and the specific adjustment value of the underlying behavior state index representation threshold value is dynamically determined based on the difference. When the difference between the user experience index representation value and the predetermined user experience index representation threshold value is greater than the predetermined difference threshold value, the behavior state index representation threshold value is reduced, thereby constructing an automatic decision and execution closed loop based on the negative feedback control principle.

[0043] Specifically, the process of determining that the target AI digital human interaction running stability and feedback both meet the standard includes: calculating the difference between the behavior state index representation value and the predetermined behavior state index representation threshold value; extracting the comparison result of the user experience index representation value and the predetermined user experience index representation threshold value; If the difference between the behavior state index representation value and the predetermined behavior state index representation threshold value is less than the predetermined difference threshold value and the user experience index representation value is less than the predetermined user experience index representation threshold value, then the target AI digital human interaction running stability and feedback both meet the standard.

[0044] In an embodiment, by requiring that real-time interaction running stability and long-term user feedback must both meet the standard, a comprehensive and rigorous quality standard is constructed, achieving high-confidence stereoscopic verification of the system health state; not only providing a clear quantitative baseline for optimization targets, but also guiding precise improvement directions through clear attribution, while serving as a key stabilizer for the entire adaptive closed loop, ensuring that the system only suspends adjustment when reaching the ideal state, effectively preventing misadjustment and performance oscillation, thereby improving experience management from fuzzy evaluation to a scientifically operated level of precise control and continuous convergence.

[0045] Please refer to Figure 4As shown, it is the structural block diagram of the AI digital human interaction system based on multi-modal emotion computing according to the embodiment of the application, and the embodiment of the application also provides an AI digital human interaction system based on multi-modal emotion computing, comprising: A state perception module is used to collect behavior state index parameters and user experience index parameters of the target AI digital human interaction model in a historical period; An emotion analysis module is connected with the state perception module and used to analyze behavior state index representation values and user experience index representation values; A monitoring and evaluation module is connected with the emotion analysis module and used to determine whether the target AI digital human interaction running stability meets the standard; If the difference between the behavior state index representation value and the predetermined behavior state index representation threshold value is less than the predetermined difference threshold value, it is determined that the target AI digital human interaction running stability meets the standard; If the difference between the behavior state index representation value and the predetermined behavior state index representation threshold value is greater than or equal to the predetermined difference threshold value, it is determined that the target AI digital human interaction running stability does not meet the standard; A decision determination module is connected with the monitoring and evaluation module and used to determine whether the target AI digital human interaction feedback is abnormal in response to the target AI digital human interaction running stability meeting the standard; If the user experience index representation value is less than the predetermined user experience index representation threshold value, it is determined that the target AI digital human interaction feedback is not abnormal; If the user experience index representation value is greater than or equal to the predetermined user experience index representation threshold value, it is determined that the target AI digital human interaction feedback is abnormal; An interaction management module is connected with the decision determination module and used to determine the processing strategy based on the difference between the user experience index representation value and the predetermined user experience index representation threshold value in response to the target AI digital human interaction feedback being abnormal; The difference between the user experience index representation value and the predetermined user experience index representation threshold value is used to determine the adjustment range of the behavior state index representation threshold value; The behavior state index parameters include speaking delay duration, line of sight contact rate, and speech speed change rate; The user experience index parameters include user active interruption rate, average conversation times, and negative feedback rate.

[0046] In the embodiment, a complete sensing, analyzing, deciding and adjusting closed loop is constructed by the state sensing module, the emotion analysis module, the monitoring and evaluation module, the decision-making module and the interaction management module. The state sensing module lays the foundation for multi-dimensional quantitative evaluation by fusing the collected process and result data. The emotion analysis module solves the fuzzy evaluation problem by integrating multiple source indicators into clear representation values. The monitoring and evaluation module realizes real-time diagnosis of the interaction process, and changes the lagging monitoring into instant control. The decision-making module accurately distinguishes between transient problems and system defects through a conditional trigger mechanism, and realizes intelligent attribution. Finally, the interaction management module executes dynamic strategies and adjusts thresholds in reverse according to the difference results, so that the system obtains adaptive evolution ability based on feedback, and realizes the whole process upgrade from comprehensive perception, accurate evaluation to automatic optimization and continuous evolution, and systematically improves the efficiency and quality of interaction experience optimization.

[0047] So far, the technical solutions of the present application have been described in combination with the embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to the specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without deviating from the principles of the present application, and the technical solutions after the changes or replacements of the embodiments will fall within the protection scope of the present application.

Claims

1. An AI digital human interaction method based on multi-modal affective computing, characterized in that, Comprising: collecting behavior state index parameters of the target AI digital human interaction model in a historical period; analyzing behavior state index representation values based on the behavior state index parameters; determining whether the target AI digital human interaction running stability meets the standard based on the difference between the behavior state index representation values and the predetermined behavior state index representation threshold values; in response to the target AI digital human interaction running stability meeting the standard, collecting user experience index parameters of the target AI digital human interaction model in the historical period; analyzing user experience index representation values based on the user experience index parameters; determining whether the target AI digital human interaction feedback is abnormal based on the comparison result of the user experience index representation values and the predetermined user experience index representation threshold values; in response to the target AI digital human interaction feedback being abnormal, determining a processing strategy based on the difference between the user experience index representation values and the predetermined user experience index representation threshold values; determining the adjustment range of the behavior state index representation threshold values; wherein the behavior state index parameters include speech delay duration, line of sight contact rate, and speech speed change rate; the user experience index parameters include user active interruption rate, average conversation times, and negative feedback rate.

2. The AI digital human interaction method based on multi-modal sentiment computing according to claim 1, characterized in that, The process of analyzing behavior state index representation values based on the behavior state index parameters includes: collecting speech delay duration, line of sight contact rate, and speech speed change rate of the target AI digital human interaction model in a historical period; calculating the ratio of the speech delay duration to the predetermined speech delay duration threshold value to determine the first behavior limiting representation parameter; calculating the ratio of the predetermined line of sight contact rate threshold value to the line of sight contact rate to determine the second behavior limiting representation parameter; calculating the ratio of the speech speed change rate to the predetermined speech speed change rate threshold value to determine the third behavior limiting representation parameter; summing the first behavior limiting representation parameter, the second behavior limiting representation parameter, and the third behavior limiting representation parameter to determine the behavior state index representation value.

3. The AI digital human interaction method based on multi-modal sentiment computing according to claim 2, characterized in that, The process of determining whether the target AI digital human interaction running stability meets the standard based on the difference between the behavior state index representation values and the predetermined behavior state index representation threshold values includes: calculating the difference between the behavior state index representation values and the predetermined behavior state index representation threshold values; if the difference between the behavior state index representation values and the predetermined behavior state index representation threshold values is less than the predetermined difference threshold value, it is determined that the target AI digital human interaction running stability meets the standard.

4. The AI digital human interaction method based on multi-modal sentiment computing according to claim 3, characterized in that, The process of determining whether the target AI digital human interaction running stability meets the standard based on the difference between the behavior state index representation values and the predetermined behavior state index representation threshold values includes: calculating the difference between the behavior state index representation values and the predetermined behavior state index representation threshold values; if the difference between the behavior state index representation values and the predetermined behavior state index representation threshold values is greater than or equal to the predetermined difference threshold value, it is determined that the target AI digital human interaction running stability does not meet the standard.

5. The AI digital human interaction method based on multi-modal sentiment computing according to claim 4, characterized in that, The process of analyzing user experience index representation values based on the user experience index parameters includes: collecting user active interruption rate, average conversation times, and negative feedback rate of the target AI digital human interaction model in a historical period; A ratio of the user active interruption rate to a predetermined user active interruption rate threshold is calculated to determine a first feedback limiting characteristic parameter; A ratio of the predetermined average session number threshold to the average session number is calculated to determine a second feedback limiting characteristic parameter; A ratio of the negative feedback rate to a predetermined negative feedback rate threshold is calculated to determine a third feedback limiting characteristic parameter; The first feedback limiting characteristic parameter, the second feedback limiting characteristic parameter, and the third feedback limiting characteristic parameter are summed to determine the user experience index characteristic value.

6. The AI digital human interaction method based on multi-modal sentiment computing according to claim 5, characterized in that, The process of determining that the target AI digital human interaction feedback is normal based on a comparison result of the user experience index characteristic value and a predetermined user experience index characteristic threshold includes: extracting the comparison result of the user experience index characteristic value and the predetermined user experience index characteristic threshold; if the user experience index characteristic value is less than the predetermined user experience index characteristic threshold, it is determined that the target AI digital human interaction feedback is normal.

7. The AI digital human interaction method based on multi-modal sentiment computing according to claim 6, characterized in that, The process of determining that the target AI digital human interaction feedback is abnormal based on a comparison result of the user experience index characteristic value and a predetermined user experience index characteristic threshold includes: extracting the comparison result of the user experience index characteristic value and the predetermined user experience index characteristic threshold; if the user experience index characteristic value is greater than or equal to the predetermined user experience index characteristic threshold, it is determined that the target AI digital human interaction feedback is abnormal.

8. The AI digital human interaction method based on multi-modal sentiment computing according to claim 7, characterized in that, The process of determining the processing strategy based on a difference result of the user experience index characteristic value and a predetermined user experience index characteristic threshold includes: calculating the difference between the user experience index characteristic value and the predetermined user experience index characteristic threshold; determining the adjustment range of the behavior state index characteristic threshold based on the difference between the user experience index characteristic value and the predetermined user experience index characteristic threshold.

9. The AI digital human interaction method based on multi-modal sentiment computing according to claim 8, characterized in that, The process of determining that the target AI digital human interaction running stability and feedback both meet the standard includes: calculating the difference between the behavior state index characteristic value and the predetermined behavior state index characteristic threshold; extracting the comparison result of the user experience index characteristic value and the predetermined user experience index characteristic threshold; if the difference between the behavior state index characteristic value and the predetermined behavior state index characteristic threshold is less than a predetermined difference threshold and the user experience index characteristic value is less than the predetermined user experience index characteristic threshold, the target AI digital human interaction running stability and feedback both meet the standard.

10. A system using the AI digital human interaction method based on multi-modal sentiment computing according to claims 1 to 9, characterized in that, It includes: a state perception module for collecting behavior state index parameters and user experience index parameters of the target AI digital human interaction model in a historical period; a sentiment analysis module connected to the state perception module for analyzing behavior state index characteristic values and user experience index characteristic values; a monitoring and evaluation module connected to the sentiment analysis module for determining whether the target AI digital human interaction running stability meets the standard; if the difference between the behavior state index characteristic value and the predetermined behavior state index characteristic threshold is less than the predetermined difference threshold, it is determined that the target AI digital human interaction running stability meets the standard; if the difference between the behavior state index characteristic value and the predetermined behavior state index characteristic threshold is greater than or equal to the predetermined difference threshold, it is determined that the target AI digital human interaction running stability does not meet the standard; a decision determination module connected with the monitoring and evaluation module, configured to determine whether the target AI digital human interaction feedback is abnormal in response to the target AI digital human interaction running stability meeting the standard; if the user experience index representation value is less than the predetermined user experience index representation threshold value, it is determined that the target AI digital human interaction feedback is not abnormal; if the user experience index representation value is greater than or equal to the predetermined user experience index representation threshold value, it is determined that the target AI digital human interaction feedback is abnormal; an interaction management module connected with the decision determination module, configured to determine a processing strategy based on the difference between the user experience index representation value and the predetermined user experience index representation threshold value in response to the target AI digital human interaction feedback being abnormal; determine the adjustment range of the behavior state index representation threshold value based on the difference between the user experience index representation value and the predetermined user experience index representation threshold value; wherein the behavior state index parameters include speaking delay duration, line of sight contact rate, and speech speed change rate; the user experience index parameters include user active interruption rate, average conversation times, and negative feedback rate.

Images