Interaction evaluation method and device for AI (artificial intelligence) digital human

Abstract

The invention discloses an artificial intelligence (AI) digital human interaction evaluation method and device. Comprising the following steps: step 1, defining evaluation indexes, wherein the evaluation indexes comprise task completion evaluation, task complexity evaluation, independent capability evaluation, attention cost evaluation, free time evaluation and lever multiple evaluation; by defining multiple evaluation indexes such as task completion, task complexity, independence capability, attention cost, free time and lever multiple, the performance of the AI digital human in different dimensions can be comprehensively and systematically evaluated, developers are helped to comprehensively understand the interaction capability of the AI digital human, and through collection and interaction analysis of user interaction data, the interaction capability of the AI digital human can be comprehensively and systematically evaluated. Problems and defects existing in the process of the AI digital human are recognized, optimization is carried out according to the evaluation result, and the interaction experience of the user can be obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence (AI) digital humans, and more specifically to an interaction evaluation method and device for an AI digital human. Background Art

[0002] The external manifestations of an AI digital human generate virtual images, actions, and sounds through technologies such as computer graphics, computer vision, and speech synthesis. To create different virtual images, an AI digital human can use videos or 3D models of real people, or methods such as generative adversarial networks (GANs); the ability of an AI digital human to communicate and converse with users depends on technologies such as natural language processing, speech recognition, image recognition, and sentiment analysis. They can understand the intentions and emotions of users, generate appropriate responses and feedback, and achieve anthropomorphic conversations and natural communication.

[0003] Traditional evaluation methods often only focus on the performance of AI digital humans in a single dimension, such as speech recognition accuracy or conversation fluency, etc., lacking a comprehensive and systematic evaluation system to comprehensively evaluate the interaction of AI digital humans. Therefore, a new technical solution is needed to solve this problem. Summary of the Invention

[0004] The purpose of the present invention is to provide an interaction evaluation method and device for an AI digital human, which solves the problem that traditional evaluation methods often only focus on the performance of AI digital humans in a single dimension, such as speech recognition accuracy or conversation fluency, etc., lacking a comprehensive and systematic evaluation system to comprehensively evaluate the interaction of AI digital humans.

[0005] To achieve the above purpose, the present invention provides the following technical solutions: an interaction evaluation method and device for an AI digital human, and the interaction evaluation method for an AI digital human includes the following steps:

[0006] Step 1: Define evaluation indicators: The evaluation indicators include task completion ability evaluation, task complexity evaluation, independent ability evaluation, attention cost evaluation, free time evaluation, and leverage multiple evaluation;

[0007] Step 2: Design evaluation scenarios: According to the application scenarios of the AI digital human, design evaluation scenarios with different task complexities and types to ensure that the evaluation scenarios can comprehensively cover all functions of the AI digital human;

[0008] Step 3: Simulate User Interaction and Data Collection: Use automated testing tools or human testers to simulate the interaction process between users and the AI digital human, ensuring that the behaviors and reactions of the simulated users are authentic and representative, and can reflect the actual interaction needs of users. During the interaction process, collect and record relevant data, such as task completion time, response content, user feedback, etc. Use data collection tools or platforms to ensure the accuracy and integrity of the data;

[0009] Step 4: Scenario Optimization and Iteration: Analyze the collected data, evaluate the performance of the AI digital human in different scenarios and tasks, identify existing problems and deficiencies, such as inaccurate answers, low task execution efficiency, etc. According to the analysis results, optimize the scenario design, such as adjusting the task difficulty, adding new scenario types, etc., to ensure that the optimized scenario design is more in line with the actual capabilities of the AI digital human and user needs;

[0010] Step 5: Evaluation and Analysis Phase: In the designed evaluation scenarios, let the AI digital human perform tasks and record relevant data. According to the evaluation indicators, quantitatively score the performance of the AI digital human, analyze the collected data, calculate the scores of various evaluation indicators, and identify the aspects where the AI digital human performs excellently and the aspects that need improvement;

[0011] Step 6: Provide feedback based on the evaluation results to assist developers in optimizing the AI digital human and improving its performance in various evaluation indicators.

[0012] As a preferred implementation manner of the present invention, in the step 1, the task completion ability evaluation realizes an automated test framework, simulates the interaction process between users and the AI digital human, records key indicators such as the time and success rate of the AI digital human performing tasks, and scores the task completion ability of the AI digital human according to the preset evaluation criteria;

[0013] The task complexity evaluation designs task scenarios with different complexities, such as simple question answering, complex reasoning, etc.

[0014] Through an automated test framework or manual testing, evaluate the performance of the AI digital human in different task scenarios, and score its task complexity processing ability according to indicators such as the success rate and response time of the AI digital human in complex tasks;

[0015] The independent ability evaluation, on the basis of introducing task complexity, simulates a situation of long-term unattended operation, observes and records the autonomous operation ability and task completion situation of the AI digital human in this situation, and scores its independent ability according to indicators such as stability and accuracy during the independent operation of the AI digital human;

[0016] Design a user interface for attention cost assessment, record the number of operations, stay time, etc. of users during the use of AI digital humans, collect the subjective feelings of users on the interaction experience of AI digital humans through user research or questionnaires, and quantitatively evaluate the attention cost of AI digital humans according to the number of user operations, stay time, and subjective feelings;

[0017] For free time assessment, calculate the saved time and energy by comparing the work processes and time consumption of users before and after using AI digital humans, design a user satisfaction survey, collect the subjective feelings of users on the improvement of work efficiency by AI digital humans, and evaluate the free time of AI digital humans by combining objective data and subjective feelings;

[0018] For leverage ratio assessment, design a multi-task concurrent test scenario, simulate the situation where users manage multiple AI digital humans simultaneously, record indicators such as operation efficiency and error rate of users under different task concurrency conditions, and evaluate the leverage ratio gain of AI digital humans according to the performance of users in multi-task concurrent management.

[0019] As a preferred embodiment of the present invention, the interaction evaluation device of the artificial intelligence AI digital human includes a data acquisition module, a processing and analysis module, an evaluation and calculation module, a result display module, a data storage module, a user interaction module, and a user interface module.

[0020] As a preferred embodiment of the present invention, the data acquisition module includes voice data acquisition, image data acquisition, and sensor data acquisition.

[0021] As a preferred embodiment of the present invention, the processing and analysis module includes a data cleaning module, a data conversion module, and a data analysis module.

[0022] As a preferred embodiment of the present invention, the evaluation and calculation module includes selecting evaluation algorithms and indicators, data preprocessing, and evaluation calculation.

[0023] As a preferred embodiment of the present invention, the result display module includes a visualization interface, an interaction control module, and an export function module.

[0024] As a preferred embodiment of the present invention, the data storage module includes a data management module, a data backup and recovery module, and a data access control module.

[0025] As a preferred embodiment of the present invention, the user interaction module includes a graphical user interface and a command line interface.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] Through the definition of multiple evaluation indicators such as task completion ability, task complexity, independent ability, attention cost, free time, and leverage multiple, the present invention can comprehensively and systematically evaluate the performance of AI digital humans in different dimensions, helping developers fully understand the interaction capabilities of AI digital humans. By collecting and analyzing user interaction data, identifying problems and deficiencies of AI digital humans in the process, and optimizing according to the evaluation results, the user interaction experience can be significantly improved, and the satisfaction and trust of users in AI digital humans can be enhanced. The data collection module can efficiently integrate data from different sources, and the processing and analysis module is responsible for cleaning, transforming, and verifying these data to ensure the quality and accuracy of the data, providing a fine data basis for subsequent evaluation work. Through leverage multiple evaluation, the performance of AI digital humans in multi-task concurrent processing can be evaluated, providing developers with the improvement of optimizing the concurrent processing ability of AI digital humans and its application effect in complex task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic flowchart of the interaction evaluation method for the AI digital human of the present invention;

[0029] Figure 2 It is a schematic diagram of the interaction evaluation device for the AI digital human of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0031] Please refer to Figure 1-2 , the present invention provides a technical solution: an interaction evaluation method and device for an AI digital human. The interaction evaluation method for an AI digital human includes the following steps:

[0032] Step 1: Define evaluation indicators: The evaluation indicators include task completion ability evaluation, task complexity evaluation, independent ability evaluation, attention cost evaluation, free time evaluation, and leverage multiple evaluation;

[0033] Step 2: Design evaluation scenarios: According to the application scenarios of AI digital humans, design evaluation scenarios containing different task complexities and types to ensure that the evaluation scenarios can comprehensively cover various functions of AI digital humans;

[0034] Step 3: Simulate User Interaction and Data Collection: Use automated testing tools or human testers to simulate the interaction process between users and the AI digital human, ensuring that the behaviors and reactions of the simulated users are authentic and representative, and can reflect the actual interaction needs of users. During the interaction process, collect and record relevant data, such as task completion time, response content, user feedback, etc. Use data collection tools or platforms to ensure the accuracy and integrity of the data;

[0035] Step 4: Scenario Optimization and Iteration: Analyze the collected data, evaluate the performance of the AI digital human in different scenarios and tasks, identify existing problems and deficiencies, such as inaccurate answers, low task execution efficiency, etc. According to the analysis results, optimize the scenario design, such as adjusting the task difficulty, adding new scenario types, etc., to ensure that the optimized scenario design better conforms to the actual capabilities of the AI digital human and user needs;

[0036] Step 5: Evaluation and Analysis Phase: In the designed evaluation scenarios, let the AI digital human execute tasks and record relevant data. According to the evaluation metrics, quantitatively score the performance of the AI digital human, analyze the collected data, calculate the scores of various evaluation metrics, and identify the aspects where the AI digital human performs well and the aspects that need improvement;

[0037] Step 6: Provide feedback based on the evaluation results to assist developers in optimizing the AI digital human and improving its performance in various evaluation metrics.

[0038] For further improvement, in Step 1, the task completion ability evaluation implements an automated testing framework to simulate the interaction process between users and the AI digital human, record key metrics such as the time and success rate of the AI digital human executing tasks, and score the task completion ability of the AI digital human according to the preset evaluation criteria;

[0039] Task complexity evaluation designs task scenarios with different complexities, such as simple Q&A, complex reasoning, etc.

[0040] Evaluate the performance of the AI digital human in different task scenarios through an automated testing framework or manual testing. According to metrics such as the success rate and response time of the AI digital human in complex tasks, score its task complexity processing ability;

[0041] Independent ability evaluation, on the basis of introducing task complexity, simulates a situation of long-term operation without human intervention, observes and records the autonomous operation ability and task completion of the AI digital human in this situation, and scores its independent ability according to metrics such as stability and accuracy during independent operation;

[0042] Design a user interface for attention cost assessment, record the number of operations, stay time, etc. of users during the use of AI digital humans, collect the subjective feelings of users on the interaction experience of AI digital humans through user research or questionnaires, and quantitatively evaluate the attention cost of AI digital humans based on the number of user operations, stay time, and subjective feelings;

[0043] For free time assessment, by comparing the work processes and time consumption of users before and after using AI digital humans, calculate the saved time and energy, design a user satisfaction survey, collect the subjective feelings of users on the improvement of work efficiency by AI digital humans, and evaluate the free time of AI digital humans by combining objective data and subjective feelings;

[0044] For leverage ratio assessment, design a multi-task concurrent test scenario to simulate the situation where users manage multiple AI digital humans simultaneously, record indicators such as operation efficiency and error rate of users under different task concurrency situations, and evaluate the leverage ratio gain of AI digital humans based on the performance of users in multi-task concurrent management.

[0045] For further improvement, as Figure 2 shown: The interaction evaluation device of the artificial intelligence AI digital human includes a data acquisition module, a processing and analysis module, an evaluation and calculation module, a result display module, a data storage module, a user interaction module, and a user interface module.

[0046] For further improvement, as Figure 2 shown: The data acquisition module includes voice data acquisition, image data acquisition, and sensor data acquisition;

[0047] Voice data acquisition: Capture the user's voice through a microphone, preprocess the voice using audio processing technology, and then transmit the processed voice data to the processing and analysis module through a software interface;

[0048] Image data acquisition: Capture the user's video through a camera, preprocess the video using image processing technology, and then transmit the processed image data to the processing and analysis module through a software interface;

[0049] Sensor data acquisition: According to the type of sensor, collect corresponding physical parameter data, such as acceleration, position information, etc., and then transmit these data to the processing and analysis module through a software interface.

[0050] For further improvement, as Figure 2 shown: The processing and analysis module includes a data cleaning module, a data conversion module, and a data analysis module;

[0051] The data cleaning module performs operations such as duplicate removal, missing value processing, outlier detection and processing on the original data;

[0052] The data conversion module converts unstructured data into structured data or performs conversions between different formats of data.

[0053] Further improved, such as Figure 2 shown: The evaluation calculation module includes selecting evaluation algorithms and metrics, data preprocessing, and evaluation calculation;

[0054] Selecting evaluation algorithms and metrics: According to the evaluation parameters set by the user, select appropriate algorithms and evaluation metrics from the evaluation algorithm library:

[0055] Data preprocessing: Further preprocess the received data to ensure it meets the requirements of the evaluation algorithm;

[0056] Evaluation calculation: Use the selected evaluation algorithms and metrics to calculate the preprocessed data and obtain the evaluation results.

[0057] Further improved, such as Figure 2 shown: The result display module includes a visualization interface, an interactive control module, and an export function module;

[0058] The visualization interface consists of a chart display, a report generation, and a dynamic update module. The chart display module uses chart forms such as bar charts, line charts, and pie charts to clearly present the change trends and comparison situations of evaluation metrics. The report generation module automatically generates a detailed evaluation report. The dynamic update module includes content such as evaluation results and analysis suggestions, which is convenient for users to consult and save. The interface can update the evaluation results in real-time or at regular intervals to ensure that users can obtain the latest evaluation information;

[0059] The interactive control module provides controls for users to interact with the displayed results, such as zooming in, zooming out, dragging, and clicking;

[0060] The export function module allows users to save the evaluation results in file form to the local for subsequent analysis and use.

[0061] Further improved, such as Figure 2 shown: The data storage module includes a data management module, a data backup and recovery module, and a data access control module;

[0062] The data management module is responsible for the storage, retrieval, update, and management of data, providing a structured way to organize data for easy access and maintenance;

[0063] The data backup and recovery module regularly backs up data automatically and tests the recovery ability of the backup data. The backup can be stored on a local or remote server to increase data security

[0064] The data access control module restricts access to data, ensuring that only authorized users or system components can access sensitive data.

[0065] Further improved, such as Figure 2 As shown: The user interaction module includes a graphical user interface and a command-line interface. The setting of these two interaction interfaces increases the convenience of user interaction.

[0066] Working principle: First, clearly define the evaluation metrics to evaluate the interaction behavior of the AI digital human, which includes task completion ability, task complexity, independence ability, attention cost, free time, and leverage ratio. These metrics together constitute the basic framework for evaluating the interaction behavior of the AI digital human. According to the actual application scenarios of the AI digital human, design evaluation scenarios with different task complexities and types. These scenarios need to comprehensively cover all functions of the AI digital human to ensure the comprehensiveness and accuracy of the evaluation. Use automated testing tools or human testers to simulate the interaction process between real users and the AI digital human. During the simulated interaction process, collect and record relevant data, such as task completion time, response content, user feedback, etc. These data will be used for subsequent analysis and evaluation. Analyze the collected data to evaluate the performance of the AI digital human in different scenarios and tasks. Identify existing problems and deficiencies, such as inaccurate answers and low task execution efficiency. According to the analysis results, optimize the evaluation scenarios, such as adjusting the task components and adding new scenario types, to improve the accuracy and efficiency of the evaluation. In the optimized evaluation scenarios, let the AI digital human execute tasks and record relevant data. According to the preset evaluation metrics, perform a custom score on the performance of the AI digital human. The data acquisition module is responsible for collecting data from different sources, including voice, image, and sensor data. These data are transmitted to the processing and analysis module through software interfaces. The processing and analysis module cleans, transforms, and analyzes the collected data to ensure the quality and accuracy of the data. The processed data will be used for subsequent evaluation calculations. The evaluation calculation module calculates the processed data according to the preset evaluation algorithms and metrics to obtain the scores of the AI digital human on various evaluation metrics. The result display module displays the evaluation results and analysis suggestions through a visual interface and interaction. Users can gain an in-depth understanding of the interaction behavior of the AI digital human through this module and perform interaction operations. The data storage module is responsible for data storage, backup, and access control to ensure the security and accessibility of the data. The user module provides a user graphical interface and a command-line interface to facilitate user interaction operations with the evaluation device.

[0067] The foregoing has shown and described the basic principles, main features and advantages of the present invention. For a person skilled in the art, it is obvious that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and without departing from the spirit or basic features of the present invention, the present invention can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

[0068] Finally, several points should be noted: First, in the description of the present application, it should be noted that unless otherwise specified and defined, the terms "installation", "connection", "connection" should be understood in a broad sense, which can be a mechanical connection or an electrical connection, or the communication inside two components, and can be directly connected. "Up", "down", "left", "right", etc. are only used to represent the relative position relationship. When the absolute position of the object being described changes, the relative position relationship may change.

[0069] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An interactive evaluation method for an AI digital human, characterized in that: The interactive evaluation method for AI digital humans includes the following steps: Step 1: Define evaluation metrics: The evaluation metrics include task completion ability evaluation, task complexity evaluation, independent ability evaluation, attention cost evaluation, free time evaluation, and leverage multiple evaluation; Step 2: Design evaluation scenarios: According to the application scenarios of AI digital humans, design evaluation scenarios with different task complexities and types to ensure that the evaluation scenarios can comprehensively cover all functions of AI digital humans; Step 3: Simulate user interaction and data collection: Use automated testing tools or human testers to simulate the interaction process between users and AI digital humans, ensuring that the behaviors and reactions of the simulated users are authentic and representative, and can reflect the actual interaction needs of users. During the interaction process, collect and record relevant data, such as task completion time, answer content, user feedback, etc. Use data collection tools or platforms to ensure the accuracy and integrity of the data; Step 4: Scenario optimization and iteration: Analyze the collected data, evaluate the performance of AI digital humans in different scenarios and tasks, identify existing problems and deficiencies, such as inaccurate answers, low task execution efficiency, etc. According to the analysis results, optimize the scenario design, such as adjusting task difficulty, adding new scenario types, etc., to ensure that the optimized scenario design is more in line with the actual capabilities of AI digital humans and user needs; Step 5: Evaluation and analysis stage: In the designed evaluation scenarios, let AI digital humans execute tasks and record relevant data. According to the evaluation metrics, quantitatively score the performance of AI digital humans, analyze the collected data, calculate the scores of each evaluation metric, and identify the aspects where AI digital humans perform well and the aspects that need improvement; Step 6: According to the evaluation results, provide feedback to assist developers in optimizing AI digital humans and improving their performance in various evaluation metrics.

2. The interactive evaluation method of an artificial intelligence AI digital human according to claim 1, characterized in that: In step 1, the task completion ability evaluation implements an automated testing framework to simulate the interaction process between users and AI digital humans, record key metrics such as the time and success rate of AI digital humans performing tasks, and score the task completion ability of AI digital humans according to the preset evaluation criteria; The task complexity evaluation designs task scenarios with different complexities, such as simple Q&A, complex reasoning, etc. Through an automated testing framework or manual testing, evaluate the performance of AI digital humans in different task scenarios, and score their task complexity processing ability according to metrics such as the success rate and response time of AI digital humans in complex tasks; On the basis of introducing task complexity, the independent ability evaluation simulates a situation without human intervention for a long time, observes and records the autonomous operation ability and task completion of AI digital humans in this situation, and scores their independent ability according to metrics such as stability and accuracy during independent operation; Design a user interface for attention cost assessment, record the number of operations and residence time of users during the use of AI digital humans, etc., collect the subjective feelings of users on the interactive experience of AI digital humans through user research or questionnaires, and quantitatively evaluate the attention cost of AI digital humans according to the number of user operations, residence time and subjective feelings; For free time assessment, calculate the saved time and energy by comparing the work processes and time consumption of users before and after using AI digital humans, design a user satisfaction survey, collect the subjective feelings of users on the improvement of work efficiency by AI digital humans, and evaluate the free time of AI digital humans by combining objective data and subjective feelings; For leverage ratio assessment, design a multi-task concurrent test scenario to simulate the situation where users manage multiple AI digital humans simultaneously, record indicators such as operation efficiency and error rate of users under different task concurrency situations, and evaluate the leverage ratio gain of AI digital humans according to the performance of users in multi-task concurrent management.

3. The interactive evaluation device for an artificial intelligence AI digital human according to claim 1, characterized in that: The interactive evaluation device of the artificial intelligence AI digital human includes a data acquisition module, a processing and analysis module, an evaluation and calculation module, a result display module, a data storage module, a user interaction module and a user interface module.

4. An interactive evaluation device for an artificial intelligence AI digital human according to claim 3, characterized in that: The data acquisition module includes voice data acquisition, image data acquisition and sensor data acquisition.

5. The interactive evaluation device for an artificial intelligence AI digital human according to claim 3, wherein: The processing and analysis module includes a data cleaning module, a data conversion module and a data analysis module.

6. The interactive evaluation device for an artificial intelligence AI digital human according to claim 3, characterized in that: The evaluation and calculation module includes selecting evaluation algorithms and indicators, data preprocessing and evaluation calculation.

7. An interactive evaluation device for an artificial intelligence AI digital human according to claim 3, characterized in that: The result display module includes a visualization interface, an interactive control module and an export function module.

8. The interactive evaluation device for an artificial intelligence AI digital human according to claim 3, characterized in that: The data storage module includes a data management module, a data backup and recovery module and a data access control module.

9. An interactive evaluation device for an artificial intelligence AI digital human according to claim 3, characterized in that: The user interaction module includes a graphical user interface and a command line interface.

Images