Evaluation methods, devices and equipment for intelligent interactive systems

By generating multi-dimensional user profiles and multimodal interaction data, the intelligent interaction system can efficiently end the simulation interaction process under complex requirements, solving the problem of insufficient service capabilities in existing technologies and improving user experience and system service capabilities.

CN123068893APending Publication Date: 2026-11-10ANT SHENGXIN (SHANGHAI) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611179841.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-04
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing intelligent interaction systems struggle to effectively terminate simulation interaction processes and attribute interaction interruptions when faced with complex user needs, resulting in insufficient service capabilities and a poor user experience.

Method used

By generating multi-dimensional user profiles, the simulation interaction process is initiated, multi-modal interaction data is generated, and the simulation interaction process is automatically terminated or the cause of the interaction interruption is attributed based on the interaction decision results, thereby improving the system service capabilities and user experience.

Benefits of technology

It has achieved efficient service capabilities and improved user experience for intelligent interactive systems under complex user needs. Through accurate simulation of multi-dimensional user profiles and multi-modal interaction data, it has improved the accuracy of evaluation and the service capabilities of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068893A_ABST
    Figure CN123068893A_ABST
Patent Text Reader

Abstract

This specification provides an evaluation method, apparatus, and device for an intelligent interactive system. The method includes: generating a multi-dimensional user profile corresponding to the intelligent interactive system to be evaluated, and initiating a simulated interaction process with the intelligent interactive system. During the simulated interaction process, multimodal interaction data corresponding to the multi-dimensional user profile is generated. Interaction decisions are made based on the multimodal interaction data and the intelligent interactive system's response data to the multimodal interaction data. These interaction decisions are used to: determine if the simulated interaction process meets the conditions for continuing interaction and continue the simulation interaction process; or determine if the simulated interaction process triggers an interaction interruption condition and terminates the simulation interaction process; or determine if the simulated interaction process meets the interaction completion condition and terminates the simulation interaction process. After multiple simulated interaction processes have ended, the intelligent interactive system is attributed to the cause of the interaction interruption based on the process data of the simulated interaction processes that triggered the interaction interruption condition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This document relates to the field of artificial intelligence technology, and in particular to an evaluation method, apparatus and equipment for an intelligent interactive system. Background Technology

[0002] With the development of artificial intelligence technology, users can leverage the service capabilities of intelligent interactive systems to obtain the services they need. For example, a user might request the system to recommend suitable financial products or plan a suitable travel route. As user needs become more complex in real-world scenarios, higher demands are being placed on the service capabilities of intelligent interactive systems. Summary of the Invention

[0003] This specification provides one or more embodiments of an evaluation method, apparatus, and device for an intelligent interactive system. It can initiate a simulation interaction process for the intelligent interactive system, terminate the simulation interaction process when an interaction interruption condition is triggered, and automatically attribute the interaction interruption to the intelligent interactive system, thereby effectively improving the service capability of the intelligent interactive system and enhancing the user's interactive experience.

[0004] To achieve the above technical solution, the embodiments in this specification are implemented as follows: This specification provides one or more embodiments of a method for evaluating an intelligent interactive system, comprising: generating a multi-dimensional user profile corresponding to the intelligent interactive system to be evaluated; and initiating a simulated interaction process with the intelligent interactive system based on the multi-dimensional user profile. In the simulated interaction process, multimodal interaction data corresponding to the multi-dimensional user profile is generated; and interaction decisions are made based on the multimodal interaction data and the intelligent interactive system's response data to the multimodal interaction data. The results of the interaction decisions include: determining that the simulated interaction process meets the conditions for continuing interaction and continuing the simulated interaction process; or determining that the simulated interaction process triggers an interaction interruption condition and ending the simulated interaction process; or determining that the simulated interaction process meets the conditions for completing interaction and ending the simulated interaction process. After multiple simulated interaction processes have ended, the intelligent interactive system is attributed to interaction interruption based on the process data of the simulated interaction processes that triggered the interaction interruption conditions.

[0005] This specification provides one or more embodiments of an evaluation device for an intelligent interactive system, comprising: a process initiation unit, which generates a multi-dimensional user profile corresponding to the intelligent interactive system to be evaluated, and initiates a simulated interaction process with the intelligent interactive system based on the multi-dimensional user profile; an interaction execution unit, which generates multimodal interaction data corresponding to the multi-dimensional user profile during the simulated interaction process, and makes an interaction decision based on the multimodal interaction data and the response data of the intelligent interactive system to the multimodal interaction data, wherein the result of the interaction decision includes: determining that the simulated interaction process meets the conditions for continuing interaction and continuing the simulated interaction process, or determining that the simulated interaction process triggers an interaction interruption condition and ending the simulated interaction process, or determining that the simulated interaction process meets the conditions for completing interaction and ending the simulated interaction process; and an interruption attribution unit, which, after multiple simulated interaction processes have ended, performs interaction interruption attribution on the intelligent interactive system based on the process data of the simulated interaction process that triggered the interaction interruption condition.

[0006] This specification provides one or more embodiments of an evaluation device for an intelligent interactive system, comprising: a processor; and a memory configured to store computer-executable instructions, wherein, when executed, the computer-executable instructions cause the processor to: generate a multi-dimensional user profile corresponding to the intelligent interactive system to be evaluated; and initiate a simulated interaction process with the intelligent interactive system based on the multi-dimensional user profile. In the simulated interaction process, multimodal interaction data corresponding to the multi-dimensional user profile is generated; and an interaction decision is made based on the multimodal interaction data and the intelligent interactive system's response data to the multimodal interaction data. The result of the interaction decision includes: determining that the simulated interaction process meets the conditions for continuing interaction and continuing the simulated interaction process; or determining that the simulated interaction process triggers an interaction interruption condition and ending the simulated interaction process; or determining that the simulated interaction process meets the conditions for completing interaction and ending the simulated interaction process. After multiple simulated interaction processes have ended, based on the process data of the simulated interaction processes that triggered the interaction interruption conditions, an interaction interruption attribution is performed on the intelligent interactive system.

[0007] This specification provides one or more embodiments of a computer-readable storage medium for storing computer-executable instructions that, when executed, implement the following process: For an intelligent interactive system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interactive system is generated; a simulated interaction process with the intelligent interactive system is initiated based on the multi-dimensional user profile. In the simulated interaction process, multimodal interaction data corresponding to the multi-dimensional user profile is generated; an interaction decision is made based on the multimodal interaction data and the intelligent interactive system's response data to the multimodal interaction data. The result of the interaction decision includes: determining that the simulated interaction process meets the conditions for continuing interaction and continuing the simulated interaction process; or determining that the simulated interaction process triggers an interaction interruption condition and ending the simulated interaction process; or determining that the simulated interaction process meets the conditions for completing interaction and ending the simulated interaction process. After multiple simulated interaction processes have ended, based on the process data of the simulated interaction processes that triggered the interaction interruption conditions, an interaction interruption attribution is performed on the intelligent interactive system.

[0008] This specification provides one or more embodiments of a computer program product, including a computer program that, when executed by a processor, implements the following process: For an intelligent interactive system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interactive system is generated; a simulated interaction process with the intelligent interactive system is initiated based on the multi-dimensional user profile. In the simulated interaction process, multimodal interaction data corresponding to the multi-dimensional user profile is generated; an interaction decision is made based on the multimodal interaction data and the intelligent interactive system's response data to the multimodal interaction data. The result of the interaction decision includes: determining that the simulated interaction process meets the conditions for continuing interaction and continuing the simulated interaction process; or determining that the simulated interaction process triggers an interaction interruption condition and ending the simulated interaction process; or determining that the simulated interaction process meets the interaction completion condition and ending the simulated interaction process. After multiple simulated interaction processes have ended, based on the process data of the simulated interaction processes that triggered the interaction interruption conditions, an interaction interruption attribution is performed on the intelligent interactive system. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1A flowchart illustrating an evaluation method for an intelligent interactive system provided in one or more embodiments of this specification; Figure 2 A flowchart illustrating another evaluation method for an intelligent interactive system provided in one or more embodiments of this specification; Figure 3 A flowchart illustrating another evaluation method for an intelligent interactive system provided in one or more embodiments of this specification; Figure 4 A flowchart illustrating another evaluation method for an intelligent interactive system provided in one or more embodiments of this specification; Figure 5 A schematic diagram of a test system provided in one or more embodiments of this specification; Figure 6 This is a schematic diagram illustrating the working process of the test system provided in one or more embodiments of this specification; Figure 7 A schematic diagram of the structure of an evaluation device for an intelligent interactive system provided in one or more embodiments of this specification; Figure 8 This is a structural schematic diagram of an evaluation device for an intelligent interactive system provided in one or more embodiments of this specification. Detailed Implementation

[0010] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0011] Currently, users can utilize the service capabilities of intelligent interactive systems to obtain the services they need. For example, a user might request the intelligent interactive system to recommend suitable financial products, such as insurance or wealth management products. To adapt to more complex user needs and improve the service capabilities of intelligent interactive systems, one or more embodiments of this specification provide an evaluation method for intelligent interactive systems. This method can initiate a simulated interaction process for the intelligent interactive system, terminate the simulated interaction process when an interaction interruption condition is triggered, and automatically attribute the interaction interruption to the intelligent interactive system, thereby effectively improving the service capabilities of the intelligent interactive system and enhancing the user's interactive experience.

[0012] Figure 1This diagram illustrates a flowchart of an evaluation method for an intelligent interactive system provided in one or more embodiments of this specification. This evaluation method can be applied to and executed by a testing system. The testing system may include a single server or a server cluster. It may include an artificial intelligence model based on AI (Artificial Intelligence) capabilities. The testing system can be viewed as an intelligent agent integrating one or more AI models for intelligent testing. Based on the capabilities of the artificial intelligence model, the testing system implements the following method flow to evaluate the intelligent interactive system. Figure 1 As shown, the method includes: Step S102: For the intelligent interaction system to be evaluated, generate a multi-dimensional user profile corresponding to the intelligent interaction system, and start the simulation interaction process between the intelligent interaction system based on the multi-dimensional user profile. Step S104: In the simulation interaction process, multimodal interaction data corresponding to the multidimensional user profile is generated. Based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, an interaction decision is made. The result of the interaction decision includes: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. Step S106: After multiple simulation interaction processes have ended, based on the process data of the simulation interaction processes that triggered the interaction interruption conditions, the interaction interruption of the intelligent interaction system is attributed.

[0013] In this embodiment, each execution of steps S102 and S104 completes one simulation interaction process. Multiple simulation interaction processes can be completed by executing steps S102 and S104 multiple times. The multi-dimensional user profiles corresponding to each simulation interaction process can be the same or different. The following description of steps S102 and S104 will take starting and completing one simulation interaction process as an example.

[0014] In step S102 above, the testing system generates a multi-dimensional user profile for the intelligent interactive system (hereinafter referred to as the intelligent interactive system) to be evaluated. In one example, the testing system receives an evaluation request for the intelligent interactive system and, in response to the request, generates a multi-dimensional user profile for the intelligent interactive system. This evaluation request is used to request evaluation of the intelligent interactive system, and can be sent to the testing system by a staff member triggering a testing component for the intelligent interactive system. For example, a staff member triggers a testing component for an intelligent teaching system within the testing system, and the testing system, in response to this triggering operation, receives the evaluation request for the intelligent interactive system.

[0015] Intelligent interactive systems are systems with interactive functions created based on AI capabilities, such as financial AI assistants and shopping AI assistants. These systems can provide users with services such as dialogue, assisted ordering, and question answering. Financial AI assistants can include either insurance AI assistants or wealth management AI assistants. In one example, the intelligent interactive system includes an insurance AI assistant that can provide users with services such as insurance product recommendations, insurance product answers, insurance claims, and insurance product purchases. In another example, the intelligent interactive system includes a wealth management AI assistant that can provide users with services such as wealth management product recommendations, wealth management product answers, and wealth management product purchases. In yet another example, the intelligent interactive system includes a shopping AI assistant that can provide users with services such as product recommendations, product answers, product purchases, and after-sales service.

[0016] In one embodiment, generating a multi-dimensional user profile corresponding to the intelligent interaction system includes: Identify the business scenario to be evaluated for the intelligent interaction system and the user group to be evaluated for the intelligent interaction system, and obtain user data of the user group to be evaluated in the business scenario to be evaluated, which corresponds to the profile dimensions; the profile dimensions include one or more of the following: user attribute dimensions, user preference dimensions, interaction habit dimensions, and historical behavior dimensions. Based on user data, generate multi-dimensional user profiles of the intelligent interaction system corresponding to the business scenarios and user groups to be evaluated.

[0017] In this embodiment, firstly, the business scenario to be evaluated corresponding to the intelligent interaction system is determined. The intelligent interaction system can be applied to one or more business scenarios. Here, the business scenario to be evaluated for the intelligent interaction system can be determined from among the various business scenarios in which the intelligent interaction system is applied. The business scenario to be evaluated is a pre-determined business scenario, including but not limited to one or more of financial scenarios and shopping scenarios. Specifically, financial scenarios include but are not limited to one or more of wealth management product scenarios and insurance product scenarios, and shopping scenarios include but are not limited to one or more of beauty product shopping scenarios and maternal and infant product shopping scenarios. Secondly, the user group to be evaluated corresponding to the intelligent interaction system is determined. The intelligent interaction system can be applied to one or more user groups. Here, the user group to be evaluated for the intelligent interaction system can be determined from among the various user groups in which the intelligent interaction system is applied. The user group to be evaluated is a pre-determined group, which can be exemplified as one or more of the following: working professionals, the elderly, students, and young people. In one example, the business scenario to be evaluated for the intelligent interaction system includes insurance product scenarios, and the user group to be evaluated for the intelligent interaction system includes working professionals. In another example, the business scenario to be evaluated for the intelligent interaction system includes insurance product scenarios, and the user group to be evaluated for the intelligent interaction system includes young people. By identifying the business scenario and user group corresponding to the intelligent interaction system, it is possible to conduct targeted evaluations of the intelligent interaction system by differentiating between the business scenario and the user group. The business scenario and user group to be evaluated in each simulation interaction process can be the same or different.

[0018] Next, user data corresponding to the profile dimensions for the user group to be evaluated within the business scenario to be evaluated is obtained. Profile dimensions include one or more of the following: user attribute dimensions, user preference dimensions, interaction habit dimensions, and historical behavior dimensions. Correspondingly, user data includes one or more of these dimensions. Specifically, user attribute data represents the basic user information of the user group to be evaluated within the business scenario; user preference data represents the business preferences of the user group in the business scenario, such as frequently purchased product types; interaction habit data represents the interaction habits of the user group in the business scenario, such as frequently sending audio data, rarely sending text data, and rarely triggering components; and historical behavior data represents the historical behavior of the user group in the business scenario, such as purchase information and product information of previously purchased products. User data corresponding to the profile dimensions for the user group to be evaluated within the business scenario can be obtained from the historical interaction data between the user group to be evaluated and the intelligent interaction system, stored in the historical business database of the intelligent interaction system.

[0019] Finally, based on user data, a multi-dimensional user profile of the intelligent interaction system is generated, corresponding to the business scenario to be evaluated and the user group to be evaluated. User data can be combined to obtain a multi-dimensional user profile. Since the above profile dimensions include one or more of user attribute dimensions, user preference dimensions, interaction habit dimensions, and historical behavior dimensions, this embodiment can construct a multi-dimensional user profile. This multi-dimensional user profile can comprehensively simulate the performance of the user group to be evaluated in the business scenario to be evaluated. The multi-dimensional user profiles for each simulated interaction process can be the same or different.

[0020] As can be seen, this embodiment can generate multi-dimensional user profiles, and based on these profiles, the service capabilities of the intelligent interaction system in the business scenarios to be evaluated, targeting the user groups to be evaluated, can be tested in a targeted manner, thereby improving the accuracy of the evaluation of the intelligent interaction system.

[0021] Of course, multi-dimensional user profiles for intelligent interaction systems can also be generated through other methods. For example, a user profile of the desired user can be constructed based on at least one profile dimension, and this user profile can be used as the multi-dimensional user profile for the intelligent interaction system. The profile dimensions include one or more of the following: user attribute dimensions, user preference dimensions, interaction habit dimensions, and historical behavior dimensions. The desired user refers to the expected user of the intelligent interaction system, and the user profile of the desired user can be set according to engineering requirements.

[0022] In step S102 above, after generating a multi-dimensional user profile, the test system initiates a simulated interaction process with the intelligent interaction system based on the multi-dimensional user profile. Simulated interaction refers to simulating the interaction between a real user corresponding to the multi-dimensional user profile and the intelligent interaction system. The real user can be the aforementioned user group to be evaluated. For example, simulating a dialogue between the aforementioned user group to be evaluated and the intelligent interaction system, during which the possible operation methods of the aforementioned user group to be evaluated are simulated to send text data, image data, audio data, video data, or trigger components associated with the intelligent interaction system, and the dialogue process is continuously advanced or terminated based on the feedback from the intelligent interaction system. The simulated interaction process refers to the process obtained based on the simulated interaction with the intelligent interaction system, such as the dialogue process described above.

[0023] In some embodiments, the test system can initiate the simulation interaction process with the intelligent interactive system by establishing a simulation interaction link. Once the simulation interaction link is successfully established, the simulation interaction process is considered to have started successfully. Of course, establishing a simulation interaction link also includes processes such as initializing simulation parameters, which will not be elaborated upon here.

[0024] In step S104 above, multimodal interaction data corresponding to the multidimensional user profile is generated during the simulation interaction process. Specifically, after the simulation interaction process is started, the test system generates multimodal interaction data corresponding to the multidimensional user profile. This multimodal interaction data is the data sent by the test system to the intelligent interaction system by the real user corresponding to the multidimensional user profile during the simulation interaction process. This data may include one or more of text data, audio data, image data, and video data, or it may include trigger data for components associated with the intelligent interaction system. Therefore, this data can be called multimodal data.

[0025] In some embodiments, generating multimodal interaction data corresponding to multidimensional user profiles includes: Determine the current interaction intent and the target interaction modality corresponding to the multi-dimensional user profile; the target interaction modality includes sending data in a specified format to the intelligent interaction system or triggering components associated with the intelligent interaction system; Based on the current interaction intent and target interaction modality, multi-modal interaction data corresponding to the multi-dimensional user profile is generated; the multi-modal interaction data includes data in a specified format or trigger data for components associated with the intelligent interaction system.

[0026] It's understandable that in a simulated interaction process, the test system repeatedly engages in multiple rounds of dialogue with the intelligent interaction system. In each round, the test system generates multimodal interaction data and sends it to the intelligent interaction system, which in turn returns response data based on this data. Therefore, the current interaction intent refers to the intent that the test system needs to convey to the intelligent interaction system in the current round of dialogue. For example, the current interaction intent might be the intent that the test system needs to convey to the intelligent interaction system in the fifth round of dialogue.

[0027] Based on this, firstly, the current interaction intent and the target interaction modality corresponding to the multi-dimensional user profile are determined. The target interaction modality includes sending data in a specified format to the intelligent interaction system or triggering components associated with the intelligent interaction system. The specified format includes one or more of text, images, videos, and audio. Therefore, the target interaction modality includes sending text data, audio data, image data, or video data to the intelligent interaction system, or triggering components associated with the intelligent interaction system. Then, based on the current interaction intent and the target interaction modality, multimodal interaction data corresponding to the multi-dimensional user profile is generated. The multimodal interaction data can convey the current interaction intent to the intelligent interaction system according to the target interaction modality. The multimodal interaction data includes one or more of text data, audio data, image data, and video data, or trigger data for components associated with the intelligent interaction system.

[0028] As can be seen, the above process can determine the current interaction intent and the target interaction modality corresponding to the multi-dimensional user profile. Since the target interaction modality includes sending data in a specified format to the intelligent interaction system or triggering components associated with the intelligent interaction system, the multimodal interaction data generated based on the current interaction intent and the target interaction modality can be data in a specified format or trigger data for components associated with the intelligent interaction system and can represent the current interaction intent. This fully simulates the behavior of real users in the process of interacting with the intelligent interaction system and accurately generates the multimodal interaction data required for evaluation.

[0029] In some embodiments, determining the current interaction intent corresponding to the multi-dimensional user profile and the target interaction modality corresponding to the current interaction intent includes: When generating the first multimodal interaction data, determine the initial interaction intent and the interaction modality corresponding to the multidimensional user profile, take the initial interaction intent as the current interaction intent, and take the interaction modality corresponding to the initial interaction intent as the target interaction modality. When generating non-first multimodal interaction data, the current interaction intent is determined based on the multi-dimensional user profile, the interaction data with the intelligent interaction system in the simulated interaction process, and the target interaction intent in the simulated interaction process. Based on the current interaction intent, the multi-dimensional user profile, and the target interaction intent, the target interaction modality is determined.

[0030] As mentioned earlier, in a simulation interaction process, the test system repeatedly engages in multiple rounds of dialogue with the intelligent interaction system. In each round, the test system generates multimodal interaction data and sends it to the intelligent interaction system, which in turn returns response data based on the multimodal interaction data. The content of each round includes the multimodal interaction data sent by the test system to the intelligent interaction system and the response data returned by the intelligent interaction system based on the multimodal interaction data. Therefore, this embodiment needs to distinguish between the case of generating the first multimodal interaction data and the case of generating subsequent multimodal interaction data. Generating the first multimodal interaction data means the test system and the intelligent interaction system conduct the first round of dialogue; generating subsequent multimodal interaction data means the test system and the intelligent interaction system conduct the second round and subsequent rounds of dialogue.

[0031] When generating the first multimodal interaction data, the initial interaction intent and the corresponding interaction modality corresponding to the multidimensional user profile are determined. The initial interaction intent is used as the current interaction intent, and the interaction modality corresponding to the initial interaction intent is used as the target interaction modality. The initial interaction intent represents the content of the first round of questions raised by the test system based on the multidimensional user profile, simulating real users (such as the user group to be evaluated mentioned above) after the simulation interaction process with the intelligent interaction system is initiated. Therefore, the initial interaction intent can also be called the seed question. This initial interaction intent can be determined based on the historical interaction data between the real users (such as the user group to be evaluated mentioned above) and the intelligent interaction system. Based on the historical interaction data, all the first round of questions raised by the user group to be evaluated after initiating the interaction with the intelligent interaction system can be determined. The first round of questions are sorted in descending order of their occurrence frequency or number of occurrences. Among the content represented by the top-ranked questions in this sorting, one content is selected as the initial interaction intent. Different initial interaction intents can be selected for each simulation interaction process.

[0032] The initial interaction intent corresponds to the interaction modality, which represents the interaction method by which the test system, based on a multi-dimensional user profile, simulates the actual user conveying the initial interaction intent to the intelligent interaction system after the simulated interaction process with the system is initiated. This interaction modality can be determined based on historical interaction data between the actual user profile (e.g., the user group to be evaluated, as mentioned above) and the intelligent interaction system. Based on historical interaction data, all the first-round questions issued by the user group to be evaluated after initiating interaction with the intelligent interaction system can be identified. These first-round questions are then sorted from highest to lowest frequency. From the content represented by the top-ranked questions, one is selected as the initial interaction intent. The interaction method by which the actual user sent the initial interaction intent (e.g., sending text data or triggering components associated with the intelligent interaction system) in the historical interaction data is then determined as the interaction modality corresponding to the initial interaction intent.

[0033] In one scenario, the initial interaction intent is represented by data in a specified format. The corresponding interaction modality includes sending data in the specified format. The test system can convey the initial interaction intent to the intelligent interaction system by sending data in the specified format. The specified format includes one or more of text, images, video, and audio. In another scenario, the initial interaction intent is represented by components associated with the intelligent interaction system. The corresponding interaction modality includes triggering components associated with the intelligent interaction system. The test system can convey the initial interaction intent to the intelligent interaction system by triggering these components.

[0034] In one example, the components associated with the intelligent interaction system include recommended guidance words. These guidance words can be located on the entry page of the intelligent interaction system, allowing users to enter the interactive page of the system after triggering them. Alternatively, the guidance words can be located within the interactive page itself. The initial interaction intent is represented by the recommended guidance words associated with the intelligent interaction system. The corresponding interaction modality includes triggering the recommended guidance words. The test system can convey the initial interaction intent to the intelligent interaction system by triggering these recommended guidance words.

[0035] When generating non-first multimodal interaction data, the current interaction intent is determined based on the multi-dimensional user profile, the interaction data already exchanged with the intelligent interaction system in the simulated interaction process, and the target interaction intent of the simulated interaction process. Based on the current interaction intent, the multi-dimensional user profile, and the target interaction intent, the target interaction modality is determined. The exchanged data includes multimodal interaction data sent by the test system to the intelligent interaction system and the response data returned by the intelligent interaction system during the simulated interaction process before generating non-first multimodal interaction data. For example, when generating multimodal interaction data for the 5th time, the exchanged data includes multimodal interaction data sent by the test system to the intelligent interaction system in rounds 1-4 of the dialogue after the start of the simulated interaction process and the response data returned by the intelligent interaction system.

[0036] The target interaction intent of the simulation interaction process refers to the purpose for which the test system, based on a multi-dimensional user profile, simulates a real user initiating the simulation interaction process and engaging in simulated interaction with the intelligent interaction system. The target interaction intent can be configured as needed before initiating the simulation interaction process. The target interaction intent can differ from the initial interaction intent mentioned above. For example, the initial interaction intent might be: "What insurance do I need to buy?", and the target interaction intent might be: "Buy insurance for everyone in my family." Alternatively, the target interaction intent can be the same as the initial interaction intent. For example, the initial interaction intent might be: "I want to make a claim," and the target interaction intent might be: "Make a claim for the insurance policy." In one example, the initial interaction intent is the content of the first question that a real user corresponding to a given multi-dimensional user profile is most likely to ask when fulfilling the target interaction intent. During the evaluation of the intelligent interaction system, the target interaction intent can remain unchanged in each simulation interaction process or be reset by the test system based on the already interacted data and multi-dimensional user profile as the simulation interaction process progresses. The initial interaction intent for each simulation interaction process can be the same or different. The target interaction intent for each simulation interaction process can be the same or different.

[0037] Based on multi-dimensional user profiles, the interaction data already exchanged with the intelligent interaction system in the simulated interaction process, and the target interaction intent of the simulated interaction process, the current interaction intent is determined. It is possible to simulate a real user corresponding to the multi-dimensional user profile, and based on the interaction data, determine the intent that needs to be expressed in this round of dialogue to achieve the target interaction intent; this intent is then taken as the current interaction intent. Therefore, the current interaction intent can be the intent that a simulated real user corresponding to the multi-dimensional user profile intends to express in this round of dialogue to achieve the target interaction intent, as determined by the interaction data. For example, if the target interaction intent is insurance claims, and the user described in the multi-dimensional user profile is a rational and efficiency-oriented user, and the interaction data includes requesting the user to provide insurance information, then the current interaction intent includes sending insurance information to the intelligent interaction system.

[0038] As can be seen, the above process enables the generation of multimodal interaction data in both the first-round and non-first-round dialogue scenarios, thereby improving the accuracy of multimodal interaction data generation in both scenarios. Furthermore, in non-first-round dialogues, the multimodal interaction data is determined based on the current interaction intent, multi-dimensional user profiles, and the target interaction intent. The current interaction intent is determined based on the multi-dimensional user profile, the interaction data already generated with the intelligent interaction system in the simulated interaction process, and the target interaction intent in the simulated interaction process. Therefore, the accuracy of determining the current interaction intent is improved, thus enhancing the accuracy of generating multimodal interaction data for non-first-round dialogues.

[0039] In some embodiments, determining the target interaction modality based on the current interaction intent, multi-dimensional user profile, and target interaction intent includes: Among multiple preset interaction modalities, the modal that can realize the current interaction intent, match the multi-dimensional user profile, and match the target interaction intent is selected as the target interaction modal.

[0040] Multiple preset interaction modalities include sending text data to the intelligent interaction system, sending audio data to the intelligent interaction system, sending image data to the intelligent interaction system, sending video data to the intelligent interaction system, and triggering components associated with the intelligent interaction system. In this step, one modality is selected from the preset interaction modalities as the target interaction modality. The target interaction modality is required to fulfill the current interaction intent and match the multi-dimensional user profile as well as the target interaction intent. It can be pre-defined that the target interaction modality fulfills the current interaction intent, has the highest degree of matching with the multi-dimensional user profile, and matches the target interaction intent.

[0041] Matching the target interaction modality with the multi-dimensional user profile means that the target interaction modality is a modality that the real user corresponding to the multi-dimensional user profile is likely to use. For example, when the real user corresponding to the multi-dimensional user profile is a young person, the possible interaction modalities include sending image data, sending text data, sending audio data, and triggering components associated with the intelligent interaction system. When the real user corresponding to the multi-dimensional user profile is an elderly person, the possible interaction modalities include sending image data, sending text data, and sending audio data. Matching the target interaction modality with the target interaction intent means that the target interaction modality is one of the modalities used to achieve the target interaction intent. For example, if the target interaction intent is insurance claims, which may involve sending personal photos and submitting a claim request, then the modalities that the target interaction intent may use include sending image data, sending text data, sending audio data, and triggering components associated with the intelligent interaction system.

[0042] You can select from various preset interaction modalities the modalities that real users might use based on the multi-dimensional user profile and that are used to achieve the target interaction intent. These can be selected as candidate modalities. From these candidate modalities, the modal that best achieves the current interaction intent is chosen as the target interaction modal. For example, in the example above, if the real user corresponding to the multi-dimensional user profile is a young person, and the target interaction intent is insurance claims, then the candidate modalities include: sending image data, sending text data, sending audio data, and triggering components associated with the intelligent interaction system. If the current interaction intent is determined to be sending an insurance policy image, then the target interaction modal can be determined to be sending image data.

[0043] When there are multiple candidate modalities that can fulfill the current interaction intent, the target interaction modal can be selected as the modal that the real user corresponding to the multi-dimensional user profile uses most frequently (the modal that best matches the multi-dimensional user profile). For example, in the above example, the real user corresponding to the multi-dimensional user profile is an elderly person, and the target interaction intent is insurance claims. The candidate modalities include: sending image data, sending text data, and sending audio data. If the current interaction intent is determined to be providing the name of an insurance product, then both sending text data and sending audio data can fulfill the current interaction intent. Since this real user frequently sends audio data to the intelligent interaction system, the target interaction modal can be determined to be sending audio data, thus providing the name of the insurance product by sending audio data.

[0044] For example, in the previous example, the real user corresponding to the multi-dimensional user profile is a young person, and the target interaction intent is insurance claim. Therefore, the candidate modalities include: sending image data, sending text data, sending audio data, and triggering components associated with the intelligent interaction system. Determining the current interaction intent as submitting a claim request, since sending text data, sending audio data, and triggering components can all fulfill this intent, all three methods can achieve the desired result. Since this real user frequently triggers components associated with the intelligent interaction system, the target interaction modal can be determined as triggering these components, thus submitting the claim request by triggering them.

[0045] In the above process, multi-dimensional user profiles, target interaction intents, and current interaction intents are considered together. The modality that can realize the current interaction intent and matches the multi-dimensional user profiles and the target interaction intent is taken as the target interaction modality, which improves the accuracy of determining the target interaction modality.

[0046] In some embodiments, multimodal interaction data corresponding to a multidimensional user profile is generated based on the current interaction intent and the target interaction modality, including: If the target interaction modality includes sending data in a specified format to the intelligent interaction system, then data representing the current interaction intent is generated in the specified format and used as multimodal interaction data. If the target interaction modality includes components associated with the intelligent interaction system, then among the components associated with the intelligent interaction system, the target component used to represent the current interaction intent is determined, and trigger data for the target component is generated as multimodal interaction data.

[0047] As mentioned earlier, the target interaction modality includes sending data in a specified format to the intelligent interaction system or including components that trigger associations with the intelligent interaction system. When the target interaction modality includes sending data in a specified format to the intelligent interaction system, data representing the current interaction intent can be generated according to the specified format, serving as multimodal interaction data. The specified format includes one or more of text, images, video, and audio. Multimodal interaction data includes one or more of text data, audio data, image data, and video data.

[0048] The components associated with an intelligent interaction system can be categorized into several types, such as feedback components, question-based components, and conversion components. Feedback components include like and dislike components; question-based components include various recommendation prompts or cards, such as insurance introduction prompts or insurance product cards; and conversion components include components used for purchasing and placing orders, such as simulated premium calculation components or insurance application components. When the target interaction modality includes triggering components associated with the intelligent interaction system, the testing system can identify the target component representing the current interaction intent from among the associated components and generate trigger data for that target component as multimodal interaction data. For example, if the current interaction intent includes asking if there are suitable insurance products, the system can first locate the question-based components, then identify the "recommended product" component as the target component, and generate trigger data for that component as multimodal interaction data.

[0049] Through the above process, the goal of accurately generating multimodal interaction data according to the target interaction modality is achieved.

[0050] After generating multimodal interaction data corresponding to multidimensional user profiles, the multimodal interaction data also needs to be sent to an intelligent interaction system for simulation testing. Therefore, in some embodiments, the aforementioned simulation interaction process initiated based on the multidimensional user profile and the intelligent interaction system includes: establishing a simulation interaction link between the multidimensional user profile and the intelligent interaction system to initiate the simulation interaction process. Correspondingly, before making interaction decisions based on the multimodal interaction data and the intelligent interaction system's response data to the multimodal interaction data, the process further includes: sending the multimodal interaction data to the intelligent interaction system through the simulation interaction link and obtaining the response data returned by the intelligent interaction system in response to the multimodal interaction data.

[0051] First, the test system establishes a simulated interaction link with the intelligent interaction system based on multi-dimensional user profiles to initiate the simulated interaction process. Then, the test system sends multimodal interaction data to the intelligent interaction system through this simulated interaction link and obtains the response data returned by the intelligent interaction system in response to the multimodal interaction data, thus performing simulated interaction. By creating a simulated interaction link with the intelligent interaction system, the test system can successfully initiate the simulated interaction process and perform simulated interaction smoothly.

[0052] As described above, in a single simulation interaction process, the test system repeatedly engages in multiple rounds of dialogue with the intelligent interaction system. In each round, the test system generates multimodal interaction data and sends it to the intelligent interaction system, which in turn returns response data based on the multimodal interaction data. Each round of dialogue includes the multimodal interaction data sent by the test system to the intelligent interaction system and the response data returned by the intelligent interaction system based on the multimodal interaction data. Therefore, in a single simulation interaction process, the test system can repeat the above process to generate multimodal interaction data for each round of dialogue. In step S104 above, taking any single simulation interaction process as an example, the test system also, for each round of dialogue, after receiving the response data from the intelligent interaction system, makes an interaction decision based on the multimodal interaction data sent and the response data received in that round. The results of the interaction decision include: determining that the simulation interaction process meets the conditions for continuing interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the interaction interruption condition and ending the simulation interaction process; or determining that the simulation interaction process meets the interaction completion condition and ending the simulation interaction process.

[0053] In some embodiments, taking any round of dialogue in any simulated interaction process as an example, interaction decisions are made based on multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, including: For the multimodal interaction data and response data of the current round, obtain the historical multimodal interaction data and historical response data corresponding to the multimodal interaction data and response data of the current round, use the historical multimodal interaction data and the multimodal interaction data of the current round as the judgment reference data, and use the response data of the current round and the historical response data as the data to be judged; Based on the judgment reference data, the degree of matching between the data to be judged and the multi-dimensional user profile, the content quality of the data to be judged, and the degree of interactive guidance of the data to be judged are determined. Based on the degree of matching, content quality, degree of interactive guidance, the degree of completion of the target interactive intent of the simulated interactive process, and the number of interactive rounds already completed in the simulated interactive process, an interaction decision is made.

[0054] Each round of dialogue includes multimodal interaction data sent from the test system to the intelligent interaction system and response data returned by the intelligent interaction system based on the multimodal interaction data. Therefore, for any simulated interaction process, there is no historical multimodal interaction data or historical response data for the first round of dialogue. For the Mth round of dialogue, the multimodal interaction data and response data of the current round are the multimodal interaction data and response data of the Mth round of dialogue, and the historical multimodal interaction data and historical response data are the multimodal interaction data and response data of the first to (M-1)th rounds of dialogue in this simulated interaction process. M is a positive integer greater than or equal to 2.

[0055] Historical multimodal interaction data and current round multimodal interaction data are used as reference data for judgment, while current round response data and historical response data are used as data to be judged. Based on the reference data, the degree of matching between the data to be judged and the multidimensional user profile, the content quality of the data to be judged, and the degree of interactive guidance of the data to be judged are determined. Here, an artificial intelligence model in the testing system can be used to determine the degree of matching between the data to be judged and the multidimensional user profile, the content quality of the data to be judged, and the degree of interactive guidance of the data to be judged.

[0056] Interaction decisions are made based on matching degree, content quality, interactive guidance degree, the degree of completion of the target interactive intent in the simulated interaction process, and the number of interaction rounds already completed in the simulated interaction process. The degree of completion of the target interactive intent in the simulated interaction process refers to the extent to which the target interactive intent for this specific simulation interaction process is achieved. The number of interaction rounds already completed in the simulated interaction process refers to the number of interaction rounds between the test system and the intelligent interaction system in this simulation interaction process.

[0057] Through the above process, we can make comprehensive and accurate interaction decisions by comprehensively considering the degree of matching between the data to be judged and the multi-dimensional user profile, the content quality of the data to be judged, the degree of interactive guidance of the data to be judged, the degree of completion of the target interaction intent of the simulated interaction process, and the number of interaction rounds of the simulated interaction process.

[0058] Taking any round of dialogue in any simulated interaction process as an example, in some embodiments, interaction decisions are made based on matching degree, content quality, interaction guidance degree, completion degree of the target interaction intent of the simulated interaction process, and the number of interaction rounds already completed in the simulated interaction process, including: If the simulation interaction process completes the target interaction intent or the number of interaction rounds reaches the maximum value, then the simulation interaction process is determined to meet the interaction completion conditions and ends. If the simulated interaction process fails to achieve the target interaction intent and the number of interaction rounds has not reached the maximum value, and the simulated interaction process meets the reasons for continuing the interaction based on the degree of matching, content quality, and degree of interaction guidance, then the simulated interaction process meets the conditions for continuing the interaction and the simulated interaction process continues. If the simulation interaction process fails to achieve the target interaction intent and the number of interaction rounds has not reached the maximum value, and the simulation interaction process meets the reason for termination of interaction based on the matching degree, content quality, and interaction guidance degree, then the simulation interaction process is determined to trigger the interaction interruption condition and end the simulation interaction process.

[0059] In this process, firstly, based on the multimodal interaction data of the current round, the response data of the current round, the historical multimodal interaction data, and the historical response data in the current simulation interaction flow, it is determined whether the current simulation interaction flow has completed its target interaction intent or whether the number of interaction rounds in the current simulation interaction flow has reached its maximum value. The maximum value for each simulation interaction flow can be the same or different. If it is determined that the current simulation interaction flow has completed its target interaction intent or the number of interaction rounds in the current simulation interaction flow has reached its maximum value, then the current simulation interaction flow is determined to meet the interaction completion conditions and ends. The interaction completion conditions include the simulation interaction flow completing its target interaction intent or the simulation interaction flow reaching its maximum value.

[0060] If the simulated interaction process fails to achieve the target interaction intent and the number of interaction rounds has not reached the maximum value, then the determination of whether the simulated interaction process satisfies the reason for continuing or terminating the interaction is based on the degree of matching, content quality, and degree of interaction guidance. A binary classification algorithm can be used to determine whether the simulated interaction process satisfies the reason for continuing or terminating the interaction based on the degree of matching, content quality, and degree of interaction guidance.

[0061] If the current simulation interaction process is determined to meet the reasons for continuing interaction, then the current simulation interaction process is determined to meet the conditions for continuing interaction and continues. The conditions for continuing interaction include that the simulation interaction process has not completed the target interaction intent, the number of interaction rounds in the simulation interaction process has not reached the maximum value, and the simulation interaction process meets the reasons for continuing interaction.

[0062] If the current simulation interaction process is determined to meet the termination reason, then the current simulation interaction process is determined to trigger the interaction interruption condition and ends. The interaction interruption condition includes: the simulation interaction process failing to complete the target interaction intent, the number of interaction rounds in the simulation interaction process not reaching the maximum value, and the simulation interaction process meeting the termination reason.

[0063] The reasons for terminating and continuing interaction in each simulated interaction process can be the same or different. In practice, reasons for continuing and terminating interaction can be preset, and a binary classification method can be used to determine whether the simulated interaction process meets the reasons for continuing or terminating interaction based on the aforementioned matching degree, content quality, and interaction guidance level. The reasons for continuing and terminating interaction are related to the aforementioned matching degree, content quality, and interaction guidance level. When the simulated interaction process meets the reasons for continuing interaction, it can be determined that the matching degree is higher than a specified degree, the content quality of the data to be judged is higher than a specified quality, and the interaction guidance level of the data to be judged is higher than a specified guidance level. When the simulated interaction process meets the reasons for terminating interaction, it can be determined that the matching degree is lower than a specified degree, or the content quality of the data to be judged is lower than a specified quality, or the interaction guidance level of the data to be judged is lower than a specified guidance level.

[0064] In one example, multiple reasons for continuing interaction can be set, and a condition for continuing interaction can be generated based on each reason. This condition includes the reason for continuing interaction. Similarly, multiple reasons for terminating interaction can be set, and an interaction termination condition can be generated based on each reason for terminating interaction. This interaction termination condition includes the reason for terminating interaction.

[0065] In one example, reasons for continuing interaction may include one or more of the following: providing structured, unexpectedly in-depth answers; setting effective "conversation hooks" at the end of responses; accurately utilizing context and user information to achieve dynamic and personalized dialogue, making users feel "understood" and "valued"; direct and affirmative "value confirmation" to build professional trust; adopting a "gradual guidance" strategy to break down complex issues into smaller, manageable ones; and providing effective answers to build trust and stimulate deeper needs. Reasons for ending interaction may include one or more of the following: misunderstanding user intent, breaking trust, and making it impossible to continue the conversation; rigid answers, resulting in a poor experience and user loss of patience; a deadlock in the conversation, leading the user to give up; asking the user a question when they want specific information, instantly negating the value of the conversation and causing the user to give up; overly comprehensive "blanket" responses that force users to digest the information on their own or think offline; answers that avoid the key points or guide the user to "take care of the next step themselves"; and triggering fallback statements.

[0066] Through the above process, the simulation interaction process ends when the target interaction intent is achieved or the number of interaction rounds reaches the maximum value; it continues when the target interaction intent is not achieved, the number of interaction rounds has not reached the maximum value, and the simulation interaction process satisfies the reason for continuing interaction; and it ends when the target interaction intent is not achieved, the number of interaction rounds has not reached the maximum value, and the simulation interaction process satisfies the reason for terminating interaction, thereby improving the precision and accuracy of interaction decisions.

[0067] Multiple simulated interaction processes with the intelligent interaction system can be initiated and completed by repeatedly executing steps S102 and S104. Each simulated interaction process involves the following parameters: the business scenario to be evaluated, the user group to be evaluated, multi-dimensional user profiles, initial interaction intent, the interaction modality of the initial interaction intent, the target interaction intent, the reason for terminating the interaction, the reason for continuing the interaction, and the maximum value of the number of interaction rounds already completed. These parameters may be the same or different in each simulated interaction process. The termination of each simulated interaction process can include: 1. The simulated interaction process terminates due to triggering an interaction interruption condition; 2. The simulated interaction process terminates due to meeting the interaction completion condition. In step S106 above, after multiple simulated interaction processes have ended, the intelligent interaction system performs interaction interruption attribution based on the process data of the simulated interaction processes that terminated due to triggering interaction interruption conditions. In this step, the simulated interaction processes that terminated due to triggering interaction interruption conditions are identified in the multiple simulated interaction processes, and the intelligent interaction system performs interaction interruption attribution based on the process data of these processes. Attributing interaction interruptions in intelligent interactive systems refers to determining the reasons for the interruption of interaction in the intelligent interactive system. The reasons for the interruption can also be referred to as the reasons for user churn.

[0068] As mentioned earlier, the interaction interruption conditions include reasons for terminating the interaction. In one approach, the simulated interaction flow that triggers the interruption condition can be identified as a problem flow. The frequency of occurrence of each reason for terminating the interaction corresponding to the problem flow is counted, and the reason for terminating the interaction with the highest frequency is identified as the reason for the interaction interruption in the intelligent interaction system. In another approach, the simulated interaction flow that triggers the interruption condition can be identified as a problem flow. The frequency of occurrence of each reason for terminating the interaction corresponding to the problem flow is counted, and different weights are configured for each reason for terminating the interaction. A score is calculated for each reason for terminating the interaction based on its frequency and weight, and the reason for terminating the interaction with the highest score is identified as the reason for the interaction interruption in the intelligent interaction system.

[0069] In some embodiments, based on the process data of the simulated interaction flow that triggers the interaction interruption condition, the intelligent interaction system performs interaction interruption attribution, including: The simulation interaction process that triggers the interaction interruption condition is identified as the problem process. Based on the process data of the problem process, the corresponding process description text is generated. Based on the semantic information of the process description text of each problem process, the problem processes are clustered to obtain at least one class of problem processes. Based on the common semantic information of the process description text corresponding to each type of problem process, determine the common reasons for the interaction interruption in the problem process of that type.

[0070] First, the simulation interaction flow that triggers the interaction interruption condition is identified as the problem flow. Based on the flow data of the problem flow, corresponding flow description text is generated. The flow description text describes the process of the problem flow, or the reason for the problem flow triggering the interaction interruption condition, or the reason and process for the problem flow triggering the interaction interruption condition. There are multiple problem flows. Then, based on the semantic information of the flow description texts of each problem flow, the problem flows are clustered to obtain at least one class of problem flows. Each class of problem flows includes one or more problem flows. The flow description texts of problem flows within the same class have similar semantic information. Here, any semantic-based clustering algorithm can be used to cluster the flow description texts of each problem flow, and the problem flows corresponding to the flow description texts that are clustered into one class are identified as the same class of problem flows. Finally, based on the common semantic information of the flow description texts corresponding to each class of problem flows, the common cause of interaction interruption in that class of problem flows is determined. For example, the common semantic information of the flow description texts corresponding to each class of problem flows is used as the common cause of interaction interruption in that class of problem flows.

[0071] In one example, there are 50 problem flows, each corresponding to a flow description text. Using any semantic-based clustering algorithm, these 50 flow description texts can be clustered into two categories: the first category contains 30 flow description texts, and the second category contains 20 flow description texts. The 30 flow description texts corresponding to the first 30 flow description texts are then identified as the first category, and the 20 flow description texts corresponding to the second 20 flow description texts are identified as the second category. For each category of flow description texts, the common semantic information is used to identify the common reasons for interaction interruptions within that category.

[0072] Taking the first type of problem flow as an example, the flow description text of some problem flows states that the intelligent interaction system's focus on providing sales cards resulted in a slow response time, thus satisfying the reason for terminating the interaction: "slow response time." Other flow description texts state that the intelligent interaction system provided sales cards too early, and although the multi-dimensional user profile showed purchase intent, no sales-related questions had been asked, thus satisfying the reason for terminating the interaction: "answers do not match questions." Still other flow description texts state that the intelligent interaction system provided sales cards too early, and the multi-dimensional user profile had not yet shown purchase intent, thus satisfying the reason for terminating the interaction: "not understanding user intent." Through this process, it can be determined that the common semantic information of the flow description texts corresponding to the first type of problem flow is "sales cards displayed too early." Therefore, the common reason for terminating the interaction of the first type of problem flow under different interaction interruption conditions (i.e., different reasons for terminating the interaction) is "sales cards displayed too early."

[0073] As can be seen, through the above process, when multiple problem flows are interrupted based on different interaction interruption conditions (i.e., different reasons for terminating interaction), process description texts of the problem flows can be generated. The problem flows are clustered based on the semantic information of the process description texts. Based on the common semantic information of the process description texts corresponding to each type of problem flow, the common reasons for the interaction interruption in that type of problem flow are determined. The common reasons are equivalent to the deep-seated reasons that trigger the interaction interruption conditions. This achieves the effect of digging deeper into the deep-seated reasons of the triggering conditions from the shallow triggering conditions, improving the depth of interaction interruption attribution in intelligent interaction systems, and enabling the service capabilities of intelligent interaction systems to be fundamentally improved based on these common reasons.

[0074] In some embodiments, based on the process data of the problem process, a process description text corresponding to the problem process is generated, including: The process data includes decision rationale data for the interactive decisions corresponding to the problem process. Based on the decision rationale data, a process description text corresponding to the problem process is generated. or, The process data includes multimodal interaction data and response data corresponding to the problem process, and the multimodal interaction data and response data corresponding to the problem process are used as the first target data; based on the first target data and the reasoning chain corresponding to the first target data, the process description text corresponding to the problem process is generated; or, The process data includes decision rationale data for interactive decisions corresponding to the problem process and multimodal interaction data and response data corresponding to the problem process. The multimodal interaction data and response data corresponding to the problem process are used as the first target data. Second target data for explaining decision rationale data is obtained from the first target data. Based on the second target data, the reasoning chain and decision rationale data corresponding to the second target data, the process description text corresponding to the problem process is generated.

[0075] The process description text corresponding to the problem process can be generated based on the process data of the problem process using one of three methods.

[0076] Method 1 The process data includes decision rationale data for the interactive decisions corresponding to the problem process. The aforementioned process of making interactive decisions based on multimodal interaction data and its response data can be executed by an artificial intelligence model in the testing system. After making the interactive decision, this AI model will also output decision rationale data, which represents the reasons for determining the conditions that trigger the interaction interruption in the simulated interactive process. In this approach, decision rationale data can be obtained, and process description text corresponding to the problem process can be generated based on this data. For example, the decision rationale data can be used as the process description text corresponding to the problem process, or the decision rationale data can be summarized to obtain the process description text corresponding to the problem process. The process description text generated through method one is used to describe the reasons why the problem process triggers the interaction interruption conditions.

[0077] Method 2 The process data includes multimodal interaction data and response data from each round of dialogue in the problem flow. This multimodal interaction data and response data are used as the first target data. Based on the first target data and its corresponding reasoning chain, a process description text corresponding to the problem flow is generated. The testing system can use an artificial intelligence model to generate multimodal interaction data through reasoning. The reasoning chain corresponding to the first target data includes the reasoning chain of the multimodal interaction data generated by the testing system and the reasoning chain of the response data output by the intelligent interaction system. The first target data and its corresponding reasoning chain can be summarized to obtain the process description text corresponding to the problem flow. The process description text generated through method two is used to describe the process of the problem flow.

[0078] Method 3 The process data includes the decision rationale data in Method 1 and the multimodal interaction data and response data in Method 2. This multimodal interaction data and response data are used as the first target data. Second target data, used to explain the decision rationale data, is obtained from the first target data. The test system determines the interaction interruption condition based on the second target data and terminates the simulation interaction process. Therefore, the second target data can be considered as the problem data that causes the interaction interruption in the simulation interaction process. The problem data includes at least one round of multimodal interaction data and the corresponding response data. The reasoning chain corresponding to the second target data includes the reasoning chain of the multimodal interaction data generated by the test system in the second target data and the reasoning chain of the response data output by the intelligent interaction system in the second target data. Based on the second target data, the corresponding reasoning chain, and the decision rationale data, a process description text corresponding to the problem process is generated. This can be a summary of the second target data, the corresponding reasoning chain, and the decision rationale data to obtain the process description text corresponding to the problem process. The process description text generated through Method 3 is used to describe the reasons and process for triggering the interaction interruption condition in the problem process.

[0079] In practice, a process description text corresponding to the problem flow can be generated using one of the three methods described above, depending on different needs. This process satisfies the need to generate process description text under different circumstances, and produces a process description text that accurately describes the problem flow.

[0080] Furthermore, Figure 2 This is a flowchart illustrating an evaluation method for another intelligent interactive system provided in one or more embodiments of this specification. This evaluation method for the intelligent interactive system can be executed by the aforementioned testing system, such as... Figure 2 As shown, the process includes: Step S202: For the intelligent interactive system to be evaluated, generate a multi-dimensional user profile corresponding to the intelligent interactive system, and start the simulation interaction process between the intelligent interactive system based on the multi-dimensional user profile. Step S204: In the simulation interaction process, multimodal interaction data corresponding to the multidimensional user profile is generated. Based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, an interaction decision is made. The result of the interaction decision includes: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. Step S206: After multiple simulation interaction processes have ended, based on the process data of the simulation interaction processes that triggered the interaction interruption conditions, the interaction interruption of the intelligent interaction system is attributed. Step S208: After multiple simulation interaction processes are completed, determine the business scenario to be evaluated corresponding to the intelligent interaction system, and determine at least one business indicator to be evaluated corresponding to the intelligent interaction system based on the business scenario to be evaluated. Step S210: Determine the indicator values ​​of the business indicators to be evaluated for the intelligent interaction system based on the multimodal interaction data and response data corresponding to each completed simulation interaction process.

[0081] The process of steps S202-S206 will not be repeated here; please refer to the description above.

[0082] In step S208, after multiple simulation interaction processes, the business scenario to be evaluated for the intelligent interaction system is determined. The intelligent interaction system can be applied to one or more business scenarios. Here, the business scenario to be evaluated for the intelligent interaction system can be determined from the various business scenarios in which the intelligent interaction system is applied. The business scenario to be evaluated is a pre-determined business scenario, which includes, but is not limited to, one or more scenarios such as financial scenarios and shopping scenarios. Based on the business scenario to be evaluated, at least one business indicator to be evaluated for the intelligent interaction system is determined. Different business scenarios to be evaluated can correspond to different business indicators to be evaluated. At least one business indicator to be evaluated for the intelligent interaction system can be determined based on the pre-established relationship between business scenarios and business indicators. The business indicator to be evaluated includes at least one of the following: the user-initiated interaction rate of the intelligent interaction system, the interaction result conversion rate of the intelligent interaction system, and the average number of interaction rounds of the intelligent interaction system.

[0083] The user-initiated interaction rate refers to the proportion of times the test system simulates a real user corresponding to a multi-dimensional user profile to initiate a second round of multi-modal interaction data after the first round of dialogue (after sending multi-modal interaction data and receiving response data in the first round), out of the total number of simulated interaction processes. The interaction result conversion rate refers to the proportion of times the test system simulates a real user corresponding to a multi-dimensional user profile to place an order for products provided by the intelligent interaction system during each simulated interaction process, out of the total number of simulated interaction processes. The average number of interaction rounds refers to the average number of rounds of dialogue between the test system's simulated multi-dimensional user profile and the intelligent interaction system during each simulated interaction process.

[0084] In step S210 above, the numerical values ​​of the business indicators to be evaluated for the intelligent interaction system are determined based on the multimodal interaction data and response data from each completed simulation interaction process. For example, based on the multimodal interaction data and response data from each completed simulation interaction process, the values ​​of the user's proactive interaction rate, the interaction result conversion rate, and the average number of interaction rounds are statistically obtained.

[0085] pass Figure 2The process described in the article can determine the indicator values ​​of the intelligent interaction system based on the multimodal interaction data and response data corresponding to each completed simulation interaction process after multiple simulation interaction processes have ended, thus realizing the accurate evaluation of the business indicators of the intelligent interaction system.

[0086] Figure 2 In this process, step S206 can be executed first, followed by steps S208 and S210; or steps S208 and S210 can be executed first, followed by step S206; or the processes of step S206, S208, and S210 can be executed simultaneously.

[0087] Furthermore, Figure 3 This is a flowchart illustrating another evaluation method for an intelligent interactive system provided in one or more embodiments of this specification. This evaluation method for the intelligent interactive system can be executed by the aforementioned testing system, such as... Figure 3 As shown, the process includes: Step S302: For the intelligent interaction system to be evaluated, generate a multi-dimensional user profile corresponding to the intelligent interaction system, and establish a simulation interaction link between the intelligent interaction system and the multi-dimensional user profile to start the simulation interaction process. Step S304: In the simulation interaction process, determine the current interaction intent and the target interaction modality corresponding to the multi-dimensional user profile, and generate multi-modal interaction data corresponding to the multi-dimensional user profile based on the current interaction intent and the target interaction modality. Step S306: Send the multimodal interaction data to the intelligent interaction system through the simulation interaction link, and obtain the response data returned by the intelligent interaction system in response to the multimodal interaction data; Step S308: Make interaction decisions based on multimodal interaction data and response data. The results of the interaction decisions include: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. Step S310: After multiple simulation interaction processes have ended, based on the process data of the simulation interaction processes that triggered the interaction interruption conditions, the intelligent interaction system is attributed to the interaction interruption. In addition, based on the multimodal interaction data and response data corresponding to each ended simulation interaction process, the indicator values ​​of the business indicators to be evaluated for the intelligent interaction system are determined.

[0088] In step S308, an interaction decision is made based on the multimodal interaction data and response data. The multimodal interaction data and response data include all multimodal interaction data and all response data that have been sent and received in this simulation interaction process. This step can be: If, based on the multimodal interaction data and response data, it is determined that the current simulation interaction process has completed the target interaction intent or that the number of interaction rounds has reached the maximum value, then the current simulation interaction process is determined to meet the interaction completion conditions and ends. If, based on the multimodal interaction data and response data, it is determined that the current simulation interaction process has not completed the target interaction intent and the number of interaction rounds has not reached the maximum value, and based on the multimodal interaction data and response data, it is determined that the current simulation interaction process meets the reasons for continuing interaction, then it is determined that the current simulation interaction process meets the conditions for continuing interaction and the current simulation interaction process continues. If, based on the multimodal interaction data and response data, it is determined that the current simulation interaction process has not completed the target interaction intent and the number of interaction rounds has not reached the maximum value, and based on the multimodal interaction data and response data, it is determined that the current simulation interaction process meets the reason for terminating the interaction, then the current simulation interaction process is determined to trigger the interaction interruption condition and end the current simulation interaction process.

[0089] Among them, determining that the simulation interaction process meets the reasons for continuing the interaction based on multimodal interaction data and response data can be as follows: For this simulated interaction process, based on the multimodal interaction data and response data of the current round, historical multimodal interaction data and historical response data corresponding to the current round's multimodal interaction data and response data are obtained. The historical multimodal interaction data and the current round's multimodal interaction data are used as judgment reference data, and the current round's response data and historical response data are used as data to be judged. Based on the judgment reference data, the degree of matching between the data to be judged and the multidimensional user profile, the content quality of the data to be judged, and the degree of interaction guidance of the data to be judged are determined. Based on the degree of matching, content quality, and degree of interaction guidance, the reasons for continuing the interaction in this simulated interaction process are determined.

[0090] Among them, determining that the simulation interaction process meets the termination reason based on multimodal interaction data and response data can be as follows: For this simulated interaction process, based on the multimodal interaction data and response data of the current round, historical multimodal interaction data and historical response data corresponding to the current round's multimodal interaction data and response data are obtained. The historical multimodal interaction data and the current round's multimodal interaction data are used as reference data for judgment, and the current round's response data and historical response data are used as data to be judged. Based on the reference data for judgment, the degree of matching between the data to be judged and the multidimensional user profile, the content quality of the data to be judged, and the degree of interaction guidance of the data to be judged are determined. Based on the degree of matching, content quality, and degree of interaction guidance, it is determined that this simulated interaction process meets the reasons for terminating the interaction.

[0091] Figure 3The other processes can be referred to in the previous description, and will not be repeated here.

[0092] Furthermore, Figure 4 This is a flowchart illustrating another evaluation method for an intelligent interactive system provided in one or more embodiments of this specification. This evaluation method for the intelligent interactive system can be executed by the aforementioned testing system, such as... Figure 4 As shown, the process includes: Step S402: For the intelligent interaction system to be evaluated, generate a multi-dimensional user profile corresponding to the intelligent interaction system, and establish a simulation interaction link between the intelligent interaction system and the multi-dimensional user profile to start the simulation interaction process. Step S404: In this simulation interaction process, determine the current interaction intent and the target interaction modality corresponding to the multi-dimensional user profile. Based on the current interaction intent and the target interaction modality, generate the Nth multi-modal interaction data corresponding to the multi-dimensional user profile; N is a positive integer greater than or equal to 1. Step S406: Send the generated multimodal interaction data to the intelligent interaction system through the simulation interaction link, and obtain the response data returned by the intelligent interaction system in response to the multimodal interaction data; Step S408: Based on the multimodal interaction data and response data of each round of interaction in this simulation interaction process, make interaction decisions to determine whether the number of interaction rounds in this simulation interaction process has reached the maximum value. If yes, proceed to step S414; if no, proceed to step S410. Step S410: Based on the multimodal interaction data and response data of each round of interaction in this simulation interaction process, make interaction decisions to determine whether the target interaction intent has been achieved in this simulation interaction process. If yes, proceed to step S414; if no, proceed to step S412. Step S412: Based on the multimodal interaction data and response data of each round of interaction in this simulation interaction process, make an interaction decision to determine whether the current simulation interaction process meets the conditions for continuing interaction or triggers the conditions for interrupting interaction. If the conditions for continuing interaction are met, return to step S404 and execute the loop. If the conditions for interrupting interaction are triggered, execute step S414.

[0093] Step S414: Confirm the end of this simulation interaction process; Step S416: Repeat steps S402-S414 to perform multiple simulation interaction processes. After the multiple simulation interaction processes are completed, based on the process data of the simulation interaction processes that triggered the interaction interruption conditions, the interaction interruption of the intelligent interaction system is attributed. Step S418: Determine the indicator values ​​of the business indicators to be evaluated for the intelligent interaction system based on the multimodal interaction data and response data corresponding to each completed simulation interaction process.

[0094] Figure 4 In this simulation, the multi-dimensional user profiles corresponding to each interaction process can be the same, while the initial interaction intents can be different. This allows for evaluation of the same business scenario and the same user group through multiple simulation interactions. It should be noted that... Figure 4 During the process, you can first determine whether the number of interaction rounds in this simulation interaction process has reached the maximum value, and then determine whether the simulation interaction process has completed the target interaction intent. Alternatively, you can first determine whether the simulation interaction process has completed the target interaction intent, and then determine whether the number of interaction rounds in this simulation interaction process has reached the maximum value. Or you can simultaneously determine whether the simulation interaction process has completed the target interaction intent and whether the number of interaction rounds has reached the maximum value. Figure 4 The first case will be used as an example for explanation. Figure 4 The process described earlier can be referred to here, and will not be repeated here.

[0095] In summary, firstly, for the intelligent interaction system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interaction system is generated. Based on the multi-dimensional user profile, a simulated interaction process with the intelligent interaction system is initiated. Then, during the simulated interaction process, multimodal interaction data corresponding to the multi-dimensional user profile is generated. Interaction decisions are made based on the multimodal interaction data and the intelligent interaction system's response data to the multimodal interaction data. The results of the interaction decisions include: determining that the simulated interaction process meets the conditions for continuing interaction and continuing the simulated interaction process; or determining that the simulated interaction process triggers the interaction interruption condition and ending the simulated interaction process; or determining that the simulated interaction process meets the interaction completion condition and ending the simulated interaction process. Finally, after multiple simulated interaction processes have ended, based on the process data of the simulated interaction process that triggered the interaction interruption condition, the intelligent interaction system is attributed to the cause of the interaction interruption. Through this embodiment, a simulated interaction process can be initiated for the intelligent interaction system, and the simulated interaction process can be terminated when the interaction interruption condition is triggered, and the cause of the interaction interruption can be automatically attributed to the intelligent interaction system, thereby effectively improving the service capability of the intelligent interaction system and enhancing the user's interactive experience.

[0096] Figure 5 A schematic diagram of the test system provided in one or more embodiments of this specification, such as Figure 5 As shown, the testing system includes: a data foundation and profile building layer, a simulation interaction layer, a full-link simulation interaction log pool, and an evaluation and diagnostic output layer. Figure 5 middle, The data foundation and user profile building layer includes a data asset and user profile building module. This module is used to retrieve user data corresponding to the profile dimensions of the user group to be evaluated in the business scenario to be evaluated from the historical business database. This data includes user attribute data, user preference data, interaction habit data, and historical behavior data. Based on this user data, a multi-dimensional user profile corresponding to the intelligent interaction system is generated. This multi-dimensional user profile can be stored in a user profile library.

[0097] The simulation interaction layer comprises a simulation interaction execution module and a behavior decision engine module. The simulation interaction execution module establishes a simulation interaction link with the intelligent interaction system to initiate the simulation interaction process. It also determines the first round of multimodal interaction data and its interaction modality based on a multi-dimensional user profile, sends this data to the intelligent interaction system via the simulation interaction link, and receives the system's response data. The behavior decision engine module makes interaction decisions based on the first round of multimodal interaction data and response data. The decision results include: determining that the simulation interaction process meets the conditions for continuing interaction and continuing the process; or determining that the simulation interaction process triggers the conditions for interruption and ending the process; or determining that the simulation interaction process meets the conditions for completion and ending the process.

[0098] If the simulation interaction process is terminated, the behavior decision engine module triggers the simulation interaction execution module to end the simulation interaction process. If the simulation interaction process is continued, the behavior decision engine module determines the current interaction intent based on the multi-dimensional user profile, the already interacted data in the simulation interaction process (the first round of multimodal interaction data and response data), and the target interaction intent of the simulation interaction process. Based on the current interaction intent, the multi-dimensional user profile, and the target interaction intent, it determines the target interaction modality. Accordingly, the simulation interaction execution module generates the second round of multimodal interaction data corresponding to the multi-dimensional user profile based on the current interaction intent and the target interaction modality, and sends it to the intelligent interaction system through the simulation interaction link.

[0099] The simulation interaction execution module also receives response data in the second round, and the behavior decision engine module repeats the interaction decision until the simulation interaction process ends.

[0100] The full-link simulation interaction log pool is used to store all multimodal interaction data and its reasoning thought chain, response data and its reasoning thought chain in each simulation interaction process.

[0101] The evaluation and diagnosis output layer includes a business conversion prediction module and an evaluation, diagnosis, and attribution module. Based on data stored in the end-to-end simulation interaction log pool, the evaluation and diagnosis output layer uses the business conversion prediction module to determine the indicator values ​​of the business metrics to be evaluated in the intelligent interaction system and outputs the predicted business indicator values. Additionally, the evaluation, diagnosis, and attribution module performs interaction interruption attribution on the intelligent interaction system and outputs a closed-loop diagnostic report. The closed-loop diagnostic report includes the reasons for the interaction interruption in the intelligent interaction system and optimization suggestions for the intelligent interaction system.

[0102] Figure 5 Each module in the system can be an AI model based on AI capabilities.

[0103] Figure 6 This is a schematic diagram illustrating the working process of the test system provided in one or more embodiments of this specification, such as... Figure 6 As shown, the working process of this testing system includes: 1. Simulation Environment Configuration and Initialization Through the configuration center at the top level of the test system, the test system receives access requests for the intelligent interaction system, loads scenario / business configurations, sets simulation parameters such as concurrency and test duration, and calls prompt word template management modules. Figure 4 The prompts for each module in the document are used to provide guidance. Figure 4 Each module in the process completes the above steps and configures the simulation interaction link, thereby completing the initialization settings.

[0104] 2. Simulation Input Loading The simulation interaction execution module generates a multi-dimensional user profile corresponding to the intelligent interaction system and loads a seed question library. Based on the seed question library, it determines the first multimodal interaction data corresponding to the intelligent interaction system as the cold start corpus for the dialogue. The seed question library includes all the aforementioned first-round questions.

[0105] 3. Call the simulation interaction link to perform simulation interaction. The simulation interaction process is initiated. The simulation interaction execution module sends the multimodal interaction data of each round of dialogue to the intelligent interaction system through the simulation interaction link, and the intelligent interaction system returns the corresponding response data. The multimodal interaction data and response data are stored in the end-to-end simulation interaction log pool.

[0106] 4. Behavioral Decision-Making and Attribution The testing system uses a behavior decision engine module to make interaction decisions. This includes determining whether the simulation interaction process triggers an interruption condition and terminates the simulation interaction process; or whether the simulation interaction process meets the conditions for continuing interaction and continues; or whether the simulation interaction process meets the conditions for completing interaction and terminates. The behavior decision engine module also determines the current interaction intent and target interaction modality if the simulation interaction continues, enabling multi-turn dialogue loops. Furthermore, after each simulation interaction process, it attributes the interaction interruption to the intelligent interaction system and determines the metric values ​​of the business indicators to be evaluated for the intelligent interaction system.

[0107] In summary, through the above process, a full-link simulation and evaluation mechanism based on multi-dimensional profiling and behavioral decision-making engines was constructed, achieving automated simulation of the entire business process. This mechanism can not only evaluate the semantic quality of dialogues but also simulate user actions in non-textual stages such as premium calculation and product card clicks. Furthermore, it can predict the performance of the intelligent interaction system in real business scenarios, such as various business indicator values, before the system goes live, enabling technical personnel to directly predict business value and improving the directionality of model iteration. Finally, it can delve into the internal logic of multi-turn dialogues and automatically attribute interaction interruptions, providing a deterministic technical basis for optimizing the intelligent interaction system.

[0108] Figure 7 This is a schematic diagram of the structure of an evaluation device for an intelligent interactive system provided in one or more embodiments of this specification, such as... Figure 7 As shown, the device includes: The process initiation unit 72 generates a multi-dimensional user profile corresponding to the intelligent interactive system to be evaluated, and initiates a simulation interaction process with the intelligent interactive system based on the multi-dimensional user profile. The interaction execution unit 74 generates multimodal interaction data corresponding to the multidimensional user profile in the simulation interaction process, and makes interaction decisions based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data. The results of the interaction decisions include: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. The interruption attribution unit 76, after the completion of multiple simulation interaction processes, performs interaction interruption attribution on the intelligent interaction system based on the process data of the simulation interaction process that triggered the interaction interruption condition.

[0109] Optionally, process initiation unit 72: determines the business scenario to be evaluated corresponding to the intelligent interaction system and the user group to be evaluated corresponding to the intelligent interaction system, and obtains user data of the user group to be evaluated in the business scenario to be evaluated corresponding to the profile dimensions; the profile dimensions include one or more of user attribute dimensions, user preference dimensions, interaction habit dimensions, and historical behavior dimensions; based on the user data, generates a multi-dimensional user profile of the intelligent interaction system corresponding to the business scenario to be evaluated and the user group to be evaluated.

[0110] Optionally, the interaction execution unit 74: determines the current interaction intent corresponding to the multi-dimensional user profile and the target interaction modality corresponding to the current interaction intent; the target interaction modality includes sending data in a specified format to the intelligent interaction system or triggering components associated with the intelligent interaction system; based on the current interaction intent and the target interaction modality, generates multi-modal interaction data corresponding to the multi-dimensional user profile; the multi-modal interaction data includes the data in the specified format or trigger data for components associated with the intelligent interaction system.

[0111] Optionally, the interaction execution unit 74: when generating the first multimodal interaction data, determines the initial interaction intent corresponding to the multidimensional user profile and the interaction modality corresponding to the initial interaction intent, takes the initial interaction intent as the current interaction intent, and takes the interaction modality corresponding to the initial interaction intent as the target interaction modality; when generating non-first multimodal interaction data, determines the current interaction intent based on the multidimensional user profile, the interaction data already performed with the intelligent interaction system in the simulated interaction process, and the target interaction intent of the simulated interaction process, and determines the target interaction modality based on the current interaction intent, the multidimensional user profile, and the target interaction intent.

[0112] Optionally, the interaction execution unit 74 selects, from a plurality of preset interaction modalities, the modal that can realize the current interaction intent and matches the multi-dimensional user profile and the target interaction intent, as the target interaction modal.

[0113] Optionally, the interaction execution unit 74: if the target interaction modality includes sending data in a specified format to the intelligent interaction system, then generates data representing the current interaction intent according to the specified format, as the multimodal interaction data; if the target interaction modality includes triggering a component associated with the intelligent interaction system, then among the components associated with the intelligent interaction system, determines a target component representing the current interaction intent, and generates trigger data for the target component as the multimodal interaction data.

[0114] Optionally, the process initiation unit 72 establishes a simulation interaction link with the intelligent interaction system based on the multi-dimensional user profile to initiate the simulation interaction process; the interaction execution unit 74, before making an interaction decision based on the multi-modal interaction data and the response data of the intelligent interaction system to the multi-modal interaction data, sends the multi-modal interaction data to the intelligent interaction system through the simulation interaction link and obtains the response data returned by the intelligent interaction system to the multi-modal interaction data.

[0115] Optionally, the interaction execution unit 74: for the multimodal interaction data and response data of the current round, obtains historical multimodal interaction data and historical response data corresponding to the multimodal interaction data and response data of the current round, uses the historical multimodal interaction data and the multimodal interaction data of the current round as judgment reference data, and uses the response data of the current round and the historical response data as data to be judged; based on the judgment reference data, determines the degree of matching between the data to be judged and the multidimensional user profile, the content quality of the data to be judged, and the degree of interaction guidance of the data to be judged; and makes an interaction decision based on the degree of matching, the content quality, the degree of interaction guidance, the degree of completion of the target interaction intent of the simulated interaction process, and the number of rounds of interaction in the simulated interaction process.

[0116] Optionally, the interaction execution unit 74: if the simulated interaction process completes the target interaction intent or the number of interaction rounds reaches the maximum value, then it is determined that the simulated interaction process meets the interaction completion condition and the simulated interaction process ends; if the simulated interaction process does not complete the target interaction intent and the number of interaction rounds does not reach the maximum value, and the simulated interaction process meets the reason for continuing interaction based on the matching degree, the content quality, and the interaction guidance degree, then it is determined that the simulated interaction process meets the condition for continuing interaction and the simulated interaction process continues; if the simulated interaction process does not complete the target interaction intent and the number of interaction rounds does not reach the maximum value, and the simulated interaction process meets the reason for terminating interaction based on the matching degree, the content quality, and the interaction guidance degree, then it is determined that the simulated interaction process triggers the interaction interruption condition and the simulated interaction process ends.

[0117] Optionally, the interruption attribution unit 76: determines the simulation interaction process that triggers the interaction interruption condition as a problem process; generates a process description text corresponding to the problem process based on the process data of the problem process; clusters each problem process based on the semantic information of the process description text of each problem process to obtain at least one class of problem processes; and determines the common cause of the interaction interruption in the problem processes of that class based on the common semantic information of the process description text corresponding to each class of problem processes.

[0118] Optionally, the interruption attribution unit 76: the process data includes decision reason data for the interactive decision corresponding to the problem process, and a process description text corresponding to the problem process is generated based on the decision reason data; or, the process data includes the multimodal interaction data and the response data corresponding to the problem process, and the multimodal interaction data and the response data corresponding to the problem process are used as first target data; a process description text corresponding to the problem process is generated based on the first target data and the reasoning chain corresponding to the first target data; or, the process data includes decision reason data for the interactive decision corresponding to the problem process, the multimodal interaction data and the response data corresponding to the problem process, and the multimodal interaction data and the response data corresponding to the problem process are used as first target data; second target data for explaining the decision reason data is obtained from the first target data, and a process description text corresponding to the problem process is generated based on the second target data, the reasoning chain corresponding to the second target data, and the decision reason data.

[0119] Optionally, it also includes an indicator determination unit: after the completion of multiple simulation interaction processes, determining the business scenario to be evaluated corresponding to the intelligent interaction system, determining at least one business indicator to be evaluated corresponding to the intelligent interaction system based on the business scenario to be evaluated, and determining the indicator value of the business indicator to be evaluated of the intelligent interaction system based on the multimodal interaction data and the response data corresponding to each completed simulation interaction process.

[0120] This embodiment enables the initiation of a simulation interaction process for an intelligent interactive system, and the termination of the simulation interaction process when an interaction interruption condition is triggered. It also automatically attributes the interaction interruption to the intelligent interactive system, thereby effectively improving the service capabilities of the intelligent interactive system and enhancing the user's interactive experience.

[0121] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0122] Since the apparatus embodiment corresponds to the method embodiment, the description is relatively simple. Figure 6 For more details on the devices used, please refer to the description of the evaluation method embodiments for intelligent interactive systems, which will not be repeated here.

[0123] Corresponding to the evaluation method for the intelligent interactive system described above, and based on the same technical concept, one or more embodiments of this specification also provide an evaluation device for an intelligent interactive system. This evaluation device is used to execute the evaluation method for the intelligent interactive system provided above. Figure 8 This is a structural schematic diagram of an evaluation device for an intelligent interactive system provided in one or more embodiments of this specification.

[0124] like Figure 8 As shown, device 800 mainly consists of a communication interface 802, a user interface 804, a processor 806, and a data storage 808. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 810. The communication interface 802 enables device 800 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 802 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 802 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 802 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 802 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0125] User interface 804 includes receiving user input and providing output to the user. Therefore, user interface 804 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 804 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 804 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 800 may support remote access from other devices via communication interface 802 or another physical interface (not shown). User interface 804 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 804 may also be configured as a display device for rendering or displaying text fragments.

[0126] Processor 806 may include one or more general-purpose processors and / or special-purpose processors. Data storage 808 may include one or more volatile and / or non-volatile storage components, and may be integrated wholly or partially with processor 806. Data storage 808 may include removable and non-removable components.

[0127] Processor 806 is capable of executing program instructions 818 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 808 to perform the various functions described herein. Data storage 808 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 800, enable device 800 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 818 by processor 806 may result in processor 806 using data 812. For example, program instructions 818 may include an operating system 822 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 800 and one or more application programs 820 (e.g., a browser, social application, or game application).

[0128] Similarly, data 812 may contain operating system data 816 and application data 814. Operating system data 816 is primarily accessible to the operating system 822, while application data 814 is primarily accessible to one or more applications 820. Application data 814 may reside in a file system visible or hidden from the user of device 800. Applications 820 may communicate with the operating system 822 through one or more application programming interfaces (APIs). These APIs facilitate applications 820 in reading and / or writing application data 814, transmitting or receiving information via communication interface 802, receiving or displaying information on user interface 804, etc. In some terms, application 820 may be simply referred to as an "app". Furthermore, application 820 may be downloaded to device 800 through one or more online app stores or app markets. However, applications may also be installed on device 800 in other ways, such as through a web browser or a physical interface on device 800 (e.g., a USB port).

[0129] In one specific embodiment, the evaluation device for the intelligent interaction system includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the evaluation device for the intelligent interaction system, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: For the intelligent interaction system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interaction system is generated, and a simulation interaction process between the intelligent interaction system is initiated based on the multi-dimensional user profile. In the simulation interaction process, multimodal interaction data corresponding to the multidimensional user profile is generated. Based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, an interaction decision is made. The result of the interaction decision includes: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. After the simulation interaction process has ended multiple times, the intelligent interaction system performs interaction interruption attribution based on the process data of the simulation interaction process that triggered the interaction interruption condition.

[0130] This embodiment enables the initiation of a simulation interaction process for an intelligent interactive system, and the termination of the simulation interaction process when an interaction interruption condition is triggered. It also automatically attributes the interaction interruption to the intelligent interactive system, thereby effectively improving the service capabilities of the intelligent interactive system and enhancing the user's interactive experience.

[0131] Since the device embodiment corresponds to the method embodiment, the description is relatively simple. For more information about the device, please refer to the description of the evaluation method embodiment for intelligent interactive systems, which will not be repeated here.

[0132] This specification provides an embodiment of a computer-readable storage medium as follows: Corresponding to the evaluation method of the intelligent interactive system described above, and based on the same technical concept, one or more embodiments of this specification also provide a computer-readable storage medium.

[0133] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed, implement the following process: For the intelligent interaction system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interaction system is generated, and a simulation interaction process between the intelligent interaction system is initiated based on the multi-dimensional user profile. In the simulation interaction process, multimodal interaction data corresponding to the multidimensional user profile is generated. Based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, an interaction decision is made. The result of the interaction decision includes: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. After the simulation interaction process has ended multiple times, the intelligent interaction system performs interaction interruption attribution based on the process data of the simulation interaction process that triggered the interaction interruption condition.

[0134] This embodiment enables the initiation of a simulation interaction process for an intelligent interactive system, and the termination of the simulation interaction process when an interaction interruption condition is triggered. It also automatically attributes the interaction interruption to the intelligent interactive system, thereby effectively improving the service capabilities of the intelligent interactive system and enhancing the user's interactive experience.

[0135] It should be noted that the embodiments of a computer-readable storage medium described in this specification and the embodiments of an evaluation method for an intelligent interactive system described in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0136] This specification provides an example of a computer program product as follows: Corresponding to the evaluation method of the intelligent interactive system described above, based on the same technical concept, one or more embodiments of this specification also provide a computer program product.

[0137] A computer program product includes a computer program / instructions that, when executed by a processor, perform the following steps: For the intelligent interaction system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interaction system is generated, and a simulation interaction process between the intelligent interaction system is initiated based on the multi-dimensional user profile. In the simulation interaction process, multimodal interaction data corresponding to the multidimensional user profile is generated. Based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, an interaction decision is made. The result of the interaction decision includes: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. After the simulation interaction process has ended multiple times, the intelligent interaction system performs interaction interruption attribution based on the process data of the simulation interaction process that triggered the interaction interruption condition.

[0138] This embodiment enables the initiation of a simulation interaction process for an intelligent interactive system, and the termination of the simulation interaction process when an interaction interruption condition is triggered. It also automatically attributes the interaction interruption to the intelligent interactive system, thereby effectively improving the service capabilities of the intelligent interactive system and enhancing the user's interactive experience.

[0139] It should be noted that the embodiments of a computer program product described in this specification and the embodiments of an evaluation method for an intelligent interactive system described in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0140] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments. For example, the device embodiment, equipment embodiment and computer-readable storage medium embodiment are all similar to the method embodiment, so the description is relatively simple. When reading the relevant content of the device embodiment, equipment embodiment and computer-readable storage medium embodiment, please refer to the description of the method embodiment.

[0141] While one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps, and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims. This specification uses specific terms to describe embodiments of this specification. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0142] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0143] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0144] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0145] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0146] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0147] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0152] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0153] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0154] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising at least one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] It should also be noted that the terms "one," "an," and "the" do not specifically refer to the singular; they can also include the plural. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to distinguish objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server. Unless explicitly stated, "receiving and sending data" does not necessarily mean direct receipt and transmission; it can be indirect. For example, A receiving data sent by B can be understood as A directly receiving data sent by B, or it can be understood as A indirectly receiving data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending data directly to A, or it can be understood as B indirectly sending data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0156] Unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is above B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0157] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0158] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A method for evaluating an intelligent interactive system, comprising: For the intelligent interaction system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interaction system is generated, and a simulation interaction process between the intelligent interaction system is initiated based on the multi-dimensional user profile. In the simulation interaction process, multimodal interaction data corresponding to the multidimensional user profile is generated. Based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, an interaction decision is made. The result of the interaction decision includes: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. After the simulation interaction process has ended multiple times, the intelligent interaction system performs interaction interruption attribution based on the process data of the simulation interaction process that triggered the interaction interruption condition.

2. The method according to claim 1, wherein generating the multi-dimensional user profile corresponding to the intelligent interaction system includes: The evaluation process involves identifying the business scenario to be evaluated corresponding to the intelligent interaction system and the user group to be evaluated corresponding to the intelligent interaction system, and obtaining user data of the user group to be evaluated in the business scenario to be evaluated, which corresponds to the profile dimensions. The profile dimensions include one or more of the following: user attribute dimensions, user preference dimensions, interaction habit dimensions, and historical behavior dimensions. Based on the user data, a multi-dimensional user profile of the intelligent interaction system is generated, corresponding to the business scenario to be evaluated and the user group to be evaluated.

3. The method according to claim 1, wherein generating the multimodal interaction data corresponding to the multidimensional user profile includes: Determine the current interaction intent corresponding to the multi-dimensional user profile and the target interaction modality corresponding to the current interaction intent; The target interaction modality includes sending data in a specified format to the intelligent interaction system or triggering components associated with the intelligent interaction system; Based on the current interaction intent and the target interaction modality, multimodal interaction data corresponding to the multidimensional user profile is generated; the multimodal interaction data includes data in the specified format or trigger data for components associated with the intelligent interaction system.

4. The method according to claim 3, wherein determining the current interaction intent corresponding to the multi-dimensional user profile and the target interaction modality corresponding to the current interaction intent comprises: When generating the first multimodal interaction data, the initial interaction intent corresponding to the multidimensional user profile and the interaction modality corresponding to the initial interaction intent are determined, and the initial interaction intent is taken as the current interaction intent and the interaction modality corresponding to the initial interaction intent is taken as the target interaction modality. When generating non-first multimodal interaction data, the current interaction intent is determined based on the multi-dimensional user profile, the interaction data with the intelligent interaction system in the simulated interaction process, and the target interaction intent of the simulated interaction process. The target interaction modality is then determined based on the current interaction intent, the multi-dimensional user profile, and the target interaction intent.

5. The method according to claim 4, wherein determining the target interaction modality based on the current interaction intent, the multi-dimensional user profile, and the target interaction intent comprises: Among multiple preset interaction modalities, the modality that can realize the current interaction intent and matches the multi-dimensional user profile and the target interaction intent is selected as the target interaction modality.

6. The method according to claim 3, wherein generating multimodal interaction data corresponding to the multidimensional user profile based on the current interaction intent and the target interaction modality includes: If the target interaction modality includes sending data in a specified format to the intelligent interaction system, then data representing the current interaction intent is generated according to the specified format and used as the multimodal interaction data; If the target interaction modality includes triggering a component associated with the intelligent interaction system, then among the components associated with the intelligent interaction system, a target component representing the current interaction intent is determined, and trigger data for the target component is generated as the multimodal interaction data.

7. The method according to claim 1, wherein initiating the simulation interaction process between the user profile and the intelligent interaction system based on the multi-dimensional user profile includes: Based on the multi-dimensional user profile, a simulation interaction link is established between the user and the intelligent interaction system to initiate the simulation interaction process. Before making interaction decisions based on the multimodal interaction data and the intelligent interaction system's response data to the multimodal interaction data, the method further includes: The multimodal interaction data is sent to the intelligent interaction system through the simulation interaction link, and the response data returned by the intelligent interaction system in response to the multimodal interaction data is obtained.

8. The method according to claim 1, wherein the step of making interaction decisions based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data includes: For the multimodal interaction data and response data of the current round, obtain the historical multimodal interaction data and historical response data corresponding to the multimodal interaction data and response data of the current round, use the historical multimodal interaction data and the multimodal interaction data of the current round as judgment reference data, and use the response data of the current round and the historical response data as data to be judged; Based on the judgment reference data, the degree of matching between the data to be judged and the multi-dimensional user profile, the content quality of the data to be judged, and the degree of interactive guidance of the data to be judged are determined. Based on the degree of matching, the content quality, the degree of interactive guidance, the degree of completion of the target interactive intent of the simulated interactive process, and the number of interactive rounds already completed in the simulated interactive process, an interaction decision is made.

9. The method according to claim 8, wherein making interaction decisions based on the matching degree, the content quality, the degree of interaction guidance, the degree of completion of the target interaction intent of the simulated interaction process, and the number of interaction rounds already completed in the simulated interaction process includes: If the simulation interaction process completes the target interaction intent or the number of interaction rounds reaches the maximum value, then the simulation interaction process is determined to meet the interaction completion condition and the simulation interaction process ends. If the simulated interaction process fails to complete the target interaction intent and the number of interaction rounds has not reached the maximum value, and the simulated interaction process is determined to meet the reason for continuing interaction based on the matching degree, the content quality, and the degree of interaction guidance, then the simulated interaction process is determined to meet the condition for continuing interaction and the simulated interaction process continues. If the simulated interaction process fails to complete the target interaction intent and the number of interaction rounds has not reached the maximum value, and the simulated interaction process meets the reason for termination of interaction based on the matching degree, the content quality, and the degree of interaction guidance, then the simulated interaction process is determined to trigger the interaction interruption condition and the simulated interaction process ends.

10. The method according to claim 1, wherein attributing the interaction interruption of the intelligent interaction system based on the process data of the simulated interaction flow that triggers the interaction interruption condition includes: The simulation interaction process that triggers the interaction interruption condition is identified as the problem process. Based on the process data of the problem process, a process description text corresponding to the problem process is generated. Based on the semantic information of the process description text of each of the problem processes, the problem processes are clustered to obtain at least one class of problem processes. Based on the common semantic information of the process description text corresponding to the problem process of each category, determine the common reasons for the interaction interruption in the problem process of that category.

11. The method according to claim 10, wherein generating the process description text corresponding to the problem process based on the process data of the problem process includes: The process data includes decision reason data for the interactive decision corresponding to the problem process, and a process description text corresponding to the problem process is generated based on the decision reason data. or, The process data includes the multimodal interaction data and the response data corresponding to the problem process, and the multimodal interaction data and the response data corresponding to the problem process are used as the first target data; Based on the first target data and the reasoning chain corresponding to the first target data, generate a process description text corresponding to the problem process; or, The process data includes decision reasoning data for the interactive decision corresponding to the problem process, multimodal interaction data and response data corresponding to the problem process, and the multimodal interaction data and response data corresponding to the problem process are used as first target data; Obtain second target data from the first target data to explain the decision rationale data, and generate a process description text corresponding to the problem process based on the second target data, the reasoning chain corresponding to the second target data, and the decision rationale data.

12. The method according to claim 1, further comprising: After the simulation interaction process is completed multiple times, the business scenario to be evaluated corresponding to the intelligent interaction system is determined, and at least one business indicator to be evaluated corresponding to the intelligent interaction system is determined based on the business scenario to be evaluated. Based on the multimodal interaction data and response data corresponding to each completed simulation interaction process, the indicator values ​​of the business indicators to be evaluated in the intelligent interaction system are determined.

13. An evaluation device for an intelligent interactive system, comprising: The process initiation unit generates a multi-dimensional user profile corresponding to the intelligent interactive system to be evaluated, and initiates a simulation interaction process with the intelligent interactive system based on the multi-dimensional user profile. The interaction execution unit generates multimodal interaction data corresponding to the multidimensional user profile in the simulation interaction process, and makes interaction decisions based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data. The results of the interaction decisions include: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. The interruption attribution unit, after the completion of multiple simulation interaction processes, performs interaction interruption attribution on the intelligent interaction system based on the process data of the simulation interaction process that triggered the interaction interruption condition.

14. An evaluation device for an intelligent interactive system, comprising: processor; And, a memory configured to store computer-executable instructions, which, when executed, cause the processor to: For the intelligent interaction system to be evaluated, a multi-dimensional user profile corresponding to the intelligent interaction system is generated, and a simulation interaction process between the intelligent interaction system is initiated based on the multi-dimensional user profile. In the simulation interaction process, multimodal interaction data corresponding to the multidimensional user profile is generated. Based on the multimodal interaction data and the response data of the intelligent interaction system to the multimodal interaction data, an interaction decision is made. The result of the interaction decision includes: determining that the simulation interaction process meets the conditions for continuing the interaction and continuing the simulation interaction process; or determining that the simulation interaction process triggers the conditions for interrupting the interaction and ending the simulation interaction process; or determining that the simulation interaction process meets the conditions for completing the interaction and ending the simulation interaction process. After the simulation interaction process has ended multiple times, the intelligent interaction system performs interaction interruption attribution based on the process data of the simulation interaction process that triggered the interaction interruption condition.