Evaluation method and device for digital assistant, equipment and storage medium

Abstract

The embodiment of the invention provides a digital assistant evaluation method and device, equipment and a storage medium. The method comprises: in response to an evaluation request for a target digital assistant, obtaining at least one group of test cases for the target digital assistant, each group of test cases comprising at least one test problem related to a chat skill of the target digital assistant; and providing the at least one group of test cases for the target digital assistant to obtain replies of the target digital assistant to the at least one group of test cases. And determining a target evaluation index for the target digital assistant at least based on the at least one group of test cases and replies of the target digital assistant to the at least one group of test cases, the target evaluation index at least comprising a first feature value indicating a chat skill score of the target digital assistant. And determining a quality evaluation result of the target digital assistant based on the target evaluation index. Therefore, the evaluation efficiency of the digital assistant can be improved, and the objectivity and consistency of the evaluation standard are met.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to evaluation methods, apparatus, devices, and storage media for digital assistants. Background Technology

[0002] Digital assistants refer to systems or applications with conversational capabilities. With the increasing prevalence of digital assistants in customer service, education, entertainment, and other fields, the quality of their interaction is becoming increasingly important. Evaluating digital assistants is crucial for ensuring their quality and performance. Through evaluation, digital assistants that meet quality and performance requirements can be recommended to users, thereby improving user experience and satisfaction. Therefore, accurately evaluating digital assistants is of paramount importance. Summary of the Invention

[0003] In a first aspect of this disclosure, an evaluation method for a digital assistant is provided. The method may include: in response to an evaluation request for a target digital assistant, obtaining at least one set of test cases for the target digital assistant, each set of test cases including at least one test question related to the chat skills of the target digital assistant; providing the at least one set of test cases to the target digital assistant to obtain responses from the target digital assistant to the at least one set of test cases; determining a target evaluation metric for the target digital assistant based at least on the at least one set of test cases and the target digital assistant's responses to the at least one set of test cases, the target evaluation metric including at least a first feature value indicating a chat skills score for the target digital assistant; and determining a quality evaluation result for the target digital assistant based on the target evaluation metric.

[0004] In a second aspect of this disclosure, an evaluation apparatus for a digital assistant is provided. The apparatus may include: a test case acquisition module configured to, in response to an evaluation request for a target digital assistant, acquire at least one set of test cases for the target digital assistant, each set of test cases including at least one test question related to the chat skills of the target digital assistant; a response acquisition module configured to provide the at least one set of test cases to the target digital assistant to obtain responses from the target digital assistant to the at least one set of test cases; a target evaluation metric determination module configured to, based at least on the at least one set of test cases and the responses from the target digital assistant to the at least one set of test cases, determine a target evaluation metric for the target digital assistant, the target evaluation metric including at least a first feature value indicating a chat skills score for the target digital assistant; and a quality evaluation result determination module configured to, based on the target evaluation metric, determine a quality evaluation result for the target digital assistant.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. A computer program is stored on the medium, which, when executed by a processor, implements the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional embodiments of one aspect of this application. In other words, when the computer instructions are executed by the processor, they implement the methods provided in various optional embodiments of one aspect of this application.

[0008] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart of a method for evaluating a digital assistant according to some embodiments of the present disclosure is shown;

[0012] Figure 3 Example diagrams of a chat skills scoring interface according to some embodiments of the present disclosure are shown;

[0013] Figure 4 A flowchart illustrating the training process of an evaluation model according to some embodiments of the present disclosure is shown;

[0014] Figure 5 An example graph showing the relevance distribution according to some embodiments of this disclosure is shown;

[0015] Figure 6A schematic structural block diagram of an apparatus for digital assistant evaluation according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device that can implement one or more embodiments of the present disclosure is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and authorization should be obtained from the relevant users. Among them, relevant users may include any type of rights holder, such as individuals, enterprises, and groups.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as the electronic device, application, server, or storage medium that performs the operation of the technical solution disclosed herein based on the prompt message.

[0023] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0026] Figure 1 A schematic diagram of an environment 100 in which embodiments of the present disclosure can be implemented is shown. A digital assistant development platform 120 provides developers 105 with an environment for creating and publishing digital assistants. For example, the digital assistant development platform 120 may provide developers 105 with various tools such as prompt words, plugins, workflows, knowledge bases, memory banks, voice, etc. In some embodiments, the digital assistant development platform 120 may be a low-code platform that provides a collection of tools for creating digital assistants. The digital assistant development platform 120 may support visual development of digital assistants, allowing developers 105 to skip the manual coding process, accelerating application development cycles and reducing costs. The digital assistant development platform 120 may support any suitable platform for users to develop digital assistants and other types of applications, such as an Application Platform as a Service (aPaaS) based platform. Such a platform enables users to efficiently develop applications, enabling operations such as application creation and application function adjustments.

[0027] The digital assistant development platform 120 can be deployed locally on the developer 105's terminal device and / or supported by a remote server. For example, the developer 105's terminal device can run a client of the digital assistant development platform 120, which can support user interaction with the digital assistant development platform 120. When the digital assistant development platform 120 runs locally on the user's terminal device, the developer 105 can directly interact with the local digital assistant development platform 120 using the client. When the digital assistant development platform 120 runs on a server device, the server device can provide services to the client running on the terminal device based on the communication connection with the terminal device. The digital assistant development platform 120 can present a corresponding interface 122 to the developer 105 based on the developer 105's operations, to output and / or receive information from the developer 105.

[0028] In some embodiments, the digital assistant development platform 120 can be associated with a corresponding database, which stores the data or information required for the digital assistant creation process supported by the digital assistant development platform 120. For example, the database can store the code and description information corresponding to the various functional modules that make up the digital assistant. The digital assistant development platform 120 can also perform operations such as calling, adding, deleting, and updating the functional modules in the database. The database can also store operations that can be performed on different functional blocks. For example, in a scenario where a digital assistant needs to be created, the digital assistant development platform 120 can call the corresponding functional blocks from the database to build the digital assistant.

[0029] In embodiments of this disclosure, developer 105 can create and publish digital assistant 121 on digital assistant development platform 120 as needed. Digital assistant 121 can be published to any suitable application platform, as long as the application platform supports the operation of digital assistant 121. After publication, digital assistant 121 can be used for conversational interaction with user 135.

[0030] After digital assistant 121 is created / published, it can be evaluated by electronic device 110 to obtain evaluation results. For digital assistants whose evaluation results meet the recommendation criteria, recommendations can be made through the recommendation interface of digital assistant recommendation platform 130. For example, digital assistant recommendation platform 130 can be integrated into electronic device 110 or a third-party platform independent of electronic device 110. The evaluation of digital assistant 121 by electronic device 110 can be based on multiple dimensions, such as evaluation indicators corresponding to chat skills, evaluation indicators corresponding to user feedback during interaction with user 135, and so on.

[0031] Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0032] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0033] Traditional digital assistant evaluation methods primarily involve operational staff trying and testing the software before making recommendations. This approach relies on manual testing and evaluation, which, while ensuring a certain level of quality, is inefficient and struggles to guarantee objectivity and consistency.

[0034] In embodiments of this disclosure, a method for evaluating a digital assistant is proposed. For a target digital assistant to be evaluated, at least one set of test cases is obtained, each set of test cases including at least one test question related to the target digital assistant's chat skills. The at least one set of test cases is provided to the target digital assistant to obtain responses from the target digital assistant to the at least one set of test cases. Based at least on the at least one set of test cases and the target digital assistant's responses to the at least one set of test cases, a target evaluation metric is determined for the target digital assistant, the target evaluation metric including at least a first feature value indicating the target digital assistant's chat skill score. Based on the target evaluation metric, a quality evaluation result for the target digital assistant is determined.

[0035] The above process enables automated evaluation of digital assistants, assessing their quality at least from the perspective of their chat skills. This automated evaluation method reduces reliance on manual testing, making the evaluation process faster, more continuous, and uninterrupted. Operations personnel no longer need to try and test each digital assistant individually, saving significant time and human resources. Furthermore, automated test cases and evaluation metrics ensure standardization of the evaluation process, avoiding subjective biases caused by individual differences and ensuring the objectivity and consistency of the evaluation results. By using multiple sets of test cases, the various functions and performance of the target digital assistant can be comprehensively evaluated, providing more detailed and accurate evaluation results.

[0036] Figure 2An example flow 200 of an evaluation method for a digital assistant according to some embodiments of the present disclosure is shown. For ease of discussion, reference will be made to... Figure 1 The process 200 is described in the context of the application environment 110. In the application environment 110, the digital assistant can be evaluated by the electronic device 110.

[0037] In box 201, electronic device 110, in response to an evaluation request for target digital assistant 121, obtains at least one set of test cases for target digital assistant 121, each set of test cases including at least one test question related to the chat skills of target digital assistant 121.

[0038] The evaluation request for the target digital assistant 121 can be varied. For example, it can be based on the target digital assistant 121 being online for a certain period of time, or it can be based on the evaluation instructions from the developers, or it can be based on the number of user feedback, and so on.

[0039] Test cases for the chat skills of the target digital assistant 121 can include multiple sets, each corresponding to a different evaluation dimension. Chat skill test cases can be generated based on prompts. For example, prompts can include the identifier of the target digital assistant 121, which could be its name or serial number. Furthermore, prompts can also include a description of the target digital assistant's functions, such as a brief introduction, function overview, and user guide, etc.

[0040] A brief description can depict the main functions of the target digital assistant 121. For example, the brief description could include: the digital assistant's role is that of a legal assistant, capable of answering various legal-related questions.

[0041] The feature description can indicate the services or functions that the target digital assistant 121 can provide. For example, the feature description may include: the digital assistant can provide users with multiple services such as legal consultation, legal document generation, legal fee calculation, and legal education.

[0042] The user guide can introduce how to interact with the target digital assistant 121. For example, the user guide could include: Users can ask me any legal-related questions they are interested in.

[0043] The evaluation dimensions can be predefined. Different evaluation dimensions are used to assess the different abilities demonstrated by the digital assistant's chat skills. In some embodiments, different evaluation dimensions may correspond to identity recognition ability, functional recognition ability, basic interaction ability, interaction ability positively correlated with the area of ​​expertise, interaction ability negatively correlated with the area of ​​expertise, and the ability to handle abnormal interactions, etc. It is understood that, depending on specific evaluation needs, other evaluation dimensions and their corresponding abilities to be evaluated can also be defined.

[0044] Test cases for identity recognition ability can be used to assess the target digital assistant 121's ability to recognize and express its own identity. For example, if the function description includes the target digital assistant 121's area of ​​expertise being law, then the test question could be something like "Who are you?" to guide the target digital assistant 121 to answer who it is. Alternatively, test questions could be things like "Are you a meeting host?" to mislead the target digital assistant 121 about its identity.

[0045] Functional cognitive ability test cases can be used to evaluate the target digital assistant 121's description and understanding of its functions, such as questions like "What can you do?", to guide the target digital assistant 121 to state its functions.

[0046] Basic interaction capability test cases can be used to evaluate the target digital assistant 121's ability to handle general dialogues, including understanding user intent and providing appropriate responses.

[0047] Interaction capability test cases that are positively correlated with the target digital assistant 121's area of ​​expertise can be used to evaluate the target digital assistant 121's performance in its area of ​​expertise. For example, the accuracy and professionalism of the target digital assistant 121 in answering legal-related questions.

[0048] Interaction capability test cases that are negatively correlated with areas of expertise can be used to examine the performance of the target digital assistant 121 on issues outside its areas of expertise, ensuring that it can reasonably guide users or acknowledge its own limitations.

[0049] Test cases for handling abnormal interactions can be used to simulate various abnormal situations or erroneous operations to evaluate how the digital assistant handles unexpected user input or abnormal requests.

[0050] For a given set of chat skills test cases, at least one round of interaction testing can be identified that is relevant to a given evaluation dimension. For example, if the feature description includes the target digital assistant's area of ​​expertise being law, then the test cases for identity recognition ability can include at least two rounds of interaction testing. The first round of interaction testing could involve questions like "Who are you?" to guide the target digital assistant to identify itself. The second round of interaction testing could involve questions like "Are you the host?" to mislead the target digital assistant about its identity.

[0051] In box 202, electronic device 110 provides at least one set of test cases to target digital assistant 121 to obtain responses from target digital assistant 121 to at least one set of test cases. Based on these responses, electronic device 110 can evaluate the performance and capabilities of target digital assistant 121 in different scenarios in subsequent processes.

[0052] In block 203, electronic device 110 determines a target evaluation metric for target digital assistant 121 based on at least one set of test cases and the responses of target digital assistant 121 to at least one set of test cases. In embodiments of this disclosure, the target evaluation metric includes at least a first feature value indicating the chat skills score of the target digital assistant.

[0053] If multiple test cases are used to test the target digital assistant 121, and each test case includes at least one test question, then the responses of the target digital assistant 121 to different test questions can be obtained. For the chat skill scores corresponding to multiple responses, the first feature value of the chat skill score of the target digital assistant 121 can be obtained by averaging, weighted averaging, or other methods.

[0054] As previously mentioned, the target evaluation metric can indicate the primary characteristic value of the target digital assistant 121's chat skills score. In addition, the target evaluation metric can also indicate characteristic values ​​for different scores such as the target digital assistant 121's response speed, fluency of speech, and personalization of responses.

[0055] In box 204, electronic device 110 determines the quality assessment result of target digital assistant 121 based on target evaluation metrics. Electronic device 110 can assign corresponding weights to different target evaluation metrics. The weights can reflect the importance of each target evaluation metric in the overall evaluation. For example, for a digital assistant, its identity recognition, functional recognition, and knowledge interaction in its areas of expertise can be given higher weights. Through this weighted processing, electronic device 110 can generate a comprehensive evaluation result that accurately reflects the overall performance of target digital assistant 121.

[0056] The evaluation methods described above overcome the limitations of existing technologies that rely on manual testing and recommendation of digital assistants by operations personnel. The automated evaluation process not only improves efficiency but also ensures objectivity and consistency. This enhances the user experience, ensures that recommended digital assistants meet user needs, and provides developers with feedback for product improvement.

[0057] As previously mentioned, the evaluation of digital assistants is based on test cases. The method for obtaining test cases is described in detail below. In some embodiments of this disclosure, the electronic device 110 obtains the prompt word information of the target digital assistant 121, which includes at least the identification information and functional description of the target digital assistant 121. It obtains the general question generation rules corresponding to each set of test cases in at least one set of test cases. Based at least on the prompt word information and the general question generation rules, it generates one or more sets of test cases for the target digital assistant 121.

[0058] The prompts from the target digital assistant 121 include at least its identification information and a functional description. The identification information uniquely identifies the target digital assistant 121, such as its name and serial number. The functional description describes the main functions and features of the target digital assistant 121, such as information about the developer, the areas of expertise of the target digital assistant 121, and the types of conversations it supports.

[0059] General question generation rules can be a set of common standards and guidelines for creating test cases. These rules ensure that the generated test cases are relevant to the prompts of the target digital assistant 121 and can effectively test its functionality and performance.

[0060] For example, general question generation rules include the following key points: During test case generation, ensure that the generated test cases are relevant to the digital assistant's prompts. Generated test cases must conform to predefined rules. In cases involving multiple rounds of interaction, continue the existing topic rather than introducing new ones to ensure the continuity of the interactions. Only output the generated test cases. Do not output any other content. Test cases must be in plain text format. Generated test cases should not contain responses to the digital assistant. Generate only one test case at a time. If there are multiple rounds of interaction, generate the next test case based on the digital assistant's responses. Avoid repeating previous test cases; maintain the uniqueness and continuity of the dialogue.

[0061] Electronic device 110 can use a test case generation model to generate one or more sets of test cases. The prompts from the target digital assistant 121 and general question generation rules can be used as input to the test case generation model, which then generates the response content. Based on the response content from the test case generation model, one or more sets of test cases for the target digital assistant 121 can be obtained.

[0062] In addition, the input information for the test case generation model can include a role description of the model. For example, the role description could be: You are a digital assistant used to generate chat test cases. Your task is to generate test cases based on the prompts from the target digital assistant.

[0063] In this embodiment, the example implementation describes the generation of input information using a Chinese language environment. Alternatively and / or additionally, the corresponding scheme for generating input information can be executed in other language environments. For example, input information can be generated in Chinese, English, Japanese, French, and other environments. For instance, based on the multilingual capabilities provided by the test case generation model, input information for the test case generation model can be generated in application environments of different languages.

[0064] In this way, the electronic device 110 can systematically generate multiple sets of test cases, providing a comprehensive and effective tool for the evaluation and optimization of the digital assistant.

[0065] Another method for obtaining test cases is as follows: Electronic device 110 obtains the prompt word information of the target digital assistant 121, which includes at least the identification information and functional description of the target digital assistant 121. Based on at least one evaluation dimension related to chat skills, at least one specific question generation rule corresponding to each evaluation dimension is determined. Based on at least the prompt word information and at least one specific question generation rule, one or more sets of test cases are generated for the target digital assistant 121.

[0066] The prompts for the target digital assistant 121 are the same as in the previous example and will not be repeated here. Assessment dimensions related to chat skills may include identity recognition ability, functional recognition ability, basic interaction ability, interaction ability positively correlated with expertise, interaction ability negatively correlated with expertise, and the ability to handle abnormal interactions, etc.

[0067] Taking the assessment dimension corresponding to identity recognition ability as an example, the specific question generation rules may include: generating an identity awareness test case based on the prompts from the target digital assistant, guiding the target digital assistant to identify itself. Furthermore, you also need to mislead the target digital assistant about its identity. The interaction test consists of two rounds.

[0068] Taking the assessment dimension corresponding to functional cognitive ability as an example, the specific question generation rules may include: generating functional test cases based on the prompts from the target digital assistant; guiding the target digital assistant to state its function; and conducting one round of interaction testing.

[0069] Taking the evaluation dimensions corresponding to basic interaction capabilities as an example, specific question generation rules may include: generating chat test cases unrelated to the target digital assistant's areas of expertise based on the prompts provided by the target digital assistant. The interaction test consists of one round.

[0070] Taking the evaluation dimension corresponding to interactive capabilities positively related to the user's area of ​​expertise as an example, specific question generation rules may include: generating test cases positively related to the user's area of ​​expertise based on the prompts from the target digital assistant. Test cases positively related to the user's area of ​​expertise should cover all functions of the target digital assistant. Both real and false input information need to be considered. Test cases positively related to the user's area of ​​expertise should not merely ask the target digital assistant if it can do something, but rather have the target digital assistant actually do it. The generated test questions positively related to the user's area of ​​expertise should not be independent of each other. They should continue the previous conversation in depth based on the chat history. Only generate test questions, do not output any information. The interaction test consists of 5 rounds.

[0071] Taking the evaluation dimension corresponding to interaction capabilities negatively correlated with the target digital assistant's area of ​​expertise as an example, specific question generation rules may include: generating functional tests completely unrelated to the target digital assistant's area of ​​expertise based on the prompts provided. Test cases for capabilities negatively correlated with the target digital assistant's area of ​​expertise should not merely ask whether the target digital assistant can do something completely unrelated to its area of ​​expertise, but rather require the target digital assistant to actually do it. The interaction test consists of two rounds.

[0072] Taking the evaluation dimension corresponding to the ability to handle abnormal interactions as an example, specific question generation rules may include: generating test cases for abnormal interactions based on the prompts from the target digital assistant. For example, if the target digital assistant requests an image, the input should be content unrelated to the image, such as text or audio. The interaction test consists of one round.

[0073] Electronic device 110 can generate one or more sets of test cases based on a test case generation model. The prompts of the target digital assistant 121 and the specific question generation rules can be used as input information for the test case generation model. Based on the response content of the test case generation model, one or more sets of test cases for the target digital assistant 121 are generated.

[0074] Through the above process, test cases can be generated based on multiple evaluation dimensions, thereby comprehensively evaluating the various capabilities of the target digital assistant 121. Specifically, the evaluation content includes identity recognition ability, function recognition ability, basic interaction ability, positive function interaction ability, negative function interaction ability, and anomaly handling ability. This method can not only evaluate the actual chat skills of the target digital assistant 121, but also identify its performance in different situations, thus providing a reliable basis for recommending high-quality digital assistants.

[0075] For example, the first feature value includes chat skill scores corresponding to at least one evaluation dimension. Figure 3 A schematic diagram of a chat skills scoring interface 300 according to some embodiments of the present disclosure is shown. (In conjunction with...) Figure 3 As shown, for each assessment dimension's test question, the target digital assistant 121 can generate a corresponding response. A scoring model can be used to generate a chat skills score for each response. For example, Figure 3 The assessment of identity recognition ability includes two test questions. The chat skills score for the first test question is 0.58, and the chat skills score for the second test question is 0.56. Similarly, the chat skills scores for test questions in other assessment dimensions such as functional recognition ability, basic interaction ability, positive functional interaction ability, negative functional interaction ability, and exception handling ability will also be considered.

[0076] In some embodiments, the first feature value in the target evaluation metric may include an aggregated value of the chat skill scores for each evaluation dimension. For example, the first feature value can be the average of the chat skill scores for each evaluation dimension.

[0077] For scenarios involving multiple rounds of interaction within a specific evaluation dimension, the process of generating test questions will differ from the previous example. The following example uses an evaluation dimension corresponding to interaction capabilities positively correlated with a region of expertise, where the interaction test consists of 5 rounds. For multi-round interaction scenarios, in the first round of interaction with the target digital assistant 121, the electronic device 110 obtains the first response from the target digital assistant 121 to the first test question. Based at least on the first response, a second test question is generated for the second round of interaction with the target digital assistant 121.

[0078] Assessing the interactive capabilities of a digital assistant within its areas of expertise requires multiple rounds of interaction testing. This is to comprehensively test the digital assistant's ability to handle complex dialogues within its areas of expertise. Taking the first round of interaction as an example, electronic device 110 uses prompt words and corresponding specific question generation rules as input information and a test case generation model to generate the first test question. Based on the target digital assistant 121's first response to the first test question, electronic device 110 can use the test case generation model to generate a second test question associated with the first response. For example, if the first response mentions the "breach of contract clause," the second question could be, "Could you explain in detail the specific content of the breach of contract clause and its actual application in this case?"

[0079] Subsequent rounds of interaction can repeat the above process. Electronic device 110 will conduct the third, fourth, and fifth rounds of interaction sequentially. Each round's test questions will be generated based on at least the responses from the previous round, ensuring the coherence and depth of the test dialogue. For example, the test questions for the third round could delve into the legal consequences of breach of contract clauses. The test questions for the fourth round could explore how to protect one's rights in a contract. The test questions for the fifth round could inquire about the application of breach of contract clauses in real-world cases.

[0080] Through this multi-round interactive testing, electronic device 110 can comprehensively evaluate the digital assistant's ability to handle complex dialogues. Each round of test questions and responses is based on the content of the previous round, simulating real-world user interaction scenarios. This testing method not only examines the target digital assistant 121's knowledge depth and response accuracy but also assesses its ability to maintain logical consistency and provide valuable information in continuous dialogue. Ultimately, these test results will be used as important indicators to evaluate the digital assistant's capabilities, helping developers 105 identify and improve its interactive performance in specific domains.

[0081] In the foregoing embodiments, the target evaluation index reflects the chat skills of the target digital assistant 121 as an example. Here, in addition to indicating the chat skills of the target digital assistant, the target evaluation index can also be determined based on the configuration information of the target digital assistant 121. In some embodiments of this disclosure, the electronic device 110 determines at least one second feature value for the target digital assistant 121 in the target evaluation index based on the configuration information of the target digital assistant 121. The target digital assistant 121 generates and presents responses based on the configuration information, and each second feature value indicates a score for the target digital assistant 121 on a configuration type.

[0082] Digital assistants can be developed based on the digital assistant development platform 120. The digital assistant development platform 120 provides various tools, such as prompt words, plugins, workflows, knowledge bases, memory banks, and voice functions. Based on this, the configuration information can reflect the tools involved in the development process of the target digital assistant 121. Each tool can correspond to a configuration type.

[0083] For example, the second feature value may include a score of the target digital assistant 121 on the configuration type. For example, the score on the configuration type may indicate the number of voices supported by the digital assistant, the number of recommended dialogues, the number of workflows, the number of plugins, the number of knowledge bases, the number of publishing platforms, whether there is a background image, the number of memory banks, the number of bound cards, whether it is open source, and so on.

[0084] The number of supported voices indicates the number of voice options supported by the target digital assistant 121, such as the number of male, female, and child voices. Providing a diverse range of voice options can improve user satisfaction and engagement.

[0085] The number of recommended conversations indicates the number of conversations that the target digital assistant 121 can recommend to the user. In other words, it indicates how many conversation topics or themes (related to the user's interests, needs, and historical conversation records, etc.) the target digital assistant 121 can provide or recommend so that the user can choose and continue interacting.

[0086] The number of workflows indicates the number of workflows the target digital assistant 121 possesses. More workflows enable more complex tasks and automated operations.

[0087] The number of plugins indicates the number of plugins that the target digital assistant 121 can integrate. By integrating plugins, the target digital assistant 121 can expand its functionality and provide more services and applications.

[0088] The number of knowledge bases indicates the number of knowledge bases that the target digital assistant 121 can access and utilize. A rich knowledge base can improve the accuracy of the digital assistant's responses and the scope of information coverage.

[0089] The number of platforms on which a target digital assistant 121 can be released indicates how many platforms it can be released on. Multi-platform release capability can expand the user base of the digital assistant, increasing its awareness and usage.

[0090] The presence of a background image indicates whether the target digital assistant 121 has a background image function. Background images can enhance visual appeal and improve the user interface experience.

[0091] The number of memory banks indicates the number of memory banks associated with the target digital assistant 121. These memory banks can be used at least to record the user's historical conversations, providing personalized services and continuous conversational context.

[0092] Whether a digital assistant is open source indicates whether it supports open-source code projects. If it is an open-source project, the development and improvement of the digital assistant can be accelerated more easily.

[0093] Furthermore, the second feature value may also include an evaluation score for the prompts from the target digital assistant 121. For example, the prompts from the target digital assistant 121 can be input into a model with natural language evaluation capabilities, which can then provide a corresponding evaluation score.

[0094] When evaluating the target digital assistant 121, its scores across the different configuration types mentioned above can be used as evaluation metrics. This provides a comprehensive evaluation framework for the target digital assistant 121, helping users identify high-quality, reliable digital assistants to better meet their needs.

[0095] In addition to the configuration information of the target digital assistant 121, the target evaluation index can also be determined based on historical interaction information related to the target digital assistant 121. In some embodiments of this disclosure, the electronic device 110 determines at least one third feature value for the target digital assistant 121 in the target evaluation index based on the historical interaction information related to the target digital assistant 121, each third feature value indicating the score of the target digital assistant 121 on a user interaction type.

[0096] Historical interaction information can reflect the real-time performance of the target digital assistant 121 and its interaction with the user 135. For example, historical interaction information includes at least one of the following: the number of users who interacted with the target digital assistant 121 within a certain period; the number of messages exchanged with the target digital assistant 121 within a certain period; and the number of at least one type of interactive behavior performed on the target digital assistant 121.

[0097] By incorporating the dynamic characteristics corresponding to these historical interaction information into evaluation metrics, such as the number of active users and chat messages over a period of time, the user engagement and interaction volume of the digital assistant can be assessed. The number of favorites, likes, and dislikes can reveal the user's level of approval and satisfaction with the digital assistant. This helps to comprehensively understand the performance of the digital assistant and user feedback during actual use. Static characteristics based on configuration information reflect the design and configuration of the digital assistant, while dynamic characteristics corresponding to historical interaction information provide interaction data of the digital assistant in actual user use. Combining these two types of characteristics allows for the evaluation of the target digital assistant 121 from multiple dimensions.

[0098] To improve the automation of quality assessment results and ensure a consistent scale for standard assessment results, the quality assessment results of the target digital assistant 121 are determined based on target assessment metrics using a trained assessment model. The quality assessment results indicate the confidence level at which the target digital assistant is recommended, and the assessment model is trained as follows: First assessment metrics for recommended digital assistants are obtained as positive samples; second assessment metrics for unrecommended digital assistants are obtained as negative samples; and the assessment model is trained using both positive and negative samples.

[0099] Figure 4 A schematic diagram of a training process 400 for an evaluation model according to some embodiments of the present disclosure is shown. A first evaluation metric for a recommended digital assistant is obtained as a positive sample. A second evaluation metric for a non-recommended digital assistant is obtained as a negative sample.

[0100] In box 401, preprocessing is first performed on the evaluation metrics for both positive and negative samples. Since the first and second evaluation metrics each include feature values ​​corresponding to multiple feature types, and the ranges of these feature values ​​can vary significantly, for example, a feature type such as the number of active users or chat messages within a certain period might have hundreds, thousands, or even tens of thousands of feature values. Conversely, a feature type such as the number of knowledge base entries typically has only single-digit values, while chat skill scores are only between 0 and 1. Therefore, preprocessing is needed to transform the feature values ​​corresponding to all evaluation metrics to the same numerical range.

[0101] In box 402, a correlation calculation needs to be performed on the evaluation indicators. That is, if the first and second evaluation indicators each include feature values ​​corresponding to multiple feature types, then the correlation between the multiple feature types in the first and second evaluation indicators can be determined. Based on the correlation between the multiple feature types, at least one feature type to be included in the target evaluation indicator is selected from the multiple feature types.

[0102] Figure 5A schematic diagram of a relevance distribution 500 according to some embodiments of the present disclosure is shown. For each feature type in the first evaluation metric and the second evaluation metric, [the following is observed]. Figure 5 Let $\mathbf{n}$ represent feature types 1 through n (where $n$ is a positive integer). The correlation distribution among multiple feature types can be determined through calculation. Based on the correlation distribution, feature types are selected for inclusion or exclusion. For example, if the correlation between two feature types is high, it may lead to multicollinearity, affecting the stability and interpretability of the model. In this case, one of the feature types can be discarded. As another example, if feature type $i$ is determined to be a key feature, and the correlation between feature type 1 and feature type $i$ (i ≤ $n$, and $i$ is a positive integer) is 0.35, while the correlation between feature type 2 and feature type $i$ is 0.8, based on the importance of the feature types, feature type 1 can be discarded while feature type 2 is retained.

[0103] Based on the correlation between various feature types, at least one feature type that should be included in the target evaluation indicator can be selected from multiple feature types. The selected feature type can be one that has a significant impact on the evaluation effect.

[0104] In box 403, a certain number of samples are randomly selected from both the positive and negative samples to ensure a balance between the number of positive and negative samples used for model training. For example, 2000 recommended digital assistants can be selected as the first evaluation metric as positive samples, and 2000 unrecommended digital assistants can be randomly selected as the second evaluation metric from the 10000 available unrecommended digital assistants.

[0105] In box 404, the evaluation model is trained. During training, Logistic Regression can be defined first as the classification algorithm for the evaluation model. This step determines the basic structure of the evaluation model, which uses logistic regression to solve a binary classification problem (recommend or not recommend). Training the evaluation model using prepared training data indicates the importance of the target evaluation metrics. For example, the target digital assistant 121 includes 10 target evaluation metrics. Based on the importance of the target evaluation metrics determined by the evaluation model, the final score is calculated, indicating whether the target digital assistant 121 is worth recommending. For example, the final score can be a value between 0 and 1. For example, a threshold value such as 0.5 or 0.75 can be used. If the score is higher than this threshold, the evaluation model will conclude that it is worth recommending. This achieves the classification function.

[0106] Through the above process, the evaluation model can effectively learn from the evaluation metrics of both recommended and unrecommended digital assistants, accurately predicting the recommendation value of new digital assistants. This not only improves the efficiency and accuracy of the recommendation system but also provides users with better choices of digital assistants.

[0107] Using a trained evaluation model, a quality evaluation result for the target digital assistant 121 can be determined based on a target evaluation metric. This quality evaluation result indicates the confidence level at which the target digital assistant 121 is recommended. In some embodiments of this disclosure, the electronic device 110 displays the target digital assistant 121 on a recommendation interface in response to the quality evaluation result meeting the recommendation criteria. A recommendation performance metric for the target digital assistant 121 after being recommended is obtained. The evaluation model is updated based on the recommendation performance metric.

[0108] When the quality assessment results meet the preset recommendation criteria (e.g., the recommendation confidence level is higher than a certain threshold), the electronic device 110 will display the target digital assistant 121 on the recommendation interface of the digital assistant recommendation platform 130. In this way, the user 135 can easily discover and use the recommended digital assistant, thereby improving the overall user experience.

[0109] After the target digital assistant 121 is recommended, the electronic device 110 will continuously monitor its recommendation performance metrics. These metrics may include, but are not limited to, the following: user click-through rate, user retention rate, user satisfaction rating, and usage frequency, etc. User click-through rate indicates the ratio of the number of times a user clicks on the target digital assistant 121 on the recommendation interface to the number of times it is displayed. User retention rate indicates the proportion of users who continue to use the assistant after using it. User satisfaction rating indicates the user's rating or feedback on the target digital assistant 121. Usage frequency indicates how often a user uses the target digital assistant 121 over a period of time.

[0110] Based on the collected recommendation performance metrics, the electronic device 110 periodically updates the evaluation model. This process includes: collecting recommendation performance metric data of the target digital assistant 121 after it has been recommended; analyzing this data to evaluate the predictive accuracy and effectiveness of the current evaluation model; adjusting the parameters of the evaluation model based on the analysis results; specific steps may include adding new features, adjusting the weights of existing features, and optimizing the model's hyperparameters; retraining the evaluation model using the updated dataset to ensure it continues to perform well in the new data environment; and deploying the updated evaluation model into the system, replacing the old model so that the new model can be used in subsequent recommendation processes.

[0111] By continuously acquiring and analyzing recommendation performance metrics, electronic device 110 can continuously improve the evaluation model, thereby enhancing recommendation accuracy and user experience, and ultimately achieving a more intelligent and personalized digital assistant recommendation system.

[0112] Figure 6A schematic structural block diagram of a device 600 for digital assistant evaluation according to some embodiments of the present disclosure is shown. The device 600 may be implemented in or included in an electronic device 110, for example. The various modules / components in the device 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0113] As shown in the figure, the device 600 includes a test case acquisition module 601, configured to acquire at least one set of test cases for the target digital assistant in response to an evaluation request for the target digital assistant, each set of test cases including at least one test question related to the chat skills of the target digital assistant. A response acquisition module 602 is configured to provide at least one set of test cases to the target digital assistant to obtain responses from the target digital assistant to the at least one set of test cases. A target evaluation metric determination module 603 is configured to determine a target evaluation metric for the target digital assistant based at least on at least one set of test cases and the target digital assistant's responses to the at least one set of test cases, the target evaluation metric including at least a first feature value indicating the chat skills score of the target digital assistant. A quality evaluation result determination module 604 is configured to determine a quality evaluation result for the target digital assistant based on the target evaluation metric.

[0114] In some embodiments of this disclosure, the test case acquisition module 601 may be specifically configured to: acquire the prompt word information of the target digital assistant, wherein the prompt word information includes at least the identification information and function description of the target digital assistant;

[0115] Obtain the general question generation rules corresponding to each test case in at least one set of test cases. Based on at least the prompt word information and the general question generation rules, generate one or more sets of test cases for the target digital assistant.

[0116] In some embodiments of this disclosure, the test case acquisition module 601 may further be specifically configured to: acquire prompt word information of the target digital assistant, the prompt word information including at least the identification information and functional description of the target digital assistant; determine at least one specific question generation rule corresponding to each of the at least one evaluation dimension related to chat skills; and generate one or more sets of test cases for the target digital assistant based at least on the prompt word information and at least one specific question generation rule.

[0117] In some embodiments of this disclosure, the first feature value includes chat skill scores corresponding to at least one evaluation dimension.

[0118] In some embodiments of this disclosure, the test case acquisition module 601 may also be configured to: acquire a first response from the target digital assistant to a first test question during a first round of interaction with the target digital assistant; and generate a second test question for a second round of interaction with the target digital assistant, based at least on the first response.

[0119] In some embodiments of this disclosure, the target evaluation index determination module 603 may be configured to: determine at least one second feature value for the target digital assistant in the target evaluation index based on the configuration information of the target digital assistant, wherein the target digital assistant generates and presents a response based on the configuration information, and each second feature value indicates the score of the target digital assistant on a configuration type.

[0120] In some embodiments of this disclosure, the target evaluation index determination module 603 may also be configured to: determine at least one third feature value for the target digital assistant in the target evaluation index based on historical interaction information related to the target digital assistant, each third feature value indicating the score of the target digital assistant on a user interaction type.

[0121] In some embodiments of this disclosure, historical interaction information includes at least one of the following: the number of users who interacted with the target digital assistant within a period of time, the number of messages that were interacted with the target digital assistant within a period of time, and the number of at least one type of interaction behavior performed on the target digital assistant.

[0122] In some embodiments of this disclosure, the quality assessment results of the target digital assistant are determined based on target assessment metrics using a trained assessment model.

[0123] In some embodiments of this disclosure, a model training module is also included. The confidence level of a target digital assistant being recommended is indicated based on the quality assessment results. The training module is configured to: obtain a first evaluation metric for digital assistants that have been recommended, as positive samples; obtain a second evaluation metric for digital assistants that have not been recommended, as negative samples; and train an evaluation model using the positive and negative samples.

[0124] In some embodiments of this disclosure, the first evaluation index and the second evaluation index each include feature values ​​corresponding to multiple feature types. The model training module can also be configured to: determine the correlation between multiple feature types in the first evaluation index and the second evaluation index; and select at least one feature type to be included in the target evaluation index from the multiple feature types based on the correlation between the multiple feature types.

[0125] In some embodiments of this disclosure, based on the confidence level indicated by the quality assessment results for recommending the target digital assistant, the model training module may also be configured to: display the target digital assistant on the recommendation interface in response to the quality assessment results meeting the recommendation conditions; obtain the recommendation performance index of the target digital assistant after it has been recommended; and update the evaluation model based on the recommendation performance index.

[0126] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The illustrated electronic device 700 may include or be implemented as Figure 1 Electronic devices 110, or Figure 6 Device 600.

[0127] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0128] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0129] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0130] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0131] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0132] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0133] According to an exemplary implementation of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform... Figure 2 The methods provided in the various optional modes are already available, so they will not be elaborated upon here.

[0134] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0136] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0138] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for evaluating digital assistants, comprising: In response to an evaluation request for a target digital assistant, at least one set of test cases is obtained for the target digital assistant, each set of test cases including at least one test question related to the chat skills of the target digital assistant; The at least one set of test cases is provided to the target digital assistant to obtain the target digital assistant's response to the at least one set of test cases; Based at least on the at least one set of test cases and the target digital assistant's responses to the at least one set of test cases, a target evaluation metric is determined for the target digital assistant, the target evaluation metric including at least a first feature value indicating the chat skills score of the target digital assistant; as well as Based on the target evaluation indicators, the quality evaluation result of the target digital assistant is determined.

2. The method of claim 1, wherein obtaining at least one set of test cases for the target digital assistant comprises: Obtain the prompt word information of the target digital assistant, wherein the prompt word information includes at least the identification information and function description of the target digital assistant; Obtain the general question generation rules corresponding to each group of test cases in the at least one group of test cases; as well as Based at least on the prompt word information and the general question generation rules, generate one or more sets of test cases for the target digital assistant.

3. The method of claim 1, wherein obtaining at least one set of test cases for the target digital assistant comprises: Obtain the prompt word information of the target digital assistant, wherein the prompt word information includes at least the identification information and function description of the target digital assistant; Based on at least one evaluation dimension related to the chat skills, determine at least one specific question generation rule corresponding to each of the at least one evaluation dimension; as well as Based at least on the prompt word information and the at least one specific question generation rule, generate one or more sets of test cases for the target digital assistant.

4. The method according to claim 3, wherein the first feature value includes chat skill scores corresponding to each of the at least one evaluation dimension.

5. The method according to claim 1, further comprising, for a given group of test cases in the at least one group of test cases: In the first round of interaction with the target digital assistant, obtain the target digital assistant's first response to the first test question; as well as Based at least on the first response, a second test question is generated for the second round of interaction with the target digital assistant.

6. The method according to claim 1, wherein determining the target evaluation index further includes: Based on the configuration information of the target digital assistant, at least one second feature value for the target digital assistant is determined in the target evaluation index. The target digital assistant generates and presents the response based on the configuration information. Each second feature value indicates the score of the target digital assistant on a configuration type.

7. The method according to claim 1, wherein determining the target evaluation index further includes: Based on historical interaction information related to the target digital assistant, at least one third feature value for the target digital assistant is determined in the target evaluation index, and each third feature value indicates the score of the target digital assistant on a user interaction type.

8. The method of claim 7, wherein the historical interaction information includes at least one of the following: The number of users who interact with the target digital assistant within a certain period of time; The number of messages exchanged with the target digital assistant within a given period of time; and The number of at least one type of interactive behavior performed on the target digital assistant.

9. The method of claim 1, wherein the quality assessment result of the target digital assistant is determined based on the target assessment index using a trained assessment model.

10. The method of claim 9, wherein the quality assessment result indicates the confidence level at which the target digital assistant is recommended, and the assessment model is trained in the following manner: Obtain the first evaluation metric of the recommended digital assistants as positive samples; Obtain a second evaluation metric for unrecommended digital assistants as a negative sample; and The evaluation model is trained using the positive and negative samples.

11. The method according to claim 10, wherein the first evaluation index and the second evaluation index each include feature values ​​corresponding to multiple feature types, and the method further includes: Determine the correlation between the multiple feature types in the first evaluation index and the second evaluation index; as well as Based on the correlation between the multiple feature types, at least one feature type to be included in the target evaluation index is selected from the multiple feature types.

12. The method of claim 1, wherein the quality assessment result indicates the confidence level at which the target digital assistant is recommended, the method further comprising: In response to the quality assessment results meeting the recommendation criteria, the target digital assistant is displayed on the recommendation interface; Obtain the recommendation performance metrics of the target digital assistant after it has been recommended; as well as The evaluation model is updated based on the recommended performance metrics.

13. An apparatus for evaluating a digital assistant, comprising: The test case acquisition module is configured to acquire at least one set of test cases for the target digital assistant in response to an evaluation request for the target digital assistant, each set of test cases including at least one test question related to the chat skills of the target digital assistant; The response acquisition module is configured to provide the at least one set of test cases to the target digital assistant in order to obtain the response from the target digital assistant to the at least one set of test cases; The target evaluation metric determination module is configured to determine a target evaluation metric for the target digital assistant based at least on the at least one set of test cases and the target digital assistant's responses to the at least one set of test cases, wherein the target evaluation metric includes at least a first feature value indicating the chat skills score of the target digital assistant; as well as The quality assessment result determination module is configured to determine the quality assessment result of the target digital assistant based on the target assessment indicators.

14. An electronic device comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processing unit.

15. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 12.

Images