Dialogue evaluation method and device

Abstract

The dialog evaluation method and device provided by the embodiments of the present specification obtain an evaluation question corresponding to at least one round of dialog of a dialog model to be evaluated, call the dialog model to be evaluated based on the evaluation question, and generate an evaluation answer corresponding to the at least one round of dialog; based on the evaluation question and the evaluation answer, determine evaluation related information corresponding to the at least one round of dialog; and based on the evaluation related information, evaluate the at least one round of dialog of the dialog model to be evaluated through a preset evaluation index.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of artificial intelligence, and in particular, to a dialogue evaluation method and an electronic device. BACKGROUND

[0002] At present, the application of large models in the field of natural language processing has made a major breakthrough. With the wide deployment of large models in the commercial and research fields, the application of large models in dialogue scenarios is becoming more and more common. Therefore, how to evaluate the performance of large models in dialogue has become the focus of attention.

[0003] In the related technical solution, the dialogue of the large model is evaluated by artificial testing. However, in this technical solution, not only is the labor cost high, but also it is impossible to realize unified evaluation for different scenarios. Therefore, how to automatically and uniformly evaluate the dialogue of large models in different scenarios has become a technical problem to be solved.

[0004] The content of the background section only represents the inventor's own knowledge and does not mean that the above information has entered the public domain before the filing date of the present disclosure, nor does it mean that it can be prior art of the present disclosure. SUMMARY

[0005] The present specification provides a dialogue evaluation method and an electronic device, which can automatically and uniformly evaluate the dialogue of large models in different scenarios.

[0006] In a first aspect, the present specification provides a dialogue evaluation method, comprising:

[0007] obtaining an evaluation question corresponding to at least one round of dialogue of a dialogue model to be evaluated;

[0008] calling the dialogue model to be evaluated based on the evaluation question to generate an evaluation answer corresponding to the at least one round of dialogue;

[0009] determining evaluation-related information corresponding to the at least one round of dialogue based on the evaluation question and the evaluation answer; and

[0010] evaluating the at least one round of dialogue of the dialogue model to be evaluated by a preset evaluation index based on the evaluation-related information.

[0011] In some example embodiments, based on the above solution, the evaluation-related information includes user intent information, slot information, and domain information.

[0012] In some example embodiments, based on the above scheme, the preset evaluation indicators include first type evaluation indicators and / or second type evaluation indicators, and the evaluation of the at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation related information by the preset evaluation indicators includes:

[0013] evaluation of the at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation related information and the first type evaluation indicators by calling a dialogue evaluation large model; and / or

[0014] evaluation of the at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation related information and the second type evaluation indicators by an evaluation script.

[0015] In some example embodiments, based on the above scheme, the evaluation of the at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation related information and the first type evaluation indicators by calling a dialogue evaluation large model includes:

[0016] generating an evaluation prompt word corresponding to the dialogue evaluation large model based on the evaluation related information and the first type evaluation indicators;

[0017] evaluating the at least one round of dialogue of the to-be-evaluated dialogue model by the dialogue evaluation large model based on the evaluation prompt word.

[0018] In some example embodiments, based on the above scheme, the second type evaluation indicators include response time, the round of dialogue includes a plurality of dialogue stages, and the evaluation of the at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation related information and the second type evaluation indicators by an evaluation script includes:

[0019] determining loading time of each of the dialogue stages of the to-be-evaluated dialogue model based on the evaluation related information and the second type evaluation indicators by an evaluation script;

[0020] evaluating the response time of the to-be-evaluated dialogue model based on the loading time of each of the dialogue stages.

[0021] In some example embodiments, based on the above scheme, the plurality of dialogue stages include a page rendering stage, a data request stage, and a content display stage.

[0022] In some example embodiments, based on the above scheme, the evaluation related information includes pre-defined standard slot information and slot information of the evaluation answer, and the evaluation of the at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation related information and the second type evaluation indicators by an evaluation script includes:

[0023] invoke the evaluation script based on the evaluation-related information and the second type of evaluation indicators, and compare the slot information of the evaluation answer with the standard slot information;

[0024] evaluate the at least one round of dialogue of the to-be-evaluated dialogue model based on the comparison result.

[0025] In some example embodiments, based on the above scheme, the evaluation question corresponding to the at least one round of dialogue of the to-be-evaluated dialogue model is obtained, comprising:

[0026] obtaining an evaluation question of at least one round of dialogue defined offline in advance for the to-be-evaluated dialogue model; or

[0027] generating an evaluation question of at least one round of dialogue corresponding to the to-be-evaluated dialogue model online based on the current dialogue content and context of the to-be-evaluated dialogue model.

[0028] In some example embodiments, based on the above scheme, before the evaluation question of at least one round of dialogue defined offline in advance for the to-be-evaluated dialogue model is obtained, the method further comprises:

[0029] generating a question generation prompt word corresponding to a pre-trained question generation large model based on question-related information;

[0030] invoke the question generation large model to generate the evaluation question of at least one round of dialogue defined offline in advance based on the question generation prompt word.

[0031] In some example embodiments, based on the above scheme, the context includes user behavior information in the at least one round of dialogue, and the evaluation question of at least one round of dialogue corresponding to the to-be-evaluated dialogue model is generated online based on the current dialogue content and context of the to-be-evaluated dialogue model, comprising:

[0032] generate an evaluation question of at least one round of dialogue corresponding to the to-be-evaluated dialogue model online based on the current dialogue content and the user behavior information of the to-be-evaluated dialogue model.

[0033] In some example embodiments, based on the above scheme, the evaluation-related information further includes question labels of the evaluation question and answer labels of the evaluation answer, and the evaluation-related information corresponding to the at least one round of dialogue is determined based on the evaluation question and the evaluation answer, comprising:

[0034] determine the question labels of the evaluation question of the at least one round of dialogue based on the evaluation question through a question labeling model;

[0035] Based on the evaluation question and the evaluation answer, the evaluation related information corresponding to the at least one round of dialogue is determined.

[0036] In some example embodiments, based on the above scheme, the question label includes one or more of a question level, a question novelty degree, and a question multi-round number, and the answer label includes one or more of an answer level, an answer novelty degree, and an answer matching degree.

[0037] In some example embodiments, based on the above scheme, the method further includes:

[0038] Obtaining an index result of each of the preset evaluation indexes of the dialogue model to be evaluated, and performing attribution analysis on the index result of each of the preset evaluation indexes.

[0039] In some example embodiments, based on the above scheme, the method further includes:

[0040] Based on the index result and the corresponding attribution analysis result, an evaluation report for the dialogue model to be evaluated is generated.

[0041] In a second aspect, the present specification also provides an electronic device, including: at least one storage medium, which stores at least one instruction set for data processing; and at least one processor, which is communicatively connected with the at least one storage medium, wherein when the electronic device is running, the at least one processor reads the at least one instruction set, and executes the dialogue evaluation method according to the indication of the at least one instruction set.

[0042] From the above technical solutions, the dialogue evaluation method and device provided by the embodiments of the present specification can automatically simulate the user dialogue process, for example, the process of single-round dialogue or multi-round dialogue, on the one hand, by calling the dialogue model to be evaluated based on the evaluation question to generate the evaluation answer corresponding to the at least one round of dialogue; on the other hand, by determining the evaluation related information corresponding to the at least one round of dialogue based on the evaluation question and the evaluation answer; and on the other hand, by evaluating the at least one round of dialogue of the dialogue model to be evaluated based on the evaluation related information through the preset evaluation index, so as to accurately and automatically evaluate the dialogue of the large model in different scenarios.

[0043] Other functions of the dialogue evaluation method and device provided by the present specification will be partially listed in the following description. According to the description, the following numbers and examples will be obvious to those skilled in the art. The creative aspects of the dialogue evaluation method and device provided by the present specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic diagram of the implementation environment of a dialogue evaluation method provided in an embodiment of this specification is shown;

[0046] Figure 2 A hardware structure diagram of an electronic device 200 provided according to an embodiment of this specification is shown;

[0047] Figure 3 A flowchart of a dialogue evaluation method provided according to some embodiments of this specification is shown;

[0048] Figure 4 The diagram illustrates evaluation questions and responses in a multi-turn dialogue provided according to some embodiments of this specification;

[0049] Figure 5 A flowchart illustrating the problem generation process provided according to some embodiments of this specification is shown;

[0050] Figure 6 A flowchart illustrating the problem annotations provided according to some embodiments of this specification is shown;

[0051] Figure 7 A schematic diagram of the model acquisition process provided according to some embodiments of this specification is shown;

[0052] Figure 8 A flowchart illustrating the answer annotations provided according to some embodiments of this specification is shown;

[0053] Figure 9 A flowchart illustrating an automated evaluation process provided according to some embodiments of this specification is shown;

[0054] Figure 10 A flowchart illustrating a dialogue evaluation method according to other embodiments of this specification is shown; and,

[0055] Figure 11 A schematic diagram of an automated evaluation process provided according to some embodiments of this specification is shown. Detailed Implementation

[0056] The following description provides specific applications and requirements of the present specification, which are intended to enable a person skilled in the art to manufacture and use the contents of the present specification. Various partial modifications of the disclosed embodiments are obvious to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the present specification. Therefore, the present specification is not limited to the embodiments shown, but is consistent with the widest scope of the claims.

[0057] The terms used herein are used only for the purpose of describing specific example embodiments, and are not limiting. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an", and "the" can also include the plural forms. When used in the present specification, the terms "comprise", "include" and / or "contain" mean that the associated whole, step, operation, element and / or component exists, but do not exclude the presence of one or more other features, whole, step, operation, element, component and / or group.

[0058] These features of the present specification and other features, as well as the operation and function of related elements of the structure, and the economy of combination and manufacture of components can be significantly improved in view of the following description. Referring to the drawings, all of which form part of the present specification. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of the present specification. It should also be understood that the drawings are not drawn to scale.

[0059] The flowchart used in the present specification shows the operation of the system implementation according to some embodiments of the present specification. It should be clearly understood that the operations of the flowchart can not be implemented in sequence. On the contrary, the operations can be implemented in reverse order or simultaneously. In addition, one or more other operations can be added to the flowchart. One or more operations can be removed from the flowchart.

[0060] First, the nomenclature related to one or more embodiments of the present specification is explained.

[0061] Large model: (Large Models) generally refers to large neural network models used in artificial intelligence (AI) and machine learning (ML). Large models contain a large number of parameters, which can reach millions to hundreds of millions or even more, making them capable of processing complex tasks and data. Large models can be applied to different fields, including natural language processing, image recognition, sound recognition, etc.

[0062] Intent: Intent represents the action or purpose the user wants to achieve when starting a conversation. For example, the user says "I want to listen to a jazz song", where the intent is "play music".

[0063] Slot: A slot is an entity or key information in a conversation that helps fulfill a specific intent. Using the example of "I want to listen to jazz music," "jazz music" is a slot that specifies the type of music to be played.

[0064] Multi-intent slot: Multi-intent slot cases represent more than one intent in a conversation, and each intent may contain one or more related slots. For example, a user says "I want to listen to jazz music and book a movie ticket for tomorrow night," which contains two intents: "play music" and "book a movie ticket."

[0065] In the related technical solution, the dialogue of the large model is evaluated by manually writing an automated script. In this technical solution, the efficiency of manually writing a script is low, and unified evaluation for different scenarios cannot be achieved.

[0066] Based on the above, the embodiments of the present specification provide a dialogue evaluation method and an electronic device. On the one hand, based on the evaluation question, the dialogue model to be evaluated is called to generate the evaluation answer corresponding to at least one round of dialogue, which can automatically simulate the user dialogue process, such as the process of single-turn dialogue or multi-turn dialogue. On the other hand, based on the evaluation question and the evaluation answer, the evaluation related information corresponding to at least one round of dialogue is determined, which can automatically obtain the evaluation related information such as intent information, slot information, and domain information in the evaluation question and the evaluation answer. On the other hand, based on the evaluation related information, at least one round of dialogue of the dialogue model to be evaluated is evaluated by the preset evaluation index, which can automatically and accurately evaluate the dialogue of the large model in different scenarios.

[0067] In the following, the technical solutions of the embodiments of the present specification will be described in detail with reference to the accompanying drawings.

[0068] Figure 1 A schematic diagram of an implementation environment of a dialogue evaluation method provided by an embodiment of the present specification is shown.

[0069] Referring to Figure 1 As shown, the implementation environment 100 can include a terminal 110, a server 130, and a database 140.

[0070] The terminal 110 is connected to the server 130 through a wireless network or a wired network 120. The terminal 110 can be a tablet computer, a notebook computer, or a desktop computer, but is not limited thereto.

[0071] The terminal 110 can store data or instructions for executing the dialogue evaluation method described in the present specification. The terminal 110 can include a hardware device with data information processing capability and the necessary programs required to drive the hardware device to work.

[0072] The server 130 is a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The server 130 provides background services for the application programs running on the terminal 110.

[0073] The server 130 is loaded with an integrated development platform. The integrated development platform, also known as an integrated development environment (IDE), is an application program used to provide a program development environment, generally including code editors, compilers, debuggers, and graphical user interfaces, and the like. Developers can write program code (i.e., program development) on the integrated development platform. The integrated development platform server can be a computing device that the integrated development platform is specifically used to implement the dialogue evaluation method. The server 130 can respectively communicate data with the terminal 110 and the database 140.

[0074] In addition, the server 130 can store data or instructions for executing the dialogue evaluation method described in the specification. The server 130 can include a hardware device with data information processing capability and the necessary programs to drive the hardware device to work. Of course, the server 130 can also be only a hardware device with data processing capability, or only a program running in a hardware device. In some embodiments, the server 130 can also be plugged in and deployed in the terminal 110, at which time the server 130 stores data or instructions for executing the dialogue evaluation method corresponding to the terminal 110 described in the specification.

[0075] The database 140 can store data and / or instructions. In some embodiments, the database 140 can store data such as test questions and test answers. In some embodiments, the database 140 can store data and / or instructions for the server 130 to execute or use to execute the dialogue test method described in this specification. The terminal 110 and the server 130 have access to the database 140, and the terminal 110 and the server 130 can access data or instructions stored in the database 140 through the network. In some embodiments, the database 140 can be directly connected to the terminal 110 and the server 130. In some embodiments, the database 140 can be part of the server 130. In some embodiments, the database 140 can include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), or the like, or any combination thereof. An example of mass storage can include a non-transitory storage medium such as a magnetic disk, an optical disk, a solid-state drive, etc. An example of removable storage can include a flash drive, a floppy disk, an optical disk, a memory card, a zip disk, a magnetic tape, etc. A typical volatile read-write memory can include a random access memory (RAM). An example of RAM can include a dynamic RAM (DRAM), a double date rate synchronous dynamic RAM (DDR SDRAM), a static RAM (SRAM), a thyristor RAM (T-RAM), and a zero-capacitor RAM (Z-RAM), etc. An example of ROM can include a mask ROM (MROM), a programmable ROM (PROM), a virtually programmable ROM (PEROM), an electronically programmable ROM (EEPROM), a compact disc (CD ROM), and a digital versatile disk ROM, etc.

[0076] A person skilled in the art can know that the number of terminals described above can be more or less. For example, the terminal described above is only one, or the terminal described above is dozens or hundreds, or more, and other terminals are included in the implementation environment described above. The number of terminals and the type of equipment are not limited in the embodiments of the present specification.

[0077] After introducing the implementation environment of the embodiments of the present specification, the application scenarios of the embodiments of the present specification will be introduced in combination with the implementation environment described above. In the following description, the terminal is the terminal 110 in the implementation environment described above, and the server is the server 130 in the implementation environment described above. The technical solutions provided by the embodiments of the present specification can be applied in various large model dialogue test scenarios, such as tourism platform dialogue large model, e-commerce platform dialogue large model, and other large model test scenarios.

[0078] With the technical solution provided by the embodiments of the present specification applied in the dialogue evaluation scene of the tourism platform large model as an example, the terminal 110 obtains an evaluation question corresponding to at least one round of dialogue of the tourism platform large model; calls a dialogue model to be evaluated based on the evaluation question, generates an evaluation answer corresponding to at least one round of dialogue; based on the evaluation question and the evaluation answer, determine the evaluation related information corresponding to at least one round of dialogue; and based on the evaluation related information, evaluate the at least one round of dialogue of the dialogue model to be evaluated through the preset evaluation index.

[0079] It should be noted that the above is an example of the technical solution provided by the embodiments of the present specification applied in the dialogue evaluation scene of the tourism platform large model. The technical solution provided by the embodiments of the present specification can also be applied in other appropriate dialogue evaluation scenes, such as the dialogue evaluation scene of the medical platform large model or the dialogue evaluation scene of the shopping platform large model, etc. The implementation process is the same as the above description, and will not be repeated here.

[0080] It should be noted that the steps in the dialogue evaluation method in the example embodiments of the present specification can be partially executed by the client, partially executed by the server, or completely executed by the server or the client. The present specification does not make special limitations on this.

[0081] Based on the implementation environment shown in the figure, the dialogue evaluation method and the electronic device provided by the embodiments of the present specification will be described in detail below. It should be noted that the above implementation environment is only shown to facilitate understanding of the spirit and principles of the present specification, and the embodiments of the present specification are not limited in this respect. On the contrary, the embodiments of the present specification can be applied to any applicable scene. Figure 1 Figure 2-10 The dialogue evaluation method and the electronic device provided by the embodiments of the present specification will be described in detail below. It should be noted that the above implementation environment is only shown to facilitate understanding of the spirit and principles of the present specification, and the embodiments of the present specification are not limited in this respect. On the contrary, the embodiments of the present specification can be applied to any applicable scene.

[0082] Figure 2 FIG. 2 is a structural schematic diagram of an electronic device 200 according to some embodiments of the present specification. The above electronic device 200 can execute the dialogue evaluation method described in the present specification. The above dialogue evaluation method is introduced in other parts of the present specification. The above electronic device 200 can be a general-purpose computer or a special-purpose computer. For example, the above electronic device 200 can be a server, a personal computer, a portable computer (such as a notebook computer, a tablet computer, etc.), or other electronic devices with computing capabilities. Of course, the above electronic device can be a terminal 110 or a server 130 in the figure, or a terminal device used by multiple developers to develop programs on an integrated development platform. Figure 1 The terminal 110 or the server 130 in the figure, or a terminal device used by multiple developers to develop programs on an integrated development platform.

[0083] ​The electronic device in the specification can include one or more of the following components: a processor 210, a memory 220, an input device 230, an output device 240, and a bus 250. The processor 210, the memory 220, the input device 230, and the output device 240 can be connected through the bus 250.

[0084] The processor 210 can include one or more processing cores. The processor 210 connects various parts within the entire electronic device by various interfaces and lines, executes the conversation evaluation method described in the specification by running or executing instructions, programs, code sets, or instruction sets stored in the memory 220, and calling data stored in the memory 220. Alternatively, the processor 210 can be implemented in at least one of hardware forms of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 210 can integrate a combination of one or several of central processing units (CPUs), graphics processing units (GPUs), and modems. Among them, the CPU is mainly used to process operating systems, user interfaces, and application programs; the GPU is used to render and draw display content; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 210, but can be realized by a separate communication chip.

[0085] The memory 220 can include random access memory (RAM) and can also include read-only memory (ROM). Alternatively, the memory 220 includes a non-transitory computer-readable storage medium. The memory 220 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 220 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as touch function, sound playing function, image playing function, etc.), instructions for implementing each method embodiment described below, etc. The operating system can be an Android system, including a system developed based on the Android system in depth, an IOS system, including a system developed based on the IOS system in depth, or other systems.

[0086] In order to enable the operating system to distinguish the specific application scenarios of the third-party application, data communication between the third-party application and the operating system needs to be enabled, so that the operating system can obtain the current scenario information of the third-party application at any time, and then perform targeted system resource adaptation based on the current scenario.

[0087] The input device 230 is configured to receive input instructions or data, and the input device 230 includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 240 is configured to output instructions or data, and the output device 240 includes but is not limited to a display device and a speaker. In an example, the input device 230 and the output device 240 can be combined, and the input device 230 and the output device 240 are a touch display screen.

[0088] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above-described drawings does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than those shown in the drawings, or combine certain components, or different component arrangements. For example, the electronic device further includes radio frequency circuitry, an input unit, a sensor, audio circuitry, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, and the like, which are not described herein.

[0089] Figure 3 A flowchart of a dialogue evaluation method according to an embodiment of the present specification is shown. As before, the electronic device 200 can perform the dialogue evaluation method of the embodiments of the present specification. Specifically, the processor 210 can read a set of instructions stored in its local storage medium, and then perform the dialogue evaluation method of the embodiments of the present specification according to the provisions of the set of instructions. In the following, steps S310 to S340 in the dialogue evaluation method will be described in detail in conjunction with the drawings.

[0090] As shown in Figure 3 In step S310, an evaluation question for at least one round of dialogue corresponding to a dialogue model to be evaluated is obtained.

[0091] In an example embodiment, the dialogue model to be evaluated can be a GPT large model in a predetermined field, such as a medical consultation large model, an e-commerce platform large model, or a tourism platform large model, etc. The electronic device 200 generates an evaluation question for at least one round of dialogue of the dialogue model to be evaluated offline or online for different dialogue scenarios in the predetermined field, and obtains an evaluation question for at least one round of dialogue corresponding to the dialogue model to be evaluated. Wherein, the at least one round of dialogue can be a single round of dialogue or multiple rounds of dialogue.

[0092] In some example embodiments, the electronic device 200 obtains an evaluation question for a pre-offline defined at least one round of dialogue of the dialogue model to be evaluated. For example, the electronic device 200 generates a question generation prompt word corresponding to the pre-training question generation large model based on question related information; and calls the question generation large model to generate the evaluation question for the pre-offline defined at least one round of dialogue based on the question generation prompt word. The question related information includes user intent information, slot information, and domain information, and the like.

[0093] In other example embodiments, the electronic device 200 generates an evaluation question for at least one round of dialogue corresponding to the dialogue model to be evaluated based on the current dialogue content and context of the dialogue model to be evaluated. The evaluation question can include a question identifier, a question content, an intent description, and a key slot. In addition, the evaluation question can also include a model response corresponding to a hypothesis or guidance of the question. Tables 1 and 2 below show the evaluation question for a single round of dialogue and the evaluation question for a multi-round dialogue.

[0094] Table 1. Evaluation question for a single round of dialogue

[0095]

[0096] Table 2. Evaluation question for a multi-round dialogue

[0097]

[0098] In step S320, the dialogue model to be evaluated is called based on the evaluation question to generate an evaluation answer corresponding to the at least one round of dialogue.

[0099] In example embodiments, the dialogue model to be evaluated can be a medical consultation large model, an e-commerce platform large model, or a tourism platform large model, and the like. The electronic device 200 calls the dialogue model to be evaluated based on the evaluation question for the at least one round of dialogue to generate an evaluation answer corresponding to the evaluation question for the at least one round of dialogue. The dialogue data is formed by collecting the evaluation question and the corresponding evaluation answer, and the dialogue data is used to test and analyze the dialogue model to be evaluated to evaluate the performance and ability of the model in processing single round and multi-round dialogues. Figure 4 An evaluation question and an evaluation answer in a multi-round dialogue in example embodiments are shown. Referring to Figure 4 As shown, the evaluation question 1 is: How do I get from Shanghai? The evaluation answer 1 is: Where do you want to go? The evaluation question 2 is: I want to go to Hangzhou. The evaluation answer 2 is: Based on your needs, the following results have been queried: Recommended route: (1) Train: 1 hour 24 minutes; D935; Shanghai Hongqiao 20:05, 53 yuan; (2) Taxi: 2 hours 11 minutes; 162.1 kilometers, about 595 yuan, all network call, expected 2 minutes to pick up.

[0100] Further, for single-turn dialogue collection, the collection module of the electronic device 200 simulates a simple question and answer scenario to collect the answers of the dialogue model to be evaluated to specific questions. For multi-turn dialogue collection, there are mainly two implementation manners:

[0101] Embodiment one:

[0102] A series of related evaluation questions for testing the dialogue model to be evaluated are defined in advance, which are designed to guide the dialogue to develop in the expected direction. For example, a preset question set is generated offline in advance by calling a large model such as a GPT model for a predetermined scene question prompt word. When collecting dialogue data, the electronic device 200 provides the preset question set to the dialogue model to be evaluated in sequence, collects the responses of the dialogue model to be evaluated, i.e., evaluation answers, and continues the next round of dialogue on this basis.

[0103] Embodiment two:

[0104] The electronic device 200 generates evaluation questions for at least one round of dialogue corresponding to the dialogue model to be evaluated based on the current dialogue content and context of the dialogue model to be evaluated. By calling a large model such as a GPT model to generate questions online, it has higher flexibility and timeliness. For example, in each round of dialogue, the collection module of the electronic device 200 proposes new questions online by calling a large model according to the content and context of the current dialogue. According to the above example embodiment, the collection module of the electronic device 200 can generate more realistic and natural multi-turn dialogues online, thereby better simulating the actual dialogue behavior mode of human users.

[0105] In step S330, based on the evaluation questions and the evaluation answers, the evaluation related information corresponding to the at least one round of dialogue is determined.

[0106] In an example embodiment, the evaluation related information includes user intent information, slot information, and domain information. The user intent is used to clearly define the goal that the user tries to achieve in the dialogue. For example, in the scenario of setting a travel plan, the user intent may involve querying a bus route, booking a ticket, or obtaining weather information; the slot information refers to the information points that need to be filled in the dialogue, such as the departure place, the destination, the travel time, etc.; the domain information is used to explicitly indicate the domain focused on by the question generation, which is particularly important when constructing domain-specific dialogue questions.

[0107] Further, the electronic device 200 determines the evaluation related information corresponding to the at least one round of dialogue based on the evaluation questions and the evaluation answers of the at least one round of dialogue of the dialogue model to be evaluated. For example, the electronic device 200 extracts the user intent information, the slot information, and the domain information corresponding to the evaluation questions and the evaluation answers from the evaluation questions and the evaluation answers of the at least one round of dialogue of the dialogue model to be evaluated.

[0108] It should be noted that although the above evaluation-related information is taken as an example for description, those skilled in the art should understand that the evaluation-related information can also include other appropriate information such as question tags of evaluation questions or answer tags of evaluation answers, and the like, which are also within the scope of the embodiments of the present specification.

[0109] For example, the evaluation-related information can also include question tags of evaluation questions or answer tags of evaluation answers. The question tags include one or more of question level, question novelty, and question multi-turn number, and the answer tags include one or more of answer level, answer novelty, and answer matching degree. The electronic device 200 determines, by a question labeling model, question tags of the evaluation questions of the at least one round of dialogue based on the evaluation questions of the at least one round of dialogue, and determines, by an answer labeling model, answer tags of the evaluation answers of the at least one round of dialogue based on the evaluation answers. The question labeling model can be a pre-trained neural network model for classifying evaluation questions, and the answer labeling model can be a pre-trained neural network model for classifying evaluation answers.

[0110] It should be noted that although the question tags are taken as an example for description, those skilled in the art should understand that the question tags can also be other appropriate tags such as multi-turn questions, multi-slot missing questions, or intent explicit questions, and the like, which are also within the scope of the embodiments of the present specification.

[0111] In step S340, at least one round of dialogue of the to-be-evaluated dialogue model is evaluated by a preset evaluation index based on the evaluation-related information.

[0112] In an example embodiment, the preset evaluation index includes a first type evaluation index and / or a second type evaluation index, the first type evaluation index is a non-objective evaluation index for evaluating the dialogue capability of the large model, and the second type evaluation index is an objective evaluation index for evaluating the dialogue capability of the large model. The first type evaluation index, i.e., the non-objective evaluation index, includes understanding accuracy, conversation coherence, natural language generation quality, and the like; and the second type evaluation index, i.e., the objective evaluation index, includes response time, user satisfaction feedback, slot missing feedback, and the like. The electronic device 200 evaluates, by calling a dialogue evaluation large model, at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation-related information and the first type evaluation index; and / or evaluates, by a evaluation script, at least one round of dialogue of the to-be-evaluated dialogue model based on the evaluation-related information and the second type evaluation index. Table 3 below shows the preset evaluation index for at least one round of dialogue in an example embodiment:

[0113] Table 3. Preset evaluation index of dialogue

[0114] Evaluation Metrics Description Understanding Accuracy How well the system understands the user's needs Conversation Coherence How the conversation remains coherent across multiple turns Information Collection Efficiency How efficiently the system collects information User Satisfaction How satisfied the user is with the answers Response Time How quickly the system answers Problem Solving Rate How well the system solves problems Error Recovery Ability How well the system handles misunderstandings or incomplete information Natural Language Generation Quality How natural and accurate the generated answers are System Adaptability How well the system adapts to different conversational contexts Data Utility How well the system learns and optimizes from conversational data

[0115] In some example embodiments, the electronic device 200 performs evaluation on at least one round of dialogue of the dialogue model to be evaluated based on the evaluation related information and the first type evaluation index by invoking the dialogue evaluation large model. For example, the electronic device 200 generates an evaluation prompt word corresponding to the dialogue evaluation large model based on the evaluation related information and the first type evaluation index, and performs evaluation on at least one round of dialogue of the dialogue model to be evaluated by the dialogue evaluation large model based on the evaluation prompt word.

[0116] In other example embodiments, the electronic device 200 performs evaluation on at least one round of dialogue of the dialogue model to be evaluated based on the evaluation related information and the second type evaluation index by the evaluation script. For example, a correspondence relationship between each evaluation index in the second type evaluation index and the evaluation script is pre-set, and the electronic device 200 performs evaluation on the target second type evaluation index of the dialogue model to be evaluated based on the evaluation related information and the target second type evaluation index based on the above correspondence relationship by invoking the evaluation script corresponding to the target second type evaluation index.

[0117] According to the technical solutions in the example embodiments, Figure 3 According to the technical solutions in the example embodiments,

[0118] Further, in the example embodiments, the second type evaluation index includes a response time, one round of dialogue includes a plurality of dialogue stages, and the electronic device 200 determines the loading time of each dialogue stage of the dialogue model to be evaluated based on the evaluation related information and the second type evaluation index by the evaluation script; and evaluates the response time of the dialogue model to be evaluated based on the loading time of each dialogue stage. Wherein, the plurality of dialogue stages include a page rendering stage, a data request stage and a content display stage.

[0119] According to the technical solutions in the example embodiments, by measuring the loading time of each stage of the dialogue, such as the time of page rendering, data request and content display, it can be identified which links may cause poor user experience, so as to optimize the code and improve the response speed and performance of the dialogue model.

[0120] Based on the above scheme, in an example embodiment, the evaluation-related information includes pre-defined standard slot information and slot information of the evaluation answer, and the electronic device 200 invokes the evaluation script based on the evaluation-related information and the second type evaluation index, compares the slot information of the evaluation answer with the standard slot information, and evaluates at least one round of dialogue of the to-be-evaluated dialogue model based on the comparison result.

[0121] According to the technical solutions in the above example embodiments, on the one hand, the comparison of the slot information of the evaluation answer with the standard slot information can accurately analyze the multi-slot information of the multi-round dialogue, not only recognizing the existence of multi-slot, but also evaluating the feedback of the model when part of the slot is missing; on the other hand, the feedback of the model when each slot missing function in the evaluation answer can be detected and compared with the pre-defined standard, which can ensure the good performance of the dialogue model intelligence when the slot is missing.

[0122] In addition, in an example embodiment, the electronic device 200 generates the evaluation question of at least one round of dialogue corresponding to the to-be-evaluated dialogue model based on the current dialogue content and the context of the to-be-evaluated dialogue model. For example, the context includes user behavior information in at least one round of dialogue, and the electronic device 200 generates the evaluation question of at least one round of dialogue corresponding to the to-be-evaluated dialogue model based on the current dialogue content and the user behavior information of the to-be-evaluated dialogue model. The user behavior information can include input operation information and click card information and other behavior information.

[0123] Based on the technical solutions in the above example embodiments, by using advanced NLP technology to analyze the typical actions of users in multi-round dialogue, such as input and click card, the behavior of real users can be better simulated, and the core function evaluation of the dialogue model can meet the service needs and user expectations.

[0124] Figure 5 A flowchart of question generation according to some embodiments of the present specification is shown.

[0125] Referring to Figure 5 As shown in step S510, a question generation task is created.

[0126] In an example embodiment, the core goal of the question generation task is to design a mechanism that can automatically generate evaluation questions related to a specific domain and user intent. These evaluation questions aim to evaluate and improve the ability of large models to handle single-round and multi-round dialogues. The electronic device 200 creates a question generation task for a predetermined scenario.

[0127] In step S520, prompt words are assembled.

[0128] In an example embodiment, the electronic device 200 assembles prompt words for generating evaluation questions for a predetermined scenario corresponding to evaluation-related information, which includes user intent information, slot information, and domain information. In the process of generating evaluation questions, assembling appropriate prompt words is a crucial step. The step of assembling prompt words involves several key elements: (1) user intent: used to clearly define the goal that the user is trying to achieve in the dialogue. For example, in the scenario of setting up a travel plan, the user intent may involve querying public transportation routes, booking tickets, or obtaining weather information; (2) slot information: refers to the information points that need to be filled in the dialogue, such as departure location, destination, travel time, etc. (3) domain information: clearly defines the field that the question generation focuses on, which is particularly important when constructing domain-specific dialogue questions.

[0129] Further, the electronic device 200 constructs a series of prompts based on user intent information, slot information, and domain information to guide the question generation large model to generate relevant questions. For example, the prompt constructed for the travel plan can be: "Generate a single-turn dialogue question about how the user can get from location A to location B, including querying different transportation options." Among them, the user intent is to query transportation options; the slot information includes the departure location, destination, etc.; and the domain information is transportation.

[0130] In step S530, the large model is called.

[0131] In an example embodiment, the electronic device 200 calls the question generation model based on the assembled prompt words. The question generation model can be a GPT model or other appropriate large model. For example, the prompt can be "Assume you are a large model testing expert, and now you need to test a multi-turn dialogue large model, where the test question is: generate a travel plan from Shanghai to Hangzhou, where slot 1: departure location; slot 2: destination; slot 3: user intent; require the model to answer in the case of multiple slots, help me generate 10 questions."

[0132] The question generation model can automatically generate a challenging question set to test and improve the performance of the dialogue model to be evaluated in real dialogue scenarios. Through the carefully designed evaluation question set for the predetermined scenario, the performance of the dialogue model to be evaluated in various scenarios can be comprehensively evaluated.

[0133] In step S540, the test question set is generated.

[0134] In the example embodiment, the electronic device 200 generates a preset question set by invoking a question generation model such as a GPT model for a predetermined scene question prompt word. For example, the electronic device 200 generates a single-round or multi-round evaluation question set according to a customized prompt word prompt using a GPT large model. Table 4 below shows 10 evaluation questions generated by the above prompt word:

[0135] Table 4: 10 generated evaluation questions

[0136] Question Classification Generated User Questions 1. Basic Travel Plan Inquiry "I want to go from Shanghai to Hangzhou, what is the fastest route?" 2. Considering Travel Cost "What is the more economical way to travel from Shanghai to Hangzhou?" 3. Travel Time Preference "I plan to leave Shanghai in the morning tomorrow, any suggestions?" 4. Specific Transportation Inquiry "Can you help me check the train times from Shanghai to Hangzhou?" 5. Traveling with Pets "I want to take my dog from Shanghai to Hangzhou, how should I arrange it?" 6. Wheelchair-Friendly Route Inquiry "I need a travel plan from Shanghai to Hangzhou suitable for wheelchair users." 7. Environmentally Friendly Travel Recommendation "I am looking for an environmentally friendly way to travel from Shanghai to Hangzhou, do you have any recommendations?" 8. Night Travel Options "I need to travel from Shanghai to Hangzhou at night, what are the safe travel options?" 9. Avoiding Peak Hours "How can I plan my trip from Shanghai to Hangzhou to avoid peak hours?" 10. Emergency Travel Needs "I need to leave Shanghai immediately for Hangzhou due to an emergency, what do you suggest?"

[0137] Further, the electronic device 200 removes at least one slot in the evaluation question by invoking the question generation model for the above-mentioned multiple evaluation questions through the question prompt word, and generates a multi-round dialogue evaluation question set. Table 5 below shows the multi-round dialogue evaluation question set generated by the prompt word:

[0138] Table 5: Multi-round dialogue evaluation question set generated

[0139]

[0140]

[0141] In the above example embodiment, the generated questions not only contain direct question and answer pairs, but also include Cartesian product questions involving multi-slot information in multi-round dialogue, and specific abnormal scenario question sets. As follows: (1) Single-round question and answer: direct questioning and answering for a specific intent or information point. (2) Multi-round multi-slot Cartesian product question: In multi-round dialogue, all possible questions are considered when different slot combinations are considered. (3) Abnormal scenario question set: edge cases, error recognition scenarios, etc. that the dialogue model to be evaluated may encounter, so as to test the robustness and abnormal handling capability of the dialogue model to be evaluated. Further, each evaluation question and its corresponding evaluation answer generated are saved to a database for subsequent evaluation process calling and analysis.

[0142] According to Figure 5 the technical solutions in the example embodiments, on the one hand, the question generation model can automatically generate a challenging question set for testing and improving the performance of the large model in actual dialogue scenarios; on the other hand, the question set generated by the above method not only guarantees the breadth and depth, but also is close to the actual dialogue scenario, which can effectively improve the performance of the large model multi-round dialogue system and user experience.

[0143] Figure 6 Fig. 4 shows a flowchart of question labeling according to some embodiments of the present specification.

[0144] Referring toFigure 6 In step S610, a question labeling task is created.

[0145] In an example embodiment, the purpose of the question labeling task is to provide accurate label information for each generated evaluation question, to facilitate subsequent model evaluation. The question labeling task includes, but is not limited to, aspects such as the type of question, the difficulty, the relevance, and the relationship with the specific multi-turn dialogue context. The electronic device 200 creates a corresponding question labeling task for the evaluation question set of the predetermined scenario.

[0146] In step S620, a question labeling model is invoked.

[0147] In an example embodiment, the electronic device 200 determines the question label of the evaluation question of at least one turn of dialogue through the question labeling model based on the evaluation question of at least one turn of dialogue. In order to ensure the accuracy and consistency of the label, the automatic generation of the label can be performed through the question labeling model. The question labeling model is usually a neural network model that can classify the evaluation question after pre-training. The question labeling model can understand the semantic content of the question and make detailed labeling of the evaluation question according to the pre-defined classifier. The question labeling model usually combines multiple NLP (Natural Language Processing) technologies, including but not limited to semantic analysis, text classification, named entity recognition, etc.

[0148] In step S630, a question label is generated.

[0149] In an example embodiment, the electronic device 200 invokes the question labeling model to generate the text label of each evaluation question for the evaluation question set of the predetermined scenario. The question label includes one or more of the question level, the question novelty, and the question multi-turn number. The following will be a detailed description of the three question labels:

[0150] (1) Classification and labeling of questions at different levels: To gain a deeper understanding of the attributes of questions, it is necessary to classify and label them at different levels. Question levels include identifying whether a question is open-ended or closed-ended, factual, inferential, or opinion-based, and whether the question depends on specific information in the context of the dialogue. Fine-grained classification of evaluation questions at the question level not only helps to assess the model's ability to respond to different types of questions, but can also be used to refine the training data of the dialogue model and improve its performance. (2) Evaluation of the novelty of questions: The evaluation of the novelty of questions is a complex process that requires the model to not only generate common standard questions, but also to create novel questions that are not explicitly mentioned by the user but contribute to the dialogue. For example, the question labeling model can be used to analyze the degree of repetition between questions and questions in the existing knowledge base, and whether questions can guide the dialogue to develop in depth or breadth. (3) Evaluation of multiple questions and multiple turns: In multi-turn dialogues, a question is often not isolated, but is related to other questions. Therefore, an efficient question labeling model also needs to evaluate the degree of correlation and complexity between questions. "Multi-turns" refers to the degree to which a question is mentioned or involved multiple times in a multi-turn dialogue. By evaluating multiple questions and multiple rounds, the model can better capture and understand the chain of questions and the flow of information in the dialogue context, thereby enabling effective question labeling.

[0151] It should be noted that although the above-mentioned labels are used as examples for the description, those skilled in the art should understand that the labels for problems can also be other appropriate labels, such as multi-round problems, multi-slot missing problems, or problems with clear intent, which are also within the scope of the embodiments in this specification.

[0152] according to Figure 6 The technical solution in the example implementation has two aspects. On the one hand, the question labeling model can automatically and accurately generate meaningful labels for each question, which will play an important role in subsequent evaluation and analysis. On the other hand, through the above-mentioned in-depth and detailed question labeling process, the multi-turn dialogue model can be evaluated and improved more accurately to provide a more intelligent and natural user interaction experience.

[0153] Figure 7 A schematic diagram of the model acquisition process provided according to some embodiments of this specification is shown.

[0154] Reference Figure 7 As shown, in step S710, a dialogue acquisition task is created.

[0155] In an example embodiment, the electronic device 200 creates a dialogue collection task for the dialogue model to be evaluated. In the dialogue collection task, the dialogue collection target and requirements need to be specified, such as being for a specific dialogue model to be evaluated, expecting to collect single-turn or multi-turn dialogue data, and specific scenarios of various dialogues. By specifying the dialogue collection target and requirements, it is helpful to design a more targeted and efficient collection process.

[0156] In step S720, the dialogue model to be evaluated is invoked.

[0157] In an example embodiment, the electronic device 200 invokes an industry large model to be evaluated, such as a GPT4.0 model, based on the set collection task. The large model to be evaluated usually has powerful natural language processing capability and can generate natural and fluent dialogue text with rich content.

[0158] In step S730, single-turn dialogue and multi-turn dialogue collection are performed.

[0159] In an example embodiment, the electronic device 200 performs single-turn dialogue and multi-turn dialogue collection for the dialogue model to be evaluated. Further, for single-turn dialogue collection, the collection module of the electronic device 200 simulates a simple question and answer scenario to collect the reply of the dialogue model to be evaluated to a specific question. For multi-turn dialogue collection, there are mainly two implementation manners of step S732 and step S734:

[0160] Step S732: Offline generation of questions. A series of related evaluation questions for testing the dialogue model to be evaluated are defined in advance, which are designed to guide the dialogue to develop in the expected direction. For example, a large model such as a GPT model is invoked to generate a preset question set offline in advance. When collecting dialogue data, the electronic device 200 provides the preset question set to the dialogue model to be evaluated in sequence, collects the response of the dialogue model to be evaluated, i.e., the evaluation answer, and continues the next round of dialogue on this basis.

[0161] Step S734: Online generation of questions. The electronic device 200 generates at least one round of dialogue evaluation questions for the dialogue model to be evaluated based on the current dialogue content and context of the dialogue model to be evaluated. By invoking a large model such as a GPT model to generate questions online, it has higher flexibility and timeliness. For example, in each round of dialogue, the collection module of the electronic device 200 proposes new questions online by invoking a large model according to the content and context of the current dialogue. According to the above example embodiment, the collection module of the electronic device 200 can generate more realistic and natural multi-turn dialogues online, thereby better simulating the actual dialogue behavior mode of human users.

[0162] In step S740, the question answer is generated.

[0163] In an example embodiment, the electronic device 200 generates question answers based on the above-mentioned evaluation questions for the to-be-evaluated dialogue model. With the two methods of online and offline generation of questions, the model collection module can effectively solve the inconsistency problem of multi-turn dialogue caused by the randomness and unpredictability of dialogue data. In particular, by calling a large model to generate questions in real time, the coherence and depth of the topic in multi-turn dialogue can be ensured, and repeated testing and analysis can also be achieved, thereby effectively evaluating the performance of the model in complex dialogue scenarios.

[0164] According to Figure 7 the technical solutions in the example embodiments, on the one hand, through the two methods of online and offline generation of questions, dialogue data can be efficiently and systematically collected from the to-be-evaluated large model, which will be used for testing and analysis to evaluate the performance and ability of the model in processing single-turn and multi-turn dialogues; on the other hand, through the efficient and flexible collection method, not only a large amount of high-quality dialogue data can be provided for evaluation activities, but also researchers and developers can deeply understand the capabilities and limitations of the model.

[0165] Figure 8 A flowchart of answer annotation is shown according to some embodiments of the present specification.

[0166] Referring to Figure 8 , in step S810, an answer annotation task is created.

[0167] In an example embodiment, the electronic device 200 generates an answer annotation task based on the above-mentioned question answers. The establishment of the answer annotation task needs to determine the target and standard of the evaluation, including the correctness, relevance, naturalness, fluency, etc. of the answer. For example, based on the above-mentioned standards of the evaluation, an annotation guide can be designed to ensure the consistency and accuracy of the annotation results.

[0168] In step S820, an answer annotation model is called.

[0169] In an example embodiment, the answer annotation model can be a pre-trained neural network model for classifying evaluation answers. For example, the answer annotation model can be an NLP expert model, which is used to automatically generate labels for answers. The answer annotation model usually combines multiple natural language processing techniques, such as text classification, sentiment analysis, semantic similarity evaluation, etc., to ensure the effectiveness and relevance of the labels. The electronic device 200 calls the answer annotation model based on the evaluation answers to determine the answer labels of the evaluation answers of at least one round of dialogue.

[0170] In step S830, an answer label is generated.

[0171] In an example embodiment, the answer label includes one or more of an answer level, an answer novelty, and an answer matching degree. The electronic device 200 invokes an answer labeling model based on the evaluation answer to determine the answer label of each evaluation answer of at least one round of conversation. The three answer labels are described in detail as follows: (1) Answer level: the answer is classified and labeled in detail to accurately describe its nature. For example, the answer is classified to distinguish factual answers, reasoning answers, and opinion answers; to identify whether the answer completely or partially responds to all key points of the question; and to evaluate the coherence of the answer in multiple rounds of interaction and the progress of the conversation. (2) Answer novelty: in order to measure the creativity of the model, it is necessary to evaluate the novelty of the answer. The answer labeling model evaluates whether the model answer provides an answer that adds new information or simply repeats previous information. Innovative answers can usually provide additional value and enrich the content of the conversation. (3) Multi-question and multi-round number label evaluation: for multiple questions raised in a multi-round conversation, the answer label module not only evaluates individual answers, but also considers the association and consistency between multiple answers. The "multi-round number" indicator can help understand the model's ability to answer multiple related questions in a coherent manner.

[0172] According to Figure 8 the technical solutions in the example embodiments, on the one hand, by generating the answer label of the evaluation answer, a systematic method can be provided to label and evaluate the answers generated by the large model; on the other hand, by accurately labeling the answer content, the performance of the model can be better understood, especially the ability to understand complex user inquiries and provide informative answers.

[0173] Figure 9 A flowchart of automatic evaluation according to some embodiments of the present specification is shown.

[0174] Referring to Figure 9 , in step S910, an evaluation task is created.

[0175] In an example embodiment, the electronic device 200 creates an evaluation task for the conversation model to be evaluated. The specific requirements of the evaluation task need to be clear. This includes determining the large model to be evaluated, i.e., the conversation system, setting the evaluation purpose, formulating the relevant evaluation indicators, and determining the weight of the evaluation indicators.

[0176] In step S920, the evaluation large model is invoked.

[0177] In an example embodiment, the electronic device 200 invokes the evaluation large model with the user intent, the slot information, and the domain knowledge. The evaluation large model synthesizes the user intent, the slot information, and the domain knowledge to simulate a real user interaction with the dialogue system. Through such an interaction, an accurate evaluation index result can be generated to reflect the performance of the large model in actual application.

[0178] In step S930, the evaluation result of the evaluation index is generated.

[0179] In an example embodiment, the electronic device 200 generates an evaluation result, such as an index score, for each evaluation index based on the evaluation related information of the dialogue model to be evaluated through the evaluation large model. The evaluation index is the key to quantitatively evaluate the performance of the large model, and the evaluation index can include accuracy, relevance of the answer, coherence of the dialogue, answer delay, user satisfaction, etc. For example, Tables 6 and 7 below provide evaluation results corresponding to evaluation indexes of single-turn dialogue and multi-turn dialogue.

[0180] Table 6. Evaluation result of single-turn dialogue

[0181]

[0182]

[0183] Table 7. Evaluation result of multi-turn dialogue

[0184]

[0185] In step S940, an attribution analysis is generated.

[0186] In an example embodiment, the electronic device 200 obtains the index result of each preset evaluation index of the dialogue model to be evaluated to perform attribution analysis on the index result of each preset evaluation index. The purpose of attribution analysis is to deeply understand the reasons for the performance of the dialogue model to be evaluated in different evaluation links. For example, the electronic device 200 automatically analyzes how the dialogue model to be evaluated understands the language expression of the user, the slot relationship, and the logical relationship in the domain to reveal the factors affecting the score of the dialogue model to be evaluated on a specific evaluation index.

[0187] In step S950, an evaluation report is generated.

[0188] In an example embodiment, the electronic device 200 generates an evaluation report for the to-be-evaluated dialogue model based on the index results of the respective evaluation indexes and the corresponding attribution analysis results. For example, the electronic device 200 collates the scores of the respective evaluation indexes and the attribution analysis to generate a detailed evaluation report. The evaluation report not only contains the comprehensive score, but also points out the advantages and disadvantages of the large model in the dialogue processing process, providing guidance for the further optimization of the model. For example, the evaluation report can be "According to the provided dialogue content, this multi-turn dialogue system maintains high understanding accuracy, conversation coherence, and information collection efficiency in the case of multiple user destination changes. In addition, it shows good system adaptability and error recovery ability, and can dynamically update information according to the latest user needs. The natural language generation quality of the dialogue is also good, providing detailed and easy-to-understand travel information. The accurate evaluation of user satisfaction, response time, and data utility requires actual user feedback and more in-depth analysis."

[0189] According to Figure 9 the technical solutions in the example embodiments, the dialogue of the to-be-evaluated dialogue model is automatically evaluated by calling the evaluation large model based on the preset evaluation indexes, so that the dialogue evaluation can be automatically and efficiently performed objectively.

[0190] Figure 10 A flowchart of a dialogue evaluation method according to another embodiment of the present specification is shown.

[0191] Referring to Figure 10 , in step S1010, an evaluation question is generated.

[0192] In an example embodiment, the electronic device 200 constructs a series of prompts based on the user intent information, the slot information, and the domain information, calls a question generation model based on the constructed prompt words, and generates a preset evaluation question set. The question generation model can be a GPT model or other appropriate large model.

[0193] For example, suppose the prompt word is "Suppose you are a dialogue model test expert, and now you need to test a multi-turn dialogue model, where the test question is: Generate a travel plan from Shanghai to Hangzhou, where slot 1: departure place; slot 2: destination; slot 3: user intent; and require the model to answer in the case of multiple slots. Help me generate 10 questions." In step S1020, the question label corresponding to the evaluation question is generated.

[0194] In an example embodiment, the electronic device 200 determines, by a question labeling model, a question label of the evaluation question of the at least one round of conversation based on the evaluation question of the at least one round of conversation. The question labeling model is usually a neural network model that can be pre-trained to classify evaluation questions. The question labeling model can understand the semantic content of the question and make a detailed label of the evaluation question according to a pre-defined classifier.

[0195] In step S1030, the conversation data of the to-be-evaluated model is collected.

[0196] In an example embodiment, the electronic device 200 calls the to-be-evaluated large model to collect single-round conversation and multi-round conversation based on the set collection task. For single-round conversation collection, the collection module of the electronic device 200 simulates a simple question and answer scenario to collect the reply of the to-be-evaluated conversation model to a specific question.

[0197] For multi-round conversation collection, there are two ways: offline generated question and online generated question. For offline generated question, when collecting conversation data, the electronic device 200 provides a preset question set to the to-be-evaluated model in sequence, collects the response of the to-be-evaluated conversation model, i.e., the evaluation answer, and continues the next round of conversation based on the evaluation answer. For online generated question, in each round of conversation, the collection module of the electronic device 200 proposes a new question by calling the large model based on the content and context of the current conversation. In step S1040, the answer label corresponding to the evaluation answer is generated.

[0198] In an example embodiment, the electronic device 200 generates an answer label based on the above question and answer by calling an answer labeling model. The answer label includes one or more of answer level, answer novelty, and answer matching degree. The answer labeling model can be a neural network model pre-trained to classify evaluation answers. For example, the answer labeling model can be an NLP expert model, which is used to automatically generate a label of the answer.

[0199] In step S1050, the to-be-evaluated conversation model is evaluated.

[0200] In an example embodiment, the electronic device 200 evaluates at least one round of conversation of the to-be-evaluated conversation model by a preset evaluation index based on the evaluation related information. The preset evaluation index includes a first type of evaluation index and a second type of evaluation index. The first type of evaluation index is a non-objective evaluation index for evaluating the conversation capability of the large model, and the second type of evaluation index is an objective evaluation index for evaluating the conversation capability of the large model. The first type of evaluation index, i.e., the non-objective evaluation index, includes understanding accuracy, conversation coherence, natural language generation quality, etc. The second type of evaluation index, i.e., the objective evaluation index, includes response time, user satisfaction feedback, slot missing feedback, etc.

[0201] According to Figure 10 the technical solutions in the example embodiments, the whole process of user dialogue is simulated through automatic technology, from a single round of dialogue to multiple rounds of dialogue, so that the dialogue speed and fluency of the dialogue model under different conditions can be accurately evaluated, and consistent experience with the user is ensured.

[0202] Figure 11 A flowchart of performing automatic evaluation according to some embodiments of the present specification is shown.

[0203] Referring to Figure 11 , in step S1110, the evaluation question is imported into the evaluation platform.

[0204] In the example embodiments, the evaluation question includes product demand questions, regression scenario questions, and abnormal scenario questions. The electronic device 200 imports the evaluation question into the evaluation platform and stores it to the data platform.

[0205] In step S1120, the evaluation task is created.

[0206] In the example embodiments, the electronic device 200 establishes an evaluation task for a predetermined number of evaluation questions. For example, 1000 evaluation tasks are established for 1000 evaluation questions in the data platform.

[0207] In step S1130, the evaluation task is executed.

[0208] In the example embodiments, the electronic device 200 executes the evaluation task through an automatic evaluation link, determines the index results of each preset evaluation index of the dialogue model to be evaluated, and performs attribution analysis on the index results of each preset evaluation index. The automatic evaluation link includes obtaining an evaluation data set, determining an evaluation model and a model to be evaluated, creating an evaluation prompt word, creating an evaluation script, and executing an evaluation task. Each step in the automatic evaluation link is similar to the implementation principle and implementation effect of the steps of the above dialogue evaluation method, and will not be repeated here.

[0209] In step S1140, the evaluation report is generated.

[0210] In the example embodiments, the electronic device 200 generates an evaluation report for the dialogue model to be evaluated based on the index results of each evaluation index and the corresponding attribution analysis results. For example, for the case of poor multi-round dialogue, the evaluation report can be "Overall, the performance of this multi-round dialogue system needs to be improved, especially in understanding the user's explicit needs and providing targeted information. The initial response fails to accurately capture the core of the user's query, resulting in the subsequent dialogue trying to correct the direction but not providing specific and useful travel information. In this case, the system's understanding accuracy, problem solving rate, and system adaptability are low."

[0211] According to Figure 11 The technical solutions in the example embodiments can automatically and accurately evaluate dialogues of large models in different scenarios.

[0212] Another aspect of the present specification provides a non-transitory storage medium storing at least one set of executable instructions for performing the dialogue evaluation. When the executable instructions are executed by a processor, the executable instructions direct the processor to implement the steps of the dialogue evaluation method described in the present specification. In some possible implementation manners, various aspects of the present specification can also be implemented in the form of a program product including program codes. When the program product is run on the electronic device 200, the program codes are used to cause the electronic device 200 to perform the steps of the dialogue evaluation method described in the present specification. The program product for implementing the above method can include the program codes in a portable compact disc read-only memory (CD-ROM) and can be run on the electronic device 200. However, the program product of the present specification is not limited to this, and in the present specification, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system. The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer readable storage medium can include a data signal carried in a baseband or as part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take on various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium can also be any readable medium that is not a storage medium, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. The program codes contained in the readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical fiber, RF, and the like, or any suitable combination of the above. The program codes for performing the operations of the present specification can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and the like, and a conventional procedural programming language such as the "C" language or a similar programming language. The program codes can be executed entirely on the electronic device 200, partially on the electronic device 200, as an independent software package, partially on the electronic device 200 and partially on a remote computing device, or entirely on a remote computing device.

[0213] The above described embodiments of the disclosure have been described. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still accomplish desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0214] In light of the above, those skilled in the art will appreciate that the foregoing detailed description of the present disclosure is susceptible to various modifications and / or revisions without departing from the spirit and scope of the present disclosure. Although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement which is calculated to achieve the same purpose can be substituted for the specific embodiments shown. This disclosure is intended to cover any adaptations or variations of the present disclosure. Therefore, it is intended that the application be protected by: the broadest interpretation of the appended claims to take into account unforeseen equivalents and alternatives based on current knowledge, or future knowledge.

[0215] In addition, certain terminology has been used to describe embodiments of the disclosure. For example, the terms "one embodiment" or "an embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase "in one embodiment" or "an embodiment", or "in at least one embodiment" in various places in the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0216] It should be understood that in the foregoing description of embodiments of the disclosure, various features can be combined in a single embodiment or in multiple embodiments of the disclosure. Such descriptions should be seen as but one embodiment of the methods made in the present disclosure and is submitted for patentability as such under 35 U.S.C. § 101. Those skilled in the art will readily recognize a method as being equivalent to one specifically described herein, and the present disclosure is intended to encompass such modified or modified methods as falling within the scope of the claims.

[0217] Each patent, patent application, publication of a patent application, and other material, for example articles, books, specifications, publications, documents, things, or the like which can be cited in the present document can be accorded with the scope of their respective copyrights. The contents of all such cited patents, patent applications, publications of patent applications, and other material are hereby incorporated by reference herein, except that for any and all that can be dependent on the history of prosecution of any application incorporated herein by reference, to the extent that such history can be inconsistent with the term of this document now or hereafter associated with it. In the event of any such inconsistency, the term of this document controls. For example, if there is any discrepancy between what can be expressed herein as to the description, definition and / or use of terms associated with any incorporated material and what is expressed in the incorporated material, the term in this document controls.

[0218] Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the present specification. Other modifications that fall within the scope of the present specification can also be made. Accordingly, the present specification discloses embodiments only as examples. One skilled in the art could, based upon the embodiments disclosed in the present specification, affect equivalent configurations without departing from the scope of the present specification. The present specification and embodiments therefore do not limit the scope of the application but rather only the specific embodiments now described.

Claims

1. A dialogue evaluation method, comprising: Obtain the evaluation question for at least one round of dialogue corresponding to the dialogue model to be evaluated; Based on the evaluation question, the dialogue model to be evaluated is invoked to generate the evaluation response corresponding to the at least one round of dialogue; Based on the evaluation questions and the evaluation answers, the evaluation-related information corresponding to the at least one round of dialogue is determined. The evaluation-related information includes user intent information, slot information, and domain information. as well as Based on the evaluation information, the at least one round of dialogue of the dialogue model to be evaluated is evaluated using preset evaluation indicators, which include a first type of evaluation indicator and / or a second type of evaluation indicator.

2. The method according to claim 1, wherein, The step of evaluating at least one round of dialogue of the dialogue model to be evaluated based on the evaluation-related information and using preset evaluation metrics includes: Based on the evaluation-related information and the first type of evaluation metrics, the at least one round of dialogue in the dialogue model to be evaluated is evaluated by invoking a large dialogue evaluation model; and / or Based on the evaluation-related information and the second type of evaluation metrics, the at least one round of dialogue of the dialogue model to be evaluated is evaluated using an evaluation script.

3. The method according to claim 2, wherein, The step of evaluating at least one round of dialogue of the dialogue model to be evaluated by calling a large dialogue evaluation model based on the evaluation-related information and the first type of evaluation metrics includes: Based on the evaluation-related information and the first type of evaluation indicators, evaluation prompt words corresponding to the large dialogue evaluation model are generated. Based on the evaluation prompts, the dialogue evaluation model is used to evaluate at least one round of dialogue in the dialogue model to be evaluated.

4. The method according to claim 2, wherein, The second type of evaluation metric includes response time. A single dialogue round includes multiple dialogue phases. The evaluation of at least one dialogue round of the dialogue model under evaluation, based on the evaluation-related information and the second type of evaluation metric, using an evaluation script, includes: Based on the evaluation-related information and the second type of evaluation metrics, the loading time of each dialogue stage of the dialogue model to be evaluated is determined by the evaluation script; The response time of the dialogue model under evaluation is assessed based on the loading time of each dialogue stage.

5. The method according to claim 4, wherein, The multiple dialogue stages include the page rendering stage, the data request stage, and the content display stage.

6. The method according to claim 2, wherein, The evaluation-related information includes predefined standard slot information and slot information of the evaluation response. The evaluation of at least one round of dialogue in the dialogue model to be evaluated, based on the evaluation-related information and the second type of evaluation metrics, using an evaluation script, includes: Based on the evaluation-related information and the second type of evaluation indicators, the evaluation script is invoked to compare the slot information of the evaluation response with the standard slot information; The at least one round of dialogue of the dialogue model to be evaluated is evaluated based on the comparison results.

7. The method according to claim 1, wherein, The step of obtaining the evaluation question for at least one round of dialogue corresponding to the dialogue model to be evaluated includes: Obtain an evaluation question for at least one pre-defined offline dialogue round for the dialogue model to be evaluated; or Based on the current dialogue content and context of the dialogue model to be evaluated, evaluation questions for at least one round of dialogue corresponding to the dialogue model to be evaluated are generated online.

8. The method according to claim 7, wherein, Before obtaining the evaluation question for at least one pre-defined offline dialogue for the dialogue model to be evaluated, the method further includes: Based on question-related information, generate question generation prompts corresponding to a pre-trained large-scale question generation model; Based on the question, prompt words are generated, and the question generation model is invoked to generate evaluation questions for at least one round of pre-defined offline dialogue.

9. The method according to claim 7, wherein, The context includes user behavior information in the at least one round of dialogue. The step of generating evaluation questions online for the at least one round of dialogue corresponding to the dialogue model to be evaluated, based on the current dialogue content and context of the dialogue model to be evaluated, includes: Based on the current dialogue content of the dialogue model to be evaluated and the user behavior information, evaluation questions for at least one round of dialogue corresponding to the dialogue model to be evaluated are generated online.

10. The method according to claim 1, wherein, The assessment-related information also includes the question tags of the assessment questions and the answer tags of the assessment answers. Determining the assessment-related information corresponding to the at least one round of dialogue based on the assessment questions and the assessment answers includes: Based on the evaluation question, the question labeling model is used to determine the question label for the evaluation question of the at least one round of dialogue; Based on the evaluation response, the response labeling model is used to determine the response label of the evaluation response in the at least one round of dialogue.

11. The method according to claim 10, wherein, The question tags include one or more of the following: question level, question novelty, and number of question rounds; the answer tags include one or more of the following: answer level, answer novelty, and answer matching degree.

12. The method according to claim 1, wherein, The method further includes: Obtain the index results of each of the preset evaluation indicators of the dialogue model to be evaluated, and perform attribution analysis on the index results of each of the preset evaluation indicators.

13. The method according to claim 12, wherein, The method further includes: Based on the results of the aforementioned indicators and the corresponding attribution analysis, an evaluation report is generated for the dialogue model to be evaluated.

14. An electronic device, comprising: At least one storage medium storing at least one instruction set for performing dialogue evaluation processing; as well as At least one processor is communicatively connected to the at least one storage medium. When the electronic device is running, the at least one processor reads the at least one instruction set and executes the dialogue evaluation method according to any one of claims 1-13.

Images