Method and device for evaluating dialogue system, electronic equipment and computer program product

Abstract

The embodiment of the invention relates to a method and device for evaluating a dialogue system, electronic equipment and a computer program product, and relates to the technical field of AB experiments, language models and large language models. The method includes obtaining a plurality of access requests for a plurality of versions of dialog systems. The method also includes assigning the plurality of access requests to the plurality of versions of the dialog system. Further, the method includes determining, using a language model, an evaluation score for each of the plurality of versions of dialog systems based on the query input in the access request, the response output of the dialog system, and a predefined evaluation indicator. Therefore, according to the embodiment of the invention, efficient and accurate online evaluation can be carried out on the dialogue systems of multiple versions through the language model, so that the evaluation efficiency is remarkably improved, and the evaluation period is shortened. In addition, the use of the language model not only can generate a more accurate scoring result, but also can quickly verify the effects of different strategies, thereby supporting the quick iteration and optimization of the dialogue system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more particularly to a method, device, electronic device, and computer program product for evaluating a dialogue system. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, conversational applications (such as chatbots, virtual assistants, customer service systems, etc.) have gradually become important tools in people's daily lives and work. Through the continuous advancement of natural language processing technology, these applications can more accurately understand user needs and achieve more intelligent and humane responses. With the popularization of smart hardware and the advancement of technology, the use scenarios of conversational applications are also constantly expanding, playing a more important role in future life.

[0003] The effectiveness evaluation of conversational applications is crucial because it is directly related to the optimization of user experience and the actual value of the application. By evaluating the application, developers can understand its performance in different scenarios. Effective evaluation can not only help identify loopholes or deficiencies in the system, but also provide directions for improvement to enhance the system's intelligence level and humanized services. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, electronic device, computer program product, and medium for evaluating a dialogue system.

[0005] According to a first aspect of the present disclosure, a method for evaluating a dialogue system based on a language model is provided. The method includes obtaining multiple access requests for multiple versions of dialogue systems, where the multiple versions of dialogue systems include at least a first version of the dialogue system and a second version of the dialogue system. The method also includes assigning the multiple access requests to the multiple versions of the dialogue systems, each of the multiple access requests corresponding to a version of the dialogue system in the multiple versions. In addition, the method also includes determining an evaluation score for each dialogue system in the multiple versions of the dialogue system using a language model based on a query input in each of the multiple access requests, a response output of the corresponding version of the dialogue system, and a predefined evaluation indicator.

[0006] According to a second aspect of the present disclosure, a device for evaluating a dialogue system based on a language model is provided. The device includes an access request acquisition module, configured to acquire multiple access requests for multiple versions of dialogue systems, the multiple versions of dialogue systems at least including a first version of the dialogue system and a second version of the dialogue system. The device also includes an access request allocation module, configured to allocate multiple access requests to multiple versions of dialogue systems, each of the multiple access requests corresponding to a version of the dialogue system in the multiple versions. In addition, the device also includes an evaluation score determination module, configured to determine the evaluation score for each dialogue system in the multiple versions of the dialogue system based on the query input in each of the multiple access requests, the response output of the corresponding version of the dialogue system, and the predefined evaluation index using the language model.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which, when executed, cause a computer to perform the steps of the method of the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.

[0010] The purpose of this Summary is to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram showing an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented;

[0013] Figure 2A flowchart of a method for evaluating a dialogue system based on a language model according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram showing another process of evaluating a dialogue system according to an embodiment of the present disclosure is shown.

[0015] Figure 4 A flowchart showing a process of creating an experiment according to an embodiment of the present disclosure;

[0016] Figure 5 A flow chart showing a process of data reporting according to an embodiment of the present disclosure;

[0017] Figure 6 A block diagram showing an apparatus for evaluating a dialogue system based on a language model according to an embodiment of the present disclosure; and

[0018] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0019] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0020] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information (such as voice) involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0023] As mentioned above, dialogue applications or dialogue systems are becoming more and more widespread, such as chatbots, virtual assistants, customer service systems, etc., so the evaluation of dialogue systems is becoming more and more important. In related technologies, the evaluation of dialogue systems relies on collecting dialogue data during system operation. After the data collection is completed, it needs to be further processed and handed over to the evaluation system for scoring. The evaluation results are used to guide the optimization of the dialogue system. This is mainly because the scoring methods of existing dialogue systems usually rely on complex calculation processes, such as rule-based matching and other manual scoring. These methods are inefficient when processing large amounts of data and are not suitable for real-time online calculations. This results in a longer cycle for data collection, processing and scoring, which limits the speed of product iteration. In addition, these methods are also less effective and cannot accurately evaluate the actual performance of the dialogue system.

[0024] As mentioned above, conversational applications (such as chatbots, virtual assistants, customer service systems, etc.) are being widely used, so the evaluation of conversational systems is becoming increasingly important. At present, the relevant scoring methods usually rely on complex calculation processes, such as rule-based matching and manual scoring methods. These methods are inefficient when processing large amounts of data and are not suitable for real-time online evaluation. Therefore, the evaluation process usually relies on collecting conversational data during system operation, further processing after data collection, and then handing it over to the evaluation system for scoring. The final evaluation results are used to guide system optimization. This makes it impossible to conduct online comparative experiments (such as AB experiments), which makes it impossible to quickly verify the effects of different strategies, limiting the rapid iteration of conversational products. In addition, these traditional evaluation methods have poor accuracy and cannot fully and accurately reflect the actual performance of the conversation system, especially in the current increasingly complex conversation scenarios.

[0025] To this end, an embodiment of the present disclosure proposes a method for evaluating a dialogue system based on a language model. First, an access request from a user is obtained, and the access request is assigned to a dialogue system in multiple versions of the dialogue system. Then, based on the query input in the access request, the response output of the corresponding version of the dialogue system, and a predefined evaluation index, the language model is used to determine the evaluation score of each version of the dialogue system.

[0026] Therefore, according to the embodiments of the present disclosure, multiple versions of the dialogue system can be evaluated online based on the language model, which significantly improves the evaluation efficiency and accuracy. Compared with the traditional offline evaluation method, the use of the language model can generate the evaluation score of the dialogue system in real time, thereby avoiding a long evaluation cycle. In addition, the use of the language model can also generate more accurate scoring results, thereby quickly and accurately verifying the effects of different strategies and promoting the rapid iteration and optimization of dialogue applications.

[0027] Figure 1Schematic diagram of an example environment in which the device and / or method according to an embodiment of the present disclosure may be implemented. Figure 1 As shown, the example environment 100 may include a computing device 110, which may be a user terminal, a mobile device, a computer, etc., or a computing system, a single server, a distributed server, or a cloud-based server. The experimental platform 114 may be deployed on the computing device, and may divert access requests from the user 112. For example, the experimental platform may be an AB experimental platform, which may be used to compare the effects of two or more strategies, functions, or versions, by randomly allocating user traffic to different experimental groups to collect data and evaluate the performance of each group. It should be understood that the user 112 refers to a large number of users who use the dialogue system, rather than a single user, and each user may initiate multiple access requests.

[0028] In some embodiments, the experiment platform 114 can distribute the access request 116 to the first version of the dialogue system 120 and distribute the access request 118 to the second version of the dialogue system 122 through the diversion service. It should be understood that two access requests are shown here as an example, and there may be fewer or more requests in fact. In addition, each user request will be assigned to a fixed version of the dialogue system. The dialogue system here may include but is not limited to a chat robot, a virtual assistant, a customer service system, etc.

[0029] The dialogue system 120 may generate a response output (i.e., an answer of the dialogue system) 124 according to the query input (i.e., the user question) in the access request 116, and the dialogue system 122 may generate a response output 126 according to the query input in the access request 118. For example, the access request may include the content of the user question, such as the user question is "What will the weather be like tomorrow?", and the dialogue system may generate an answer to the user question, i.e., a response output.

[0030] The experiment platform 114 may call the language model 128 to evaluate the answer of the dialogue system, and generate an evaluation score 130 and an evaluation score 132. For example, the evaluation indicator may be predefined as factual consistency, and the language model may generate an evaluation score for factual consistency. In some embodiments, multiple indicators may be configured, such as answer relevance, context relevance, context recall, context accuracy, answer correctness, and answer similarity, so that evaluation scores for multiple indicators may be generated.

[0031] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0032] The following will combine Figures 2 to 7 The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0033] Figure 2 A flowchart of a method 200 for evaluating a dialog system based on a language model according to an embodiment of the present disclosure is shown. At block 202, multiple access requests for multiple versions of dialog systems may be obtained, where the multiple versions of dialog systems include at least a first version of the dialog system and a second version of the dialog system. For example, referring to Figure 1 , the experiment platform 114 may obtain the access request 116 and the access request 118, and the multiple versions of the dialog system include at least the first version of the dialog system 120 and the second version of the dialog system 122. At block 204, the multiple access requests may be assigned to the multiple versions of the dialog system, each of the access requests corresponding to one of the multiple versions of the dialog system. Figure 1 , the experiment platform 114 can distribute the access request 116 and the access request 118 to the dialogue system 120 and the dialogue system 122 respectively.

[0034] At block 206, an evaluation score for each of the multiple versions of the dialogue system may be determined using a language model based on a query input in each of the multiple access requests, a response output of the corresponding version of the dialogue system, and a predefined evaluation metric. Figure 1 The experimental platform 114 uses the language model 128 to determine an evaluation score 130 for the dialogue system 120 and an evaluation score 132 for the dialogue system 122 based on the query input in each access request in the multiple access requests, the response output of the corresponding version of the dialogue system, and the predefined evaluation indicators.

[0035] Therefore, according to the method 200 of the embodiment of the present disclosure, multiple versions of the dialogue system can be evaluated online based on the language model, which significantly improves the evaluation efficiency and accuracy. Compared with the traditional offline evaluation method, the use of the language model can generate the evaluation score of the dialogue system in real time, thereby avoiding a long evaluation cycle. In addition, the use of the language model can also generate more accurate scoring results, thereby quickly and accurately verifying the effects of different strategies and promoting the rapid iteration and optimization of the dialogue system.

[0036] Figure 3 FIG. 3 is a schematic diagram of another process 300 of evaluating a dialog system according to an embodiment of the present disclosure. Figure 3As shown, user 302 can initiate a dialogue request to interact with dialogue application 304. Dialogue application 304 can be any form of dialogue system, including but not limited to chatbots, intelligent customer service, tool assistants, etc. Dialogue application 304 can be deployed on multiple platforms, such as mobile applications, PC applications, or provide services in the form of web pages. User 302 can interact with dialogue application 304 through different terminal devices, including but not limited to smartphones, computers, and tablet devices, etc.

[0037] When user 302 uses dialog application 304, dialog application 304 may send the access request of user 302 to experimental platform 306. For example, experimental platform 306 may be an AB experimental platform for online evaluation and comparative experiments of multiple versions of dialog systems. Different system policies may be configured on experimental platform 306 to process these access requests, and each system policy corresponds to a version of the dialog system. In other words, different versions of the dialog system are distinguished by different system policies. For example, the system policy may specify the use of different dialog models, different system prompt words, or even different training data or other configurations.

[0038] In some embodiments, the system policy can specify the dialogue model called by the dialogue application. For example, 80% of user requests call the first dialogue model through the benchmark strategy, and the remaining 20% ​​of requests call the second dialogue model through the experimental strategy. This traffic distribution strategy can be used to verify whether the second dialogue model is better than the first dialogue model. In addition, in the case where the dialogue application uses a large language model, the system policy can also specify the system prompt words of the large language model called by the dialogue application. For example, some user requests use the system prompt words in the benchmark strategy, and other user requests use the system prompt words in the experimental strategy. In this way, it can be verified whether the prompt words corresponding to the experimental strategy can bring better results.

[0039] In addition to the dialogue model and system prompt words, the system policy can also configure a variety of other parameters, such as dialogue turns, the length of the generated dialogue, the vector library called, etc., especially in the dialogue system using the retrieval augmented generation (RAG) architecture. When the dialogue system adopts the RAG architecture, the system policy can also specify the vector library called and the data set in the vector library, which may affect the generation effect of the dialogue. Therefore, the system policy described in the embodiments of the present disclosure is flexible and extensible, can cover a variety of possible configurations, and does not limit the specific implementation details.

[0040] Continue to refer Figure 3, the experiment platform 306 may include a diversion service 308, an indicator service 310, a report service 312, and an evaluation service 314. The diversion service 308 may distribute the access request of the user 302 to different versions of the dialogue system. For example, the user request may be distributed to a certain version of the dialogue system based on the hash value of the user ID or device ID, ensuring that all user requests of a certain user will only be distributed to a fixed dialogue system, so that experiments will not interfere with each other.

[0041] The indicator service 310 can be used to configure and manage evaluation indicators of different versions of dialogue systems. Through the indicator service 310, users can customize and configure multiple evaluation indicators, which are used to measure the performance of the dialogue system in different scenarios. Evaluation indicators may include but are not limited to fact consistency, answer relevance, context relevance, context recall, context accuracy, answer correctness, and answer similarity. These indicators can accurately reflect the multi-dimensional performance of the dialogue system and ensure that the evaluation results have wide applicability. For example, the fact consistency indicator can be configured in the indicator service 310 to measure the degree of consistency between the answers generated by the dialogue system and the facts. After the configuration is completed, the evaluation service 314 can calculate the scores of the baseline version dialogue system and the experimental version dialogue system in fact consistency online based on the indicator, so as to compare the performance of the two in this specific evaluation dimension. In some embodiments, multiple indicators can be configured at the same time using the indicator service 310, and then the evaluation service 314 will calculate the evaluation scores of each version of the dialogue system for multiple evaluation indicators, so as to facilitate the generation of more comprehensive evaluation results.

[0042] The reporting service 312 can summarize the evaluation results and generate a report, and display it on the front-end page. Based on the processing results of the evaluation data, the reporting service 312 automatically generates a detailed experimental report, showing the scoring data 318 of different indicators of each dialogue system, such as the score of factual consistency, the score of answer relevance, and the score of context relevance, etc. These reports can show the comparison between the evaluation scores of each version of the dialogue system, which is convenient for analysts to make optimization decisions. As mentioned above, different versions of the dialogue system are distinguished by different system strategies. If the evaluation score of the experimental version of the dialogue system is higher, then its system strategy is better than the system strategy of the baseline version. In subsequent product iterations, the experimental strategy can replace the baseline strategy as the new baseline strategy.

[0043] The evaluation service 314 can use the language model 316 to evaluate the evaluation score of the dialogue system. For example, if it is necessary to evaluate the performance of the baseline dialogue system and the experimental dialogue system on the fact consistency index, the language model 316 can be used to evaluate the user's query input (that is, the user's question) and the dialogue system's response output (the dialogue system's answer). For example, in order to calculate the fact consistency score using the language model 316, the prompt words shown in Table 1 can be used.

[0044] Table 1. Prompt words used to evaluate the fact consistency score of the dialogue system

[0045]

[0046] When evaluating the score of factual consistency, the prompt words shown in Table 1 can be used, and the user's question and the answer of the dialogue system can be input into the language model 316 at the same time, and the factual consistency score of the question and answer pair can be calculated. Thus, the factual consistency score can be determined quickly and accurately, avoiding the conventional evaluation method, and the text of the question and answer needs to be segmented, vectorized and processed in other ways to obtain the evaluation score. In addition, it should be understood that the evaluation score of factual consistency can be calculated for all question and answer pairs generated by the experimental version in a certain period of time (for example, 1 day) (for example, the order of magnitude can be in the millions), so that the factual consistency score of the dialogue system of the experimental version in this time period can be determined. It should be understood that the prompt words for the same indicator can be different, and other prompt words can be set for other evaluation indicators. As mentioned above, the evaluation service 314 can also calculate multiple evaluation indicators at the same time, so that the language model can be evaluated more comprehensively. In some embodiments, multiple indicator scores can be fused using different weights to generate a comprehensive score, and these weights are configurable.

[0047] In some embodiments, the large language model can be pre-trained and fine-tuned for specific evaluation tasks. In the pre-training stage, the language model is trained using text data to learn common language patterns and structures. In the fine-tuning stage, the language model is trained according to specific evaluation indicators and task requirements using labeled question-answer pairs, evaluation standards, and corpora in related fields, so that the model can more accurately evaluate the performance of the dialogue system. During the fine-tuning process, the accuracy of the model for specific evaluation indicators (such as factual consistency, answer relevance, context relevance, etc.) can be improved by introducing training data related to the evaluation task, selecting a suitable loss function, and adjusting the training strategy. In this way, the fine-tuned language model can provide more accurate evaluation scores in practical applications, ensuring that the performance of the dialogue system under different versions and strategies can be accurately evaluated.

[0048] The experimental platform 306 can use the database 320 for data storage. For example, the database 320 can store access requests from the dialogue system, query content input by the user, response output generated by the dialogue system, and evaluation scores calculated for each question-answer pair. In addition, the database 320 can also store predefined evaluation indicators for evaluation, configuration information of different versions of the dialogue system, and traffic distribution data of each experimental version.

[0049] Figure 4 FIG. 4 is a flow chart showing a process 400 for creating an experiment according to an embodiment of the present disclosure. Figure 4 As shown, at box 402, an experimental version can be created. As mentioned above, different versions of the dialogue system are distinguished by different system strategies. When no experiment is conducted, all user requests are assigned to the baseline dialogue system through the baseline strategy. For example, in the baseline strategy, the dialogue system can be specified to call the first dialogue model. When the developer has trained the second dialogue model in some way and wants to verify whether the second dialogue model is better than the first dialogue model, an experimental strategy can be created, and the dialogue system can be configured to call the second dialogue model in the experimental strategy. In this case, the dialogue system of the first version is the dialogue system of the baseline version, which is configured with a baseline strategy, indicating that the dialogue system calls the first dialogue model to generate answers for the user; the dialogue system of the second version is the dialogue system of the experimental version, which is configured with an experimental strategy, indicating that the dialogue system calls the second dialogue model to generate answers for the user. In addition, the system strategy is not limited to the model called by the dialogue system, but can configure any strategy related to the dialogue system. In some embodiments, multiple experimental versions can be created, for example, another experimental version can be created, which is configured with another experimental strategy, indicating that the dialogue system calls the third dialogue model.

[0050] At block 404, the experimental traffic may be allocated. Figure 3 The traffic proportion of each experiment is configured in the configuration file of the experimental platform 306 shown in the figure. For example, 80% of the user access traffic can be configured for the baseline version, and 20% of the user access traffic can be configured for the experimental version. In other words, 80% of the user access requests will be diverted to the baseline version, and 20% of the user access requests will be diverted to the experimental version.

[0051] At block 406, an evaluation metric may be selected. As previously described, the metric service of the experiment platform may be used to configure one or more metrics to be evaluated. For example, one or more evaluation metrics may be selected, including but not limited to metrics such as factual consistency, answer relevance, and context relevance.

[0052] At block 408, a target user may be selected. In an AB experiment, the selection of a target user allows the experiment platform to define which user groups will participate in the experiment, also known as audience users. The filter criteria for the target user may be the device type. When conducting an AB experiment on users using Android devices, a filter criterion may be set in the configuration file of the experiment platform, specifying the target user as "device = Android", thereby ensuring that user requests of Android devices will be assigned to the experimental version of the dialogue system for evaluation.

[0053] Figure 5 FIG. 5 is a flow chart showing a data reporting process 500 according to an embodiment of the present disclosure. Figure 5 As shown, at box 502, the user can access the dialogue application. For example, the user can initiate a dialogue request through a smart phone, tablet computer, computer and other devices to access dialogue applications deployed on different platforms, such as chat robots, virtual assistants, intelligent customer service systems, etc. The user's access request can be submitted in the form of a mobile application, a PC application or a web page. The request contains the user's input query, and the system generates a response through the dialogue application and interacts with the user. At this time, the query content entered by the user and the answer output by the system will serve as the basis for subsequent evaluation and data analysis.

[0054] At block 504, user requests may be diverted. The experimental platform may distribute user requests to different versions of the dialog system according to a predetermined traffic distribution strategy. The diversion strategy may ensure that a user request is only distributed to a fixed version of the dialog system by hashing the user ID. For example, requests from user 1 and user 2 may only be diverted to version 1 506, requests from user 3 and user 4 may only be diverted to version 2 508, requests from user 5 and user 6 may only be diverted to version N 510, and so on.

[0055] At box 512, each version of the dialogue system can be evaluated online. For example, the experimental platform can score each version of the dialogue system using predefined evaluation indicators based on the query content in the user request and the response output of the corresponding version of the dialogue system. These evaluation indicators may include but are not limited to factual consistency, answer relevance, context relevance, etc. As mentioned above, the evaluation process can use a language model for automatic evaluation, and calculate the evaluation score for each version of the dialogue system for the specified indicator based on the user input and the dialogue system output. In some embodiments, the inter-group differences and significance of the evaluation scores can be calculated based on statistical methods. For example, the experimental platform can use statistical test methods (such as t-test, analysis of variance, etc.) to compare the evaluation scores of different versions of the dialogue system to determine whether there are significant performance differences between different versions. In this way, the reliability of the evaluation results can be ensured.

[0056] At box 514, the evaluation score can be displayed on the report page. For example, the experimental platform can generate a comprehensive report to show the scores of each version of the dialogue system under different evaluation indicators. The report page can include the scores of various indicators of each version of the dialogue system (such as factual consistency, answer relevance, context relevance, etc.), as well as the inter-group differences and significance results obtained through statistical analysis. Through data visualization, the report page can intuitively display the evaluation results, such as using bar charts, line charts, etc. to show the performance of each version on different indicators. In this way, developers can easily compare the performance of each version of the dialogue system, identify the best performing system strategy or configuration, and provide data support for subsequent optimization and decision-making.

[0057] At box 516, the winning experiment can be determined based on the evaluation score. For example, the experimental platform marks the experimental version with the best performance on the display page based on the evaluation scores of each version of the dialogue system. For example, the factual consistency of a dialogue system of a certain experimental version has the highest score, which can be highlighted in the chart. In addition, developers also need to consider the significance index of the evaluation score. In some embodiments, it can be based on a comprehensive score of multiple indicators. For example, the scores of each indicator can be weighted according to the set weights to ensure that the selection of the winning version does not rely solely on a single evaluation dimension, but takes into account the comprehensive performance of multiple dimensions. In some embodiments, developers can configure weights for different evaluation indicators to customize the comprehensive evaluation score according to needs. For example, if the developer believes that the factual consistency of the dialogue system is more important, the weight of the indicator can be increased. When a certain experimental version wins, its corresponding system strategy can be used as a new baseline strategy. It should be understood that it is also possible that the effects of all experimental versions are not as good as the baseline version, so the baseline strategy can continue to be used.

[0058] Figure 6 FIG. 6 is a block diagram of an apparatus 600 for evaluating a dialog system based on a language model according to some embodiments of the present disclosure. Figure 6 As shown, the apparatus 600 includes an access request acquisition module 602, which is configured to acquire multiple access requests for multiple versions of the dialogue system, and the multiple versions of the dialogue system include at least a first version of the dialogue system and a second version of the dialogue system. The apparatus 600 also includes an access request allocation module 604, which is configured to allocate the multiple access requests to the multiple versions of the dialogue system, and each access request in the multiple access requests corresponds to a version of the dialogue system in the multiple versions. In addition, the apparatus 600 also includes an evaluation score determination module 606, which is configured to determine the evaluation score for each dialogue system in the multiple versions of the dialogue system based on the query input in each access request in the multiple access requests, the response output of the corresponding version of the dialogue system, and the predefined evaluation index, using a language model.

[0059] Figure 7 A block diagram of an electronic device 700 is shown in accordance with certain embodiments of the present disclosure. Figure 7 FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a co-processor.

[0060] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0061] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0062] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0063] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0065] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0066] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0067] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0068] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0069] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for evaluating a dialogue system based on a language model, comprising: Acquire multiple access requests for multiple versions of the dialog system, where the multiple versions of the dialog system include at least a first version of the dialog system and a second version of the dialog system; assigning the plurality of access requests to the plurality of versions of the dialog system, each access request in the plurality of access requests corresponding to a version of the dialog system in the plurality of versions; as well as Based on the query input in each of the multiple access requests, the response output of the corresponding version of the dialogue system, and the predefined evaluation index, the language model is used to determine the evaluation score for each dialogue system in the multiple versions of the dialogue system.

2. The method according to claim 1, wherein determining the evaluation score for each of the multiple versions of the dialogue systems using the language model comprises: Based on the predefined evaluation index, determining a prompt word corresponding to the evaluation index; as well as Based on the prompt words, the evaluation score for each dialogue system is determined using the language model.

3. The method of claim 1, wherein the predefined evaluation metric comprises a plurality of evaluation metrics, and determining the evaluation score comprises: Based on the multiple evaluation indicators, determining multiple prompt words corresponding to the multiple evaluation indicators; Based on the multiple prompt words, determine multiple evaluation scores for each dialogue system using the language model; as well as Based on the multiple evaluation scores, a comprehensive score for each dialog system is determined.

4. The method according to claim 3, wherein the evaluation indicators include one or more of the following: fact consistency, answer relevance, context relevance, context recall, context precision, answer correctness, or answer similarity.

5. The method according to claim 1, wherein the first version of the dialogue system is a baseline version of the dialogue system and the second version of the dialogue system is an experimental version of the dialogue system, and the method further comprises: Acquire an experimental strategy for creating the experimental version of the dialog system, wherein the experimental strategy is used to configure the experimental version of the dialog system, and the baseline version of the dialog system is configured using the baseline strategy; Based on the predetermined traffic setting, setting a traffic allocation ratio for the multiple versions of the dialogue system; as well as Select one or more evaluation metrics for multiple versions of the dialogue system.

6. The method according to claim 5, further comprising: For the one or more evaluation indicators, obtaining corresponding training data; as well as The language model is fine-tuned based on the training data.

7. The method according to claim 5, further comprising: Based on the screening condition, target users for the multiple versions of the dialog system are determined.

8. The method of claim 5, wherein the experimental strategy is different from the baseline strategy and the experimental strategy specifies at least one of the following: The dialogue model called by the dialogue system; System prompts for the dialogue system; or A library of query vectors for dialogue systems.

9. The method according to claim 5, further comprising: Based on the evaluation score of each dialogue system in the multiple versions of the dialogue systems, determining a mean and a variance of the evaluation score for each dialogue system; as well as Based on the mean and variance of the evaluation scores for each dialogue system, a score significance for each dialogue system is determined.

10. The method according to claim 9, further comprising: The evaluation score and the score significance of each dialog system in the multiple versions of the dialog systems are displayed.

11. The method according to claim 10, further comprising: Determining a winning strategy among the benchmark strategy and the experimental strategy based on the evaluation score and the score significance; as well as The benchmark version of the dialogue system is configured using the winning strategy.

12. An apparatus for evaluating a dialogue system based on a language model, comprising: An access request acquisition module, configured to acquire multiple access requests for multiple versions of the dialogue system, wherein the multiple versions of the dialogue system include at least a first version of the dialogue system and a second version of the dialogue system; an access request allocation module configured to allocate the plurality of access requests to the plurality of versions of the dialog system, each of the plurality of access requests corresponding to a version of the dialog system among the plurality of versions; as well as The evaluation score determination module is configured to determine the evaluation score for each dialogue system in the multiple versions of the dialogue system using a language model based on the query input in each access request in the multiple access requests, the response output of the corresponding version of the dialogue system, and the predefined evaluation indicators.

13. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 11.

14. A computer program product tangibly stored on a non-transitory computer readable medium and comprising computer executable instructions for performing the method according to any one of claims 1 to 11.

Images