A testing method and device for multi-round dialogue capability of a large model

Abstract

This specification discloses a testing method and device for the multi-round dialogue capability of a large model. Obtain the topic question text, and determine the number of dialogue rounds required to test the model to be tested. According to the number of dialogue rounds, perform multi-round input operations for the model to be tested, wherein, for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested, and the answer text output by the model to be tested in the round of dialogue is obtained, and the answer text output by the model to be tested in the round of dialogue and the preset prompt sentence are input into the preset auxiliary model, so as to obtain the input data input into the model to be tested by the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the topic question text. The input data corresponding to each round of dialogue and the preset evaluation text are input into the preset evaluation model to obtain the test results for the model to be tested, and the evaluation text records the evaluation rules for evaluating the model to be tested.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a method and device for testing the multi-round dialogue capability of a large model. Background Art

[0002] At present, with the rapid development of artificial intelligence, the large language models trained in this way have achieved great success in the fields of virtual assistants, customer service systems, and chatbots. For example, virtual assistants with built-in large language models can provide services and assistance to users by simulating natural language communication.

[0003] However, the huge success is also accompanied by a problem: although many large language models are designed with the ability of multiple rounds of dialogue into consideration and can maintain a certain dialogue history, as the number of dialogue rounds increases, the large language model may gradually forget the previous dialogue content, resulting in incoherent answers or loss of previously discussed details, or even errors.

[0004] Therefore, in order to ensure the multi-round conversation capability of the trained large language model and provide better services to users, the multi-round conversation capability of the large language model needs to be tested and evaluated before it is deployed online.

[0005] In the prior art, the multi-round dialogue capability of a large language model is usually tested and evaluated manually, which consumes huge manpower costs and is inefficient.

[0006] To this end, this specification provides a method and device for testing the multi-round dialogue capability of a large model. Summary of the invention

[0007] This specification provides a method and device for testing the multi-round dialogue capability of a large model to partially solve the above-mentioned problems existing in the prior art.

[0008] This manual adopts the following technical solutions:

[0009] This manual provides a method for testing the multi-round dialogue capability of a large model, including:

[0010] Get the topic question text and determine the number of conversation turns required to test the model under test;

[0011] According to the number of dialogue rounds, multiple rounds of input operations are performed for the model to be tested, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested, and the answer text output by the model to be tested in the round of dialogue is obtained, and the answer text output by the model to be tested in the round of dialogue and the preset prompt sentence are input into the preset auxiliary model, so as to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the topic question text;

[0012] The input data corresponding to each round of dialogue and the preset evaluation text are input into the preset evaluation model to obtain the test result for the model to be tested, and the evaluation text records the evaluation rules for evaluating the model to be tested.

[0013] Optionally, the method further includes:

[0014] The test result is returned to the terminal device used by the designated user, so that when the designated user determines that the model to be tested needs to be adjusted based on the test result displayed by the terminal device, the designated user performs an adjustment operation in the terminal device to adjust the model to be tested.

[0015] Optionally, the method further includes:

[0016] According to the test results, construct a test case corresponding to the subject question text and save it;

[0017] The method further comprises:

[0018] The model to be tested is tested according to the saved test cases.

[0019] Optionally, the model to be tested is tested according to the saved test case, specifically including:

[0020] Reading a saved test case, and generating a test case fine-tuning instruction according to the saved test case;

[0021] Parsing the test case according to the use case fine-tuning instruction to parse out a question text contained in the test case as a target question text, and identifying keywords contained in the target question text;

[0022] Determine synonyms corresponding to the keywords from a preset text library, and replace the keywords contained in the target question text according to the synonyms to obtain a replacement question text;

[0023] The model to be tested is retested according to the replaced question text.

[0024] Optionally, retesting the model to be tested according to the replaced question text specifically includes:

[0025] Determine the number of dialogue rounds corresponding to the saved test case as the target number of rounds;

[0026] According to the target number of rounds, multiple rounds of input operations are performed for the model to be tested, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and a preset prompt sentence are input into a preset auxiliary model to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the replacement question text;

[0027] The answer text output by the model to be tested in each round obtained by executing multiple rounds of dialogue for the target number of rounds is compared with the answer text output by each round corresponding to the saved test case to obtain a comparison result, so as to determine the test result for the model to be tested based on the comparison result.

[0028] Optionally, the method further includes:

[0029] According to the test result, an interference question text is generated for the subject question text, wherein the content to be asked by the interference question text is the same as that of the subject question text, but the corresponding subject content in the subject question text is obscured;

[0030] Inputting the subject question text into the model to be tested to obtain the answer text of the model to be tested for the subject question text, and inputting the interference question text into the dialog window in which the model to be tested outputs the answer text for the subject question text to obtain the answer text for the interference question text output by the model to be tested when the question and answer of the subject question text has been completed;

[0031] The test result of the model to be tested is obtained according to the answer text of the interference question text.

[0032] Optionally, the method further includes:

[0033] According to the test result, the number of dialogue rounds required for retesting the model to be tested is adjusted to obtain an adjusted number of dialogue rounds;

[0034] The model to be tested is retested according to the adjusted number of dialogue rounds.

[0035] This specification provides a test device for the multi-round dialogue capability of a large model, including:

[0036] An acquisition module is used to obtain the topic question text and determine the number of conversation turns required to test the model under test;

[0037] A dialogue module, configured to perform multiple rounds of input operations for the model to be tested according to the number of dialogue rounds, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and a preset prompt sentence are input into a preset auxiliary model to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the subject question text;

[0038] The evaluation module is used to input the input data corresponding to each round of dialogue and the preset evaluation text into the preset evaluation model to obtain the test result for the model to be tested. The evaluation text records the evaluation rules for evaluating the model to be tested.

[0039] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned testing method for the multi-round dialogue capability of a large model.

[0040] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for testing the multi-round dialogue capability of a large model when executing the program.

[0041] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0042] The test method for the multi-round dialogue capability of a large model provided in this specification first obtains the subject question text and determines the number of dialogue rounds required to test the model to be tested. According to the number of dialogue rounds, a multi-round input operation for the model to be tested is performed, wherein, for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested, and the answer text output by the model to be tested in the round of dialogue is obtained, and the answer text output by the model to be tested in the round of dialogue and the preset prompt sentence are input into the preset auxiliary model, so as to obtain the input data input into the model to be tested by the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the subject question text. The input data corresponding to each round of dialogue and the preset evaluation text are input into the preset evaluation model to obtain the test results for the model to be tested, and the evaluation text records the evaluation rules for evaluating the model to be tested.

[0043] Through the answer text output by the model to be tested, the auxiliary model generates the input data for the next round of dialogue, and then the data generated by the auxiliary model is input into the model to be tested to obtain the answer text. After multiple rounds of cycles, the content of multiple rounds of dialogue based on the first model is obtained. In this way, according to the content of the multiple rounds of dialogue of the model to be tested, the multiple round dialogue ability of the model to be tested can be accurately evaluated through the evaluation model and the preset scoring text. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0045] Figure 1 A flowchart of a method for testing the multi-round dialogue capability of a large model provided in an embodiment of this specification;

[0046] Figure 2 A schematic diagram of a multi-round dialogue provided in this specification;

[0047] Figure 3 A comparison diagram of the replacement subject question text provided for this specification;

[0048] Figure 4 A schematic diagram of a test device for multi-round dialogue capability of a large model provided in this specification;

[0049] Figure 5 A method corresponding to the Figure 1 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0052] Figure 1 A flowchart of a method for testing the multi-round dialogue capability of a large model provided in an embodiment of this specification includes the following steps:

[0053] S100: Obtain the topic question text and determine the number of conversation rounds required to test the model to be tested.

[0054] In the present specification, the process of testing the multi-round dialogue capability of the large model involves the processing of text data. In the embodiments of the present specification, the test process of the multi-round dialogue capability of the large model can be performed by a server. Of course, the present specification does not limit which device is used to perform the test process of the multi-round dialogue capability of the large model. For example, a personal computer and a mobile terminal can also be used to test the multi-round dialogue capability of the large model. For the convenience of description, the following description is based on the server as the execution subject.

[0055] In one or more embodiments of the present specification, the server may determine a topic for starting a multi-round conversation from a preset topic set, wherein the preset topic set may include a variety of topics that a user may start or a fixed topic question template.

[0056] When the server determines a topic from a preset topic set, it can derive a topic question text related to the topic based on the determined topic. The derivation method can be to input the topic and the prompt words related to the derivation into the pre-trained universal big model. For example, the prompt words related to the derivation can be "Based on the input topic, ask a reasonable Chinese question and directly return the question content without any unnecessary explanation, and only provide a Chinese question." The pre-trained universal big model can provide a Chinese question based on the prompt words. For example, if the topic is "global warming", the pre-trained universal big model may provide a topic question text of "Please introduce the impact of global warming."

[0057] Of course, the server can also select a fixed theme question template included in a preset theme set as the theme question text, for example, the fixed theme question template is "Please introduce the impact of global warming" or "Please introduce your favorite literary work" and so on.

[0058] In this specification, the requirement for the number of dialogue rounds is as many as possible. The current large model can bring a good experience to users when conducting dialogues with users within two rounds. However, when the number of rounds gradually increases to ten rounds or more, the large model can no longer bring users a good dialogue question and answer experience. The answers given by the large model may be repeated or the context is not fully understood. Therefore, it is necessary to test the multi-round dialogue capability of the large model to ensure that it can bring users a good user experience after the large model is launched. Furthermore, after determining the topic question text, the server can determine the number of dialogue rounds required to test the multi-round dialogue capability of the model to be tested.

[0059] S102: According to the number of dialogue rounds, perform multiple rounds of input operations for the model to be tested, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and a preset prompt sentence are input into a preset auxiliary model to obtain the input data for the next round of dialogue input into the model to be tested through the auxiliary model. When the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the topic question text.

[0060] In one or more embodiments of the present specification, after the server determines the subject question text, it can start simulating multiple rounds of conversations with the user based on the subject question text, and execute multiple rounds of input operations of the model to be tested based on the determined number of conversation rounds.

[0061] The specific process of multiple rounds of input operations is that for each round of operations, the server inputs the input data corresponding to that round of dialogue into the model to be tested, obtains the answer text output by the model to be tested in that round of dialogue, and inputs the answer text output by the model to be tested in that round of dialogue and the preset prompt sentence into the preset auxiliary model, so as to obtain the input data input into the model to be tested for the next round of dialogue through the auxiliary model. When this round of dialogue is the first round of dialogue, the input data corresponding to this round of dialogue is the topic question text.

[0062] The preset prompt sentence of the server input auxiliary model can be "Please ask a new reasonable Chinese question based on the historical question content, and directly return the question content without any extra explanation." The prompt sentence is used to guide the auxiliary model so as to determine the question text (i.e., input data) required for the next round of dialogue.

[0063] Figure 2 This is a schematic diagram of a multi-round dialogue provided in this specification. Figure 2 As shown in the figure, in the model to be tested shown on the left, the subject question text is input, and the answer text given by the model to be tested for the subject question text is obtained. Then the subject question text and the answer text for the subject question text are input into the auxiliary model together, so that the auxiliary model outputs the input data for the next round of dialogue according to the preset prompt sentence. The input data is input into the model to be tested, and the model to be tested outputs the answer text for the input data, and this cycle is repeated to complete multiple rounds of dialogue.

[0064] S104: Inputting the input data corresponding to each round of dialogue, the answer text corresponding to each round of dialogue, and the preset evaluation text into a preset evaluation model to obtain a test result for the model to be tested, wherein the evaluation text records the evaluation rules for evaluating the model to be tested.

[0065] In one or more embodiments of the present specification, when the number of conversation rounds between the server and the model to be tested reaches a preset number of conversation rounds, the server can evaluate the multi-round conversation capability of the model to be tested based on the question text and answer text in each round of conversation, and determine the multi-round conversation capability of the model to be tested.

[0066] Specifically, the server inputs the input data corresponding to each round of dialogue, the answer text corresponding to each round of dialogue, and the preset evaluation text into the preset evaluation model to obtain the test results for the model to be tested. The evaluation text records the evaluation rules for evaluating the model to be tested.

[0067] Among them, the preset evaluation text may include scoring rules for multiple dimensions related to the ability of turn-by-turn dialogue, and the multiple dimensions include at least dialogue fluency, context understanding, response accuracy, diversity, innovation, error rate, and user interaction. For example, the specific content of the evaluation text may be: "Role setting: You are an expert, proficient in the use of various large language models, have rich practical experience, and are good at comprehensively summarizing and summarizing the use scenarios and functions of the models. Roles can be modified.

[0068] The skills you have are as follows:

[0069] Skill 1: Overall scoring: 1) Comprehensively evaluate the performance of the model and give an overall score of 1-10; 2) When scoring, comprehensively consider dimensions such as dialogue fluency, context understanding, response accuracy, diversity and innovation, error rate, and user interaction.

[0070] Skill 2: Dimension scoring: 1) For the dimension of conversation fluency, a score of 1-10 is given based on the natural coherence of the conversation and the consistency of the context logic; 2) For the dimension of context understanding, a score of 1-10 is given based on the model's correct understanding and memory of the conversation context information, as well as its consistent performance in multiple rounds of conversations; 3) For the dimension of response accuracy, a score of 1-10 is given based on the accuracy and relevance of the model's answers, whether it can correctly respond to questions and provide appropriate suggestions; 4) For the dimension of diversity and innovation, a score of 1-10 is given based on the diversity and richness of the answers, the avoidance of repetitive and monotonous situations, and whether it can provide innovative and interesting responses; 5) For the dimension of error rate, a score of 1-10 is given based on the frequency of grammatical, logical and other errors in the model in multiple rounds of conversations; 6) For the dimension of user interaction, a score of 1-10 is given based on the model's ability to guide user interaction and keep the conversation active. Dimension scoring can be modified.

[0071] Restrictions: 1) All scoring must be done strictly according to the given criteria and scoring range; 2) The output content must be organized according to the given format and must not deviate. Restrictions support modification. ".

[0072] According to the scoring rules in the above evaluation text, the server can use the scoring text as a prompt statement for the evaluation model to guide the evaluation model to score the multi-round dialogue capability of the model to be tested.

[0073] Afterwards, the server can determine the test results based on the score of the multi-round dialogue capability of the model to be tested, and judge whether the model to be tested can be deployed online to meet the user's good multi-round dialogue experience.

[0074] In addition, in one or more embodiments of the present specification, after the server obtains the test results output by the evaluation model, it can return the test results to the terminal device used by the specified user, so that when the specified user determines that the model to be tested needs to be adjusted based on the test results displayed by the terminal device, the specified user can perform adjustment operations in the terminal device to adjust the model to be tested.

[0075] In one or more embodiments of the present specification, the server may also construct and save a test case corresponding to the subject question text based on the test results.

[0076] Among them, the server can filter and construct test cases according to the scores of each dimension in the test results. The way to filter and construct test cases can be that the server can use the input data corresponding to each round of dialogue and the answer text corresponding to each round of dialogue in the multiple rounds of dialogue caused by the theme question text as the content of the multiple rounds of dialogue corresponding to the theme question text. After the content of the multiple rounds of dialogue of the theme question text and the evaluation text are input into the evaluation model, the test results corresponding to the theme question text of the model to be tested are obtained. According to the preset thresholds of each dimension, it is judged whether the scores of each dimension in the test results corresponding to the theme question text output by the evaluation model have reached the threshold. If so, the content of the multiple rounds of dialogue corresponding to this theme question text is used as a successful case. If not, the content of the multiple rounds of dialogue corresponding to the theme question text is used as a failed case. Of course, both successful cases and failed cases are test cases.

[0077] Afterwards, after the designated user has adjusted the model to be tested, the saved test case can be used to test the adjusted model to determine the multi-round dialogue capability of the adjusted model to be tested. Of course, the saved test case can also be used to test the model to be tested that has not been adjusted by the designated user. Not only can the context understanding capability of the model to be tested be further determined through the question text that has been asked in the test case, but also the test results can be compared with the adjusted model to quantitatively analyze whether the adjustment operation of the designated user is effective and in which dimensions there is improvement or regression.

[0078] Furthermore, in order to conduct a more comprehensive test on the capabilities of the model to be tested to ensure the performance of the model to be tested, in this specification, the model to be tested can be tested through a saved test case. The specific process can be that the server reads the saved test case and generates a case fine-tuning instruction based on the saved test case. Then, the test case is parsed according to the case fine-tuning instruction to parse out the question text contained in the test case as the target question text, and to identify the keywords contained in the target question text. Among them, since the test case contains the content of multiple rounds of dialogue, each round of dialogue has a question text and an answer text, and the target question text can be one of multiple question texts, such as a topic question text.

[0079] Afterwards, the server determines the synonyms corresponding to the keywords contained in the target question text from the preset text library, and replaces the keywords contained in the target question text according to the synonyms to obtain a replacement question text with the same or similar semantics as the target question text.

[0080] Finally, the server can retest the model to be tested based on the replaced question text. Alternatively, the server generates a corresponding replacement question text for each question text in the read test case, and then replaces each question text in the read test case with the replacement question text to regenerate the test case. And according to the regenerated test case, each replacement question text in the regenerated test case is input into the model to be tested in turn to obtain the answer text corresponding to each replacement question text. The answer text corresponding to each replacement question text is compared with each answer text in the test case initially read by the server to determine the test result of the model to be tested.

[0081] Specifically, the process of the server retesting the model to be tested according to the replacement question text is that the server determines the number of dialogue rounds corresponding to the test case read and saved as the target number of rounds. Then, according to the target number of rounds, multiple rounds of input operations for the model to be tested are performed, wherein, for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and the preset prompt sentence are input into the preset auxiliary model, so as to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the replacement question text. Finally, the server compares the answer text output by the model to be tested in each round obtained by executing the multiple rounds of dialogue with the answer text output in each round corresponding to the saved test case to obtain the comparison result, and determines the test result for the model to be tested according to the comparison result.

[0082] By comparing the answers of the model to be tested before and after the replacement of the same question text, the accuracy and stability of the model to be tested in understanding questions and responses can be accurately measured. In addition, by checking whether the answers of the model to be tested are consistent in different texts but with the same semantics, it is ensured that it can stably provide high-quality services or information. If the model to be tested gives completely different answers to similar inputs, it may indicate that the model to be tested has certain unstable factors and needs to be adjusted and improved.

[0083] Figure 3 This is a comparison diagram of the replacement subject question text provided in this specification. Figure 3 As shown, the subject question text in the test case is replaced with a synonym to obtain a replacement question text, and the question text is replaced with this, and multiple rounds of dialogue are conducted through the model to be tested to obtain multiple rounds of dialogue content based on the replacement question text. Finally, the answer text corresponding to the subject question text and the answer text corresponding to the replacement question text, and the answer text corresponding to input data 1 and input data 1' are compared until the last pair of input data, and the test result of the model to be tested can be determined.

[0084] In one or more embodiments of the present specification, in daily conversations with the big model, a situation may occur where, after the user opens the window for conversation with the big model, he asks the big model a question, and the big model then gives a targeted answer based on the question. After the big model answers the question, the user can continue to ask questions in the current window. When the user continues to ask questions with roughly the same text as the previous question, but does not clearly indicate the object to which the question corresponds, the answer given by the big model may be targeted at the object to which the previous question corresponds, so that the content texts of the two answers are extremely similar. Even if the accompanying materials for reference by the big model are different when the two questions are asked, the answers to the two questions will be the same, which cannot meet the needs of the user and reflects the limitations of the big model.

[0085] Therefore, the server can generate interference question texts for the theme question text according to the evaluation results output by the evaluation model. Among them, the content to be asked in the interference question text is the same as that in the theme question text, but the corresponding subject content in the theme question text is blurred. For example, the theme question text is "Please introduce the impact of global warming on this island country, and the attached materials are the historical sea level data of island country A." The interference question text is "Please introduce the impact of global warming on this island country, and the attached materials are the historical sea level data of island country B." The interference question text has the same content as the theme question text, but the subject is different. The subject in the theme question text is island country A, and the subject in the interference question text is island country B.

[0086] Afterwards, the server inputs the subject question text into the model to be tested, obtains the answer text of the model to be tested for the subject question text, and inputs the interference question text into the dialogue window where the model to be tested outputs the answer text for the subject question text, and obtains the answer text for the interference question text output by the model to be tested when the question and answer of the subject question text has been completed. Among them, the dialogue window for inputting the subject question text and the interference question text is different from the dialogue window for conducting multiple rounds of dialogue for the subject question text. In order to avoid the influence of the content of the multiple rounds of dialogue in the dialogue window of the multiple rounds of dialogue, the dialogue window for inputting the subject question text and the interference question text is a newly opened dialogue window.

[0087] Finally, the server can obtain the test results of the model to be tested based on the answer text of the interference question text. If the answer text of the interference question text is the same or similar to the answer text of the subject question text, then the test results of the model to be tested are obviously not good and are affected by the subject question text.

[0088] In one or more embodiments of the present specification, after obtaining the test results for the model to be tested, the server may adjust the number of dialogue rounds required for retesting the model to be tested according to the test results to obtain the adjusted number of dialogue rounds, and then retest the model to be tested according to the adjusted number of dialogue rounds.

[0089] In this way, the multi-round dialogue capability of the model to be tested can be evaluated more accurately. For example, after ten rounds of dialogue, if it is found that the answer text is repeated, the number of dialogue rounds can be gradually reduced to determine the number of rounds after which repetition occurs and determine the test results.

[0090] In one or more embodiments of the present specification, the model to be tested, the auxiliary model and the evaluation model may be a Generative Pre-Trained Transformer (GPT) model, such as GPT3, GPT3.5 and GPT4 models, which can be used in this solution. There is no restriction on the GPT model version, which can be set according to actual conditions.

[0091] The above is a test method for the multi-round dialogue capability of a large model provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding test device for the multi-round dialogue capability of a large model, such as Figure 4 shown.

[0092] Figure 4 A schematic diagram of a test device for multi-round dialogue capability of a large model provided in this specification, specifically including:

[0093] An acquisition module 400 is used to acquire the subject question text and determine the number of dialogue turns required to test the model to be tested;

[0094] The dialogue module 402 is used to perform multiple rounds of input operations for the model to be tested according to the number of dialogue rounds, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and the preset prompt sentence are input into the preset auxiliary model to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model. When the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the topic question text;

[0095] The evaluation module 404 is used to input the input data corresponding to each round of dialogue and the preset evaluation text into the preset evaluation model to obtain the test result for the model to be tested. The evaluation text records the evaluation rules for evaluating the model to be tested.

[0096] Optionally, the device further includes an adjustment module 406;

[0097] The adjustment module 406 is used to return the test results to the terminal device used by the designated user, so that when the designated user determines that the model to be tested needs to be adjusted based on the test results displayed by the terminal device, the designated user performs an adjustment operation in the terminal device to adjust the model to be tested.

[0098] Optionally, the device further includes a retest module 408;

[0099] The retest module 408 is used to construct a test case corresponding to the subject question text according to the test result, save the test case, and test the model to be tested according to the saved test case.

[0100] Optionally, the retest module 408 is used to read a saved test case, and generate a case fine-tuning instruction based on the saved test case, parse the test case according to the case fine-tuning instruction to parse out the question text contained in the test case as the target question text, and identify the keywords contained in the target question text, determine the synonyms corresponding to the keywords from a preset text library, and replace the keywords contained in the target question text according to the synonyms to obtain a replacement question text, and retest the model to be tested according to the replacement question text.

[0101] Optionally, the retest module 408 is used to determine the number of dialogue rounds corresponding to the saved test case as the target number of rounds, and perform multiple rounds of input operations for the model to be tested according to the target number of rounds, wherein, for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and a preset prompt statement are input into a preset auxiliary model to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the replacement question text, and the answer text output by the model to be tested in each round obtained by executing multiple rounds of dialogue with the target number of rounds is compared with the answer text output in each round corresponding to the saved test case to obtain a comparison result, so as to determine the test result for the model to be tested according to the comparison result.

[0102] Optionally, the device further includes an interference module 410;

[0103] The interference module 410 is used to generate an interference question text for the subject question text according to the test result, wherein the content to be asked in the interference question text is the same as that of the subject question text, but the corresponding subject content in the subject question text is blurred, the subject question text is input into the model to be tested, and an answer text of the model to be tested for the subject question text is obtained, and the interference question text is input into the dialog window in which the model to be tested outputs the answer text for the subject question text, and the answer text for the interference question text output by the model to be tested when the question and answer of the subject question text is completed is obtained, and the test result of the model to be tested is obtained according to the answer text of the interference question text.

[0104] Optionally, the device further includes a round number adjustment module 412;

[0105] The round number adjustment module 412 is used to adjust the number of dialogue rounds required for retesting the model to be tested according to the test result to obtain an adjusted number of dialogue rounds, and retest the model to be tested according to the adjusted number of dialogue rounds.

[0106] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A testing method for the multi-round dialogue capability of a large model is provided.

[0107] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The testing method for the multi-round dialogue capability of a large model.

[0108] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0109] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0110] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0111] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0112] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0113] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0118] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0119] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0121] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0123] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for testing the multi-round dialogue capability of a large model, characterized in that: include: Get the topic question text and determine the number of conversation turns required to test the model under test; According to the number of dialogue rounds, multiple rounds of input operations are performed for the model to be tested, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested, and the answer text output by the model to be tested in the round of dialogue is obtained, and the answer text output by the model to be tested in the round of dialogue and the preset prompt sentence are input into the preset auxiliary model, so as to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the topic question text; Inputting the input data corresponding to each round of dialogue, the answer text corresponding to each round of dialogue, and the preset evaluation text into a preset evaluation model to obtain a test result for the model to be tested, wherein the evaluation text records the evaluation rules for evaluating the model to be tested; According to the test result, an interference question text is generated for the subject question text, wherein the content to be asked by the interference question text is the same as that of the subject question text, but the corresponding subject content in the subject question text is obscured; Inputting the subject question text into the model to be tested to obtain the answer text of the model to be tested for the subject question text, and inputting the interference question text into the dialog window in which the model to be tested outputs the answer text for the subject question text to obtain the answer text for the interference question text output by the model to be tested when the question and answer of the subject question text has been completed; The test result of the model to be tested is obtained according to the answer text of the interference question text.

2. The method according to claim 1, characterized in that The method further comprises: The test result is returned to the terminal device used by the designated user, so that when the designated user determines that the model to be tested needs to be adjusted based on the test result displayed by the terminal device, the designated user performs an adjustment operation in the terminal device to adjust the model to be tested.

3. The method according to claim 1, characterized in that The method further comprises: According to the test results, construct a test case corresponding to the subject question text and save it; The method further comprises: The model to be tested is tested according to the saved test cases.

4. The method according to claim 3, characterized in that According to the saved test cases, the model to be tested is tested, specifically including: Reading a saved test case, and generating a test case fine-tuning instruction according to the saved test case; Parsing the test case according to the use case fine-tuning instruction to parse out a question text contained in the test case as a target question text, and identifying keywords contained in the target question text; Determine synonyms corresponding to the keywords from a preset text library, and replace the keywords contained in the target question text according to the synonyms to obtain a replacement question text; The model to be tested is retested according to the replaced question text.

5. The method according to claim 4, characterized in that Retesting the model to be tested according to the replaced question text, specifically including: Determine the number of dialogue rounds corresponding to the saved test case as the target number of rounds; According to the target number of rounds, multiple rounds of input operations are performed for the model to be tested, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and a preset prompt sentence are input into a preset auxiliary model to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the replacement question text; The answer text output by the model to be tested in each round obtained by executing multiple rounds of dialogue for the target number of rounds is compared with the answer text output by each round corresponding to the saved test case to obtain a comparison result, so as to determine the test result for the model to be tested based on the comparison result.

6. The method according to claim 1, characterized in that The method further comprises: According to the test result, the number of dialogue rounds required for retesting the model to be tested is adjusted to obtain an adjusted number of dialogue rounds; The model to be tested is retested according to the adjusted number of dialogue rounds.

7. A testing device for multi-round dialogue capability of a large model, characterized in that: include: An acquisition module is used to obtain the topic question text and determine the number of conversation turns required to test the model under test; A dialogue module, configured to perform multiple rounds of input operations for the model to be tested according to the number of dialogue rounds, wherein for each round of dialogue, the input data corresponding to the round of dialogue is input into the model to be tested to obtain the answer text output by the model to be tested in the round of dialogue, and the answer text output by the model to be tested in the round of dialogue and a preset prompt sentence are input into a preset auxiliary model to obtain the input data input into the model to be tested in the next round of dialogue through the auxiliary model, and when the round of dialogue is the first round of dialogue, the input data corresponding to the round of dialogue is the subject question text; An evaluation module, used to input the input data corresponding to each round of dialogue and a preset evaluation text into a preset evaluation model to obtain a test result for the model to be tested, wherein the evaluation text records the evaluation rules for evaluating the model to be tested; An interference module is used to generate an interference question text for the subject question text according to the test result, wherein the content to be asked in the interference question text is the same as that in the subject question text, but the corresponding subject content in the subject question text is blurred, the subject question text is input into the model to be tested, and an answer text of the model to be tested for the subject question text is obtained, and the interference question text is input into a dialog window in which the model to be tested outputs the answer text for the subject question text, and an answer text for the interference question text is output by the model to be tested when the question and answer of the subject question text is completed, and a test result of the model to be tested is obtained according to the answer text of the interference question text.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Images