Method and device for model evaluation, equipment and storage medium
By performing multiple rounds of dialogue interactions between machine learning models, collecting interaction information and evaluating performance levels, the problem of difficulty in effectively evaluating model performance in the prior art is solved, and automated and efficient model evaluation is achieved.
Patent Information
- Application Number
- CN202311433037.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-31
- Publication Date
- 2025-05-02
AI Technical Summary
It is difficult for the prior art to effectively evaluate the performance of machine learning models, especially in multi-round dialogue interaction scenarios. Traditional methods rely on manual evaluation and are costly, and it is difficult to construct a fixed data set to evaluate model performance.
By controlling at least two models to perform multiple rounds of dialogue interaction, interactive information for multiple rounds of dialogue is collected, and the performance level of the model is determined based on these interaction information.
Automatic model evaluation is realized, reducing the cost of obtaining evaluation data sets, improving the quality and efficiency of evaluation data generation, and being able to more accurately and comprehensively evaluate model dialogue performance.
Smart Images
Figure CN119918691A_ABST
Abstract
Description
Technical Field
[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly to methods, devices, apparatuses, and computer-readable storage media for model evaluation. Background Art
[0002] Machine learning and deep learning technologies have been widely used in many fields. Machine learning models can be configured and trained to generate corresponding model outputs based on given model inputs. Machine learning models can be applied to natural language processing, machine translation, speech synthesis, image generation and other fields, and in the future they can be applied to more fields, such as medical care, finance, education, etc. After model training, it is usually expected to accurately evaluate the performance of the model to determine the training effect, subsequent application methods, etc. Summary of the invention
[0003] In a first aspect of the present disclosure, a method for model evaluation is provided. The method includes: controlling at least a first model and a second model to perform multiple rounds of dialogue interaction, in each round of dialogue interaction, using one of the first model and the second model to provide a first dialogue message to the other model, and using the other model to generate a second dialogue message based on at least the first dialogue message; obtaining interaction information in the multiple rounds of interaction, the interaction information including multiple dialogue messages in the multiple rounds of interaction; and determining the performance level of at least one of the first model and the second model based on the interaction information.
[0004] In a second aspect of the present disclosure, a device for model evaluation is provided. The device includes: a dialogue control module, configured to control at least a first model and a second model to perform multiple rounds of dialogue interaction, in each round of dialogue interaction, using one of the first model and the second model to provide a first dialogue message to the other model, and using the other model to generate a second dialogue message based on at least the first dialogue message; an information acquisition module, configured to obtain interaction information in multiple rounds of interaction, the interaction information including multiple dialogue messages in the multiple rounds of interaction; and a performance determination module, configured to determine the performance level of at least one of the first model and the second model based on the interaction information.
[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.
[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.
[0007] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;
[0010] Figure 2 A block diagram of a model evaluation system according to some embodiments of the present disclosure is shown;
[0011] Figure 3 shows a multi-round dialogue interaction process between models according to some embodiments of the present disclosure;
[0012] Figure 4 A flowchart showing a process for model evaluation according to some embodiments of the present disclosure is shown;
[0013] Figure 5 A block diagram showing an apparatus for model evaluation according to some embodiments of the present disclosure; and
[0014] Figure 6 An electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION
[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.
[0017] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0018] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0019] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.
[0020] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0021] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0022] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.
[0023] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application stage, the trained model can be used to process the actual model input based on the parameter values obtained through training to determine the corresponding model output.
[0024] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, a model training system 110 is configured to perform training of a model 105 using a training data set 112. The model 105 can be configured with a corresponding model structure based on the task to be processed.
[0025] Before training, the model 105 may have initial parameter values. The training process is to update the parameter values of the model 105 to the expected values based on the training data. During the training process, one or more training tasks may be designed. The training tasks are used to help update the parameters of the model 105. In the training phase 102, the model 105 may learn a strong generalization capability through large-scale training data. After the training is completed, the parameter values of the model 105 have been updated and have the trained parameter values.
[0026] In the environment 100, the model evaluation system 120 is configured to perform a model performance evaluation on the trained model 105 to determine the quality of the model output. When performing the model evaluation, the model evaluation system 120 uses an evaluation data set 122 to evaluate the performance of the model 105. Generally, the evaluation data set 122 includes at least the input and output of the model 105. The test set used in the model evaluation scheme proposed in the embodiment of the present disclosure will be described in detail below.
[0027] Model 105 is configured to provide corresponding output 124 by processing input 122. Depending on the model configuration, model 105 input 122 may include content in text mode, and output 124 is also content in text mode. For example, model 105 can be based on a language model and can be used as a natural language processing model. Such a model can generate a text sequence of output based on a text sequence of input. In some example embodiments below, for the purpose of discussion, a text generation model is used as an example for explanation.
[0028] In some embodiments, the model 105 input 122 may also additionally or alternatively include other modalities, such as image modality, video modality, audio modality, etc., and the output 124 may also include other models other than the text modality, such as image modality, video modality, audio modality. For example, the model 105 may be an image generation model that can generate new images based on input text and / or images. The model 105 may also be a speech synthesis model that can generate speech based on input text. The model 105 may be any appropriate unimodal or multimodal content generation model.
[0029] In some embodiments, a question-answering system may be constructed based on the model 105 , where the model input may be considered as a question, and the model output may be considered as an answer generated by the model 105 to the input question.
[0030] In some embodiments, model 105 may be a generative model. A generative model refers to a content generation technology implemented by a machine learning model. A generative model can automatically generate meaningful and coherent content based on input content, including text content or content in other modalities such as images, audio, and video. A language model refers to a model trained on the basis of a large-scale corpus and having a certain understanding and generation capability for a language. For text content generation, it mainly relies on a language model to execute. Driven by large-scale pre-trained models, the generation quality and speed of generative models have become increasingly satisfactory.
[0031] In environment 100, the model training system 110 and the model evaluation system 120 can be any type of terminal device or server device. Examples of terminal devices include mobile terminals, fixed terminals or portable terminals, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the model evaluation system 120 can also support any type of interface for the user (such as "wearable" circuits, etc.). Examples of server devices include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, and the like.
[0032] It should be understood that Figure 1The components and arrangements in the environment 100 shown are merely examples, and a computing system suitable for implementing the exemplary implementations described in the present disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model training system 110 and the model evaluation system 120 may be integrated into the same system or device, or distributed in a cloud computing environment. The implementations of the present disclosure are not limited in this respect.
[0033] Model evaluation is mainly to test the performance or ability of the trained machine learning model to determine whether to continue to adjust the model, whether and / or how to apply the model.
[0034] Generative models refer to content generation technology implemented by language models. Generative models can automatically generate meaningful and coherent content based on input content, including natural language text or images, audio, video, etc. Driven by large-scale pre-trained models, generative models have been applied in many fields. Although generative models have made significant progress in generating content, how to objectively and effectively evaluate the quality of the content generated by the model remains a key challenge. The main reason is that for generative models, there is usually no objective and unique answer to the evaluation of their model outputs. Therefore, model evaluation needs to be based on rule judgment rather than comparison with fixed answers. In addition to generative models, there are other types of machine learning models whose model outputs are difficult to evaluate by comparing with fixed truth model outputs.
[0035] Traditionally, the performance evaluation of such models usually relies on manual evaluation, which is very costly. Some automated evaluation schemes have been proposed, which mainly rely on evaluation datasets for various tasks. The samples in these evaluation datasets include model input and corresponding reference model output. For example, some classic evaluation datasets for language models use question-answering format (including short answers), where the title of the question is the model input and the answer is the model output. The model performance is evaluated by calculating the accuracy of the model on the evaluation dataset, or the error between the generated answer and the standard answer.
[0036] However, such an evaluation scheme has limitations. On the one hand, there is no standard and unique reference output for some questions. If the evaluation method is to compare the answer output by the model with the reference answer of the test set, it is difficult for the evaluation result to be accurate and comprehensive. Because in a conversation, one question may correspond to multiple answers, it is also difficult to build a fixed data set to evaluate model performance.
[0037] On the other hand, if the trained model is used for conversation, it is difficult to accurately evaluate the model performance based on the results of a single question-answer interaction, because the subsequent model answers are diverse for each user input to the model, so the next question cannot be fixed in the dataset.
[0038] According to an example embodiment of the present disclosure, an improved model scheme is provided. In this scheme, the interaction information of multiple rounds of dialogues is collected by controlling multiple models to perform multiple rounds of dialogue interactions. The performance level of one, some or all of the multiple models is determined based on such interaction information. Through this scheme, multiple rounds of dialogue messages can be automatically obtained as evaluation data for evaluating the performance of the model in multiple rounds of dialogue. This can avoid the difficulty of manually constructing a complete dialogue data set and achieve a comprehensive evaluation of the model dialogue performance. Moreover, through automation and model-based technology, the quality and efficiency of evaluation data generation can be improved while reducing the burden of manual sample annotation.
[0039] Some example embodiments of the present disclosure will be described below with reference to the accompanying drawings.
[0040] Figure 2 1 shows a block diagram of a model evaluation system 120 according to some embodiments of the present disclosure. Figure 1 environment to describe.
[0041] like Figure 2 As shown, the model evaluation system 120 includes a multi-round dialogue subsystem 220 and a performance evaluation subsystem 230. The multi-round dialogue subsystem 220 is configured to control multiple models to perform multi-round dialogue interactions. Figure 2 In the example of FIG. 2 , model 210 (sometimes referred to as a “first model”) and model 212 (sometimes referred to as a “second model”) are shown. Among the multiple models performing a multi-round dialog interaction, at least one model is a model to be evaluated.
[0042] In each round of dialogue interaction, the multi-round dialogue subsystem 220 is configured to use one model among the multiple models to provide a first dialogue message to another model, and use the other model to generate a second dialogue message based on at least the first dialogue message. The dialogue message last generated by one model in the previous round of dialogue interaction will be provided to the next round of dialogue interaction for providing to another model. In this way, multiple models can achieve a dialogue effect.
[0043] Figure 3A multi-round dialogue interaction process 300 between models according to some embodiments of the present disclosure is shown. In this example, the multi-round interaction between model 210 and model 212 is taken as an example for explanation. As shown in the figure, model 210 generates dialogue message 1 and provides dialogue message 1 to model 212; model 212 generates dialogue message 2 based on dialogue message 1. In this way, model 210 and model 212 complete a round of dialogue interaction. Next, dialogue message 2 generated by model 212 is provided to model 210; model 212 generates dialogue message 3 based on dialogue message 2, and provides dialogue message 3 to model 210; model 210 generates subsequent dialogue messages based on dialogue message 3.
[0044] In some embodiments, the model 210 and the model 212 can be configured to be connected to each other, so that the model output of this time can be automatically provided to the other model as input, and the model output from the other model can be received as its own input. In some embodiments, the multi-round dialogue subsystem 220 can be used as an intermediate forwarding role, responsible for transmitting conversation messages between the models 210 and 212.
[0045] In some embodiments, in the initial dialogue interaction between models 210 and 212, the multi-turn interaction subsystem 220 is configured to input an initial dialogue message to model 210 or model 212 to trigger a dialogue between models 210 and 212. The initial dialogue message can be a question selected by the multi-turn interaction subsystem 220 from an existing question-answering dataset or a question suitable for initiating a dialogue determined from any other appropriate data source. The embodiments of the present disclosure are not limited in this respect.
[0046] In some embodiments, when another model generates another dialogue message based on a dialogue message output by a model, the dialogue message may not be directly used as a model input, but prompt words may be constructed based on the dialogue message output by the model, and then the constructed prompt words may be input to another model to obtain the dialogue message output by the other model. Prompt words can be considered as guiding information of the model. Different constructions of prompt words will also affect the quality of model output. In this way, by constructing prompt words to perform multiple rounds of dialogue interaction, it is also possible to evaluate whether the construction of prompt words is appropriate in the future.
[0047] In some embodiments, because of the involvement of the conversation context, in a round of conversation interaction, historical conversation messages in previous conversation interactions may also be input into each model, so that the model can determine the conversation message to be output in combination with the conversation context.
[0048] Note that although Figure 3Several rounds of dialogue interaction are shown, but the number of rounds each dialogue lasts may be different depending on the initial dialogue message, the configuration and performance of models 210 and 212, and the different messages generated in each round of dialogue interaction. In some embodiments, a preset upper limit on the number of rounds can be configured to prevent the dialogue from lasting too long. In some embodiments, model 210 and / or model 212 can decide when to end the dialogue, because the timing of dialogue termination can also be used to evaluate the performance of model 210 and / or model 212.
[0049] In this article, "model" may also be referred to as a machine learning model or agent. In some embodiments, models 210 and 212 may be configured as generative models that can generate expected output content based on model input. In some embodiments, models 210 and 212 have at least the content generation capability of text modality. Based on the text modality, model 210 and / or model 212 may also support the content generation capability of other modalities (e.g., images, videos, etc.). In some embodiments, models 210 and 212 are configured as models with conversation capabilities, which is also one of the characteristics of a generative model, particularly a text generative model. In the embodiments of the present disclosure, the specific configuration, parameters, and training methods of models 210 and 212 are not limited.
[0050] exist Figure 2 and Figure 3 The multi-round dialogue interaction of two models is specifically shown. In some embodiments, the multi-round dialogue interaction can also be generalized to more models. The multi-round dialogue subsystem 220 can also use three or more models to perform multi-round dialogue interaction. In the dialogue interaction including the first model, the second model and the third model, the third model generates a third dialogue message based on the dialogue message generated by the first model or the second model, so that the first model or the second model is used to generate another dialogue message.
[0051] In a conversation including three or more models, each model will determine whether to reply to a conversation message, whether to actively start the next conversation, etc. based on the understanding and judgment of the conversation context. The multi-round conversation interaction of multiple models can simulate the user group chat conversation, which can also determine the conversation ability of the model, such as whether it can support simultaneous conversations with multiple users, whether it can determine when and how to cut into the ongoing conversation, etc. In some embodiments, when three or more models are used to perform multi-round conversation interactions, a decision module can also be added to each model to determine the timing of joining the conversation.
[0052] In some embodiments, among the multiple models used for multi-round dialogue interaction, one or more models may be models with a performance level exceeding a threshold performance, and another one or more models may be models to be evaluated. Figure 2 and Figure 3 In the example, model 210 may be a model with a known higher performance level, while the performance level of model 212 is to be evaluated. By using a model that is generally considered to be better to communicate with the model to be evaluated, it is easier to determine whether the performance of the model to be evaluated is lower than, equal to, or higher than the model with known performance.
[0053] In some embodiments, multiple models used for multi-round dialogue interactions, such as models 210 and 212, may be the same model to be evaluated. In this way, the dialogue capabilities of the models in the multi-round dialogue interaction are equal, and the capabilities of the models can be evaluated from the perspectives of both or multiple parties in the dialogue.
[0054] Continue to refer Figure 2 The multi-round dialogue subsystem 220 obtains interaction information 230 between the model 210 and the model 212 in the multi-round interaction, and the interaction information includes multiple dialogue messages in the multi-round interaction. The obtained interaction information 230 is used as an evaluation data set to evaluate the performance of the model 210 and / or the model 212. The interaction information 230 is provided to the performance evaluation subsystem 240.
[0055] In some embodiments, to comprehensively evaluate the model, the multi-turn dialogue subsystem 220 can trigger the model 210 and the model 212, or the model to be evaluated (model 210 or 212) and another model to perform more multi-turn dialogue interactions, thereby obtaining more interaction information in different dialogues to perform model evaluation. This depends on the data volume required for model evaluation.
[0056] The performance evaluation subsystem 240 is configured to determine the performance level of at least one of the models 210 and 212 based on the interaction information 230 of the multi-round dialogue interaction. Specifically, if the model 210 and the model 212 are the same model to be evaluated, the performance evaluation subsystem 240 can evaluate the model using the interaction information 230. If the model 212 is the model to be evaluated, the performance evaluation subsystem 240 can evaluate the model 212 using the interaction information 230.
[0057] In some embodiments, for the interaction information 230 of multiple rounds of dialogue interaction, the performance evaluation subsystem 240 can determine whether the interaction information 230 meets one or more evaluation criteria, or the score under the corresponding evaluation criteria, based on one or more predetermined evaluation criteria. In some embodiments, the performance evaluation subsystem 240 can help automatically evaluate the obtained interaction information 230 with the help of a higher performance model. In addition to the model, the performance evaluation subsystem 240 can also use other automated tools to complete the model evaluation. The embodiments of the present disclosure do not specifically limit the specific evaluation criteria and automated evaluation tools used.
[0058] Compared to the need for manual dialogue with the model to be evaluated to obtain an evaluation data set, the automatic multi-round dialogue of multiple models can greatly reduce the cost of obtaining the evaluation data set, and the automatic multi-round dialogue data is used for model evaluation. Through the selection of models and the setting of model prompt words, the model capability can also be better evaluated. Furthermore, since the model evaluation is based on multi-round dialogue interaction information, rather than being performed on a specific question and answer pair, compared to the traditional pre-built dialogue evaluation data set, the solution disclosed in the present invention is more in line with the evaluation requirements of the model dialogue capability, and can more accurately and comprehensively evaluate the model capability.
[0059] In some embodiments, the method of performing multiple rounds of dialogue interactions using multiple models can be applied not only to model evaluation, but also to model training. In some embodiments, after the performance levels of models 210 and 212 (and possibly other models) have been determined, at least these models can be controlled to perform additional multiple rounds of dialogue interactions, thereby obtaining interaction information of additional multiple rounds of dialogue interactions. Other models (e.g., a fourth model) are trained using at least the interaction information of the additional multiple rounds of dialogue interactions. The fourth model can be a model to be trained to have dialogue capabilities.
[0060] In some embodiments, when collecting the interaction information of multiple rounds of dialogue interactions as training data for the model, a suitable model can be selected according to the requirements of the model training task, the initial conversation message that triggers the model interaction, the construction of the model prompt words, and the filtering of the obtained interaction information. After the performance level of the models 210 and 212 used is determined, it can be determined whether the collected interaction information of multiple rounds of dialogue interactions can be used as a positive sample or a negative sample in the model training process according to the performance level of the models 210 and 212. By collecting the multiple rounds of dialogue information automatically generated by the model, the model can be better learned to have better dialogue capabilities during the training stage.
[0061] Figure 4 FIG. 4 is a flowchart of a process 400 for model evaluation according to some embodiments of the present disclosure. The process 400 may be implemented in Figure 1 The model evaluation system 120 is provided everywhere.
[0062] In box 410, the model evaluation system 120 controls at least the first model and the second model to perform multiple rounds of dialogue interaction. In each round of dialogue interaction, one of the first model and the second model is used to provide a first dialogue message to the other model, and the other model is used to generate a second dialogue message based on at least the first dialogue message.
[0063] In block 420 , the model evaluation system 120 obtains interaction information in multiple rounds of interaction, where the interaction information includes multiple dialog messages in the multiple rounds of interaction.
[0064] At block 430 , the model evaluation system 120 determines a performance level of at least one of the first model and the second model based on the interaction information.
[0065] In some embodiments, controlling at least a first model and a second model to perform multiple rounds of conversation interactions includes: controlling the first model, the second model, and the third model to perform multiple rounds of conversation interactions, wherein in each round of conversation interaction, the third model generates a third conversation message based on a conversation message generated by the first model or the second model, so as to be used by the first model or the second model to generate another conversation message.
[0066] In some embodiments, generating the second dialog message based at least on the first dialog message using another model includes: generating the second dialog message based on the first dialog message and a session message in a previous dialog interaction using another model.
[0067] In some embodiments, generating a second dialog message based on at least the first dialog message using another model includes: constructing prompt words based on the first dialog message; and inputting the constructed prompt words to another model to obtain a second dialog message output by the other model.
[0068] In some embodiments, in an initial dialog interaction of a multi-round dialog interaction, an initial dialog message is input to one of the first model and the second model.
[0069] In some embodiments, the first model and the second model are the same model to be evaluated.
[0070] In some embodiments, the first model is a model having a performance level exceeding a threshold performance, and the second model is a model to be evaluated. In some embodiments, determining the performance level of at least one of the first model and the second model includes determining the performance level of the second model.
[0071] In some embodiments, process 400 also includes: after the performance levels of the first model and the second model have been determined, controlling at least the first model and the second model to perform additional multi-round dialogue interactions; obtaining interaction information of the additional multi-round dialogue interactions; and training a fourth model using at least the interaction information of the additional multi-round dialogue interactions.
[0072] In some embodiments, the first model and the second model are configured as models with conversation capabilities.
[0073] Figure 5 : shows a block diagram of an apparatus 500 for model evaluation according to some embodiments of the present disclosure. The apparatus 500 may be implemented as or included in Figure 1 The model evaluation system 120 of the embodiment of the present invention. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware or any combination thereof.
[0074] As shown in the figure, the device 500 includes a dialogue control module 510, which is configured to control at least a first model and a second model to perform multiple rounds of dialogue interaction. In each round of dialogue interaction, one of the first model and the second model is used to provide a first dialogue message to the other model, and the other model is used to generate a second dialogue message based on at least the first dialogue message. The device 500 also includes an information acquisition module 520, which is configured to obtain interaction information in multiple rounds of interaction, and the interaction information includes multiple dialogue messages in the multiple rounds of interaction. The device 500 also includes a capability determination module 530, which is configured to determine the performance level of at least one of the first model and the second model based on the interaction information.
[0075] In some embodiments, the conversation control module 510 is configured to: control the first model, the second model, and the third model to perform multiple rounds of conversation interactions, in which the third model generates a third conversation message based on the conversation message generated by the first model or the second model in each round of conversation interaction, so as to be used by the first model or the second model to generate another conversation message.
[0076] In some embodiments, the dialog control module 510 is configured to generate a second dialog message based on the first dialog message and a conversation message in a previous dialog interaction using another model.
[0077] In some embodiments, the dialog control module 510 is configured to: construct prompt words based on the first dialog message; and input the constructed prompt words to another model to obtain a second dialog message output by the other model.
[0078] In some embodiments, in an initial dialog interaction of a multi-round dialog interaction, an initial dialog message is input to one of the first model and the second model.
[0079] In some embodiments, the first model and the second model are the same model to be evaluated.
[0080] In some embodiments, the first model is a model having a performance level exceeding a threshold performance, and the second model is a model to be evaluated. In some embodiments, determining the performance level of at least one of the first model and the second model includes determining the performance level of the second model.
[0081] In some embodiments, the device 500 also includes: a further interaction module, configured to control at least the first model and the second model to perform additional multi-round dialogue interactions after the performance levels of the first model and the second model have been determined; obtain interaction information of the additional multi-round dialogue interactions; and a model training module, configured to train a fourth model using at least the interaction information of the additional multi-round dialogue interactions.
[0082] In some embodiments, the first model and the second model are configured as models with conversation capabilities.
[0083] Figure 6 1 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to implement Figure 1 Model evaluation system 120. Figure 6 The electronic device 600 shown can be used to implement Figure 5 Device 500.
[0084] like Figure 6 As shown, the electronic device 600 is in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 600.
[0085] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 630 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 600.
[0086] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 626 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.
[0087] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.
[0088] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).
[0089] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.
[0090] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0091] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0092] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0093] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.
[0094] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.
Claims
1. A method for model evaluation, comprising: Controlling at least a first model and a second model to perform multiple rounds of dialogue interaction, in each round of dialogue interaction, using one of the first model and the second model to provide a first dialogue message to the other model, and using the other model to generate a second dialogue message based on at least the first dialogue message; Obtaining interaction information in the multiple rounds of interaction, wherein the interaction information includes multiple dialogue messages in the multiple rounds of interaction; as well as A performance level of at least one of the first model and the second model is determined based on the interaction information.
2. The method according to claim 1, wherein controlling at least the first model and the second model to perform multiple rounds of dialogue interaction comprises: Control the first model, the second model and the third model to perform the multiple rounds of dialogue interaction, wherein in each round of dialogue interaction, the third model generates a third dialogue message based on the dialogue message generated by the first model or the second model, so as to be used by the first model or the second model to generate another dialogue message.
3. The method of claim 1 , wherein generating a second dialog message based at least on the first dialog message using the other model comprises: A second dialog message is generated based on the first dialog message and a session message in a previous dialog interaction using the other model.
4. The method of claim 1, wherein generating a second dialog message based at least on the first dialog message using the other model comprises: constructing a prompt word based on the first conversation message; as well as The constructed prompt words are input to the other model to obtain the second dialog message output by the other model.
5. The method according to claim 1, wherein in an initial dialog interaction of the multi-round dialog interaction, an initial dialog message is input to one of the first model and the second model.
6. The method according to claim 1, wherein the first model is a model having a performance level exceeding a threshold performance, and the second model is a model to be evaluated, and Wherein determining the performance level of at least one of the first model and the second model comprises: A performance level of the second model is determined. The method of claim 1 , wherein the first model and the second model are the same model to be evaluated.
8. The method according to claim 1, further comprising: After the performance levels of the first model and the second model have been determined, controlling at least the first model and the second model to perform additional multiple rounds of dialog interactions; Obtaining interaction information of the additional multiple rounds of dialogue interactions; as well as The fourth model is trained using at least the interaction information of the additional multiple rounds of dialogue interactions.
9. A device for model evaluation, comprising: a dialogue control module configured to control at least a first model and a second model to perform multiple rounds of dialogue interaction, wherein in each round of dialogue interaction, one of the first model and the second model is used to provide a first dialogue message to the other model, and the other model is used to generate a second dialogue message based on at least the first dialogue message; An information acquisition module, configured to obtain interaction information in the multiple rounds of interaction, wherein the interaction information includes multiple dialogue messages in the multiple rounds of interaction; as well as A performance determination module is configured to determine a performance level of at least one of the first model and the second model based on the interaction information.
10. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.
11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.