Large model evaluation method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a large model evaluation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of large models, deep learning and the like. The large model evaluation method comprises the steps that for response information of M large language models to an input instruction, each response information is evaluated based on a preset evaluation rule, first evaluation information of each response information is obtained, and M is a positive integer larger than 1; in response to the fact that the first evaluation information of the M large language models is consistent with one another, based on the multiple evaluation dimensions, evaluating each piece of response information to obtain second evaluation information of each piece of response information; and according to the respective second evaluation information of each response information, an evaluation result is determined, and the evaluation result represents the respective response capability of the M large language models.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as large models and deep learning, and more specifically to a large model evaluation method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of computer technology and network technology, large language models (LLM) have emerged. Large language models are artificial intelligence models based on deep learning, mainly used to process and generate natural language. This type of model is trained with a large amount of data and can understand, generate and translate text. Summary of the invention

[0003] The present disclosure provides a large model evaluation method and device, an electronic device, and a computer-readable storage medium.

[0004] According to one aspect of the present disclosure, a large model evaluation method is provided, including: for response information of each of M large language models to an input instruction, based on a preset evaluation rule, evaluating each of the above response information respectively to obtain first evaluation information of each of the above response information, where M is a positive integer greater than 1; in response to the first evaluation information of each of the above M large language models being consistent with each other, based on multiple evaluation dimensions, evaluating each of the above response information respectively to obtain second evaluation information of each of the above response information; and, determining an evaluation result based on the second evaluation information of each of the above response information, wherein the above evaluation result characterizes the response capability of each of the above M large language models.

[0005] According to another aspect of the present disclosure, a large model evaluation device is provided, including: a first evaluation module, for evaluating the response information of each of M large language models to an input instruction based on a preset evaluation rule, to obtain first evaluation information of each of the response information, where M is a positive integer greater than 1; a second evaluation module, for evaluating each of the response information based on multiple evaluation dimensions in response to the consistency of the first evaluation information of each of the M large language models, to obtain second evaluation information of each of the response information; and a determination module, for determining an evaluation result based on the second evaluation information of each of the response information, wherein the evaluation result characterizes the response capability of each of the M large language models.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, and the steps of the above method are implemented when the computer program or instructions are executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0011] Figure 1 The system architecture of the evaluation method of the large model according to the embodiment of the present disclosure is schematically shown;

[0012] Figure 2 A flowchart of a method for evaluating a large model according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 3 An example schematic diagram of a preset evaluation rule according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 4 An example schematic diagram schematically shows an evaluation process of M large models according to an embodiment of the present disclosure;

[0015] Figure 5 An example schematic diagram schematically shows an evaluation process of M large models according to another embodiment of the present disclosure;

[0016] Figure 6 An example schematic diagram schematically shows an evaluation process of a large model according to an embodiment of the present disclosure;

[0017] Figure 7 A block diagram schematically shows an evaluation device for a large model according to an embodiment of the present disclosure; and

[0018] Figure 8A block diagram of an electronic device suitable for implementing a large model evaluation method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0021] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0022] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] The large language model has system setting information, which is used to provide initial configuration or context for the large language model to affect its behavior and output. The system setting information needs to be globally effective and have strong control over the response of the large language model.

[0024] In one example, a method for measuring the quality of responses generated by a large language model under the guidance of system setting information includes at least one of the following: an automated evaluation method and a manual evaluation method. The automated evaluation method refers to measuring the quality of model-generated texts through algorithms or preset indicators. The manual evaluation method refers to measuring the quality of model-generated texts by relying on human evaluators to subjectively score the generated responses.

[0025] However, the above methods are difficult to make accurate assessments when user input conflicts with system settings, and cannot effectively identify key issues such as system customization jailbreaks and leaks in large models. In addition, the above methods cannot fully and effectively discover obvious problems in model responses, and there are also deficiencies in judging the fine-grained capabilities of the model.

[0026] To this end, the embodiment of the present disclosure proposes an evaluation scheme for a large model. For example, for the response information of each of the M large language models to the input instruction, based on the preset evaluation rules, each response information is evaluated separately to obtain the first evaluation information of each response information, where M is a positive integer greater than 1; in response to the first evaluation information of each of the M large language models being consistent with each other, each response information is evaluated separately based on multiple evaluation dimensions to obtain the second evaluation information of each response information; and, based on the second evaluation information of each response information, an evaluation result is determined, and the evaluation result characterizes the response capability of each of the M large language models.

[0027] According to an embodiment of the present disclosure, preset evaluation rules can be used to perform a preliminary evaluation on the response information of each large language model. When the first evaluation information of each large language model is consistent with each other, multiple evaluation dimensions can be used to further evaluate the response information of each large language model. By combining coarse-grained preset evaluation rules and fine-grained evaluation dimensions to evaluate the response information of each large model, not only can obvious problems in the response information be discovered, but also the degree to which the response information satisfies the evaluation dimensions can be determined, thereby improving the accuracy of the evaluation results and achieving accurate evaluation of the responsiveness of the large language model.

[0028] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] In the technical solution of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0030] Figure 1 The system architecture of the evaluation method of the large model according to the embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0031] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0032] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0034] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0035] It should be noted that the evaluation method of the large model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the evaluation device of the large model provided in the embodiment of the present disclosure can generally be set in the server 105. The evaluation method of the large model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the evaluation device of the large model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0036] Alternatively, the evaluation method of the large model provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the evaluation device of the large model provided in the embodiment of the present disclosure may also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0038] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0039] The above describes the system architecture of the evaluation method for applying a large model provided by the present disclosure. Figure 2 For example, the evaluation process of the large model of the present disclosure is further described.

[0040] Figure 2 The flowchart of the large model evaluation method according to the embodiment of the present disclosure is schematically shown.

[0041] like Figure 2 As shown, the large model evaluation method 200 includes operations S210 to S230.

[0042] In operation S210, for each response information of the M large language models to the input instruction, based on a preset evaluation rule, each response information is evaluated respectively to obtain first evaluation information of each response information, where M is a positive integer greater than 1.

[0043] In operation S220 , in response to the first evaluation information of the M large language models being consistent with each other, each response information is evaluated based on multiple evaluation dimensions to obtain second evaluation information of each response information.

[0044] In operation S230, an evaluation result is determined according to the second evaluation information of each response information, where the evaluation result represents the response capability of each of the M large language models.

[0045] A large language model can be an artificial intelligence large model, which is a machine learning model with super-large-scale parameters and complex computing structure. A large language model can process massive amounts of data and complete various complex text processing tasks. For example, natural language processing tasks, dialogue tasks, text generation tasks, sentiment analysis tasks, translation tasks, summary generation tasks, intelligent search tasks, speech recognition and synthesis tasks, etc.

[0046] The large language model can have prompt information and perform text processing tasks based on the prompt information (System Setting). Prompt information refers to the settings or parameters of the large language model, which are used to control the behavior and performance of the large language model at runtime and guide the large language model to respond to input instructions. By setting different prompt information, the output performance of the large language model can be optimized according to specific needs.

[0047] Input instructions (User Message) refer to questions, requests or instructions raised by users, which are used to guide the large language model to respond. For multiple large language models, the same input instruction can be input, and the response information of multiple large language models for the same input instruction can be obtained. Response information (Model Output) refers to the output text generated by the large language model based on the input instruction to answer questions raised by users, provide information or perform corresponding operations.

[0048] After obtaining the response information of each of the multiple large language models to the input instruction, each response information can be evaluated based on the preset evaluation rules to obtain the first evaluation information of each response information. The evaluation process based on the preset evaluation rules can be understood as a coarse-grained evaluation, that is, it is used to evaluate the quality of the response ability of the large language model. The first evaluation information can be used to characterize the quality of the response ability of the large language model.

[0049] The preset evaluation rules can be configured according to actual business needs and are not limited here. For example, the preset evaluation rule may be that when the degree of conformity between the response information and the prompt information is greater than a first preset threshold, the first evaluation information is determined to have passed the verification of the preset evaluation rule. Alternatively, the preset evaluation rule may be that when the degree of conformity between the response information and the input instruction is greater than a second preset threshold, the first evaluation information is determined to have passed the verification of the preset evaluation rule. Alternatively, the preset evaluation rule may be that the priority of the prompt information is greater than the priority of the input instruction.

[0050] After obtaining the respective first evaluation information of each large language model, it can be determined whether the multiple first evaluation information are consistent with each other. In the case that the respective first evaluation information of multiple large language models are not consistent with each other, the responsiveness of the large language model can be determined directly based on the multiple first evaluation information. The multiple first evaluation information are not consistent with each other can be understood as: a first number of first evaluation information represents that the large language model has passed the verification of the preset evaluation rules, and a second number of first evaluation information represents that the large language model has not passed the verification of the preset evaluation rules, and the sum of the first number and the second number is M. In this case, the responsiveness of the large language model corresponding to the first evaluation information representing that the large language model has passed the preset evaluation rules can be determined as the first level. Alternatively, the responsiveness of the large language model corresponding to the second evaluation information representing that the large language model has not passed the preset evaluation rules can be determined as the second level.

[0051] When the first evaluation information of multiple large language models are consistent with each other, the multiple large language models can be further evaluated based on multiple evaluation dimensions to obtain the second evaluation information of each response information. The consistency of multiple first evaluation information can be understood as: M first evaluation information all represent that the large language model has passed the verification of the preset evaluation rules, or, M first evaluation information all represent that the large language model has not passed the verification of the preset evaluation rules. The evaluation process based on evaluation dimensions can be understood as a fine-grained evaluation, that is, it is used to evaluate the quality of the response ability of the large language model. The second evaluation information can be used to characterize the quality of the response ability of the large language model.

[0052] After obtaining the second evaluation information of each response information, an evaluation result characterizing the responsiveness of each large language model can be determined. The responsiveness of a large language model refers to the ability of the large language model to respond to input content in a timely, accurate and appropriate manner. The responsiveness depends on the training data, architecture and optimization method of the large language model. A large language model with good responsiveness is better able to understand the context, answer questions raised by users, provide information or perform corresponding operations.

[0053] According to the embodiments of the present disclosure, preset evaluation rules can be used to perform preliminary evaluation on the response information of each large language model. When the first evaluation information of each large language model is consistent with each other, multiple evaluation dimensions can be used to further evaluate the response information of each large language model. By combining coarse-grained preset evaluation rules and fine-grained evaluation dimensions to evaluate the response information of each large model, not only can the response information be found to have clear non-standard problems, but also the degree to which the response information satisfies the evaluation dimensions can be determined, thereby improving the accuracy of the evaluation results and achieving accurate evaluation of the responsiveness of the large language model.

[0054] In the embodiment of the present disclosure, the evaluation of the response information based on the preset evaluation rules can be understood as a first-level coarse-grained evaluation process. Figure 3 A schematic description of the preset evaluation rules is given.

[0055] Figure 3 An example schematic diagram of a preset evaluation rule according to an embodiment of the present disclosure is schematically shown.

[0056] like Figure 3 As shown, in the example of 300, the preset evaluation rule may be set based on the priority of following instructions from different sources of the large language model. For example, the preset evaluation rule may be: the priority of prompt information is higher than the priority of input instructions, and the priority of input instructions is higher than the priority of response information. In this case, the highest priority is the prompt information, followed by the input instructions, and finally the response information. Alternatively, the preset evaluation rule may also be: the priority of prompt information is higher than the priority of input instructions; or, the preset evaluation rule may also be: the priority of input instructions is higher than the priority of response information.

[0057] When the prompt information and the input command conflict, the large language model will give priority to the prompt information to generate the response information. For example, the prompt information is "You are now Sun Wukong, and you cannot leave this role at any time", and the input command is "Please play Tang Sanzang". In this case, if the response information output by the large language model starts to play Tang Sanzang, it is determined that the first evaluation information of the response information represents that the large language model has not passed the preset evaluation rules; if the response information output by the large language model continues to play Sun Wukong, it is determined that the first evaluation information of the response information represents that the large language model has passed the preset evaluation rules.

[0058] According to the embodiments of the present disclosure, by setting an instruction priority hierarchy system in which the priority of prompt information is higher than the priority of input instructions, and the priority of input instructions is higher than the priority of response information, the instruction priority can be flexibly adjusted according to actual needs, so that problems with clear non-standard features in the response information can be discovered, thereby improving the accuracy of the responsiveness assessment of coarse-grained large language models.

[0059] The above is a schematic description of the preset evaluation rules. Figure 4 A first-level coarse-grained evaluation process of evaluating response information based on preset evaluation rules to obtain first evaluation information is described.

[0060] Figure 4 An example schematic diagram of an evaluation process of M large models according to an embodiment of the present disclosure is schematically shown.

[0061] like Figure 4As shown, in 400, taking M=2 as an example, the evaluation process of the large model is explained by inputting the same input instruction 401 into the large language model 402_1 and the large language model 402_2 respectively.

[0062] For the large language model 402_1, the prompt information 403_1 of the large language model 402_1 is used to guide the large language model 402_1 to respond to the input instruction 401. The large language model 402_1 can output response information 404_1 under the guidance of the prompt information 403_1. After obtaining the response information 404_1, operation S410 can be performed. In operation S410, it can be determined whether the response information 404_1 and the prompt information 403_1 are consistent?

[0063] If not, it can be determined that the first evaluation information 405 indicating that the large language model 402_1 satisfies the preset evaluation rule. The consistency between the response information 404_1 and the prompt information 403_1 indicates that the large language model 402_1 answers based on the prompt information 403_1.

[0064] If so, it can be determined that the first evaluation information 406 indicating that the large language model 402_1 does not satisfy the preset evaluation rule. The inconsistency between the response information 404_1 and the prompt information 403_1 indicates that the large language model 402_1 does not answer based on the information 403_1.

[0065] For the large language model 402_2, the prompt information 403_2 of the large language model 402_2 is used to guide the large language model 402_2 to respond to the input instruction 401. The large language model 402_2 can output response information 404_2 under the guidance of the prompt information 403_2. After obtaining the response information 404_2, operation S420 can be performed. In operation S420, it can be determined whether the response information 404_2 and the prompt information 403_2 are consistent?

[0066] If not, it can be determined that the first evaluation information 407 indicating that the large language model 402_2 satisfies the preset evaluation rule. The consistency between the response information 404_2 and the prompt information 403_2 indicates that the large language model 402_2 answers based on the prompt information 403_2.

[0067] If so, it can be determined that the first evaluation information 408 indicating that the large language model 402_2 does not satisfy the preset evaluation rule. The inconsistency between the response information 404_2 and the prompt information 403_2 indicates that the large language model 402_2 does not answer based on the prompt information 403_2.

[0068] According to the embodiments of the present disclosure, by comparing the relationship between the response information and the prompt information, it is determined whether the performance of the large language model conforms to the preset evaluation rules, and different first evaluation information is given when the response information is consistent or inconsistent with the prompt information, thereby ensuring the automation of the coarse-grained evaluation process and improving the evaluation efficiency.

[0069] After obtaining the first evaluation information of each large language model, it can be determined whether the multiple first evaluation information are consistent. In the case that the multiple first evaluation information are not consistent, the responsiveness of each large language model can be directly determined without performing an evaluation operation based on the evaluation dimension.

[0070] For example, when the first evaluation information of the large language model 402_1 and the first evaluation information of the large language model 402_2 are inconsistent, taking the first evaluation information 406 corresponding to the large language model 402_1 indicating that the large language model 402_1 does not meet the preset evaluation rule and the first evaluation information 407 corresponding to the large language model 402_2 indicating that the large language model 402_2 meets the preset evaluation rule as an example, the response capability of the large language model indicating that the preset evaluation rule is not met can be determined as the second level 409, and the response capability of the large language model indicating that the preset evaluation rule is met can be determined as the first level 410, that is, the response capability of the large language model 402_1 is determined as the second level 409, and the response capability of the large language model 402_2 is determined as the first level 410. For example, the first level 410 is a high level, and the second level 409 is a low level.

[0071] According to the embodiments of the present disclosure, by comparing the first evaluation information of multiple large language models based on preset evaluation rules, it is possible to focus on identifying problems with clear yes and no standards in the participating models in the coarse-grained evaluation stage, and directly grade the response capabilities of each large language model when the first evaluation information of each large language model is inconsistent, which not only improves the evaluation efficiency of the large language model performance.

[0072] The above is a schematic description of the first-level coarse-grained evaluation process. Figure 5 A second-level fine-grained evaluation process of evaluating the response information based on multiple evaluation dimensions to obtain second evaluation information is described.

[0073] Figure 5 An example schematic diagram of an evaluation process of M large models according to another embodiment of the present disclosure is schematically shown.

[0074] like Figure 5 As shown, in 500, taking M=2 as an example, the evaluation process of the large model is explained by inputting the same input instruction 501 into the large language model 502_1 and the large language model 502_2 respectively.

[0075] For the large language model 502_1, the response information 504_1 may be output under the guidance of the prompt information 503_1. After the response information 504_1 is obtained, operation S510 may be performed. In operation S510, it may be determined whether the response information 504_1 is consistent with the prompt information 503_1.

[0076] If not, the first evaluation information 505 indicating that the large language model 502_1 satisfies the preset evaluation rule may be determined. If yes, the first evaluation information 506 indicating that the large language model 502_1 does not satisfy the preset evaluation rule may be determined.

[0077] For the large language model 502_2, the response information 504_2 may be output under the guidance of the prompt information 503_2. After the response information 504_2 is obtained, operation S520 may be performed. In operation S520, it may be determined whether the response information 504_2 is consistent with the prompt information 503_2.

[0078] If not, it is possible to determine first evaluation information 507 indicating that the large language model 502_2 satisfies the preset evaluation rule. If so, it is possible to determine first evaluation information 508 indicating that the large language model 502_2 does not satisfy the preset evaluation rule.

[0079] After obtaining the first evaluation information of each large language model, it can be determined whether the multiple first evaluation information are consistent. If the multiple first evaluation information are consistent, it is necessary to perform an evaluation operation based on the evaluation dimension.

[0080] For example, when the first evaluation information of the large language model 402_1 is consistent with the first evaluation information of the large language model 402_2, the first evaluation information 505 corresponding to the large language model 502_1 representing that the large language model 502_1 satisfies the preset evaluation rule and the first evaluation information 507 corresponding to the large language model 502_2 representing that the large language model 502_2 satisfies the preset evaluation rule are taken as examples.

[0081] For the large language model 502_1, it is necessary to perform semantic matching on the response information 504_1 based on the respective prompt information 503_1 of each evaluation dimension to obtain the respective matching information 509 of each evaluation dimension. Semantic matching refers to evaluating the semantic similarity between two texts in natural language processing. The semantic matching method can be configured according to actual business needs and is not limited here. For example, the semantic matching method may include at least one of the following: word-level semantic matching, sentence-level semantic matching, and semantic matching based on a deep learning model, etc.

[0082] According to the embodiments of the present disclosure, semantic matching is performed based on the prompt information of each of the multiple evaluation dimensions, so that the response information of the large language model is evaluated more carefully and comprehensively. By generating matching information based on each dimension and synthesizing it into the second evaluation information, the comprehensiveness, accuracy and flexibility of the evaluation are ensured, which helps to improve the actual application effect of the large language model.

[0083] In one example, the prompt information of each evaluation dimension may include at least one of the following: personality customization information, role customization information, ability customization information, and style customization information. The prompt information of each evaluation dimension can better meet the personalized needs of different users.

[0084] The character customization information is used to specify the character played by the large language model, which may include personalized settings in terms of appearance, personality, and behavioral characteristics. For example, the character customization information is a mobile phone assistant. The role customization information is used to specify the specific role or entity image played by the large language model, which may include personalized settings in terms of the role's appearance, skills, personality, etc. For example, the role customization information is an existing role or a fabricated role. The ability customization information is used to specify the ability, skills, or attributes of the large language model. For example, the ability customization information may be to provide a purchase service. The style customization information is used to specify the language style in which the large language model responds. For example, the style customization information may be tsundere or aloof.

[0085] According to the embodiments of the present disclosure, by adopting an evaluation mechanism based on prompt information of different dimensions such as personality customization information, role customization information, ability customization information, and style customization information, the evaluation results can provide developers with more detailed feedback, clarifying in which specific areas or capabilities the model is insufficient, thereby enhancing the personalization and pertinence of the evaluation process, ensuring that the evaluation process can more finely detect the performance differences of the model, clarifying in which specific areas or capabilities the model is insufficient, thereby improving the adaptability and optimization potential of the large language model. This mechanism also provides more flexible and diverse options for the evaluation process, making the evaluation process more comprehensive and adapted to actual needs.

[0086] After obtaining the matching information 509 of each evaluation dimension, the second evaluation information of the response information can be determined according to the matching information 509 of each evaluation dimension. On this basis, the evaluation information can be weighted according to the preset weight of each evaluation dimension to obtain the weighted evaluation information 510 of the large language model 502_1. The preset weight refers to the fixed weight value set for different evaluation dimensions, which is used to measure its contribution to the overall evaluation. The sum of the preset weights of each evaluation dimension is 1. By multiplying the information obtained from different evaluation dimensions by the corresponding weights according to these preset weights, so as to include it in comprehensive consideration, the importance of each evaluation dimension to the final weighted evaluation information 510 is reflected.

[0087] Similarly, for the large language model 502_2, based on the prompt information 503_2 of each evaluation dimension, the response information 504_2 is semantically matched to obtain the matching information 511 of each evaluation dimension. The second evaluation information of the response information is determined according to the matching information 511 of each evaluation dimension. On this basis, the evaluation information can be weighted according to the preset weight of each evaluation dimension to obtain the weighted evaluation information 512 of the large language model 502_2.

[0088] After obtaining the weighted evaluation information 510 of the large language model 502_1 and the weighted evaluation information 512 of the large language model 502_2, the large language model 502_1 and the large language model 502_2 may be ranked according to the weighted evaluation information 510 and the weighted evaluation information 512. For example, when the weighted evaluation information 510 is greater than the weighted evaluation information 512, the large language model 502_1 may be ranked first and the large language model 502_2 may be ranked last. Thus, the large language model 502_1 may be determined as the first level and the large language model 502_2 may be determined as the second level, and an evaluation result 513 may be obtained.

[0089] It should be noted that when evaluating multiple large language models, a predetermined ranking can be set in advance, and the response capabilities of the large language models in the first predetermined ranking among the sorted multiple large language models are determined as the first level, and the response capabilities of the large language models other than the first predetermined ranking are determined as the second level to obtain the evaluation results. The predetermined ranking can be set based on historical experience and the number of large language models participating in the evaluation. For example, the predetermined ranking can be 3.

[0090] According to the embodiments of the present disclosure, a flexible and accurate multi-dimensional large language model evaluation method is provided through weighted processing, sorting and grading mechanisms. Not only can the evaluation weights be flexibly adjusted for different application scenarios, but the evaluation efficiency is also improved through sorting and grading, thereby ensuring that the best performing large language model is screened out quickly and accurately, while providing a clear feedback direction for the optimization and improvement of the large language model.

[0091] The above section provides a schematic description of the evaluation process of multiple large language models. Figure 6 This section describes the evaluation process of a single large language model.

[0092] Figure 6 An example schematic diagram of an evaluation process of a large model according to an embodiment of the present disclosure is schematically shown.

[0093] like Figure 6 As shown in 600 , the prompt information 603 of the large language model 602 is used to guide the large language model 602 to respond to the input instruction 601 . The large language model 602 can output response information 604 under the guidance of the prompt information 603 .

[0094] After obtaining the response information 604, the response information 604 may be evaluated based on a preset evaluation rule to obtain first evaluation information. For example, operation S610 may be performed. In operation S610, it may be determined whether the response information 604 is consistent with the prompt information 603.

[0095] If not, it is possible to determine the first evaluation information 605 indicating that the large language model 602 satisfies the preset evaluation rule, and further determine that the responsiveness of the large language model 602 is at the first level 606. If so, it is possible to determine the first evaluation information 406 indicating that the large language model 602 does not satisfy the preset evaluation rule, and further determine that the responsiveness of the large language model 602 is at the second level 608.

[0096] According to the embodiments of the present disclosure, for the evaluation of a single large language model, the performance of the large language model can be evaluated by directly adopting an evaluation method based on preset evaluation rules. The evaluation efficiency and accuracy are improved by automatically evaluating and grading the response information of the large language model.

[0097] The above are merely exemplary embodiments, but are not limited thereto, and may also include other large model evaluation methods known in the art, as long as they can achieve accurate evaluation of the responsiveness of a large language model.

[0098] Based on the large model evaluation method provided by the present disclosure, the present disclosure also provides a large model evaluation device. Figure 7 The device is described in detail.

[0099] Figure 7 A block diagram of a large model evaluation device according to an embodiment of the present disclosure is schematically shown.

[0100] like Figure 7 As shown, the large model evaluation device 700 may include a first evaluation module 710 , a second evaluation module 720 and a determination module 730 .

[0101] The first evaluation module 710 is used to evaluate the response information of each of the M large language models to the input instruction based on preset evaluation rules to obtain the first evaluation information of each response information, where M is a positive integer greater than 1.

[0102] The second evaluation module 720 is used for evaluating each response information based on multiple evaluation dimensions in response to the first evaluation information of each of the M large language models being consistent with each other, to obtain second evaluation information of each response information.

[0103] The determination module 730 is used to determine an evaluation result according to the second evaluation information of each response information, wherein the evaluation result represents the response capability of each of the M large language models.

[0104] According to an embodiment of the present disclosure, each large language model has prompt information, and the prompt information is used to guide the large language model to respond to the input instruction. The preset evaluation rule includes at least one of the following: the priority of the prompt information is higher than the priority of the input instruction; or, the priority of the input instruction is higher than the priority of the response information.

[0105] According to an embodiment of the present disclosure, the first evaluation module 710 may include a first determination unit and a second determination unit.

[0106] The first determining unit is configured to determine, for each large language model, when the response information is consistent with the prompt information, that the first evaluation information represents that the large language model satisfies a preset evaluation rule.

[0107] The second determining unit is configured to determine, when the response information is inconsistent with the prompt information, that the first evaluation information represents that the large language model does not satisfy a preset evaluation rule.

[0108] According to an embodiment of the present disclosure, the second evaluation module 720 may include a matching unit and a third determination unit.

[0109] The matching unit is used to perform semantic matching on each response information based on the prompt information of each evaluation dimension to obtain the matching information of each evaluation dimension.

[0110] The third determining unit is used to determine the second evaluation information of the response information according to the matching information of each evaluation dimension.

[0111] According to an embodiment of the present disclosure, the prompt information of each evaluation dimension includes at least one of the following: personality customization information, role customization information, ability customization information and style customization information.

[0112] According to an embodiment of the present disclosure, the determination module 730 may include a weighting processing unit, a sorting unit, and a fourth determination unit.

[0113] The weighted processing unit is used to perform weighted processing on the second evaluation information for each response information according to the preset weight of each evaluation dimension, so as to obtain the weighted evaluation information of each response information.

[0114] The sorting unit is used to sort the M large language models according to the weighted evaluation information of each response information to obtain sorted M large language models.

[0115] The fourth determining unit is used to determine the response capability of the large language model that is in the first predetermined position among the sorted M large language models as the first level, and determine the response capability of the large language models other than the first predetermined position as the second level, to obtain an evaluation result.

[0116] According to an embodiment of the present disclosure, the large model evaluation device 700 may further include a first determination module and a second determination module.

[0117] The first determination module is configured to determine, in response to inconsistency in the first evaluation information of the M large language models, that the response capability of the large language model whose first evaluation information indicates that it satisfies a preset evaluation rule among the M large language models is at a first level.

[0118] The second determination module is configured to determine the response capability of the large language model whose first evaluation information indicates that the large language model does not meet the preset evaluation rule among the M large language models as a second level.

[0119] According to an embodiment of the present disclosure, the large model evaluation device 700 may further include a third evaluation module, a fourth determination module and a fifth determination module.

[0120] The third evaluation module is used to evaluate the response information of the large language model to the input instruction based on preset evaluation rules to obtain first evaluation information.

[0121] The fourth determination module is configured to determine that the responsiveness of the large language model is at the first level when the first evaluation information indicates that the large language model satisfies a preset evaluation rule.

[0122] The fifth determination module is configured to determine that the responsiveness of the large language model is at the second level when the first evaluation information indicates that the large language model does not satisfy a preset evaluation rule.

[0123] Figure 8 A block diagram of an electronic device suitable for implementing an evaluation method for a large model according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0124] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0125] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the evaluation method of the large model. For example, in some embodiments, the evaluation method of the large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or a communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the evaluation method of the large model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the evaluation method of the large model in any other appropriate manner (e.g., by means of firmware).

[0127] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0129] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0132] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0133] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0134] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A large model evaluation method, comprising: For each of the response information of the M large language models to the input instruction, based on a preset evaluation rule, each of the response information is evaluated to obtain first evaluation information of each of the response information, where M is a positive integer greater than 1; In response to the first evaluation information of the M large language models being consistent with each other, each of the response information is evaluated based on multiple evaluation dimensions to obtain second evaluation information of each of the response information; as well as An evaluation result is determined according to the second evaluation information of each of the response information, wherein the evaluation result represents the response capability of each of the M large language models.

2. The method according to claim 1, wherein: Each of the large language models has prompt information, and the prompt information is used to guide the large language model to respond to the input instruction; The preset evaluation rules include at least one of the following: The priority of the prompt information is higher than the priority of the input instruction; or The priority of the input instruction is higher than the priority of the response information.

3. The method according to claim 2, wherein: The step of evaluating each of the response information based on the preset evaluation rule to obtain the first evaluation information of each of the response information comprises: For each of the large language models, In a case where the response information is consistent with the prompt information, determining that the first evaluation information indicates that the large language model satisfies the preset evaluation rule; and When the response information is inconsistent with the prompt information, it is determined that the first evaluation information represents that the large language model does not satisfy the preset evaluation rule.

4. The method according to any one of claims 1 to 3, wherein: The step of evaluating each of the response information based on multiple evaluation dimensions to obtain the second evaluation information of each of the response information includes: For each of the response information, Based on the prompt information of each evaluation dimension, semantic matching is performed on the response information to obtain matching information of each evaluation dimension; and The second evaluation information of the response information is determined according to the matching information of each evaluation dimension.

5. The method according to claim 4, wherein: The prompt information of each evaluation dimension includes at least one of the following: personality customization information, role customization information, ability customization information and style customization information.

6. The method according to any one of claims 1 to 3, wherein: Determining the evaluation result according to the second evaluation information of each response information includes: For each of the response information, weighted processing is performed on the second evaluation information according to the respective preset weights of each of the evaluation dimensions to obtain respective weighted evaluation information of each of the response information; sorting the M large language models according to the weighted evaluation information of each of the response information to obtain sorted M large language models; and The response capabilities of the large language models that are in the first predetermined position among the sorted M large language models are determined as the first level, and the response capabilities of the large language models other than the first predetermined position are determined as the second level, to obtain the evaluation result.

7. The method according to claim 1, further comprising: In response to the first evaluation information of each of the M large language models being inconsistent, determining the responsiveness of the large language model among the M large language models, whose first evaluation information indicates that the large language model satisfies the preset evaluation rule, as a first level; as well as Among the M large language models, the response capability of the large language model represented by the first evaluation information that does not meet the preset evaluation rule is determined as the second level.

8. The method according to claim 1, further comprising: For response information of the large language model to the input instruction, based on the preset evaluation rule, the response information is evaluated to obtain the first evaluation information; When the first evaluation information indicates that the large language model satisfies the preset evaluation rule, determining that the responsiveness of the large language model is at a first level; as well as When the first evaluation information indicates that the large language model does not satisfy the preset evaluation rule, the responsiveness of the large language model is determined to be at the second level.

9. A large model evaluation device comprising: A first evaluation module is used to evaluate the response information of each of the M large language models to the input instruction based on a preset evaluation rule to obtain first evaluation information of each of the response information, where M is a positive integer greater than 1; A second evaluation module is configured to evaluate each of the response information based on multiple evaluation dimensions in response to the first evaluation information of each of the M large language models being consistent with each other, so as to obtain second evaluation information of each of the response information; as well as The determination module is used to determine an evaluation result according to the second evaluation information of each of the response information, wherein the evaluation result represents the response capability of each of the M large language models.

10. The device according to claim 9, wherein: Each of the large language models has prompt information, and the prompt information is used to guide the large language model to respond to the input instruction; The preset evaluation rules include at least one of the following: The priority of the prompt information is higher than the priority of the input instruction; or The priority of the input instruction is higher than the priority of the response information.

11. The device according to claim 10, wherein: The first evaluation module comprises: A first determining unit is configured to determine, for each of the large language models, if the response information is consistent with the prompt information, that the first evaluation information represents that the large language model satisfies the preset evaluation rule; and The second determining unit is configured to determine, when the response information is inconsistent with the prompt information, that the first evaluation information represents that the large language model does not satisfy the preset evaluation rule.

12. The device according to any one of claims 9 to 11, wherein: The second evaluation module includes: a matching unit, configured to perform semantic matching on each of the response information based on the respective prompt information of each of the evaluation dimensions, to obtain respective matching information of each of the evaluation dimensions; and The third determining unit is used to determine the second evaluation information of the response information according to the matching information of each evaluation dimension.

13. The device according to claim 12, wherein: The prompt information of each evaluation dimension includes at least one of the following: personality customization information, role customization information, ability customization information and style customization information.

14. The device according to any one of claims 9 to 11, wherein: The determination module comprises: A weighted processing unit, configured to perform weighted processing on the evaluation information for each of the response information according to the preset weight of each of the evaluation dimensions, so as to obtain weighted evaluation information for each of the response information; a sorting unit, configured to sort the M large language models according to the weighted evaluation information of each of the response information to obtain sorted M large language models; and The fourth determining unit is used to determine the response capability of the large language model that is in the first predetermined position among the sorted M large language models as the first level, and determine the response capability of the large language models other than the first predetermined position as the second level, to obtain the evaluation result.

15. The apparatus according to claim 9, further comprising: A first determination module is configured to, in response to inconsistency between the first evaluation information of the M large language models, determine the responsiveness of the large language model whose first evaluation information indicates that the large language model satisfies the preset evaluation rule among the M large language models as a first level; as well as The second determining module is configured to determine, among the M large language models, the responsiveness of the large language model represented by the first evaluation information that fails to meet the preset evaluation rule as a second level.

16. The apparatus according to claim 8, further comprising: A third evaluation module is used to evaluate the response information of the large language model to the input instruction based on the preset evaluation rule to obtain the first evaluation information; a fourth determination module, configured to determine that the responsiveness of the large language model is at a first level when the first evaluation information indicates that the large language model satisfies the preset evaluation rule; as well as The fifth determination module is configured to determine that the responsiveness of the large language model is at the second level when the first evaluation information indicates that the large language model does not satisfy the preset evaluation rule.

17. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, The method is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

18. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images