Large language model evaluation method, device, electronic device and readable storage medium

Abstract

The present disclosure provides a large language model evaluation method, device, electronic device, and readable storage medium. The large language model evaluation method includes: obtaining a sample to be evaluated corresponding to the large language model to be evaluated; inputting the sample to be evaluated into an evaluation planning agent to obtain a target evaluation plan output by the evaluation planning agent; inputting the target evaluation plan into a tool verification agent to obtain target evaluation tool information for at least one target evaluation dimension output by the tool verification agent; using a target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result for the at least one target evaluation dimension based on the sample to be evaluated; inputting the sample to be evaluated and the first evaluation result for the at least one target evaluation dimension into a result verification agent to obtain an evaluation conclusion for the large language model to be evaluated output by the result verification agent.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, particularly to artificial intelligence technologies such as natural language processing, large language models, deep learning, and big data. A large language model evaluation method, device, electronic device, and readable storage medium are provided. Background Art

[0002] With the rapid development of artificial intelligence (AI), large language models have demonstrated remarkable performance in a variety of fields. However, their application requires evaluation to determine their performance in the corresponding tasks. Therefore, accurately evaluating large language models is a pressing technical challenge. Summary of the Invention

[0003] According to a first aspect of the present disclosure, a method for evaluating a large language model is provided, comprising: obtaining a sample to be evaluated corresponding to the large language model to be evaluated, the sample to be evaluated including model input content and model output content; inputting the sample to be evaluated into an evaluation planning agent to obtain a target evaluation plan output by the evaluation planning agent, the target evaluation plan including at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension; inputting the target evaluation plan into a tool verification agent to obtain target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent; using a target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated; inputting the sample to be evaluated and the first evaluation result of the at least one target evaluation dimension into a result verification agent to obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0004] According to a second aspect of the present disclosure, an evaluation device for a large language model is provided, comprising: an acquisition unit for acquiring a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content; a planning unit for inputting the sample to be evaluated into an evaluation planning agent to obtain a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension; a first verification unit for inputting the target evaluation plan into a tool verification agent to obtain target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent; a processing unit for using a target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated; and a second verification unit for inputting the sample to be evaluated and the first evaluation result of the at least one target evaluation dimension into a result verification agent to obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0006] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method as described above.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0008] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0010] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0011] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0014] Figure 5 The block diagram is a block diagram of an electronic device for implementing the large language model evaluation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, and various details of the embodiments of the present disclosure are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and mechanisms are omitted in the following description.

[0016] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the evaluation method of the large language model of this embodiment specifically includes the following steps:

[0017] S101: Obtain a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content;

[0018] S102: Input the sample to be evaluated into an evaluation planning agent, and obtain a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information for the at least one target evaluation dimension;

[0019] S103: Input the target evaluation plan into a tool verification agent, and obtain target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent;

[0020] S104: using the target assessment tool corresponding to the target assessment tool information, obtaining a first assessment result of the at least one target assessment dimension according to the sample to be assessed;

[0021] S105. Input the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent, and obtain the evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0022] The large language model evaluation method of this embodiment, on the one hand, completes the evaluation of the large language model to be evaluated through the interaction between the evaluation planning agent, the tool verification agent and the result verification agent. With the help of the planning and decision-making capabilities of each agent, it can achieve the purpose of evaluating the large language model in different scenarios, thereby improving the scope of application of the evaluation. On the other hand, the evaluation planning agent generates a target evaluation plan in real time according to the input sample to be evaluated, avoiding the need to preset evaluation dimensions before evaluation or the low accuracy and poor flexibility that exist when evaluating the large language model only based on fixed evaluation dimensions, thereby improving the evaluation accuracy and evaluation flexibility.

[0023] In this embodiment, a large language model (LLM) refers to a deep learning model trained using a large amount of text data; the large language model can generate natural language text or understand the meaning of natural language text, and is thus used to process a variety of natural language tasks, such as text classification, text question and answer, and dialogue.

[0024] In this embodiment, an agent refers to an agent that can perceive the environment and take actions to achieve specific goals. The agent perceives changes in the environment (for example, through sensors or data input), makes plans and decisions based on its own learned knowledge and algorithms, and then performs actions to influence the environment or achieve predetermined goals.

[0025] When executing S101 in this embodiment, the large language model to be evaluated may be determined first, and then samples to be evaluated may be acquired based on the determined large language model to be evaluated.

[0026] In this embodiment, when executing S101 to determine the large language model to be evaluated, a preset large language model can be used as the large language model to be evaluated, or the large language model corresponding to the input identification information (such as the name and type of the large language model) input from the input end can be used as the large language model to be evaluated.

[0027] After determining the large language model to be evaluated, this embodiment can obtain samples to be evaluated based on the determined large language model to be evaluated. The samples to be evaluated obtained by executing S101 in this embodiment include model input content and model output content. The model output content is the response result obtained by the large language model to be evaluated in response to the model input content (e.g., a query entered by the user); in this embodiment, the data format of the model input content and the model output content can be text.

[0028] For example, the model input content included in the sample to be evaluated obtained during S101 in this embodiment may be "Give the following text an attractive title and summary, extract all characters that appear, and output them in the following format: {'Title': , 'Summary': , 'Characters': []}." "Song A" is a classic song written and performed by singer A. Not only has it had a profound musical impact, its lyrics also deeply depict the true nature of life, resonating widely. The following is a detailed description of "Song A": 1. Text content corresponding to the creative background, e.g., "Song A" is a song written, composed, and performed by singer A...; 2. Text content corresponding to the song content, e.g., "Song A" has well-balanced lyrics and is full of majestic energy...; 3. Text content corresponding to the musical style, e.g., "Song A" combines pop and Chinese elements, with a smooth and infectious melody...; 4. Text content corresponding to social impact and cover songs, e.g., "Song A" has been widely acclaimed since its release, becoming one of singer A's representative works. The song's profound lyrics, beautiful melody, and soulful performance resonated strongly with listeners... etc. Fifth, the textual content summarized for Song A. For example, "Song A" is a song full of philosophy and emotion. It examines life from an ordinary perspective, expressing a profound insight into life and a unique understanding of life through vivid lyrics and a beautiful melody... etc.

[0029] When executing S101, this embodiment inputs the above-mentioned model input content into the large language model to be evaluated, and the obtained model output content can be "{'Title': "Song A": A classic work by singer A, which deeply depicts the true face of life', 'Abstract': "Song A" is a classic song composed and sung by singer A. The lyrics deeply depict the true face of life, showing the hardships and helplessness of ordinary people, reflecting on emotions such as love and friendship, and proposing a profound reflection on the meaning of life. The song combines elements of pop and Chinese style, with a beautiful melody and affectionate singing, which has generated strong resonance. It has been covered by many singers in different versions and has become a timeless classic.', 'Character': [Singer A]}".

[0030] When executing S101 to obtain the sample to be evaluated corresponding to the large language model to be evaluated, this embodiment can obtain the content input in real time at the input end as the model input content, and after inputting the model input content into the large language model to be evaluated, use the output result of the large language model to be evaluated as the model output content; when executing S101, this embodiment can also obtain the model input content and model output content corresponding to the large language model to be evaluated as the sample to be evaluated according to a preset corresponding relationship.

[0031] After executing S101 to obtain the sample to be evaluated, this embodiment executes S102 to input the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent; in this embodiment, the obtained target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of at least one target evaluation dimension.

[0032] In this embodiment, the evaluation planning agent is used to plan in real time the evaluation dimensions that need to be evaluated for the large language model to be evaluated based on the input samples to be evaluated, and output candidate evaluation tool information corresponding to different evaluation dimensions.

[0033] That is to say, in this embodiment, the evaluation planning agent obtains the target evaluation plan for evaluating the large language model in real time based on the input samples to be evaluated, without the need to pre-set different evaluation dimensions, thereby improving the real-time and flexibility in obtaining the target evaluation plan, and avoiding the problem of being unable to evaluate the large language model in the absence of preset evaluation dimensions.

[0034] When executing S102, this embodiment can input the acquired samples to be evaluated into the evaluation planning agent, use at least one evaluation dimension output by the evaluation planning agent as at least one target evaluation dimension, and use the evaluation tool information corresponding to the corresponding evaluation dimension as candidate evaluation tool information of the target evaluation dimension.

[0035] For example, if the sample to be evaluated includes the model input content and model output content in the above example, the target evaluation plan obtained by executing S102 in this embodiment may include an instruction understanding evaluation dimension (used to evaluate whether the model output content understands the intention of the model input content), an instruction compliance evaluation dimension (used to evaluate whether the model output content conforms to the format required by the model input content), a text quantity evaluation dimension (used to evaluate the number of words contained in the model output content) and a text completeness evaluation dimension (used to evaluate whether the model output content is complete), etc.

[0036] Among them, the candidate tool information corresponding to the instruction understanding evaluation dimension can be tool information 1 (tool information 1 corresponds to the intention understanding evaluation tool), the candidate tool information corresponding to the instruction compliance evaluation dimension can be tool information 2 (tool information 2 corresponds to the format verification evaluation tool), the candidate tool information corresponding to the text quantity evaluation dimension can be tool information 3 (tool information 3 corresponds to the word count evaluation tool), and the candidate tool information corresponding to the text completeness evaluation dimension can be tool information 4 (tool information 4 corresponds to the content integrity evaluation tool), etc.

[0037] After executing S102 to obtain the target evaluation plan output by the evaluation planning agent, this embodiment executes S103 to input the target evaluation plan into the tool verification agent to obtain target evaluation tool information of at least one target evaluation dimension output by the tool verification agent; in this embodiment, different target evaluation dimensions correspond to different target evaluation tool information.

[0038] In this embodiment, the tool verification agent is used to verify the input target evaluation plan, that is, to determine whether the evaluation tool corresponding to the candidate evaluation tool information can complete the evaluation of the corresponding target evaluation dimension, for example, whether the intention understanding evaluation tool corresponding to tool information 1 can complete the evaluation of the instruction understanding evaluation dimension.

[0039] In this embodiment, if the evaluation tool corresponding to the candidate evaluation tool information can complete the evaluation of the corresponding target evaluation dimension, the verification of the tool information passes, and the target evaluation tool information corresponding to the target evaluation dimension output by the tool verification agent is the candidate evaluation tool information.

[0040] In this embodiment, if the evaluation tool corresponding to the candidate evaluation tool information cannot complete the evaluation of the corresponding target evaluation dimension, the verification of the tool information fails, and the tool verification intelligent body will regenerate the evaluation tool information based on the target evaluation dimension, and then use the regenerated evaluation tool information as the target evaluation tool information corresponding to the target evaluation dimension.

[0041] In addition, in addition to the ability to verify tool information, the tool verification agent in this embodiment also has the ability to generate tools, that is, the tool verification agent can generate a target evaluation tool corresponding to a certain target evaluation dimension or a certain target evaluation tool information in real time based on the received evaluation tool acquisition request.

[0042] In this embodiment, after receiving the evaluation tool acquisition request, the tool verification intelligent agent first generates a corresponding code according to the first pending evaluation dimension in the evaluation tool acquisition request, and then obtains the initial evaluation tool corresponding to the first pending evaluation dimension based on the generated code. Finally, the obtained initial evaluation tool is tested, for example, whether the initial evaluation tool can operate normally, whether it can obtain evaluation results similar to the evaluation samples (this embodiment can preset evaluation samples corresponding to different evaluation dimensions, and the evaluation samples include input content and its corresponding evaluation results), etc. After passing the test, the initial evaluation tool is used as the target evaluation tool corresponding to the first pending evaluation dimension.

[0043] After executing S103 to obtain the target evaluation tool information of at least one target evaluation dimension output by the tool verification intelligent agent, this embodiment executes S104 to use the target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result of at least one target evaluation dimension based on the sample to be evaluated; in this embodiment, different target evaluation dimensions correspond to different first evaluation results.

[0044] That is to say, this embodiment uses different target evaluation tools to complete the evaluation of the large language model on different target evaluation dimensions based on the samples to be evaluated; specifically, when executing S104, this embodiment inputs the samples to be evaluated into different target evaluation tools respectively, and then obtains the output results of each target evaluation tool as the first evaluation results corresponding to different target evaluation dimensions.

[0045] When executing S104, this embodiment can first obtain the target evaluation tool based on the target evaluation tool information, and then use the obtained target evaluation tool to obtain the corresponding first evaluation result, such as the first evaluation result of the instruction understanding evaluation dimension, the first evaluation result of the text quantity evaluation dimension, etc.

[0046] For example, after executing S104 and inputting the sample to be evaluated into the intention understanding evaluation tool, this embodiment obtains a first evaluation result output by the intention understanding evaluation tool, indicating that the instruction understanding is accurate or inaccurate; after inputting the sample to be evaluated into the word count evaluation tool, the first evaluation result output by the word count evaluation tool, indicating that the number of words meets the requirements or the number of words does not meet the requirements, etc.

[0047] In this embodiment, when executing S104 to obtain the target evaluation tool according to the target evaluation tool information, a search may be first performed in the preset tool library. If the tool is found, the evaluation tool found is used as the target evaluation tool. If the tool is not found, an open source tool is searched for on the Internet (for example, on GitHub). If the tool is found, the open source tool found is used as the target evaluation tool.

[0048] When executing S104, this embodiment may further include the following contents if the corresponding target evaluation tool cannot be obtained according to the target evaluation tool information: using the target evaluation dimension corresponding to the target evaluation tool information for which the target evaluation tool cannot be obtained as the first evaluation dimension to be processed; generating an evaluation tool acquisition request based on the first evaluation dimension to be processed; and sending the obtained evaluation tool acquisition request to the tool verification agent to obtain the target evaluation tool corresponding to the first evaluation dimension to be processed output by the tool verification agent.

[0049] That is to say, in this embodiment, when the target evaluation tool with corresponding target evaluation tool information is not obtained through the tool library or the Internet, the tool verification intelligent agent generates the target evaluation tool in real time according to the corresponding target evaluation dimension, avoiding the problem of being unable to complete the evaluation of a certain target evaluation dimension due to the failure to obtain the target evaluation tool, ensuring the integrity of the evaluation, and further improving the accuracy of the evaluation conclusions obtained.

[0050] It is understandable that, after executing S104 to obtain the target evaluation tool output by the tool verification agent, this embodiment can also add the target evaluation tool to the preset tool library for use in subsequent evaluations of other large language models.

[0051] After executing S104 to obtain a first evaluation result of at least one target evaluation dimension based on the sample to be evaluated, this embodiment executes S105 to input the sample to be evaluated and the first evaluation result of at least one target evaluation dimension into the result verification agent to obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent; wherein the obtained evaluation conclusion can be the score of the large language model to be evaluated, or it can be whether the model output content in the sample to be evaluated is correct or incorrect.

[0052] In this embodiment, the result verification agent is used to perform result verification on at least one first evaluation result based on the input sample to be evaluated to determine whether the first evaluation result is accurate, and then, when it is determined that all the first evaluation results are accurate, obtain the evaluation conclusion of the corresponding large language model to be evaluated based on all the first evaluation results.

[0053] Specifically, when executing S105, this embodiment inputs the first evaluation results of the sample to be evaluated and at least one target evaluation dimension into the result verification agent, and obtains the evaluation conclusion of the large language model to be evaluated output by the result verification agent. The implementation method that can be adopted is: the result verification agent performs result verification on at least one first evaluation result based on the sample to be evaluated; when it is determined that at least one first evaluation result has all passed the result verification, the result verification agent obtains the evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

[0054] In this embodiment, the result verification agent performs result verification on the first evaluation result based on the sample to be evaluated, that is, determines whether the first evaluation result corresponding to a certain target evaluation dimension is consistent with the model input content or model output content in the sample to be evaluated; for example, if the first evaluation result of the corresponding text quantity evaluation dimension output by the word count evaluation tool is "a large number of words, meeting the word count requirement", but in fact the model output content contains fewer words, it indicates that the first evaluation result is inconsistent with the model output content, and the first evaluation result corresponding to the text quantity evaluation dimension fails the result verification.

[0055] When executing S105, this embodiment may also include the following contents: using the target evaluation dimension corresponding to the first evaluation result that fails the result verification as the second evaluation dimension to be processed; using the target evaluation tool corresponding to the second evaluation dimension to be processed to obtain the second evaluation result of the second evaluation dimension to be processed based on the sample to be processed; the result verification agent performs result verification on the second evaluation result based on the input sample to be evaluated; when it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second evaluation dimension to be processed is updated to the second evaluation result, and then the result verification agent obtains the evaluation conclusion based on the updated first evaluation result.

[0056] That is to say, this embodiment can obtain a second evaluation result for updating the first evaluation result by repeatedly calling the corresponding target evaluation tool when the first evaluation result fails the result verification, thereby ensuring the accuracy of the first evaluation result used by the result verification agent to obtain the evaluation conclusion.

[0057] It can be understood that when executing S105, this embodiment can continue to call the corresponding target evaluation tool to obtain the second evaluation result again, and then the result verification agent performs result verification on the second evaluation result obtained again, and this cycle is repeated until it is determined that the second evaluation result obtained passes the result verification or the number of times the second evaluation result fails the result verification reaches a preset number.

[0058] In addition, when executing S105, this embodiment may also include the following contents: when it is determined that the second evaluation result has failed the result verification for a preset number of times, the result verification agent generates a plan acquisition request based on the second evaluation result, and the generated plan acquisition request includes error information of the second evaluation result, problems that occur, etc.; the generated plan acquisition request is sent to the evaluation planning agent; the evaluation planning agent re-obtains the target evaluation plan based on the sample to be evaluated and the plan acquisition request, and uses the re-obtained target evaluation plan to re-evaluate the large language model to be evaluated.

[0059] That is to say, in this embodiment, when the target evaluation tool is called multiple times and the second evaluation result that passes the result verification is still not obtained, the result verification agent generates a plan acquisition request and sends it to the evaluation planning agent, so that the evaluation planning agent can re-plan to obtain the target evaluation plan again, thereby re-evaluating the large language model to be evaluated. Through the interaction between the two agents, the problem of not being able to obtain an evaluation conclusion due to an erroneous first evaluation result is avoided, and the accuracy of the obtained evaluation conclusion is further improved.

[0060] Figure 2 Schematic diagram of the second embodiment of the present disclosure. Figure 2 As shown, when executing S102 "inputting the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent", this embodiment may include the following steps:

[0061] S201: Input the sample to be evaluated into the evaluation planning agent, and obtain an initial evaluation plan output by the evaluation planning agent, wherein the initial evaluation plan includes at least one initial evaluation dimension and initial evaluation tool information of the at least one initial evaluation dimension;

[0062] S202: input the initial assessment plan into the assessment planning agent, and the assessment planning agent updates the at least one initial assessment dimension and / or the initial assessment tool information of the at least one initial assessment dimension;

[0063] S203: When it is determined that the update of the initial evaluation plan meets the preset requirements, the target evaluation plan is obtained according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

[0064] That is to say, this embodiment obtains the target evaluation plan by updating the initial evaluation dimensions and / or initial tool information in the initial evaluation plan through the evaluation planning agent. Since the current initial evaluation plan will be checked and supplemented by the evaluation planning agent during the update, there will be no errors or omissions in the obtained target evaluation plan, thereby further improving the accuracy of the obtained target evaluation plan.

[0065] When executing S202 in this embodiment, the evaluation planning agent updates the initial evaluation plan at least once, that is, with the help of the evaluation planning agent's own planning and decision-making capabilities, the initial evaluation plan is continuously reflected upon to determine whether the initial evaluation dimensions are accurate and complete, and whether the initial evaluation tool information corresponding to the initial evaluation dimensions is accurate.

[0066] In addition, when executing S202, this embodiment can also input the sample to be evaluated into the evaluation planning agent, so that the evaluation planning agent can update at least one initial evaluation dimension and / or the initial evaluation tool information of at least one initial evaluation dimension in combination with the sample to be evaluated.

[0067] In this embodiment, when the initial evaluation plan is updated at least once, the result of the update of the initial evaluation plan is used as the input of the next evaluation plan agent, and this cycle is repeated until the update meets the preset requirements.

[0068] When executing S203 to determine whether the update of the initial assessment plan meets the preset requirements, this embodiment may determine that the preset requirements are met when it is determined that the number of updates to the initial assessment plan reaches the preset number of updates, or may determine that the preset requirements are met when it is determined that there are no errors in the updated initial assessment plan (initial assessment dimension errors or initial assessment tool information errors) or the number of initial assessment dimensions reaches the preset number.

[0069] After executing S203 to determine that the update of the initial evaluation plan meets the preset requirements, this embodiment obtains the target evaluation plan based on the result of the last update of the initial evaluation plan by the evaluation planning agent, for example, using the evaluation dimension obtained by the last update as the target evaluation dimension, and using the evaluation tool information corresponding to the corresponding evaluation dimension obtained by the last update as the candidate tool information of the corresponding target evaluation dimension.

[0070] Figure 3 Schematic diagram of the third embodiment of the present disclosure. Figure 3 As shown in , this embodiment shows an evaluation framework for a large language model; in this evaluation framework, the evaluation planning agent, the tool verification agent and the result verification agent are all obtained through pre-training and have corresponding capabilities, for example, the evaluation planning agent has the ability to generate an evaluation plan, the tool verification agent has the ability to verify evaluation tool information, and the result verification agent has the ability to verify evaluation results and generate evaluation conclusions; after obtaining the samples to be evaluated corresponding to the large language model to be evaluated, this embodiment inputs the obtained samples to be evaluated into the evaluation framework, and can obtain the evaluation conclusion of the large language model to be evaluated output by the evaluation framework, such as the evaluation score. The higher the obtained evaluation score, the more accurately the large language model to be evaluated will respond to the input content.

[0071] Figure 4 Schematic diagram of the fourth embodiment of the present disclosure. Figure 4 As shown, the large language model evaluation device 400 of this embodiment includes:

[0072] An acquisition unit 401 is configured to acquire a sample to be evaluated corresponding to a large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content;

[0073] The planning unit 402 is configured to input the sample to be evaluated into an evaluation planning agent, and obtain a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information for the at least one target evaluation dimension;

[0074] The first verification unit 403 is configured to input the target evaluation plan into a tool verification agent, and obtain target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent;

[0075] The processing unit 404 is configured to obtain a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated using the target evaluation tool corresponding to the target evaluation tool information;

[0076] The second verification unit 405 is used to input the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent, and obtain the evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0077] The acquisition unit 401 may first determine a large language model to be evaluated, and then acquire samples to be evaluated according to the determined large language model to be evaluated.

[0078] When determining the large language model to be evaluated, the acquisition unit 401 can use a preset large language model as the large language model to be evaluated, or it can use the large language model corresponding to the input identification information (such as the name, type, and other information of the large language model) as the large language model to be evaluated based on the identification information of the large language model input from the input end.

[0079] After determining the large language model to be evaluated, this embodiment can then acquire samples to be evaluated based on the determined large language model to be evaluated. The samples to be evaluated acquired by acquisition unit 401 include model input content and model output content. The model output content is the response result obtained by the large language model to be evaluated in response to the model input content (e.g., a query entered by a user); in this embodiment, the data format of the model input content and the model output content can be text.

[0080] When acquiring the sample to be evaluated corresponding to the large language model to be evaluated, the acquisition unit 401 can acquire the content input in real time at the input end as the model input content, and after inputting the model input content into the large language model to be evaluated, use the output result of the large language model to be evaluated as the model output content; the acquisition unit 401 can also acquire the model input content and model output content corresponding to the large language model to be evaluated as the sample to be evaluated according to a preset corresponding relationship.

[0081] In this embodiment, after the acquisition unit 401 acquires the sample to be evaluated, the planning unit 402 inputs the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent; in this embodiment, the obtained target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of at least one target evaluation dimension.

[0082] In this embodiment, the evaluation planning agent is used to plan in real time the evaluation dimensions that need to be evaluated for the large language model to be evaluated based on the input samples to be evaluated, and output candidate evaluation tool information corresponding to different evaluation dimensions.

[0083] That is to say, in this embodiment, the evaluation planning agent obtains the target evaluation plan for evaluating the large language model in real time based on the input samples to be evaluated, without the need to pre-set different evaluation dimensions, thereby improving the real-time and flexibility in obtaining the target evaluation plan, and avoiding the problem of being unable to evaluate the large language model in the absence of preset evaluation dimensions.

[0084] The planning unit 402 can input the acquired samples to be evaluated into the evaluation planning agent, use at least one evaluation dimension output by the evaluation planning agent as at least one target evaluation dimension, and use the evaluation tool information corresponding to the corresponding evaluation dimension as candidate evaluation tool information of the target evaluation dimension.

[0085] In addition, when the planning unit 402 inputs the sample to be evaluated into the evaluation planning agent and obtains the target evaluation plan output by the evaluation planning agent, it can also include the following contents: inputting the sample to be evaluated into the evaluation planning agent to obtain the initial evaluation plan output by the evaluation planning agent, the initial evaluation plan including at least one initial evaluation dimension and initial evaluation tool information of at least one initial evaluation dimension; inputting the initial evaluation plan into the evaluation planning agent, and having the evaluation planning agent update at least one initial evaluation dimension and / or initial evaluation tool information of at least one initial evaluation dimension; when it is determined that the update of the initial evaluation plan meets the preset requirements, obtaining the target evaluation plan according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

[0086] That is to say, the planning unit 402 obtains the target evaluation plan by updating the initial evaluation dimensions and / or initial tool information in the initial evaluation plan through the evaluation planning agent. Since the evaluation planning agent will check for omissions and fill in the gaps in the current initial evaluation plan during the update, there will be no errors or omissions in the obtained target evaluation plan, thereby further improving the accuracy of the obtained target evaluation plan.

[0087] In this embodiment, after the planning unit 402 obtains the target evaluation plan output by the evaluation planning agent, the first verification unit 403 inputs the target evaluation plan into the tool verification agent to obtain target evaluation tool information of at least one target evaluation dimension output by the tool verification agent; in this embodiment, different target evaluation dimensions correspond to different target evaluation tool information.

[0088] In this embodiment, the tool verification agent is used to verify the input target evaluation plan, that is, to determine whether the evaluation tool corresponding to the candidate evaluation tool information can complete the evaluation of the corresponding target evaluation dimension.

[0089] In this embodiment, after the first verification unit 403 obtains the target evaluation tool information of at least one target evaluation dimension output by the tool verification intelligent agent, the processing unit 404 uses the target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result of at least one target evaluation dimension based on the sample to be evaluated; in this embodiment, different target evaluation dimensions correspond to different first evaluation results.

[0090] That is to say, this embodiment uses different target evaluation tools to complete the evaluation of the large language model on different target evaluation dimensions based on the samples to be evaluated; specifically, the processing unit 404 inputs the samples to be evaluated into different target evaluation tools respectively, and then obtains the output results of each target evaluation tool as the first evaluation results corresponding to different target evaluation dimensions.

[0091] The processing unit 404 can first obtain the target evaluation tool based on the target evaluation tool information, and then use the obtained target evaluation tool to obtain the corresponding first evaluation result, such as the first evaluation result of the instruction understanding evaluation dimension, the first evaluation result of the text quantity evaluation dimension, etc.

[0092] When the processing unit 404 obtains the target evaluation tool based on the target evaluation tool information, it can first search in the preset tool library. If the evaluation tool is found, the evaluation tool found is used as the target evaluation tool; if the tool is not found, the open source tool is searched through the Internet (for example, on GitHub). If the tool is found, the open source tool found is used as the target evaluation tool.

[0093] If the corresponding target evaluation tool cannot be obtained according to the target evaluation tool information, the processing unit 404 may also include the following contents: using the target evaluation dimension corresponding to the target evaluation tool information for which the target evaluation tool cannot be obtained as the first evaluation dimension to be processed; generating an evaluation tool acquisition request based on the first evaluation dimension to be processed; sending the obtained evaluation tool acquisition request to the tool verification agent to obtain the target evaluation tool corresponding to the first evaluation dimension to be processed output by the tool verification agent.

[0094] That is to say, in this embodiment, when the target evaluation tool with corresponding target evaluation tool information is not obtained through the tool library or the Internet, the tool verification intelligent agent generates the target evaluation tool in real time according to the corresponding target evaluation dimension, avoiding the problem of being unable to complete the evaluation of a certain target evaluation dimension due to the failure to obtain the target evaluation tool, ensuring the integrity of the evaluation, and further improving the accuracy of the evaluation conclusions obtained.

[0095] It is understandable that, after obtaining the target evaluation tool output by the tool verification agent, the processing unit 404 can also add the target evaluation tool to the preset tool library for use in subsequent evaluations of other large language models.

[0096] In this embodiment, after the processing unit 404 obtains the first evaluation result of at least one target evaluation dimension based on the sample to be evaluated, the second verification unit 405 inputs the sample to be evaluated and the first evaluation result of at least one target evaluation dimension into the result verification agent to obtain the evaluation conclusion of the large language model to be evaluated output by the result verification agent; wherein the obtained evaluation conclusion can be the score of the large language model to be evaluated, or it can be whether the model output content in the sample to be evaluated is correct or incorrect.

[0097] In this embodiment, the result verification agent is used to perform result verification on at least one first evaluation result based on the input sample to be evaluated to determine whether the first evaluation result is accurate, and then, when it is determined that all the first evaluation results are accurate, obtain the evaluation conclusion of the corresponding large language model to be evaluated based on all the first evaluation results.

[0098] Specifically, when the second verification unit 405 inputs the first evaluation results of the sample to be evaluated and at least one target evaluation dimension into the result verification agent and obtains the evaluation conclusion of the large language model to be evaluated output by the result verification agent, the implementation method that can be adopted is: the result verification agent performs result verification on at least one first evaluation result based on the sample to be evaluated; when it is determined that at least one first evaluation result has passed the result verification, the result verification agent obtains the evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

[0099] The second verification unit 405 can also perform the following: use the target evaluation dimension corresponding to the first evaluation result that fails the result verification as the second evaluation dimension to be processed; use the target evaluation tool corresponding to the second evaluation dimension to be processed to obtain the second evaluation result of the second evaluation dimension to be processed based on the sample to be processed; the result verification agent performs result verification on the second evaluation result based on the input sample to be evaluated; when it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second evaluation dimension to be processed is updated to the second evaluation result, and then the result verification agent obtains the evaluation conclusion based on the updated first evaluation result.

[0100] That is to say, this embodiment can obtain a second evaluation result for updating the first evaluation result by repeatedly calling the corresponding target evaluation tool when the first evaluation result fails the result verification, thereby ensuring the accuracy of the first evaluation result used by the result verification agent to obtain the evaluation conclusion.

[0101] It can be understood that if the second evaluation result still fails to pass the result verification of the result verification agent, the second verification unit 405 can continue to call the corresponding target evaluation tool to obtain the second evaluation result again, and then the result verification agent performs result verification on the second evaluation result obtained again, and this cycle is repeated until it is determined that the second evaluation result obtained passes the result verification or the number of times the second evaluation result fails the result verification reaches a preset number.

[0102] The second verification unit 405 can also perform the following: when it is determined that the second evaluation result has failed the result verification for a preset number of times, the result verification agent generates a plan acquisition request based on the second evaluation result, and the generated plan acquisition request includes error information of the second evaluation result, problems that occur, etc.; the generated plan acquisition request is sent to the evaluation planning agent; the evaluation planning agent re-obtains the target evaluation plan based on the sample to be evaluated and the plan acquisition request, and uses the re-obtained target evaluation plan to re-evaluate the large language model to be evaluated.

[0103] That is to say, in this embodiment, when the target evaluation tool is called multiple times and the second evaluation result that passes the result verification is still not obtained, the result verification agent generates a plan acquisition request and sends it to the evaluation planning agent, so that the evaluation planning agent can re-plan to obtain the target evaluation plan again, thereby re-evaluating the large language model to be evaluated. Through the interaction between the two agents, the problem of not being able to obtain an evaluation conclusion due to an erroneous first evaluation result is avoided, and the accuracy of the obtained evaluation conclusion is further improved.

[0104] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0105] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0106] like Figure 5 , is a block diagram of an electronic device for the evaluation method of a large language model according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0107] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0108] Various components in device 500 are connected to I / O interface 505, including: an input unit 506, such as a keyboard, mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, optical disk, etc.; and a communication unit 509, such as a network card, modem, wireless communication transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 501 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the evaluation method of the large language model. For example, in some embodiments, the evaluation method of the large language model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508.

[0110] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the large language model evaluation method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the large language model evaluation method in any other appropriate manner (e.g., by means of firmware).

[0111] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable large language model evaluation device so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service system that addresses the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Servers," or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0117] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0118] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A large language model evaluation method, comprising: Obtaining a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content; Inputting the sample to be evaluated into an evaluation planning agent, obtaining a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension; Inputting the target assessment plan into a tool verification agent, and obtaining target assessment tool information of the at least one target assessment dimension output by the tool verification agent; Obtaining a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated using the target evaluation tool corresponding to the target evaluation tool information; Inputting the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into a result verification agent, and obtaining an evaluation conclusion of the large language model to be evaluated output by the result verification agent; The step of inputting the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent includes: Inputting the sample to be evaluated into the evaluation planning agent to obtain an initial evaluation plan output by the evaluation planning agent, wherein the initial evaluation plan includes at least one initial evaluation dimension and initial evaluation tool information of the at least one initial evaluation dimension; Inputting the initial assessment plan into the assessment planning agent, and having the assessment planning agent update the at least one initial assessment dimension and / or initial assessment tool information of the at least one initial assessment dimension; When it is determined that the update of the initial evaluation plan meets the preset requirements, the target evaluation plan is obtained according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

2. The method according to claim 1, wherein Inputting the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent, and obtaining the evaluation conclusion of the large language model to be evaluated output by the result verification agent includes: The result verification agent performs result verification on at least one first evaluation result based on the sample to be evaluated; When it is determined that all of the at least one first evaluation result passes the result verification, the result verification agent obtains an evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

3. The method according to claim 2, further comprising: The target evaluation dimension corresponding to the first evaluation result that fails the result verification is used as the second evaluation dimension to be processed; Using a target assessment tool corresponding to the second pending assessment dimension, obtaining a second assessment result of the second pending assessment dimension based on the sample to be assessed; The result verification agent performs result verification on the second evaluation result based on the sample to be evaluated; When it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second to-be-processed evaluation dimension is updated to the second evaluation result.

4. The method according to claim 3, further comprising: When it is determined that the second evaluation result has failed the result verification a number of times reaching a preset number, the result verification agent generates a plan acquisition request based on the second evaluation result; Sending the plan acquisition request to the evaluation planning agent; The evaluation planning agent re-obtains a target evaluation plan based on the sample to be evaluated and the plan acquisition request, so as to re-evaluate the large language model to be evaluated based on the re-obtained target plan request.

5. The method according to claim 1, further comprising: The target evaluation dimension corresponding to the target evaluation tool information for which the target evaluation tool cannot be obtained is used as the first evaluation dimension to be processed; generating an evaluation tool acquisition request according to the first to-be-processed evaluation dimension; The evaluation tool acquisition request is sent to the tool verification agent to obtain the target evaluation tool corresponding to the first evaluation dimension to be processed output by the tool verification agent.

6. A large language model evaluation device, comprising: An acquisition unit, configured to acquire a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content; A planning unit, configured to input the sample to be evaluated into an evaluation planning agent, and obtain a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information for the at least one target evaluation dimension; A first verification unit is configured to input the target evaluation plan into a tool verification agent, and obtain target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent; a processing unit, configured to obtain a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated, using the target evaluation tool corresponding to the target evaluation tool information; A second verification unit is configured to input the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into a result verification agent, and obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent; When the planning unit inputs the sample to be evaluated into the evaluation planning agent and obtains the target evaluation plan output by the evaluation planning agent, it specifically performs the following steps: Inputting the sample to be evaluated into the evaluation planning agent to obtain an initial evaluation plan output by the evaluation planning agent, wherein the initial evaluation plan includes at least one initial evaluation dimension and initial evaluation tool information of the at least one initial evaluation dimension; Inputting the initial assessment plan into the assessment planning agent, and having the assessment planning agent update the at least one initial assessment dimension and / or initial assessment tool information of the at least one initial assessment dimension; When it is determined that the update of the initial evaluation plan meets the preset requirements, the target evaluation plan is obtained according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

7. The device according to claim 6, wherein When the second verification unit inputs the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent and obtains the evaluation conclusion of the large language model to be evaluated output by the result verification agent, the second verification unit specifically performs: The result verification agent performs result verification on at least one first evaluation result based on the sample to be evaluated; When it is determined that all of the at least one first evaluation result passes the result verification, the result verification agent obtains an evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

8. The apparatus according to claim 7, wherein the second verification unit is further configured to perform: The target evaluation dimension corresponding to the first evaluation result that fails the result verification is used as the second evaluation dimension to be processed; Using a target assessment tool corresponding to the second pending assessment dimension, obtaining a second assessment result for the second pending assessment dimension based on the pending assessment sample; The result verification agent performs result verification on the second evaluation result based on the sample to be evaluated; When it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second to-be-processed evaluation dimension is updated to the second evaluation result.

9. The apparatus according to claim 8, wherein the second verification unit is further configured to perform: When it is determined that the second evaluation result has failed the result verification a number of times reaching a preset number, the result verification agent generates a plan acquisition request based on the second evaluation result; Sending the plan acquisition request to the evaluation planning agent; The evaluation planning agent re-obtains a target evaluation plan based on the sample to be evaluated and the plan acquisition request, so as to re-evaluate the large language model to be evaluated based on the re-obtained target plan request.

10. The apparatus according to claim 6, wherein the processing unit is further configured to execute: The target evaluation dimension corresponding to the target evaluation tool information for which the target evaluation tool cannot be obtained is used as the first evaluation dimension to be processed; generating an evaluation tool acquisition request according to the first to-be-processed evaluation dimension; The evaluation tool acquisition request is sent to the tool verification agent to obtain the target evaluation tool corresponding to the first evaluation dimension to be processed output by the tool verification agent.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images