Assessment method and device of large language model, electronic equipment and readable storage medium

Abstract

The invention provides a large language model evaluation method and device, electronic equipment and a readable storage medium. The assessment method for the large language model comprises the steps of obtaining a to-be-assessed sample corresponding to a to-be-assessed large language model; inputting the to-be-evaluated sample into an evaluation planning agent to obtain a target evaluation plan output by the evaluation planning agent; inputting the target evaluation plan into a tool verification intelligent agent to obtain target evaluation tool information of the at least one target evaluation dimension output by the tool verification intelligent agent; obtaining a first evaluation result of the at least one target evaluation dimension according to the to-be-evaluated sample by using a target evaluation tool corresponding to the target evaluation tool information; and inputting the to-be-evaluated sample and the first evaluation result of the at least one target evaluation dimension into a result verification agent to obtain an evaluation conclusion of the to-be-evaluated large language model output by the result verification agent.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, in particular to the field of artificial intelligence technology such as natural language processing, large language models, deep learning, big data, etc. A large language model evaluation method, device, electronic device and readable storage medium are provided. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models have demonstrated excellent performance in many fields. In the application process of large language models, it is necessary to evaluate the large language models to determine whether the large language models have good performance when completing the corresponding tasks. Therefore, how to accurately evaluate large language models is a technical problem that needs to be solved urgently. Summary of the invention

[0003] According to a first aspect of the present disclosure, a method for evaluating a large language model is provided, comprising: obtaining a sample to be evaluated corresponding to the large language model to be evaluated, the sample to be evaluated including model input content and model output content; inputting the sample to be evaluated into an evaluation planning agent to obtain a target evaluation plan output by the evaluation planning agent, the target evaluation plan including at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension; inputting the target evaluation plan into a tool verification agent to obtain target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent; using a target evaluation tool corresponding to the target evaluation tool information, obtaining a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated; inputting the sample to be evaluated and the first evaluation result of the at least one target evaluation dimension into a result verification agent to obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0004] According to a second aspect of the present disclosure, there is provided an evaluation device for a large language model, comprising: an acquisition unit, for acquiring a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content; a planning unit, for inputting the sample to be evaluated into an evaluation planning agent, and obtaining a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension; a first verification unit, for inputting the target evaluation plan into a tool verification agent, and obtaining target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent; a processing unit, for using a target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated; and a second verification unit, for inputting the sample to be evaluated and the first evaluation result of the at least one target evaluation dimension into a result verification agent, and obtaining an evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0006] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.

[0008] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0010] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0011] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0014] Figure 5 The present invention is a block diagram of an electronic device for implementing the large language model evaluation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and mechanisms is omitted in the following description.

[0016] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the evaluation method of the large language model of this embodiment specifically includes the following steps:

[0017] S101, obtaining a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content;

[0018] S102, inputting the sample to be evaluated into the evaluation planning agent, obtaining a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension;

[0019] S103, inputting the target evaluation plan into the tool verification agent, and obtaining the target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent;

[0020] S104, using the target evaluation tool corresponding to the target evaluation tool information, obtaining a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated;

[0021] S105. Input the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into a result verification agent, and obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0022] The large language model evaluation method of the present embodiment, on the one hand, completes the evaluation of the large language model to be evaluated through the interaction between the evaluation planning agent, the tool verification agent and the result verification agent. With the help of the planning and decision-making capabilities of each agent, it can achieve the purpose of evaluating the large language model in different scenarios, thereby improving the scope of application of the evaluation. On the other hand, the evaluation planning agent generates a target evaluation plan in real time according to the input sample to be evaluated, avoiding the need to preset evaluation dimensions before evaluation or the low accuracy and poor flexibility that exist when evaluating the large language model only according to fixed evaluation dimensions, thereby improving the evaluation accuracy and evaluation flexibility.

[0023] In this embodiment, a large language model (LLM) refers to a deep learning model trained using a large amount of text data; the large language model can generate natural language text or understand the meaning of natural language text, and is thus used to process a variety of natural language tasks, such as text classification, text question and answer, and dialogue.

[0024] In this embodiment, an agent refers to an agent that can perceive the environment and take actions to achieve specific goals; the agent perceives changes in the environment (for example, through sensors or data input), makes plans and decisions based on the knowledge and algorithms it has learned, and then performs actions to affect the environment or achieve predetermined goals.

[0025] When executing S101 in this embodiment, a large language model to be evaluated may be determined first, and then samples to be evaluated may be acquired according to the determined large language model to be evaluated.

[0026] When executing S101 to determine the large language model to be evaluated, this embodiment can use a preset large language model as the large language model to be evaluated, or can use the large language model corresponding to the input identification information (such as the name and type of the large language model) as the large language model to be evaluated based on the identification information of the large language model input by the input end.

[0027] After the large language model to be evaluated is determined, the present embodiment can obtain samples to be evaluated based on the determined large language model to be evaluated. The samples to be evaluated obtained by executing S101 in the present embodiment include model input content and model output content, and the model output content is the response result obtained by the large language model to be evaluated responding to the model input content (such as the query input by the user); in the present embodiment, the data format of the model input content and the model output content can be text.

[0028] For example, the model input content included in the sample to be evaluated obtained by executing S101 in this embodiment can be "Give the following text an attractive title and summary, and extract all the characters that appear, and output them in the following format: {'title':, 'summary':, 'character': []}; "Song A" is a classic song created and sung by singer A. It not only has a profound influence in music, but its lyrics also deeply depict the true face of life and arouse widespread resonance. The following is a detailed introduction to Song A: 1. Text content corresponding to the creation background, for example, Song A is a song written, composed and sung by singer A... etc.; 2. Text content corresponding to the content of the song, for example, the lyrics of Song A are well-balanced and full of majestic spirit... etc.; 3. Text content corresponding to the music style, for example, the music style of Song A combines elements of pop and Chinese style, and the melody is smooth and infectious... etc.; 4. Text content corresponding to social influence and cover songs, for example, Song A has been widely praised since its release and has become one of the representative works of singer A. The profound content of the lyrics, the beautiful melody, and the affectionate singing made this song resonate strongly among the audience...etc.; 5. The text content summarized in the corresponding song A, for example, "Song A" is a song full of philosophy and emotion. It examines life from the perspective of ordinary people, and expresses a deep insight into life and a unique understanding of life with vivid lyrics and beautiful melody...etc. ".

[0029] When executing S101, this embodiment inputs the above model input content into the large language model to be evaluated, and the obtained model output content can be "{'Title': "Song A": A classic work by singer A, which deeply depicts the true face of life', 'Abstract': "Song A" is a classic song created and sung by singer A. The lyrics deeply depict the true face of life, show the hardships and helplessness of ordinary people, think about emotions such as love and friendship, and put forward a profound reflection on the meaning of life. The song combines elements of pop and Chinese style, with a beautiful melody and affectionate singing, which has generated strong resonance. It has been covered in different versions by many singers and has become a timeless classic.', 'Character': [Singer A]}".

[0030] When executing S101 to obtain the sample to be evaluated corresponding to the large language model to be evaluated, this embodiment can obtain the content of the real-time input of the input end as the model input content, and after inputting the model input content into the large language model to be evaluated, use the output result of the large language model to be evaluated as the model output content; when executing S101, this embodiment can also obtain the model input content and the model output content corresponding to the large language model to be evaluated as the sample to be evaluated according to the preset corresponding relationship.

[0031] After executing S101 to obtain the sample to be evaluated, this embodiment executes S102 to input the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent; in this embodiment, the obtained target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of at least one target evaluation dimension.

[0032] In this embodiment, the evaluation planning agent is used to plan in real time the evaluation dimensions that need to be evaluated for the large language model to be evaluated this time according to the input samples to be evaluated, and output the candidate evaluation tool information corresponding to different evaluation dimensions.

[0033] That is to say, in this embodiment, the evaluation planning agent obtains the target evaluation plan for evaluating the large language model in real time according to the input samples to be evaluated, without the need to pre-set different evaluation dimensions, thereby improving the real-time and flexibility in obtaining the target evaluation plan, and avoiding the problem of being unable to evaluate the large language model in the absence of preset evaluation dimensions.

[0034] When executing S102, this embodiment can input the acquired samples to be evaluated into the evaluation planning agent, use at least one evaluation dimension output by the evaluation planning agent as at least one target evaluation dimension, and use the evaluation tool information corresponding to the corresponding evaluation dimension as candidate evaluation tool information of the target evaluation dimension.

[0035] For example, if the sample to be evaluated includes the model input content and model output content in the above example, the target evaluation plan obtained by executing S102 in this embodiment may include an instruction understanding evaluation dimension (used to evaluate whether the model output content understands the intention of the model input content), an instruction compliance evaluation dimension (used to evaluate whether the model output content conforms to the format required by the model input content), a text quantity evaluation dimension (used to evaluate the number of words contained in the model output content) and a text completeness evaluation dimension (used to evaluate whether the model output content is complete), etc.

[0036] Among them, the candidate tool information corresponding to the instruction understanding evaluation dimension can be tool information 1 (tool information 1 corresponds to the intention understanding evaluation tool), the candidate tool information corresponding to the instruction compliance evaluation dimension can be tool information 2 (tool information 2 corresponds to the format verification evaluation tool), the candidate tool information corresponding to the text quantity evaluation dimension can be tool information 3 (tool information 3 corresponds to the word count evaluation tool), and the candidate tool information corresponding to the text completeness evaluation dimension can be tool information 4 (tool information 4 corresponds to the content completeness evaluation tool), etc.

[0037] After executing S102 to obtain the target evaluation plan output by the evaluation planning agent, this embodiment executes S103 to input the target evaluation plan into the tool verification agent to obtain target evaluation tool information of at least one target evaluation dimension output by the tool verification agent; in this embodiment, different target evaluation dimensions correspond to different target evaluation tool information.

[0038] In this embodiment, the tool verification agent is used to verify the input target evaluation plan, that is, to determine whether the evaluation tool corresponding to the candidate evaluation tool information can complete the evaluation of the corresponding target evaluation dimension, for example, whether the intention understanding evaluation tool corresponding to the tool information 1 can complete the evaluation of the instruction understanding evaluation dimension.

[0039] In this embodiment, if the evaluation tool corresponding to the candidate evaluation tool information can complete the evaluation of the corresponding target evaluation dimension, the verification of the tool information passes, and the target evaluation tool information corresponding to the target evaluation dimension output by the tool verification agent is the candidate evaluation tool information.

[0040] In this embodiment, if the evaluation tool corresponding to the candidate evaluation tool information cannot complete the evaluation of the corresponding target evaluation dimension, the verification of the tool information fails, and the tool verification intelligent body will regenerate the evaluation tool information according to the target evaluation dimension, and then use the regenerated evaluation tool information as the target evaluation tool information corresponding to the target evaluation dimension.

[0041] In addition, the tool verification agent in this embodiment has the ability to verify tool information as well as the ability to generate tools, that is, the tool verification agent can generate a target evaluation tool corresponding to a certain target evaluation dimension or corresponding to a certain target evaluation tool information in real time based on the received evaluation tool acquisition request.

[0042] In this embodiment, after receiving the evaluation tool acquisition request, the tool verification agent first generates a corresponding code according to the first to-be-processed evaluation dimension in the evaluation tool acquisition request, then obtains the initial evaluation tool corresponding to the first to-be-processed evaluation dimension according to the generated code, and finally tests the obtained initial evaluation tool, for example, testing whether the initial evaluation tool can operate normally, whether it can obtain evaluation results similar to the evaluation samples (in this embodiment, evaluation samples corresponding to different evaluation dimensions can be preset, and the evaluation samples include input content and its corresponding evaluation results), etc. After passing the test, the initial evaluation tool is used as the target evaluation tool corresponding to the first to-be-processed evaluation dimension.

[0043] After executing S103 to obtain target evaluation tool information of at least one target evaluation dimension output by the tool verification agent, this embodiment executes S104 to use the target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result of at least one target evaluation dimension based on the sample to be evaluated; in this embodiment, different target evaluation dimensions correspond to different first evaluation results.

[0044] That is to say, this embodiment uses different target evaluation tools to complete the evaluation of the large language model on different target evaluation dimensions based on the samples to be evaluated; specifically, when executing S104, this embodiment inputs the samples to be evaluated into different target evaluation tools respectively, and then obtains the output results of each target evaluation tool as the first evaluation results corresponding to different target evaluation dimensions.

[0045] When executing S104, this embodiment can first obtain the target evaluation tool according to the target evaluation tool information, and then use the obtained target evaluation tool to obtain the corresponding first evaluation result, such as the first evaluation result of the instruction understanding evaluation dimension, the first evaluation result of the text quantity evaluation dimension, etc.

[0046] For example, after executing S104 and inputting the sample to be evaluated into the intention understanding evaluation tool, this embodiment obtains a first evaluation result output by the intention understanding evaluation tool, indicating that the instruction understanding is accurate or the instruction understanding is inaccurate; after inputting the sample to be evaluated into the word count evaluation tool, the first evaluation result output by the word count evaluation tool, indicating that the number of words meets the requirement or the number of words does not meet the requirement, etc.

[0047] When executing S104 to obtain the target evaluation tool according to the target evaluation tool information, this embodiment can first search in the preset tool library. If the evaluation tool is found, the evaluation tool found is used as the target evaluation tool. If the evaluation tool is not found, the open source tool is searched through the Internet (for example, searching on GitHub). If the evaluation tool is found, the open source tool found is used as the target evaluation tool.

[0048] When executing S104, this embodiment may further include the following contents if the corresponding target evaluation tool cannot be obtained according to the target evaluation tool information: using the target evaluation dimension corresponding to the target evaluation tool information for which the target evaluation tool cannot be obtained as the first evaluation dimension to be processed; generating an evaluation tool acquisition request according to the first evaluation dimension to be processed; and sending the obtained evaluation tool acquisition request to the tool verification agent to obtain the target evaluation tool corresponding to the first evaluation dimension to be processed output by the tool verification agent.

[0049] That is to say, in the present embodiment, when the target evaluation tool whose corresponding target evaluation tool information is not obtained through the tool library or the Internet, the tool verification intelligent agent generates the target evaluation tool in real time according to the corresponding target evaluation dimension, thereby avoiding the problem of being unable to complete the evaluation of a certain target evaluation dimension due to the failure to obtain the target evaluation tool, ensuring the integrity of the evaluation, and further improving the accuracy of the evaluation conclusions obtained.

[0050] It is understandable that, after executing S104 to obtain the target evaluation tool output by the tool verification agent, this embodiment can also add the target evaluation tool to the preset tool library for use in subsequent evaluations of other large language models.

[0051] After executing S104 to obtain a first evaluation result of at least one target evaluation dimension based on the sample to be evaluated, this embodiment executes S105 to input the sample to be evaluated and the first evaluation result of at least one target evaluation dimension into the result verification agent to obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent; wherein the obtained evaluation conclusion can be the score of the large language model to be evaluated, or it can be whether the model output content in the sample to be evaluated is correct or incorrect.

[0052] In this embodiment, the result verification agent is used to verify at least one first evaluation result based on the input sample to be evaluated to determine whether the first evaluation result is accurate, and then when it is determined that all the first evaluation results are accurate, obtain the evaluation conclusion of the corresponding large language model to be evaluated based on all the first evaluation results.

[0053] Specifically, when executing S105, the present embodiment inputs the first evaluation results of the sample to be evaluated and at least one target evaluation dimension into the result verification agent, and obtains the evaluation conclusion of the large language model to be evaluated output by the result verification agent. The implementation method that can be adopted is: the result verification agent performs result verification on at least one first evaluation result based on the sample to be evaluated; when it is determined that at least one first evaluation result has all passed the result verification, the result verification agent obtains the evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

[0054] In this embodiment, the result verification agent verifies the first evaluation result based on the sample to be evaluated, that is, determines whether the first evaluation result corresponding to a certain target evaluation dimension is consistent with the model input content or model output content in the sample to be evaluated; for example, if the first evaluation result of the corresponding text quantity evaluation dimension output by the word count evaluation tool is "the number of words is large, meeting the word count requirement", but in fact the model output content contains fewer words, indicating that the first evaluation result is inconsistent with the model output content, and the first evaluation result of the corresponding text quantity evaluation dimension fails the result verification.

[0055] When executing S105, this embodiment may also include the following contents: using the target evaluation dimension corresponding to the first evaluation result that fails the result verification as the second evaluation dimension to be processed; using the target evaluation tool corresponding to the second evaluation dimension to be processed to obtain a second evaluation result of the second evaluation dimension to be processed according to the sample to be processed; the result verification agent performs result verification on the second evaluation result according to the input sample to be evaluated; when it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second evaluation dimension to be processed is updated to the second evaluation result, and then the result verification agent obtains an evaluation conclusion according to the updated first evaluation result.

[0056] That is to say, when the first evaluation result fails the result verification, this embodiment can obtain the second evaluation result for updating the first evaluation result by repeatedly calling the corresponding target evaluation tool, thereby ensuring the accuracy of the first evaluation result used by the result verification agent to obtain the evaluation conclusion.

[0057] It can be understood that when executing S105, this embodiment can continue to call the corresponding target evaluation tool to obtain the second evaluation result again, and then the result verification agent performs result verification on the second evaluation result obtained again, and this cycle is repeated until it is determined that the second evaluation result obtained passes the result verification or it is determined that the number of times the second evaluation result fails the result verification reaches a preset number.

[0058] In addition, when executing S105, this embodiment may also include the following contents: when it is determined that the number of times the second evaluation result fails the result verification reaches a preset number, the result verification agent generates a plan acquisition request based on the second evaluation result, and the generated plan acquisition request includes error information of the second evaluation result, problems that occur, etc.; the generated plan acquisition request is sent to the evaluation planning agent; the evaluation planning agent re-obtains the target evaluation plan based on the sample to be evaluated and the plan acquisition request, so as to use the re-obtained target evaluation plan to re-evaluate the large language model to be evaluated.

[0059] That is to say, in this embodiment, when the target evaluation tool has been called multiple times and the second evaluation result that has not passed the result verification is still not obtained, the result verification agent can generate a plan acquisition request and send it to the evaluation planning agent, so that the evaluation planning agent can re-plan to obtain the target evaluation plan again, thereby re-evaluating the large language model to be evaluated. Through the interaction between the two agents, the problem of not being able to obtain an evaluation conclusion due to an erroneous first evaluation result is avoided, and the accuracy of the obtained evaluation conclusion is further improved.

[0060] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown, when executing S102 "inputting the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent", this embodiment may include the following steps:

[0061] S201, inputting the sample to be evaluated into the evaluation planning agent, obtaining an initial evaluation plan output by the evaluation planning agent, wherein the initial evaluation plan includes at least one initial evaluation dimension and initial evaluation tool information of the at least one initial evaluation dimension;

[0062] S202, inputting the initial assessment plan into the assessment planning agent, and the assessment planning agent updates the at least one initial assessment dimension and / or the initial assessment tool information of the at least one initial assessment dimension;

[0063] S203: When it is determined that the update of the initial evaluation plan meets the preset requirements, the target evaluation plan is obtained according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

[0064] That is to say, this embodiment obtains the target evaluation plan by having the evaluation planning agent update the initial evaluation dimensions and / or initial tool information in the initial evaluation plan. Since the evaluation planning agent will check for omissions and fill in the gaps in the current initial evaluation plan during the update, there will be no errors or omissions in the obtained target evaluation plan, thereby further improving the accuracy of the obtained target evaluation plan.

[0065] When executing S202 in this embodiment, the evaluation planning agent updates the initial evaluation plan at least once, that is, with the help of the evaluation planning agent's own planning and decision-making capabilities, the initial evaluation plan is continuously reflected upon to determine whether the initial evaluation dimensions are accurate and complete, and whether the initial evaluation tool information corresponding to the initial evaluation dimensions is accurate.

[0066] In addition, when executing S202, the present embodiment may also input the sample to be evaluated into the evaluation planning agent, so that the evaluation planning agent can update at least one initial evaluation dimension and / or initial evaluation tool information of at least one initial evaluation dimension in combination with the sample to be evaluated.

[0067] In this embodiment, when the initial evaluation plan is updated at least once, the result of the update of the initial evaluation plan is used as the input of the next evaluation plan agent, and this cycle is repeated until the update meets the preset requirements.

[0068] When executing S203 to determine whether the update of the initial assessment plan meets the preset requirements, this embodiment may determine that the preset requirements are met when it is determined that the number of updates of the initial assessment plan reaches the preset number of updates, or may determine that the preset requirements are met when it is determined that there are no errors in the updated initial assessment plan (initial assessment dimension errors or initial assessment tool information errors) or the number of initial assessment dimensions reaches the preset number.

[0069] After executing S203 to determine that the update of the initial evaluation plan meets the preset requirements, this embodiment obtains the target evaluation plan based on the result of the last update of the initial evaluation plan by the evaluation planning agent, for example, using the evaluation dimension obtained by the last update as the target evaluation dimension, and using the evaluation tool information corresponding to the corresponding evaluation dimension obtained by the last update as the candidate tool information of the corresponding target evaluation dimension.

[0070] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure. Figure 3 As shown in , this embodiment shows an evaluation framework for a large language model; in this evaluation framework, the evaluation planning agent, the tool verification agent and the result verification agent are all obtained through pre-training and have corresponding capabilities, for example, the evaluation planning agent has the ability to generate an evaluation plan, the tool verification agent has the ability to verify evaluation tool information, and the result verification agent has the ability to verify evaluation results and generate evaluation conclusions; after obtaining the samples to be evaluated corresponding to the large language model to be evaluated, this embodiment inputs the obtained samples to be evaluated into the evaluation framework, and the evaluation conclusion of the corresponding large language model to be evaluated output by the evaluation framework, such as an evaluation score, can be obtained. The higher the evaluation score obtained, the more accurately the large language model to be evaluated will respond to the input content.

[0071] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 As shown, the large language model evaluation device 400 of this embodiment includes:

[0072] The acquisition unit 401 is used to acquire a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content;

[0073] The planning unit 402 is used to input the sample to be evaluated into the evaluation planning agent, and obtain the target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension;

[0074] The first verification unit 403 is used to input the target evaluation plan into the tool verification agent to obtain the target evaluation tool information of the at least one target evaluation dimension output by the tool verification agent;

[0075] The processing unit 404 is configured to obtain a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated by using the target evaluation tool corresponding to the target evaluation tool information;

[0076] The second verification unit 405 is used to input the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent, and obtain the evaluation conclusion of the large language model to be evaluated output by the result verification agent.

[0077] The acquisition unit 401 may first determine a large language model to be evaluated, and then acquire samples to be evaluated according to the determined large language model to be evaluated.

[0078] When determining the large language model to be evaluated, the acquisition unit 401 may use a preset large language model as the large language model to be evaluated, or may use the large language model corresponding to the input identification information (such as the name and type of the large language model) as the large language model to be evaluated based on the identification information of the large language model input at the input end.

[0079] After the large language model to be evaluated is determined, the present embodiment can acquire samples to be evaluated based on the determined large language model to be evaluated. The samples to be evaluated acquired by the acquisition unit 401 include model input content and model output content, and the model output content is the response result obtained by the large language model to be evaluated responding to the model input content (such as the query input by the user); in the present embodiment, the data format of the model input content and the model output content can be text.

[0080] When acquiring the sample to be evaluated corresponding to the large language model to be evaluated, the acquisition unit 401 can acquire the content input in real time at the input end as the model input content, and after inputting the model input content into the large language model to be evaluated, use the output result of the large language model to be evaluated as the model output content; the acquisition unit 401 can also acquire the model input content and the model output content corresponding to the large language model to be evaluated as the sample to be evaluated according to a preset corresponding relationship.

[0081] In this embodiment, after the acquisition unit 401 acquires the sample to be evaluated, the planning unit 402 inputs the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent; in this embodiment, the obtained target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of at least one target evaluation dimension.

[0082] In this embodiment, the evaluation planning agent is used to plan in real time the evaluation dimensions that need to be evaluated for the large language model to be evaluated this time according to the input samples to be evaluated, and output the candidate evaluation tool information corresponding to different evaluation dimensions.

[0083] That is to say, in this embodiment, the evaluation planning agent obtains the target evaluation plan for evaluating the large language model in real time according to the input samples to be evaluated, without the need to pre-set different evaluation dimensions, thereby improving the real-time and flexibility in obtaining the target evaluation plan, and avoiding the problem of being unable to evaluate the large language model in the absence of preset evaluation dimensions.

[0084] The planning unit 402 can input the acquired samples to be evaluated into the evaluation planning agent, use at least one evaluation dimension output by the evaluation planning agent as at least one target evaluation dimension, and use the evaluation tool information corresponding to the corresponding evaluation dimension as candidate evaluation tool information of the target evaluation dimension.

[0085] In addition, when the planning unit 402 inputs the sample to be evaluated into the evaluation planning agent and obtains the target evaluation plan output by the evaluation planning agent, it can also include the following contents: inputting the sample to be evaluated into the evaluation planning agent to obtain the initial evaluation plan output by the evaluation planning agent, the initial evaluation plan including at least one initial evaluation dimension and initial evaluation tool information of at least one initial evaluation dimension; inputting the initial evaluation plan into the evaluation planning agent, and the evaluation planning agent updates at least one initial evaluation dimension and / or the initial evaluation tool information of at least one initial evaluation dimension; when it is determined that the update of the initial evaluation plan meets the preset requirements, obtaining the target evaluation plan according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

[0086] That is to say, the planning unit 402 obtains the target evaluation plan by updating the initial evaluation dimensions and / or initial tool information in the initial evaluation plan through the evaluation planning agent. Since the evaluation planning agent will check for omissions and fill in the gaps in the current initial evaluation plan during the update, there will be no errors or omissions in the obtained target evaluation plan, thereby further improving the accuracy of the obtained target evaluation plan.

[0087] In this embodiment, after the planning unit 402 obtains the target evaluation plan output by the evaluation planning agent, the first verification unit 403 inputs the target evaluation plan into the tool verification agent to obtain target evaluation tool information of at least one target evaluation dimension output by the tool verification agent; in this embodiment, different target evaluation dimensions correspond to different target evaluation tool information.

[0088] In this embodiment, the tool verification agent is used to verify the input target evaluation plan, that is, to determine whether the evaluation tool corresponding to the candidate evaluation tool information can complete the evaluation of the corresponding target evaluation dimension.

[0089] In this embodiment, after the first verification unit 403 obtains the target evaluation tool information of at least one target evaluation dimension output by the tool verification intelligent agent, the processing unit 404 uses the target evaluation tool corresponding to the target evaluation tool information to obtain a first evaluation result of at least one target evaluation dimension according to the sample to be evaluated; in this embodiment, different target evaluation dimensions correspond to different first evaluation results.

[0090] That is to say, this embodiment uses different target evaluation tools to complete the evaluation of the large language model on different target evaluation dimensions based on the samples to be evaluated; specifically, the processing unit 404 inputs the samples to be evaluated into different target evaluation tools respectively, and then obtains the output results of each target evaluation tool as the first evaluation results corresponding to different target evaluation dimensions.

[0091] The processing unit 404 can first obtain the target evaluation tool according to the target evaluation tool information, and then use the obtained target evaluation tool to obtain the corresponding first evaluation result, such as the first evaluation result of the instruction understanding evaluation dimension, the first evaluation result of the text quantity evaluation dimension, etc.

[0092] When the processing unit 404 obtains the target evaluation tool according to the target evaluation tool information, it can first search in the preset tool library. If the evaluation tool is found, the evaluation tool found is used as the target evaluation tool; if the tool is not found, the open source tool is searched through the Internet (for example, searching on GitHub). If the tool is found, the open source tool found is used as the target evaluation tool.

[0093] If the corresponding target evaluation tool cannot be obtained according to the target evaluation tool information, the processing unit 404 may also include the following contents: using the target evaluation dimension corresponding to the target evaluation tool information for which the target evaluation tool cannot be obtained as the first evaluation dimension to be processed; generating an evaluation tool acquisition request according to the first evaluation dimension to be processed; and sending the obtained evaluation tool acquisition request to the tool verification agent to obtain the target evaluation tool corresponding to the first evaluation dimension to be processed output by the tool verification agent.

[0094] That is to say, in the present embodiment, when the target evaluation tool whose corresponding target evaluation tool information is not obtained through the tool library or the Internet, the tool verification intelligent agent generates the target evaluation tool in real time according to the corresponding target evaluation dimension, thereby avoiding the problem of being unable to complete the evaluation of a certain target evaluation dimension due to the failure to obtain the target evaluation tool, ensuring the integrity of the evaluation, and further improving the accuracy of the evaluation conclusions obtained.

[0095] It is understandable that, after obtaining the target evaluation tool output by the tool verification agent, the processing unit 404 can also add the target evaluation tool to the preset tool library for use in subsequent evaluations of other large language models.

[0096] In this embodiment, after the processing unit 404 obtains the first evaluation result of at least one target evaluation dimension based on the sample to be evaluated, the second verification unit 405 inputs the sample to be evaluated and the first evaluation result of at least one target evaluation dimension into the result verification agent to obtain the evaluation conclusion of the large language model to be evaluated output by the result verification agent; wherein the obtained evaluation conclusion can be the score of the large language model to be evaluated, or it can be whether the model output content in the sample to be evaluated is correct or incorrect.

[0097] In this embodiment, the result verification agent is used to verify at least one first evaluation result based on the input sample to be evaluated to determine whether the first evaluation result is accurate, and then when it is determined that all the first evaluation results are accurate, obtain the evaluation conclusion of the corresponding large language model to be evaluated based on all the first evaluation results.

[0098] Specifically, when the second verification unit 405 inputs the first evaluation results of the sample to be evaluated and at least one target evaluation dimension into the result verification agent, and obtains the evaluation conclusion of the large language model to be evaluated output by the result verification agent, the implementation method that can be adopted is: the result verification agent performs result verification on at least one first evaluation result based on the sample to be evaluated; when it is determined that at least one first evaluation result has all passed the result verification, the result verification agent obtains the evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

[0099] The second verification unit 405 can also execute the following contents: use the target evaluation dimension corresponding to the first evaluation result that fails the result verification as the second evaluation dimension to be processed; use the target evaluation tool corresponding to the second evaluation dimension to be processed to obtain the second evaluation result of the second evaluation dimension to be processed according to the sample to be processed; the result verification agent performs result verification on the second evaluation result according to the input sample to be evaluated; when it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second evaluation dimension to be processed is updated to the second evaluation result, and then the result verification agent obtains the evaluation conclusion according to the updated first evaluation result.

[0100] That is to say, when the first evaluation result fails the result verification, this embodiment can obtain the second evaluation result for updating the first evaluation result by repeatedly calling the corresponding target evaluation tool, thereby ensuring the accuracy of the first evaluation result used by the result verification agent to obtain the evaluation conclusion.

[0101] It can be understood that if the second evaluation result still fails to pass the result verification of the result verification agent, the second verification unit 405 can continue to call the corresponding target evaluation tool to obtain the second evaluation result again, and then the result verification agent performs result verification on the second evaluation result obtained again, and this cycle is repeated until it is determined that the second evaluation result obtained passes the result verification or it is determined that the number of times the second evaluation result fails the result verification reaches a preset number.

[0102] The second verification unit 405 can also perform the following contents: when it is determined that the second evaluation result has failed the result verification for a preset number of times, the result verification agent generates a plan acquisition request based on the second evaluation result, and the generated plan acquisition request includes error information of the second evaluation result, problems that occur, etc.; the generated plan acquisition request is sent to the evaluation planning agent; the evaluation planning agent re-obtains the target evaluation plan based on the sample to be evaluated and the plan acquisition request, so as to use the re-obtained target evaluation plan to re-evaluate the large language model to be evaluated.

[0103] That is to say, in this embodiment, when the target evaluation tool has been called multiple times and the second evaluation result that has not passed the result verification is still not obtained, the result verification agent can generate a plan acquisition request and send it to the evaluation planning agent, so that the evaluation planning agent can re-plan to obtain the target evaluation plan again, thereby re-evaluating the large language model to be evaluated. Through the interaction between the two agents, the problem of not being able to obtain an evaluation conclusion due to an erroneous first evaluation result is avoided, and the accuracy of the obtained evaluation conclusion is further improved.

[0104] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0105] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0106] like Figure 5 , is a block diagram of an electronic device according to an evaluation method of a large language model of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0107] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0108] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the evaluation method of the large language model. For example, in some embodiments, the evaluation method of the large language model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508.

[0110] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the evaluation method of the large language model described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the evaluation method of the large language model in any other appropriate manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable large language model evaluation device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for evaluating a large language model, comprising: Obtaining a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content; Inputting the sample to be evaluated into an evaluation planning agent, obtaining a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension; Inputting the target assessment plan into a tool verification agent to obtain target assessment tool information of at least one target assessment dimension output by the tool verification agent; Using the target assessment tool corresponding to the target assessment tool information, obtaining a first assessment result of the at least one target assessment dimension according to the sample to be assessed; The first evaluation result of the sample to be evaluated and the at least one target evaluation dimension is input into a result verification agent to obtain an evaluation conclusion of the large language model to be evaluated output by the result verification agent.

2. The method according to claim 1, wherein: The step of inputting the sample to be evaluated into the evaluation planning agent to obtain the target evaluation plan output by the evaluation planning agent comprises: Inputting the sample to be evaluated into the evaluation planning agent to obtain an initial evaluation plan output by the evaluation planning agent, wherein the initial evaluation plan includes at least one initial evaluation dimension and initial evaluation tool information of the at least one initial evaluation dimension; Inputting the initial assessment plan into the assessment planning agent, and having the assessment planning agent update the at least one initial assessment dimension and / or the initial assessment tool information of the at least one initial assessment dimension; When it is determined that the update of the initial evaluation plan meets the preset requirements, the target evaluation plan is obtained according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

3. The method according to claim 1, wherein: The step of inputting the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent, and obtaining the evaluation conclusion of the large language model to be evaluated output by the result verification agent comprises: The result verification agent performs result verification on at least one first evaluation result according to the sample to be evaluated; When it is determined that all of the at least one first evaluation result passes the result verification, the result verification agent obtains an evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

4. The method according to claim 3, further comprising: The target evaluation dimension corresponding to the first evaluation result that fails the result verification is used as the second evaluation dimension to be processed; Using a target assessment tool corresponding to the second to-be-processed assessment dimension, obtaining a second assessment result of the second to-be-processed assessment dimension according to the to-be-assessed sample; The result verification agent performs result verification on the second evaluation result according to the sample to be evaluated; When it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second evaluation dimension to be processed is updated to the second evaluation result.

5. The method according to claim 4, further comprising: When it is determined that the number of times that the second evaluation result fails the result verification reaches a preset number, the result verification agent generates a plan acquisition request according to the second evaluation result; Sending the planning acquisition request to the evaluation planning agent; The evaluation planning agent re-obtains a target evaluation plan according to the sample to be evaluated and the plan acquisition request, so as to re-evaluate the large language model to be evaluated according to the re-obtained target planning request.

6. The method according to claim 1, further comprising: The target assessment dimension corresponding to the target assessment tool information for which the target assessment tool cannot be obtained is used as the first assessment dimension to be processed; Generate an evaluation tool acquisition request according to the first evaluation dimension to be processed; The evaluation tool acquisition request is sent to the tool verification agent to obtain the target evaluation tool output by the tool verification agent corresponding to the first evaluation dimension to be processed.

7. An evaluation device for a large language model, comprising: An acquisition unit, used to acquire a sample to be evaluated corresponding to the large language model to be evaluated, wherein the sample to be evaluated includes model input content and model output content; A planning unit, configured to input the sample to be evaluated into an evaluation planning agent, and obtain a target evaluation plan output by the evaluation planning agent, wherein the target evaluation plan includes at least one target evaluation dimension and candidate evaluation tool information of the at least one target evaluation dimension; A first verification unit is used to input the target evaluation plan into a tool verification agent to obtain target evaluation tool information of at least one target evaluation dimension output by the tool verification agent; A processing unit, configured to obtain a first evaluation result of the at least one target evaluation dimension according to the sample to be evaluated, using the target evaluation tool corresponding to the target evaluation tool information; The second verification unit is used to input the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent to obtain the evaluation conclusion of the large language model to be evaluated output by the result verification agent.

8. The device according to claim 7, wherein: When the planning unit inputs the sample to be evaluated into the evaluation planning agent and obtains the target evaluation plan output by the evaluation planning agent, the planning unit specifically performs: Inputting the sample to be evaluated into the evaluation planning agent to obtain an initial evaluation plan output by the evaluation planning agent, wherein the initial evaluation plan includes at least one initial evaluation dimension and initial evaluation tool information of the at least one initial evaluation dimension; Inputting the initial assessment plan into the assessment planning agent, and having the assessment planning agent update the at least one initial assessment dimension and / or the initial assessment tool information of the at least one initial assessment dimension; When it is determined that the update of the initial evaluation plan meets the preset requirements, the target evaluation plan is obtained according to the result of the last update of the initial evaluation plan by the evaluation planning agent.

9. The device according to claim 7, wherein: When the second verification unit inputs the first evaluation result of the sample to be evaluated and the at least one target evaluation dimension into the result verification agent and obtains the evaluation conclusion of the large language model to be evaluated output by the result verification agent, the second verification unit specifically performs: The result verification agent performs result verification on at least one first evaluation result according to the sample to be evaluated; When it is determined that all of the at least one first evaluation result passes the result verification, the result verification agent obtains an evaluation conclusion of the large language model to be evaluated based on the at least one first evaluation result.

10. The device according to claim 9, wherein the second verification unit is further configured to execute: The target evaluation dimension corresponding to the first evaluation result that fails the result verification is used as the second evaluation dimension to be processed; Using a target assessment tool corresponding to the second assessment dimension to be processed, obtaining a second assessment result of the second assessment dimension to be processed according to the assessment sample to be processed; The result verification agent performs result verification on the second evaluation result according to the sample to be evaluated; When it is determined that the second evaluation result passes the result verification, the first evaluation result corresponding to the second evaluation dimension to be processed is updated to the second evaluation result.

11. The device according to claim 10, wherein the second verification unit is further configured to execute: When it is determined that the number of times that the second evaluation result fails the result verification reaches a preset number, the result verification agent generates a plan acquisition request according to the second evaluation result; Sending the planning acquisition request to the evaluation planning agent; The evaluation planning agent re-obtains a target evaluation plan according to the sample to be evaluated and the plan acquisition request, so as to re-evaluate the large language model to be evaluated according to the re-obtained target planning request.

12. The apparatus according to claim 7, wherein the processing unit is further configured to execute: The target assessment dimension corresponding to the target assessment tool information for which the target assessment tool cannot be obtained is used as the first assessment dimension to be processed; Generate an evaluation tool acquisition request according to the first evaluation dimension to be processed; The evaluation tool acquisition request is sent to the tool verification agent to obtain the target evaluation tool output by the tool verification agent corresponding to the first evaluation dimension to be processed.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images