Method and device for evaluating text classification performance of large language model

Abstract

The present disclosure provides a method and device for evaluating the text classification performance of a large language model, and relates to the fields of artificial intelligence technology such as large language models, natural language processing, and deep learning. The method for evaluating the text classification performance of a large language model includes: obtaining an evaluation data set corresponding to a source evaluation task and an evaluation data set corresponding to at least one sub-evaluation task; obtaining evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, inputting the evaluation input data into the large language model to be evaluated, and using the output result of the large language model to be evaluated as the predicted answer corresponding to the evaluation text for different evaluation task types; obtaining the source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to at least one sub-evaluation task according to the labeled answer and the predicted answer corresponding to the same evaluation task type of the evaluation text; obtaining the text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to the field of artificial intelligence technology such as large language models, natural language processing, and deep learning. Provided are a method, device, electronic device, and readable storage medium for evaluating the text classification performance of a large language model. Background Art

[0002] The Large Language Model (LLM) is a generative model that can generate corresponding response content based on input data and can handle a variety of natural language tasks including text classification.

[0003] How to evaluate the performance of a large language model in text classification is a fundamental issue. Existing technologies usually only evaluate a large language model by setting a text classification task for execution. However, this method is a coarse-grained evaluation, and the accuracy of the text classification performance obtained by the evaluation is low. Summary of the invention

[0004] According to a first aspect of the present disclosure, a method for evaluating the text classification performance of a large language model is provided, comprising: obtaining an evaluation data set corresponding to a source evaluation task and an evaluation data set corresponding to at least one sub-evaluation task, wherein the source evaluation task is a text classification task, the sub-evaluation tasks include an entity recognition task and a summary generation task, and different evaluation data sets include evaluation texts and annotated answers of the evaluation texts corresponding to different evaluation task types; obtaining evaluation input data according to the evaluation texts and the evaluation task types corresponding to the evaluation texts, inputting the evaluation input data into a large language model to be evaluated, and using the output results of the large language model to be evaluated as predicted answers of the evaluation texts corresponding to different evaluation task types; obtaining a source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to the at least one sub-evaluation task according to the annotated answers and predicted answers of the evaluation texts corresponding to the same evaluation task type; and obtaining the text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result.

[0005] According to the second method of the present disclosure, there is provided an evaluation device for the text classification performance of a large language model, comprising: an acquisition unit, for acquiring an evaluation data set corresponding to a source evaluation task, and an evaluation data set corresponding to at least one sub-evaluation task, wherein the source evaluation task is a text classification task, the sub-evaluation tasks include an entity recognition task and a summary generation task, and different evaluation data sets include evaluation texts and annotated answers corresponding to different evaluation task types of the evaluation texts; a prediction unit, for obtaining evaluation input data according to the evaluation texts and the evaluation task types corresponding to the evaluation texts, inputting the evaluation input data into a large language model to be evaluated, and using the output results of the large language model to be evaluated as predicted answers corresponding to different evaluation task types of the evaluation texts; a processing unit, for obtaining a source evaluation result corresponding to the source evaluation task, and at least one sub-evaluation result corresponding to the at least one sub-evaluation task, according to the annotated answers and predicted answers corresponding to the same evaluation task type of the evaluation texts; and an evaluation unit, for obtaining the text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0012] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0017] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0018] Figure 8 The block diagram of an electronic device used to implement the method for evaluating the text classification performance of a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and mechanisms is omitted in the following description.

[0020] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the text classification performance evaluation method of the large language model of this embodiment specifically includes the following steps:

[0021] S101, obtaining an evaluation data set corresponding to a source evaluation task and an evaluation data set corresponding to at least one sub-evaluation task, wherein the source evaluation task is a text classification task, and the sub-evaluation tasks include an entity recognition task and a summary generation task, and different evaluation data sets include evaluation texts and annotated answers corresponding to different evaluation task types of the evaluation texts;

[0022] S102, obtaining evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, inputting the evaluation input data into the large language model to be evaluated, and using the output result of the large language model to be evaluated as the predicted answer corresponding to the evaluation text for different evaluation task types;

[0023] S103, obtaining a source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to the at least one sub-evaluation task according to the annotated answers and the predicted answers corresponding to the same evaluation task type of the evaluation text;

[0024] S104: Obtain text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result.

[0025] The method for evaluating the text classification performance of a large language model of this embodiment achieves the purpose of evaluating the text classification performance of a large language model by introducing entity recognition tasks and / or summary generation tasks in addition to text classification tasks in the evaluation process. Since the classification accuracy of a large language model when performing text classification depends to a large extent on the large language model's understanding of the text, the higher the understanding of the text, the more accurate the classification accuracy of the large language model will be. Therefore, in addition to using the text classification task as the source evaluation task, this embodiment additionally sets a variety of sub-evaluation tasks related to text understanding, thereby performing fine-grained evaluation of the large language model from different angles, which can improve the evaluation accuracy of text classification performance.

[0026] In this embodiment, the large language model to be evaluated is a generative large language model (LLM). The large language model is a deep learning model trained using a large amount of text data, which can handle a variety of natural language tasks such as text classification, text generation, and entity recognition.

[0027] The different evaluation data sets obtained by executing S101 in this embodiment correspond to different evaluation task types, and the evaluation task types include text classification tasks, entity recognition tasks, and summary generation tasks. The text classification task can be further divided into binary classification tasks and multi-classification tasks. In addition to at least one evaluation text, each evaluation data set also includes at least one annotated answer corresponding to the current evaluation task type.

[0028] The evaluation data set corresponding to the source evaluation task obtained by executing S101 in this embodiment may be referred to as a source evaluation data set, and the evaluation data set for the sub-evaluation task may be referred to as a sub-evaluation data set.

[0029] For example, if the evaluation dataset corresponding to the source evaluation task is dataset 1, the labeled answers contained in dataset 1 are the classification labels corresponding to each evaluation text; if the sub-evaluation task is an entity recognition task, if the evaluation dataset corresponding to the entity recognition task is dataset 2, the labeled answers contained in dataset 2 are the entities contained in each evaluation text; if the sub-evaluation task is a summary generation task, if the evaluation dataset corresponding to the summary generation task is dataset 3, the labeled answers contained in dataset 3 are the summary texts generated based on each evaluation text.

[0030] The source evaluation task in this embodiment can also correspond to different classification scenarios. For example, in the text classification scenario of product evaluation, the classification labels included in the evaluation data set corresponding to the source evaluation task may be "product advantage evaluation", "product defect feedback", "product improvement suggestions", "shopping experience sharing" and "shopping question help", etc.; in the text classification scenario of content security, the classification labels included in the evaluation data set corresponding to the source evaluation task may be "contains sensitive words" and "does not contain sensitive words".

[0031] It can be understood that the annotated answers contained in the evaluation data set obtained by executing S101 in this embodiment can be obtained by processing the corresponding evaluation text through a large language model, or can be obtained through manual annotation, or can be obtained by combining a large language model with manual annotation.

[0032] That is to say, when evaluating the text classification performance of the large language model, this embodiment, in addition to setting the text classification task as the source evaluation task, will also additionally set the entity recognition task and / or summary generation task related to text understanding as sub-evaluation tasks, so as to comprehensively evaluate the text classification performance of the large language model in terms of entity recognition and / or summary generation, and compare with the method of only setting the text classification task to evaluate the large language model, this embodiment can effectively improve the evaluation accuracy of the text classification performance.

[0033] When executing S101 in this embodiment, only the entity recognition task may be used as a sub-evaluation task, only the summary generation task may be used as a sub-evaluation task, or both the entity recognition task and the summary generation task may be used as sub-evaluation tasks.

[0034] After executing S101 to obtain evaluation data sets corresponding to different evaluation task types, this embodiment executes S102 to obtain evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, inputs the evaluation input data into the large language model to be evaluated, and uses the output result of the large language model to be evaluated as the predicted answer for the evaluation text corresponding to the different evaluation task types; wherein the evaluation task type corresponding to the evaluation text is the evaluation task type corresponding to the evaluation data set to which the evaluation text belongs.

[0035] It can be understood that, when executing S102, this embodiment can select a large language model as the large language model to be evaluated, or can select multiple large language models as the large language models to be evaluated, that is, this embodiment can evaluate the text classification performance of only one large language model, or can evaluate the text classification performance of multiple large language models at the same time; this embodiment can display multiple large language models, and then use the large language model selected at the input end as the large language model to be evaluated.

[0036] When executing S102 to obtain evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, this embodiment can adopt the following implementation method: determine the evaluation task type corresponding to the evaluation text, the evaluation task type in this embodiment is one of a text classification task, an entity recognition task and a summary generation task; obtain a prompt text according to the determined evaluation task type, the prompt text is prompt; obtain the evaluation input data according to the evaluation text and the obtained prompt text.

[0037] For example, if the evaluation task type is a text classification task, the prompt text obtained by executing S102 in this embodiment may be "Please classify the following text data and output the classification label"; if the evaluation task type is an entity recognition task, the enhanced text obtained by executing S102 in this embodiment may be "Please find the entity from the following text data and output it"; if the evaluation task type is a summary generation task, the prompt text obtained by executing S102 in this embodiment may be "Please generate a summary based on the following text data and output it".

[0038] For example, if the evaluation text is "This mobile phone has a high cost performance, runs smoothly, and takes great photos!", if the obtained prompt text is "Please classify the following text data and output the classification label", the evaluation input data obtained by executing S102 in this embodiment can be "Please classify the following text data and output the classification label: This mobile phone has a high cost performance, runs smoothly, and takes great photos!".

[0039] When executing S102 to obtain the evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, this embodiment can also use the evaluation task type corresponding to the evaluation text as a suffix to splice it with the evaluation text, and use the splicing result as the evaluation input data, for example, "evaluation text + text classification" as the evaluation input data, "evaluation text + entity recognition" as the evaluation input data, and "evaluation text + summary generation" as the evaluation input data.

[0040] In this embodiment, after executing S102 to input the obtained evaluation input data into the large language model to be evaluated, the large language model to be evaluated can perform corresponding processing according to the input evaluation input data, thereby outputting predicted answers corresponding to different evaluation task types of the evaluation text.

[0041] After executing S102 to obtain predicted answers for evaluation texts corresponding to different evaluation task types, this embodiment executes S103 to obtain source evaluation results corresponding to the source evaluation task and at least one sub-evaluation result corresponding to at least one sub-evaluation task based on the labeled answers and predicted answers for the same evaluation task type corresponding to the evaluation text.

[0042] When executing S103, this embodiment can first obtain the annotation answers corresponding to the evaluation text from the evaluation data set to which the evaluation text belongs, and then obtain the evaluation results corresponding to different evaluation task types based on the obtained annotation answers and the predicted answers output by the large language model to be evaluated.

[0043] For example, when executing S103 to obtain the source evaluation result of the corresponding source evaluation task, this embodiment can use evaluation indicators such as accuracy, precision, recall, F1-score, etc. to match the annotated answer and the predicted answer corresponding to the evaluation text, and use the calculation result as the source evaluation result of the corresponding source evaluation task.

[0044] For example, if the sub-evaluation task is an entity recognition task, when executing S103 to obtain the sub-evaluation result of the corresponding sub-evaluation task, this embodiment can use evaluation indicators such as accuracy and recall to match the annotated answer and the predicted answer corresponding to the evaluation text, and use the calculation result as the sub-evaluation result of the corresponding entity recognition task.

[0045] For example, if the sub-evaluation task is a summary generation task, when executing S103 to obtain the sub-evaluation result of the corresponding sub-evaluation task, this embodiment can use evaluation indicators such as Rouge (Recall Oriented Understudy for GistingEvaluation) and Bert_score_f1 to match the annotated answer and the predicted answer corresponding to the evaluation text, and use the calculation result as the sub-evaluation result of the corresponding summary generation task.

[0046] After executing S103 to obtain the source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to at least one sub-evaluation task, this embodiment executes S104 to obtain the text classification performance of the large language model to be evaluated based on the source evaluation result and the at least one sub-evaluation result.

[0047] In this embodiment, when executing S104 to obtain the text classification performance of the large language model to be evaluated based on the source evaluation result and at least one sub-evaluation result, the addition result between the source evaluation result and the at least one sub-evaluation result can be used as the text classification performance of the large language model to be evaluated.

[0048] When executing S104 to obtain the text classification performance of the large language model to be evaluated based on the source evaluation results and at least one sub-evaluation result, this embodiment may also include the following contents: obtaining weight values ​​corresponding to different evaluation task types; obtaining the text classification performance of the large language model to be evaluated based on the weight value and source evaluation result of the corresponding source evaluation task, and the weight value and at least one sub-evaluation result of the corresponding at least one sub-evaluation task.

[0049] That is to say, this embodiment more accurately obtains the text classification performance of the large language model based on the evaluation results corresponding to different evaluation tasks by setting weight values ​​corresponding to different evaluation tasks.

[0050] In order to further highlight the significance of text classification tasks in evaluating the text classification performance of large language models, in this embodiment, the weight value corresponding to the source evaluation task is greater than the weight value corresponding to the sub-evaluation task; the weight values ​​corresponding to different sub-evaluation tasks can be the same or different.

[0051] For example, if the weight value of the corresponding source evaluation task is weight value 1, the weight value of the corresponding entity recognition task is weight value 2, and the weight value of the corresponding summary generation task is weight value 3, if the source evaluation result of the corresponding source evaluation task is result 1, the sub-evaluation result of the corresponding entity recognition task is result 2, and the sub-evaluation result of the corresponding summary generation task is result 3, when executing S104, this embodiment can use the calculation result obtained by (result 1×weight value 1+result 2×weight value 2+result 3×weight value 3) as the text classification performance of the large language model to be evaluated.

[0052] In this embodiment, there may be multiple large language models to be evaluated when performing text classification performance evaluation. Therefore, after executing S104 to obtain the text classification performance of each large language model to be evaluated, this embodiment can also select one with the best text classification performance (for example, the highest score) from the multiple large language models to be evaluated according to the obtained text classification performance, as the target large language model; when executing S104, this embodiment can further display the selected target large language model at the input end.

[0053] In addition, after executing S104 to obtain the text classification performance of the large language model to be evaluated, in addition to outputting the obtained text classification performance, this embodiment can also output a source evaluation result and at least one sub-evaluation result together.

[0054] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown, this embodiment may also include the following contents:

[0055] S201, obtaining evaluation text corresponding to the target classification scenario;

[0056] S202, obtaining initial input data according to the evaluation text and different evaluation task types, inputting the initial input data into a candidate large language model, and obtaining annotated answers corresponding to different evaluation task types output by the candidate large language model for the evaluation text;

[0057] S203 , obtaining evaluation data sets corresponding to different evaluation task types according to the evaluation text and the annotated answers of the evaluation text corresponding to different evaluation task types.

[0058] In this embodiment, the target classification scenario may be a text classification scenario of product reviews; or it may be a text classification scenario of content security, that is, classifying whether the text contains sensitive words.

[0059] If the target classification scenario is a text classification scenario of product evaluation, this embodiment can obtain the text of the user's evaluation after completing the purchase of the product or service as the evaluation text when executing S201; if the target classification scenario is a text classification scenario with safe content, this embodiment can obtain the user's posting content or comments on the Internet as the evaluation text when executing S201.

[0060] When executing S201, this embodiment can also perform text preprocessing on the obtained evaluation text, such as text conversion between traditional and simplified Chinese, full-width and half-width, uppercase and lowercase, removal of URL symbols, replacement of continuous spaces, etc., to obtain a high-quality evaluation text.

[0061] In this embodiment, different evaluation task types include at least one of an entity recognition task and a summary generation task, and a text classification task.

[0062] When executing S202 to obtain initial input data according to the evaluation text and different evaluation task types, this embodiment can obtain the initial input data according to the evaluation text and prompt texts corresponding to different evaluation task types, or directly obtain the initial input data according to the evaluation text and different evaluation task types.

[0063] Since the output data of the large language model may not be standardized when it outputs the corresponding output data based on the initial input data, when executing S202 to obtain the annotation answers corresponding to different evaluation task types output by the candidate large language model for the evaluation text, this embodiment can also manually review the output results of the candidate large language model, and then use the output results that have passed the review as the annotation answers corresponding to different evaluation task types for the evaluation text.

[0064] In this embodiment, the number of evaluation texts can be multiple. Therefore, when executing S203, this embodiment can obtain an evaluation data set corresponding to the current evaluation task type for each evaluation task type based on different evaluation texts and the annotated answers of different evaluation texts corresponding to the current evaluation task type.

[0065] The evaluation data set corresponding to the text classification task contains evaluation texts corresponding to different classification labels. If the number of evaluation texts corresponding to two classification labels differs greatly, it may affect the evaluation accuracy of the large model to be evaluated.

[0066] Therefore, after executing S203 to obtain the evaluation data set corresponding to the text classification task, this embodiment performs clustering uniform sampling on the evaluation texts of all classification labels, thereby ensuring that the number of evaluation texts corresponding to each classification label in the evaluation data set is balanced.

[0067] When executing S203, this embodiment can first determine the cluster center of the evaluation text corresponding to each classification label, and then select a preset number of evaluation texts close to the cluster center as the evaluation texts corresponding to each classification label, for example, select 5000 evaluation texts close to the cluster center.

[0068] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure. Figure 3 As shown, in this embodiment, when executing S102 "using the output result of the large language model to be evaluated as the predicted answer corresponding to different evaluation task types of the evaluation text", the following contents may also be included:

[0069] S301, constructing a label syntax tree according to the classification labels corresponding to the target classification scenario;

[0070] S302: When it is determined that the evaluation task type is a text classification task, perform text parsing on the output result of the large language model to be evaluated to obtain a parsing result;

[0071] S303: Extract the part of the parsing result that matches the label syntax tree from the output result as the predicted answer.

[0072] When the large language model to be evaluated in this embodiment performs a text classification task, the output result obtained based on the input data may be only the text corresponding to the classification label, or may include other texts in addition to the text corresponding to the classification label. Therefore, when this embodiment determines that the evaluation task type is a text classification task, the text corresponding to the classification label is accurately obtained from the output result as the predicted answer by matching the parsing result of the output result and the label syntax tree.

[0073] In this embodiment, different text classification scenarios correspond to different classification labels. When executing S301, this embodiment can first determine the target classification scenario, then obtain the classification label corresponding to the determined target classification scenario, and finally construct a label syntax tree based on the obtained classification label.

[0074] For example, if the target classification scenario is a text classification scenario of product evaluation, classification labels such as "product advantage evaluation", "product defect feedback", "product improvement suggestions", "shopping experience sharing" and "shopping question help" can be obtained. The label syntax tree constructed according to the above classification labels in executing S301 in this embodiment can be:

[0075] S->Product|Shopping

[0076] Product->"Product"Attribute

[0077] Attribute->"Advantage Evaluation"|"Defect Feedback"|"Improvement Suggestions"

[0078] Shopping->"Shopping"Attribute

[0079] Attribute->"Experience Sharing"|"Questions and Help"

[0080] When executing S302 in this embodiment, a syntax analyzer may be used to perform text parsing on the output result, thereby obtaining a parsing result corresponding to the output result.

[0081] For example, if the output result of the large language model to be evaluated is "Based on your input, the classification label of the above text is product advantage evaluation", after matching the parsed result of the output result with the above label syntax tree, only the "product advantage evaluation" that matches the two is extracted from the output result as the predicted answer.

[0082] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 As shown, after executing S104 "obtaining the text classification performance of the large language model to be evaluated", this embodiment may also include the following contents:

[0083] S401, when it is determined that the large language model to be evaluated is a target large language model according to the text classification performance, obtaining a text to be classified corresponding to a target classification scenario;

[0084] S402: Input the text to be classified into the target large language model for text classification, and use the output result of the target large language model as the classification result of the text to be classified.

[0085] That is to say, after completing the evaluation of the text classification performance of the large language model to be evaluated, this embodiment can determine whether the large language model to be evaluated is the target large language model based on the obtained text classification performance, and then after the determination, input the acquired text to be classified corresponding to the target classification scenario into the target large language model for text classification, which can improve the accuracy of the obtained text classification results.

[0086] When this embodiment executes S401, if there is only one large language model to be evaluated, then when it is determined that the text classification performance of the large language model to be evaluated exceeds a preset threshold, the large language model to be evaluated can be determined as the target large language model; if there are multiple large language models to be evaluated, when this embodiment executes S401, the large language model to be evaluated with the best text classification performance can be used as the target large language model.

[0087] It can be understood that when executing S402, this embodiment can obtain classification output data based on the text to be classified and the text classification task, and then input the classification input data into the target large language model; and when the target large language model has been set to only perform text classification tasks, then only the text to be classified can be input into the target large language model.

[0088] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure. Figure 5 The flowchart of this embodiment when constructing evaluation data corresponding to different evaluation task types is shown: S501, obtaining evaluation text; S502, performing text preprocessing on the obtained evaluation text; S503, when the obtained evaluation text already has a corresponding classification label, clustering and uniformly sampling the evaluation texts belonging to different classification labels; S504, setting different types of evaluation tasks; S505, setting prompt texts corresponding to different evaluation task types; S506, using the candidate large language model, fine-grainedly annotating the evaluation text for different types of evaluation tasks; S507, according to the output results of the candidate large language model, obtaining the annotation answers of the evaluation text corresponding to different evaluation task types; S508, constructing the evaluation data set corresponding to different evaluation task types according to the evaluation text and its corresponding annotation answers.

[0089] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure. Figure 6The flowchart of this embodiment when evaluating the text classification performance of a large language model is shown; S601, selecting different types of evaluation tasks; S602, selecting a large language model to be evaluated; S603, reading an evaluation data set corresponding to the selected evaluation task; S604, calling the selected large language model to be evaluated, and generating a predicted answer for the evaluation text in the evaluation data set; S605, in the case of determining that the evaluation task type is a text classification task, constructing a label syntax tree according to the classification label; S606, if the evaluation task type is not a text classification task, using the output result of the large language model to be evaluated as Predict the answer; S607. When it is determined that the output result of the large language model to be evaluated is valid according to the label syntax tree (i.e., determine whether there is content matching the label syntax tree in the parsed result of the output result), select the matching part from the output result as the predicted answer, otherwise the output result is invalid (the label missing can be used to record it); S608. According to the annotated answers and predicted answers corresponding to the evaluation text, obtain the evaluation results corresponding to different evaluation task types; S609. According to the evaluation results corresponding to different evaluation task types, obtain the text classification performance of the large language model to be evaluated.

[0090] Figure 7 is a schematic diagram according to the seventh embodiment of the present disclosure. Figure 7 As shown, the text classification performance evaluation device 700 of the large language model of this embodiment includes:

[0091] An acquisition unit 701 is used to acquire an evaluation data set corresponding to a source evaluation task and an evaluation data set corresponding to at least one sub-evaluation task, wherein the source evaluation task is a text classification task, and the sub-evaluation tasks include an entity recognition task and a summary generation task, and different evaluation data sets include evaluation texts and annotated answers corresponding to different evaluation task types of the evaluation texts;

[0092] The prediction unit 702 is used to obtain evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, input the evaluation input data into the large language model to be evaluated, and use the output result of the large language model to be evaluated as the prediction answer corresponding to the evaluation text for different evaluation task types;

[0093] The processing unit 703 is used to obtain a source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to the at least one sub-evaluation task according to the annotated answers and the predicted answers corresponding to the same evaluation task type of the evaluation text;

[0094] The evaluation unit 704 is used to obtain the text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result.

[0095] The different evaluation data sets obtained by the acquisition unit 701 correspond to different evaluation task types. The evaluation task types include text classification tasks, entity recognition tasks, and summary generation tasks. The text classification task can be further divided into binary classification tasks and multi-classification tasks. In addition to containing at least one evaluation text, each evaluation data set also contains at least one annotated answer corresponding to the current evaluation task type of the evaluation text.

[0096] The evaluation data set corresponding to the source evaluation task acquired by the acquisition unit 701 may be referred to as a source evaluation data set, and the evaluation data set corresponding to the sub-evaluation task may be referred to as a sub-evaluation data set.

[0097] The source evaluation task in this embodiment can also correspond to different classification scenarios. For example, in the text classification scenario of product evaluation, the classification labels included in the evaluation data set corresponding to the source evaluation task may be "product advantage evaluation", "product defect feedback", "product improvement suggestions", "shopping experience sharing" and "shopping question help", etc.; in the text classification scenario of content security, the classification labels included in the evaluation data set corresponding to the source evaluation task may be "contains sensitive words" and "does not contain sensitive words".

[0098] It can be understood that the annotated answers contained in the evaluation data set obtained by the acquisition unit 701 can be obtained by processing the corresponding evaluation text through a large language model, or can be obtained through manual annotation, or can be obtained by combining a large language model with manual annotation.

[0099] The acquisition unit 701 may only use the entity recognition task as a sub-evaluation task, or only use the summary generation task as a sub-evaluation task, or may use both the entity recognition task and the summary generation task as sub-evaluation tasks.

[0100] In this embodiment, after the acquisition unit 701 acquires the evaluation data sets corresponding to different evaluation task types, the prediction unit 702 obtains the evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, inputs the evaluation input data into the large language model to be evaluated, and uses the output result of the large language model to be evaluated as the predicted answer for the evaluation text corresponding to the different evaluation task types; wherein the evaluation task type corresponding to the evaluation text is the evaluation task type corresponding to the evaluation data set to which the evaluation text belongs.

[0101] It can be understood that the prediction unit 702 can select a large language model as the large language model to be evaluated, or can select multiple large language models as the large language models to be evaluated, that is, this embodiment can evaluate the text classification performance of only one large language model, or can evaluate the text classification performance of multiple large language models at the same time; this embodiment can display multiple large language models, and then use the large language model selected at the input end as the large language model to be evaluated.

[0102] When the prediction unit 702 obtains the evaluation input data based on the evaluation text and the evaluation task type corresponding to the evaluation text, the following implementation method can be adopted: determine the evaluation task type corresponding to the evaluation text, the evaluation task type in this embodiment is one of a text classification task, an entity recognition task and a summary generation task; obtain a prompt text according to the determined evaluation task type, the prompt text is prompt; obtain the evaluation input data according to the evaluation text and the obtained prompt text.

[0103] When the prediction unit 702 obtains the evaluation input data based on the evaluation text and the evaluation task type corresponding to the evaluation text, it can also use the evaluation task type corresponding to the evaluation text as a suffix to concatenate it with the evaluation text, and use the concatenation result as the evaluation input data, for example, using "evaluation text + text classification" as the evaluation input data, using "evaluation text + entity recognition" as the evaluation input data, and using "evaluation text + summary generation" as the evaluation input data.

[0104] After the prediction unit 702 inputs the obtained evaluation input data into the large language model to be evaluated, the large language model to be evaluated can perform corresponding processing according to the input evaluation input data, thereby outputting predicted answers corresponding to different evaluation task types of the evaluation text.

[0105] When the prediction unit 702 uses the output result of the large language model to be evaluated as the predicted answer for the evaluation text corresponding to different evaluation task types, it may also include the following contents: constructing a label syntax tree according to the classification label corresponding to the target classification scenario; when it is determined that the evaluation task type is a text classification task, performing text parsing on the output result of the large language model to be evaluated to obtain the parsed result; extracting the part of the parsed result that matches the label syntax tree from the output result as the predicted answer.

[0106] In this embodiment, different text classification scenarios correspond to different classification labels. The prediction unit 702 may first determine the target classification scenario, then obtain the classification label corresponding to the determined target classification scenario, and finally construct a label syntax tree according to the obtained classification label.

[0107] The prediction unit 702 may use a syntax analyzer to perform text analysis on the output result, thereby obtaining a parsing result corresponding to the output result.

[0108] In this embodiment, after the prediction unit 702 obtains the predicted answers of the evaluation text corresponding to different evaluation task types, the processing unit 703 obtains the source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to at least one sub-evaluation task based on the labeled answers and the predicted answers of the evaluation text corresponding to the same evaluation task type.

[0109] The processing unit 703 may first obtain the annotation answer corresponding to the evaluation text from the evaluation data set to which the evaluation text belongs, and then obtain the evaluation results corresponding to different evaluation task types based on the obtained annotation answer and the predicted answer output by the large language model to be evaluated.

[0110] In this embodiment, after the processing unit 703 obtains the source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to at least one sub-evaluation task, the evaluation unit 704 obtains the text classification performance of the large language model to be evaluated based on the source evaluation result and the at least one sub-evaluation result.

[0111] When the evaluation unit 704 obtains the text classification performance of the large language model to be evaluated based on the source evaluation result and at least one sub-evaluation result, the addition result between the source evaluation result and the at least one sub-evaluation result can be used as the text classification performance of the large language model to be evaluated.

[0112] When the evaluation unit 704 obtains the text classification performance of the large language model to be evaluated based on the source evaluation results and at least one sub-evaluation result, it can also include the following contents: obtaining weight values ​​corresponding to different evaluation task types; obtaining the text classification performance of the large language model to be evaluated based on the weight value and source evaluation result of the corresponding source evaluation task, and the weight value and at least one sub-evaluation result of the corresponding at least one sub-evaluation task.

[0113] In order to further highlight the significance of text classification tasks in evaluating the text classification performance of large language models, in this embodiment, the weight value corresponding to the source evaluation task is greater than the weight value corresponding to the sub-evaluation task; the weight values ​​corresponding to different sub-evaluation tasks can be the same or different.

[0114] In this embodiment, there may be multiple large language models to be evaluated when performing text classification performance evaluation. Therefore, after obtaining the text classification performance of each large language model to be evaluated, the evaluation unit 704 may also select one with the best text classification performance (for example, the highest score) from the multiple large language models to be evaluated according to the obtained text classification performance as the target large language model; the evaluation unit 704 may further display the selected target large language model at the input end.

[0115] In addition, after obtaining the text classification performance of the large language model to be evaluated, the evaluation unit 704 may output a source evaluation result and at least one sub-evaluation result together with the obtained text classification performance.

[0116] The text classification performance evaluation device 700 of the large language model of this embodiment may also include a construction unit 705, which is used to execute the following contents: obtaining an evaluation text corresponding to a target classification scenario; obtaining initial input data according to the evaluation text and different evaluation task types, inputting the initial input data into a candidate large language model, and obtaining annotated answers corresponding to different evaluation task types output by the candidate large language model for the evaluation text; and obtaining evaluation data sets corresponding to different evaluation task types according to the evaluation text and the annotated answers of the evaluation text corresponding to different evaluation task types.

[0117] In this embodiment, the target classification scenario may be a text classification scenario of product reviews; or it may be a text classification scenario of content security, that is, classifying whether the text contains sensitive words.

[0118] If the target classification scenario is a text classification scenario of product evaluation, the construction unit 705 can obtain the text of the user's evaluation after completing the purchase of the product or service as the evaluation text; if the target classification scenario is a text classification scenario of content security, the construction unit 705 can obtain the user's posting content or comments on the Internet as the evaluation text.

[0119] The construction unit 705 can also perform text preprocessing on the acquired evaluation text, such as text conversion between traditional and simplified Chinese, full-width and half-width, uppercase and lowercase, removing URL symbols, replacing continuous spaces, etc., to obtain high-quality evaluation text.

[0120] In this embodiment, different evaluation task types include at least one of an entity recognition task and a summary generation task, and a text classification task.

[0121] When constructing unit 705 obtains initial input data according to the evaluation text and different evaluation task types, it can obtain the initial input data according to the evaluation text and prompt texts corresponding to different evaluation task types, or it can directly obtain the initial input data according to the evaluation text and different evaluation task types.

[0122] Since the output data of the large language model may not be standardized when it outputs the corresponding output data based on the initial input data, when the construction unit 705 obtains the annotation answers corresponding to different evaluation task types output by the candidate large language model for the evaluation text, the output results of the candidate large language model can also be manually reviewed, and the output results that have passed the review can be used as the annotation answers for the evaluation text corresponding to different evaluation task types.

[0123] In this embodiment, there may be multiple evaluation texts, so the construction unit 705 can obtain an evaluation data set corresponding to the current evaluation task type for each evaluation task type based on different evaluation texts and the annotated answers of different evaluation texts corresponding to the current evaluation task type.

[0124] The evaluation data set corresponding to the text classification task contains evaluation texts corresponding to different classification labels. If the number of evaluation texts corresponding to two classification labels differs greatly, it may affect the evaluation accuracy of the large model to be evaluated.

[0125] Therefore, after obtaining the evaluation data set corresponding to the text classification task, the construction unit 705 performs clustering uniform sampling on the evaluation texts of all classification labels, thereby ensuring that the number of evaluation texts corresponding to each classification label in the evaluation data set is balanced.

[0126] The construction unit 705 may first determine the cluster center of the evaluation text corresponding to each classification label, and then select a preset number of evaluation texts close to the cluster center as the evaluation texts corresponding to each classification label, for example, select 5000 evaluation texts close to the cluster center.

[0127] The apparatus 700 for evaluating the text classification performance of a large language model of this embodiment may further include a classification unit 706 for executing the following contents: when the large language model to be evaluated is determined to be a target large language model according to the text classification performance, obtaining a text to be classified corresponding to the target classification scenario; inputting the text to be classified into the target large language model for text classification, and using the output result of the target large language model as the classification result of the text to be classified.

[0128] That is to say, after completing the evaluation of the text classification performance of the large language model to be evaluated, the classification unit 706 can also determine whether the large language model to be evaluated is the target large language model based on the obtained text classification performance, and then after the determination, input the obtained text to be classified corresponding to the target classification scenario into the target large language model for text classification, which can improve the accuracy of the obtained text classification results.

[0129] If there is only one large language model to be evaluated, the classification unit 706 can determine that the large language model to be evaluated is the target large language model if it is determined that the text classification performance of the large language model to be evaluated exceeds a preset threshold; if there are multiple large language models to be evaluated, the classification unit 706 can use the large language model to be evaluated with the best text classification performance as the target large language model.

[0130] It can be understood that the classification unit 706 can obtain classification output data based on the text to be classified and the text classification task, and then input the classification input data into the target large language model; and when the target large language model has been set to only perform text classification tasks, only the text to be classified can be input into the target large language model.

[0131] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0132] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0133] like Figure 8 , is a block diagram of an electronic device according to a method for evaluating the text classification performance of a large language model according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0134] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0135] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0136] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the evaluation method of the text classification performance of the large language model. For example, in some embodiments, the evaluation method of the text classification performance of the large language model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808.

[0137] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for evaluating the text classification performance of the large language model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the method for evaluating the text classification performance of the large language model in any other appropriate manner (e.g., by means of firmware).

[0138] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable large language model text classification performance evaluation device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0142] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0143] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0145] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for evaluating the text classification performance of a large language model, comprising: Obtain an evaluation data set corresponding to a source evaluation task and an evaluation data set corresponding to at least one sub-evaluation task, wherein the source evaluation task is a text classification task, and the sub-evaluation tasks include an entity recognition task and a summary generation task, and different evaluation data sets include evaluation texts and annotated answers corresponding to different evaluation task types of the evaluation texts; Obtaining evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, inputting the evaluation input data into the large language model to be evaluated, and using the output result of the large language model to be evaluated as the predicted answer corresponding to the evaluation text for different evaluation task types; Obtaining a source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to the at least one sub-evaluation task according to the annotated answers and the predicted answers corresponding to the same evaluation task type of the evaluation text; Obtaining text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result; Wherein, obtaining the text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result includes: Obtaining weight values ​​corresponding to different evaluation task types, wherein the weight value of the source evaluation task is greater than the weight value of the sub-evaluation task; The text classification performance of the large language model to be evaluated is obtained according to the weight value corresponding to the source evaluation task and the source evaluation result, and the weight value corresponding to the at least one sub-evaluation task and the at least one sub-evaluation result.

2. The method according to claim 1, wherein: The evaluation input data obtained according to the evaluation text and the evaluation task type corresponding to the evaluation text includes: Determining the evaluation task type corresponding to the evaluation text; Obtaining a prompt text according to the evaluation task type; The evaluation input data is obtained according to the evaluation text and the prompt text.

3. The method according to claim 1, further comprising: Obtain the evaluation text corresponding to the target classification scenario; Obtaining initial input data according to the evaluation text and different evaluation task types, inputting the initial input data into a candidate large language model, and obtaining annotated answers corresponding to different evaluation task types output by the candidate large language model for the evaluation text; According to the evaluation text and the annotated answers of the evaluation text corresponding to different evaluation task types, evaluation data sets corresponding to different evaluation task types are obtained.

4. The method according to claim 1, wherein: The method of using the output result of the large language model to be evaluated as the predicted answer of the evaluation text corresponding to different evaluation task types includes: Construct a label syntax tree according to the classification labels corresponding to the target classification scenario; When it is determined that the evaluation task type is a text classification task, performing text parsing on the output result of the large language model to be evaluated to obtain a parsing result; The portion of the parsing result that matches the label syntax tree is extracted from the output result as the predicted answer.

5. The method according to claim 1, further comprising: When it is determined that the large language model to be evaluated is a target large language model according to the text classification performance, obtaining a text to be classified corresponding to a target classification scenario; The text to be classified is input into the target large language model for text classification, and the output result of the target large language model is used as the classification result of the text to be classified.

6. A device for evaluating the text classification performance of a large language model, comprising: An acquisition unit is used to acquire an evaluation data set corresponding to a source evaluation task and an evaluation data set corresponding to at least one sub-evaluation task, wherein the source evaluation task is a text classification task, and the sub-evaluation tasks include an entity recognition task and a summary generation task, and different evaluation data sets include evaluation texts and annotated answers corresponding to different evaluation task types of the evaluation texts; A prediction unit, used to obtain evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, input the evaluation input data into the large language model to be evaluated, and use the output result of the large language model to be evaluated as the prediction answer of the evaluation text corresponding to different evaluation task types; A processing unit, configured to obtain a source evaluation result corresponding to the source evaluation task and at least one sub-evaluation result corresponding to the at least one sub-evaluation task according to the annotated answers and the predicted answers corresponding to the same evaluation task type of the evaluation text; An evaluation unit, configured to obtain the text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result; Wherein, when the evaluation unit obtains the text classification performance of the large language model to be evaluated according to the source evaluation result and the at least one sub-evaluation result, it specifically performs: Obtaining weight values ​​corresponding to different evaluation task types, wherein the weight value of the source evaluation task is greater than the weight value of the sub-evaluation task; The text classification performance of the large language model to be evaluated is obtained according to the weight value corresponding to the source evaluation task and the source evaluation result, and the weight value corresponding to the at least one sub-evaluation task and the at least one sub-evaluation result.

7. The device according to claim 6, wherein: When the prediction unit obtains the evaluation input data according to the evaluation text and the evaluation task type corresponding to the evaluation text, the prediction unit specifically performs: Determining the evaluation task type corresponding to the evaluation text; Obtaining a prompt text according to the evaluation task type; The evaluation input data is obtained according to the evaluation text and the prompt text.

8. The apparatus according to claim 6, further comprising a construction unit for executing: Obtain the evaluation text corresponding to the target classification scenario; Obtaining initial input data according to the evaluation text and different evaluation task types, inputting the initial input data into a candidate large language model, and obtaining annotated answers corresponding to different evaluation task types output by the candidate large language model for the evaluation text; According to the evaluation text and the annotated answers of the evaluation text corresponding to different evaluation task types, evaluation data sets corresponding to different evaluation task types are obtained.

9. The device according to claim 6, wherein: When the prediction unit uses the output result of the large language model to be evaluated as the predicted answer of the evaluation text corresponding to different evaluation task types, the prediction unit specifically performs: Construct a label syntax tree according to the classification labels corresponding to the target classification scenario; When it is determined that the evaluation task type is a text classification task, performing text parsing on the output result of the large language model to be evaluated to obtain a parsing result; The portion of the parsing result that matches the label syntax tree is extracted from the output result as the predicted answer.

10. The apparatus according to claim 6, further comprising a classification unit, configured to perform: When it is determined that the large language model to be evaluated is a target large language model according to the text classification performance, obtaining a text to be classified corresponding to a target classification scenario; The text to be classified is input into the target large language model for text classification, and the output result of the target large language model is used as the classification result of the text to be classified.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images