Large model evaluation data generation method and system based on prompt learning

Abstract

The invention discloses a large model evaluation data generation method and system based on prompt learning, and belongs to the field of machine learning models, and the method comprises the following steps: obtaining an initial prompt template from a prompt template set; performing mutation operation on the initial prompt template; receiving an input; combining the mutated prompt template with the received input to form a plurality of complete prompts, and inputting the complete prompts into the generative model to obtain evaluation data; and the generation quality of the evaluation data is evaluated, and the prompt template capable of generating high-quality evaluation data is put into a prompt template set for next evaluation data generation. Compared with the prior art, the method has the advantages that the evaluation data generation quality is high, the evaluation data generation cost is low, the evaluation data generation speed is high, evaluation scenes are rich, and evaluation data generation is easy to achieve.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning models, and specifically relates to a method and system for generating large model evaluation data based on prompt learning. Background Art

[0002] Large models are machine learning models with large-scale parameters and complex computational structures. Large models are usually built with deep neural networks and have billions or even hundreds of billions of parameters. Large models are designed to improve the expressiveness and predictive performance of the model and to handle more complex tasks and data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems. Large models learn complex patterns and features by training massive amounts of data, have stronger generalization capabilities, and can make accurate predictions on unseen data. Large model evaluation is the process of performance evaluation and quality inspection of large models. This process aims to measure the performance of the model on specific tasks and data sets to ensure its effectiveness and reliability in practical applications.

[0003] Prompt learning is an emerging technique in natural language processing that aims to guide pre-trained language models to generate specific outputs or perform specific tasks by designing and using prompts. Prompt learning is different from traditional supervised learning methods in that it relies more on the pre-trained knowledge of the language model rather than on a large amount of labeled data. The key to prompt learning is how to design effective prompts so that the model can understand and follow these prompts to complete the target task. Among them, prompts can be natural language questions, task descriptions, or examples to help the model understand and complete tasks. The advantages of prompt learning are that it reduces the demand for labeled data, efficiently utilizes the knowledge of pre-trained language models, and flexibly adapts to multiple tasks. Prompt learning can achieve good performance in zero-shot or few-shot situations, allowing the model to perform well in data-scarce environments. Prompt learning methods have been widely used in text generation, question-answering systems, text classification, translation, and summarization. For example, by designing appropriate prompts, the model can generate text on a specific topic, answer questions, perform sentiment analysis, or translate text.

[0004] At present, there are two main methods for generating large model evaluation data: one is to train and fine-tune the generative model and use the generative model to generate evaluation data; the other is to directly use manually designed evaluation data. Among them, for training and fine-tuning the generative model, first prepare a large or appropriate amount of large model evaluation data, train a model from scratch or fine-tune it on the basis of a pre-trained language model that already has good generation capabilities, so as to obtain a generative model with good capabilities in the specific task of generating large model evaluation data. Model training is the process of providing a large amount of training data and target outputs, adjusting model parameters, and enabling the model to learn the relationship between input and output from the data. This process includes the following steps: first, data collection and preprocessing to ensure that the data quality and format are suitable for training; next is model initialization, selecting the appropriate model architecture and setting the initial parameters; then the training process, by inputting data into the model, calculating the error (loss) between the predicted value and the actual value, and adjusting the model parameters using the back propagation algorithm to reduce the error. During the training process, a validation set is usually used to evaluate the model performance and adjust the hyperparameters to optimize the generalization ability of the model. Model fine-tuning is to further adjust the pre-trained language model to improve its performance on a specific task or dataset. Usually, fine-tuning involves using a pre-trained model (for example, a deep learning model trained on a large-scale dataset) and then retraining it on a smaller task-specific dataset. The fine-tuning process involves freezing some model layers and only adjusting the parameters of the last few layers or specific layers to prevent overfitting and speed up the training process. Through this method of training and fine-tuning the generative model, the knowledge of the pre-trained language model can be used to significantly reduce training time and computing resources while improving the performance of the model. However, for training generative models, training a model from scratch requires a large amount of data, and it is difficult to obtain enough task-related data. If the amount of data obtained is insufficient or the data quality is not high, there will be a risk of overfitting, resulting in poor performance of the generative model in practical applications. In addition, the training of the model requires a lot of computing resources and time, and if the data is updated, it needs to be trained multiple times, which is costly. For fine-tuning generative models, although the data requirement is much lower than that for training models, the effect of fine-tuning is very dependent on the quality of the pre-fine-tuning data. If the pre-trained model is trained on a dataset that is very different from the target task, then fine-tuning may not be well adapted to the target task, resulting in performance degradation.

[0005] For manually designed evaluation data, the large model evaluation data is mainly designed and written manually by evaluators. Evaluators fully consider the complexity and variability of application scenarios and tasks, independently design evaluation data, or obtain samples from various public data sets and actual application data, and expand and enrich the evaluation data set through data enhancement techniques (such as synonym replacement, sentence reconstruction, etc.). At the same time, adding some edge cases and abnormal data can effectively test the robustness and generalization of large models. For manually designed evaluation data, the construction of high-quality data may be difficult, requiring a high level of professional knowledge and skills, and manually designed data is susceptible to the subjective bias and errors of the designer. In addition, the human resource cost required for manually designed data is high, and the design process may be very time-consuming. Summary of the invention

[0006] In order to solve the problem that large models may output harmful content, the present invention provides a large model evaluation data generation method and system based on prompt learning in order to generate high-quality and diverse evaluation data, build and generate a wide-coverage and fine-grained evaluation data set to meet model evaluation needs in different scenarios.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] The present invention provides a method for generating large model evaluation data based on prompt learning, which mainly includes the following steps:

[0009] Step S1: obtaining an initial prompt template from a prompt template set;

[0010] Step S2: performing mutation operation on the initial prompt template;

[0011] Step S3: receiving input;

[0012] Step S4: combining the mutated prompt template and the received input to form a number of complete prompts, and inputting these complete prompts into the generation model to obtain evaluation data;

[0013] Step S5: Evaluate the generation quality of the evaluation data, and put the prompt template that can generate high-quality evaluation data into the prompt template set for use in the next evaluation data generation.

[0014] Furthermore, the prompts in the prompt template set are manually written or collected from the Internet.

[0015] Furthermore, the mutation operation is text grammar transformation and text paraphrase generation.

[0016] Furthermore, the received input includes keywords, fields and external knowledge; the keywords are required, indicating which word the evaluation data revolves around; the fields are optional, indicating that the evaluation data is limited to a specific range; the external knowledge is optional. If the evaluation data needs to involve some uncommon knowledge or knowledge that the generation model does not have, it is necessary to add external knowledge for use by the generation model through manual input or using a knowledge graph.

[0017] The present invention provides a large model evaluation data generation system based on prompt learning, which is used to implement the large model evaluation data generation method based on prompt learning. The system includes:

[0018] The input module is used to receive input content, perform a validity check on the input content, and, when the input content needs to use external knowledge in the knowledge graph, call the knowledge graph module to obtain relevant knowledge text;

[0019] The knowledge graph module is used to store and maintain the knowledge graph, provide operations of adding, deleting, modifying and searching entities and relations, and to find all relevant entity-relationship-entity triples from the knowledge graph according to the keywords and fields passed by the input module, and to convert several groups of entity-relationship-entity triples into coherent knowledge texts using a data-to-text generation method;

[0020] The prompt module is used to store and maintain the prompt template set, provide add, delete, modify and query operations for the prompt template, select an initial prompt template from the prompt template set, and perform text grammar transformation and text retelling generation mutation operations on the initial prompt template;

[0021] The generation model module is used to combine the input content with the prompt template to form a complete prompt, and is used to adjust the prompt structure according to the text format of the model input and output when the generation model changes, and is used to call the generation model and input the prompt, receive the output of the generation model and perform format checking and modification.

[0022] Furthermore, the structure of the prompt template is shown in formula (1):

[0023] P0NP1[NP i ] n Formula (1)

[0024] Where N represents the blank position to be filled, P0, P1, P i They all represent prompt text, [·] represents an optional part, and the subscript n indicates that the optional part [·] can be repeated n times. The size of n is determined by the grammatical structure of the prompt and the amount of information entered by the user, n∈{0, 1, 2}.

[0025] Furthermore, the selection of the initial prompt template from the prompt template set is implemented by using the algorithm shown in formula (2):

[0026]

[0027] Where S represents the score of a certain prompt template, the first item represents the average of the past scores of the prompt template, that is, it tends to select the prompt template with a higher score, s i Indicates the score of the prompt template in the i-th iteration; in the second item Tend to select the prompt template that has been selected less times, c represents a constant, which is used to balance the two goals, N1 represents the total number of iterations, and n1 represents the number of times the prompt template has been selected;

[0028] Each time the initial prompt template is selected, the score of each prompt template in the prompt template set is calculated using formula (2), and then the m prompt templates with the highest scores are selected as the initial prompt templates. The size of m is determined by the amount of required evaluation data.

[0029] Furthermore, if t is used to represent the initial prompt template, t mut represents the mutated prompt template, S(·) represents the semantic quantization of the text, and T(·) represents the grammatical structure quantization of the text. The effect of the mutation operation is shown in formula (3) and formula (4):

[0030] |S(t)-S(t mut )|<ε Formula (3)

[0031] |T(t)-T(t mut )|>δ Formula (4)

[0032] Where ε represents a sufficiently small positive integer, δ represents a sufficiently large positive integer, formula (3) indicates that the semantic difference between the hint template before and after the mutation should be sufficiently small, and formula (4) indicates that the grammatical difference between the hint template before and after the mutation should be sufficiently large.

[0033] Furthermore, if the content input by the user has n parts, and n∈{1,2,3}, they are I1,...,I n , combining the input content with the prompt template shown in formula (1), and transforming I1, ..., I n Replace the blank position N, and the combined complete prompt is shown in formula (5):

[0034] P0I1P1...I n P n Formula (5).

[0035] The beneficial effects of the present invention are:

[0036] 1. High-quality evaluation data generation: As a generative model, the pre-trained language model has excellent language comprehension and expression capabilities, and can output high-quality answers when faced with rich and appropriate prompts.

[0037] 2. Low cost of generating evaluation data: Prompt learning only involves the reasoning of the generative model and does not require training, so the required computing power and time are very small, and the labor cost is negligible. The variation of the prompt template and the evaluation of the quality of the evaluation data require only a small amount of running and labor costs.

[0038] 3. Fast evaluation data generation: Prompt learning only requires building prompt and inference generation models, which requires very little runtime compared to model training.

[0039] 4. Rich evaluation scenarios: The pre-trained language model has a rich knowledge reserve. Using the pre-trained language model as a generation model, the generated evaluation data can cover many evaluation fields.

[0040] 5. Evaluation data generation is easy to implement: the diversity of prompt templates can support the implementation of multiple mutation algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of a method for generating large model evaluation data based on prompt learning provided by the present invention.

[0042] Figure 2 A structural block diagram of a large model evaluation data generation system based on prompt learning provided by the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0044] In a first aspect, the present invention provides a method for generating large model evaluation data based on prompt learning.

[0045] See also Figure 1 To illustrate, a large model evaluation data generation method based on prompt learning of the present invention has the following specific implementation process:

[0046] Step S1: obtaining an initial prompt template from a prompt template set;

[0047] First, a prompt template is prepared, and an initial prompt template is selected from a pre-prepared prompt template set, wherein the prompts in the prompt template set can be manually written or collected from the Internet.

[0048] Step S2: initial prompt template mutation operation;

[0049] Since the output of the model is very sensitive to the input prompts, in order to improve the diversity of generated evaluation data, it is necessary to enrich the prompt templates and perform mutation operations such as text grammar transformation and text retelling generation on the initial prompt templates.

[0050] Among them, text grammar transformation is a technology in natural language processing that aims to change the grammatical structure of a sentence or text without changing its original core meaning. Text grammar transformation involves adjusting language elements such as word order, sentence pattern, and clause structure to generate a new text that is semantically identical or similar to the original text but has a different grammatical structure.

[0051] Among them, text paraphrase generation is a technology in the field of natural language processing (NLP). Its core task is to restate the input text into an output text with the same or similar content but different wording and expression. The purpose of text paraphrase generation is to re-express the information by changing the sentence structure, replacing synonyms, adjusting the word order, or using different rhetorical techniques while maintaining the original meaning, so as to generate text with diverse language styles.

[0052] When preparing prompt templates, the present invention uses methods such as text grammar transformation and text paraphrase generation to mutate the templates, which significantly increases the diversity of prompts and further improves the diversity of evaluation data generated by the generation model.

[0053] Step S3: receiving input;

[0054] The received input mainly includes keywords, fields, and external knowledge. Knowledge in text form can be directly input, or keywords and fields can be used to query the knowledge graph to construct knowledge text.

[0055] Among them, keywords are required, indicating which word the evaluation data revolves around; fields are optional, indicating that the evaluation data is limited to a specific range, such as the economic field or the cultural field; external knowledge is also optional. If the evaluation data needs to involve some uncommon knowledge or knowledge that the generation model does not have, external knowledge needs to be input for the generation model to use.

[0056] In the present invention, in addition to necessary keywords and optional fields, the input content can also choose to add external knowledge through manual input or the use of knowledge graphs to deal with the situation where certain knowledge does not exist in the generated model or the amount of knowledge is insufficient, effectively broadening the coverage scenarios of the evaluation data.

[0057] Among them, the knowledge graph is a structured data model for representing and managing knowledge. It intuitively displays entities and their relationships in the form of nodes and edges. Each node represents an entity or concept, and the edge represents the relationship between these entities. The knowledge graph not only contains the connections between entities, but also carries semantic information about these entities, such as the categories, features of the entities, and the specific relationships between them. Knowledge graphs are widely used in search engines, recommendation systems, question-answering systems and other fields, which can help the system better understand the intention of user queries and provide more accurate and relevant answers.

[0058] The specific implementation process of step S3 is: first input keywords, then determine whether there is a restricted field, if so, input the field and then determine whether external knowledge is needed, if not, directly determine whether external knowledge is needed; determine whether external knowledge is needed, if so, directly determine whether a knowledge graph is needed, if not, combine the mutated prompt template with the received input to form several complete prompts; determine whether a knowledge graph is needed, if so, first query the knowledge graph, then construct the knowledge text, and finally input the knowledge text, combine the mutated prompt template with the received input to form several complete prompts, if not, directly input the knowledge text, and then combine the mutated prompt template with the received input to form several complete prompts;

[0059] Step S4: Combine the mutated prompt template and the received input to form a number of complete prompts, and input these complete prompts into the generation model to obtain evaluation data.

[0060] The specific implementation process of step S4 is: firstly, the mutated prompt template is combined with the received input to form a number of complete prompts, and then one of the prompts is taken and input into the generation model to generate evaluation data.

[0061] Step S5: Finally, the generation quality of the evaluation data is evaluated, and the prompt template that can generate high-quality evaluation data is placed in the prompt template set for use in the next evaluation data generation. After obtaining the evaluation data, the present invention iteratively optimizes the initial prompt template set by evaluating the generation quality of the evaluation data to improve the generation quality of the next generation task.

[0062] The specific implementation process of step S5 is: evaluate the generation quality of the evaluation data. If the generation quality of the evaluation data is high, put the corresponding prompt into the prompt template set, and then determine whether there are more prompts. If the generation quality of the evaluation data is not high, directly determine whether there are more prompts. Determine whether there are more prompts. If so, return to step S4 and take one of the prompts to input into the generation model to generate evaluation data. If not, end the process directly.

[0063] The present invention uses a prompt learning-based method to generate large model evaluation data, which has the advantages of fast speed and low cost compared to previous methods based on training, fine-tuning or manual construction. When preparing the prompt template, the template is mutated using methods such as text grammar transformation and text paraphrase generation, which significantly increases the diversity of prompts and further improves the diversity of evaluation data generated by the generation model. After obtaining the evaluation data, the initial prompt template set is iteratively optimized by evaluating the quality of the evaluation data to improve the generation quality of the next generation task. In addition to the necessary keywords and optional fields, the input content can also choose to add external knowledge by manual input or using knowledge graphs to deal with the situation where certain knowledge does not exist in the generation model or the amount of knowledge is insufficient, effectively broadening the coverage scenarios of the evaluation data.

[0064] In a second aspect, the present invention provides a large model evaluation data generation system based on prompt learning, which is used to implement a large model evaluation data generation method based on prompt learning provided in the first aspect.

[0065] See also Figure 2 To illustrate, the present invention is a large model evaluation data generation system based on prompt learning, which specifically includes the following modules: input module, knowledge graph module, prompt module and generation model module.

[0066] In the present invention, the input module is mainly used for receiving input modules, and the input module has the following functions:

[0067] (1) Receive input content and perform a legality check on the input content to ensure that the input content does not contain sensitive information, illegal information, malicious information, etc.;

[0068] (2) When the input content needs to use external knowledge in the knowledge graph, it is responsible for calling the knowledge graph module to obtain relevant knowledge text.

[0069] In the present invention, the knowledge graph module is mainly used to provide a module for operations such as knowledge graph access and maintenance, knowledge text generation, etc. The knowledge graph module has the following functions:

[0070] (1) Store and maintain knowledge graphs, and provide operations for adding, deleting, modifying, and querying entities and relationships;

[0071] (2) Search all relevant “entity-relationship-entity” triplets from the knowledge graph based on the keywords and fields passed by the input module;

[0072] (3) Use data-to-text generation methods to convert several groups of "entity-relation-entity" triplets into coherent knowledge text.

[0073] In the present invention, the prompt module is mainly used to provide a module for operations such as prompt template access and maintenance, prompt template variation, etc. The prompt module has the following functions:

[0074] (1) Store and maintain a collection of prompt templates, and provide operations for adding, deleting, modifying, and querying prompt templates. The structure of a prompt template is shown in formula (1):

[0075] P0NP1[NP i ] n Formula (1)

[0076] Where N represents the blank position to be filled, P0, P1, P i All of them represent prompt text, [·] represents optional part, and subscript n represents optional part [·] can be repeated n times. The size of n is determined by the grammatical structure of the prompt and the amount of information input by the user. Since the amount of information that the user can input is limited, it only includes three parts: keywords, fields, and external knowledge. Therefore, the value of n is n∈(0, 1, 2};

[0077] (2) selecting an initial prompt template from a prompt template set;

[0078] Common selection strategies mainly include: random selection, sequential selection or cyclic selection, etc., but these methods cannot dynamically adjust the selection according to the generation results of the generative model, and lack flexibility. Therefore, the following algorithm is used to construct a flexible and efficient selection strategy to dynamically adjust the score of each prompt template according to the quality of the generated evaluation data. The algorithm used is shown in formula (2):

[0079]

[0080] In formula (2), S represents the score of a certain prompt template. The first term represents the average of the past scores of the prompt template, that is, it tends to select the prompt template with a higher score, s i Indicates the score of the prompt template in the i-th iteration; in the second item Tend to select the prompt template that has been selected less times, c represents a constant, which is used to balance the two goals. Where N1 represents the total number of iterations, and n1 represents the number of times the prompt template has been selected.

[0081] Each time the initial prompt template is selected, the score of each prompt template in the prompt template set is first calculated using the above formula (2), and then the m prompt templates with the highest scores are selected as the initial prompt templates, where the size of m is determined by the amount of required evaluation data.

[0082] (3) Perform mutation operations such as text grammar transformation and text retelling generation on the initial prompt template. Let t represent the initial prompt template, t mutrepresents the mutated prompt template, S(·) represents the semantic quantization of the text, and T(·) represents the grammatical structure quantization of the text. The effect of the mutation is shown in formula (3) and formula (4):

[0083] |S(t)-S(t mut )|<ε Formula (3)

[0084] |T(t)-T(t mut )|>δ Formula (4)

[0085] Where ε represents a sufficiently small positive integer, δ represents a sufficiently large positive integer, formula (3) indicates that the semantic difference between the prompt templates before and after the mutation should be sufficiently small, and formula (4) indicates that the grammatical structure difference between the prompt templates before and after the mutation should be sufficiently large, so as to achieve prompt diversification.

[0086] In the present invention, the generation model module is mainly used to provide a module for operations such as generation model access and maintenance. The generation model module has the following functions: storing and maintaining the generation model. Specifically:

[0087] (1) Combine the input content with the prompt template to form a complete prompt. If the user input content has n parts, and n∈{1,2,3}, they are I1,...,I n , combining the input content with the prompt template shown in formula (1), and transforming I1, ..., I n Replace the blank position N, and the combined complete prompt is shown in formula (5):

[0088] P0I1P1...I n P n Formula (5)

[0089] (2) When the generative model changes, the prompt structure is adjusted according to the text format of the model input and output (for example, when the generative model is Bert, the answer space in the prompt is in the middle of the sentence, and when the generative model is the traditional Transformer, the answer space in the prompt is at the end of the sentence).

[0090] (3) Call the generative model and input prompts, receive the output of the generative model and perform format checking and modification, including deleting extra blank lines and spaces, deleting extra prefixes and suffixes, etc.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein, but these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating large model evaluation data based on prompt learning, characterized in that: The following steps are involved: Step S1: obtaining an initial prompt template from a prompt template set; Step S2: performing mutation operation on the initial prompt template; Step S3: receiving input; Step S4: combining the mutated prompt template and the received input to form a number of complete prompts, and inputting these complete prompts into the generation model to obtain evaluation data; Step S5: Evaluate the generation quality of the evaluation data, and put the prompt template that can generate high-quality evaluation data into the prompt template set for use in the next evaluation data generation.

2. According to the method for generating large model evaluation data based on prompt learning according to claim 1, it is characterized in that: The prompts in the prompt template set are manually written or collected from the Internet.

3. According to the method for generating large model evaluation data based on prompt learning according to claim 1, it is characterized in that: The mutation operation is text grammar transformation and text paraphrase generation.

4. The method for generating large model evaluation data based on prompt learning according to claim 1, characterized in that: The received input includes keywords, fields and external knowledge; the keywords are required, indicating which word the evaluation data revolves around; the fields are optional, indicating that the evaluation data is limited to a specific range; the external knowledge is optional. If the evaluation data needs to involve some uncommon knowledge or knowledge that the generation model does not have, it is necessary to add external knowledge for use by the generation model through manual input or the use of knowledge graphs.

5. A large model evaluation data generation system based on prompt learning, characterized in that: The system is used to implement a method for generating large model evaluation data based on prompt learning as described in any one of claims 1 to 4, comprising: The input module is used to receive input content, perform a validity check on the input content, and, when the input content needs to use external knowledge in the knowledge graph, call the knowledge graph module to obtain relevant knowledge text; The knowledge graph module is used to store and maintain the knowledge graph, provide operations of adding, deleting, modifying and searching entities and relations, and to find all relevant entity-relationship-entity triples from the knowledge graph according to the keywords and fields passed by the input module, and to convert several groups of entity-relationship-entity triples into coherent knowledge texts using a data-to-text generation method; The prompt module is used to store and maintain the prompt template set, provide add, delete, modify and query operations for the prompt template, select an initial prompt template from the prompt template set, and perform text grammar transformation and text retelling generation mutation operations on the initial prompt template; The generation model module is used to combine the input content with the prompt template to form a complete prompt, and is used to adjust the prompt structure according to the text format of the model input and output when the generation model changes, and is used to call the generation model and input the prompt, receive the output of the generation model and perform format checking and modification.

6. A large model evaluation data generation system based on prompt learning according to claim 5, characterized in that: The structure of the prompt template is shown in formula (1): P0NP1[NP i ] n Formula(1) Where N represents the blank position to be filled, P0, P1, P i They all represent prompt text, [·] represents an optional part, and the subscript n indicates that the optional part [·] can be repeated n times. The size of n is determined by the grammatical structure of the prompt and the amount of information entered by the user, n∈{0,1,2}.

7. The method for generating large model evaluation data based on prompt learning according to claim 5, characterized in that: The selection of the initial prompt template from the prompt template set is implemented by using the algorithm shown in formula (2): Where S represents the score of a certain prompt template, the first item represents the average of the past scores of the prompt template, that is, it tends to select the prompt template with a higher score, s i Indicates the score of the prompt template in the i-th iteration; in the second item Tend to select the prompt template that has been selected less times, c represents a constant, which is used to balance the two goals, N1 represents the total number of iterations, and n1 represents the number of times the prompt template has been selected; Each time the initial prompt template is selected, the score of each prompt template in the prompt template set is calculated using formula (2), and then the m prompt templates with the highest scores are selected as the initial prompt templates. The size of m is determined by the amount of required evaluation data.

8. The system for generating large model evaluation data based on prompt learning according to claim 5, characterized in that: If t represents the initial prompt template, t mut represents the mutated prompt template, S(·) represents the semantic quantization of the text, and T(·) represents the grammatical structure quantization of the text. The effect of the mutation operation is shown in formula (3) and formula (4): |S(t)-S(t mut )|< ε Equation (3) |T(t)-T(t mut )|>δ Formula (4) Where ε represents a sufficiently small positive integer, δ represents a sufficiently large positive integer, formula (3) indicates that the semantic difference between the hint template before and after the mutation should be sufficiently small, and formula (4) indicates that the grammatical difference between the hint template before and after the mutation should be sufficiently large.

9. A large model evaluation data generation system based on prompt learning according to claim 6, characterized in that: If the user input has n parts, and n∈{1,2,3}, they are I1,…,I n , combine the input content with the prompt template shown in formula (1), and transform I1,…,I n Replace the blank position N, and the combined complete prompt is shown in formula (5): P0I1P1…I n P n Formula (5).

Images