Model instruction quality evaluation method and apparatus, computer device and storage medium

Abstract

The present application relates to the field of machine learning, in particular to a model instruction quality evaluation method and apparatus, a computer device, a storage medium and a computer program product. The method comprises: acquiring a model instruction and a plurality of sample sets (step 202); for each of the plurality of sample sets, separately fusing the model instruction and each quality evaluation sample in the sample set, so as to obtain a model input set corresponding to the sample set (step 204); acquiring a machine learning model corresponding to the model instruction, and, on the basis of the model input set corresponding to the sample set, triggering the machine learning model to execute a specified task, so as to obtain a task result set (step 206); according to the accuracy of each task execution result in the task result set, determining a quality evaluation score of the model instruction obtained by means of evaluation based on the sample set (step 208); and, according to the quality evaluation score obtained by means of separate evaluation based on each sample set, determining an overall quality evaluation score of the model instruction (step 210).

Description

Model instruction quality assessment method, device, computer equipment and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 2024103836748, filed on March 29, 2024, entitled “Quality Assessment Method, Apparatus, Computer Equipment and Storage Medium for Model Instructions,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for evaluating the quality of model instructions. Background Art

[0004] With the advancement of science and technology, large models have emerged. Large models are a type of model in machine learning and artificial intelligence characterized by a large number of parameters and complex structure. These models are typically constructed from multi-layer neural networks, with each layer containing a large number of neurons. Large models are often able to capture complex patterns and relationships in data, achieving excellent performance in various tasks such as image recognition and natural language processing.

[0005] In practical applications, the expression of model instructions has a significant impact on the generation results of large models. A model instruction (prompt) typically refers to a prompt word or phrase. It is a text instruction or description entered by the user to the large model to guide the large model in generating the corresponding response. When constructing a large number of task-related model instructions, it is extremely valuable to automatically evaluate the model instructions, grade their quality, and then put high-quality model instructions into production to guide the large model to complete tasks more accurately.

[0006] Currently, the quality assessment of model instructions mainly relies on manual labor, but the efficiency of manual quality assessment of model instructions is low. Summary of the Invention

[0007] According to various embodiments provided in this application, a method, apparatus, computer device, computer-readable storage medium, and computer program product for evaluating the quality of a model instruction are provided.

[0008] In a first aspect, the present application provides a method for evaluating the quality of a model instruction, performed by a computer device, the method comprising:

[0009] Get model instructions and multiple sample sets;

[0010] For each sample set in the multiple sample sets, fusing the model instruction with each quality assessment sample in the sample set to obtain a model input set corresponding to the sample set;

[0011] Obtaining a machine learning model corresponding to the model instruction, triggering the machine learning model to perform the specified task based on a model input set corresponding to the sample set, and obtaining a task result set; the specified task is the task that the machine learning model is specified to perform by the model instruction;

[0012] Determining a quality assessment score of the model instruction obtained by evaluating the sample set according to the accuracy of each task execution result in the task result set;

[0013] The comprehensive quality evaluation score of the model instruction is determined according to the quality evaluation score obtained by evaluating each sample set respectively.

[0014] In a second aspect, the present application further provides a model instruction quality assessment device, the device comprising:

[0015] A model input set generation module is configured to obtain a model instruction and a plurality of sample sets; for each sample set in the plurality of sample sets, the model instruction is respectively fused with each quality assessment sample in the sample set to obtain a model input set corresponding to the sample set;

[0016] a quality assessment score determination module configured to obtain a machine learning model corresponding to the model instruction, trigger the machine learning model to perform the specified task based on a model input set corresponding to the sample set, and obtain a task result set; the specified task being the task specified by the model instruction for the machine learning model to perform; and determine, based on the accuracy of each task execution result in the task result set, a quality assessment score for the model instruction obtained by evaluating the sample set;

[0017] The comprehensive quality evaluation score determination module is used to determine the comprehensive quality evaluation score of the model instruction according to the quality evaluation score obtained by evaluating each sample set respectively.

[0018] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps in the quality assessment method of any model instruction provided in the embodiments of the present application.

[0019] In a fourth aspect, the present application also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps in the quality assessment method of any model instruction provided in the embodiments of the present application are implemented.

[0020] In a fifth aspect, the present application also provides a computer program product, which includes computer-readable instructions. When the computer-readable instructions are executed by a processor, they implement the steps in the quality assessment method of any model instruction provided in the embodiments of the present application.

[0021] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a diagram illustrating an application environment of a method for evaluating the quality of model instructions according to an embodiment;

[0023] FIG2 is a schematic flow chart of a method for evaluating the quality of model instructions according to one embodiment;

[0024] FIG3 is a schematic diagram of determining a quality assessment score in one embodiment;

[0025] FIG4 is a schematic diagram of determining a comprehensive quality assessment score in one embodiment;

[0026] FIG5 is a schematic diagram of generating a sample set in one embodiment;

[0027] FIG6 is a schematic diagram of determining sample weights in one embodiment;

[0028] FIG7 is a schematic diagram of determining a sample score in one embodiment;

[0029] FIG8 is a schematic diagram of set weights in one embodiment;

[0030] FIG9 is a schematic diagram showing the overall determination of set weights in one embodiment;

[0031] FIG10 is a schematic diagram of generating multiple comprehensive quality scores in one embodiment;

[0032] FIG11 is a schematic diagram illustrating the application of high-quality model instructions according to an embodiment;

[0033] FIG12 is a flow chart of a method for evaluating the quality of model instructions according to a specific embodiment;

[0034] FIG13 is a block diagram of a quality assessment apparatus for model instructions according to one embodiment;

[0035] FIG14 is a diagram showing the internal structure of a computer device according to another embodiment;

[0036] FIG15 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0038] The quality assessment method of the model instruction provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. Both the terminal 102 and the server 104 can be used alone to execute the quality assessment method of the model instruction provided in the embodiment of the present application. The terminal 102 and the server 104 can also be used together to execute the quality assessment method of the model instruction provided in the embodiment of the present application. Taking the example of the terminal 102 and the server 104 being used to jointly execute the quality assessment method of the model instruction provided in the embodiment of the present application, the terminal 102 can obtain the input model instruction and send the model instruction to the server 104, so that the server 104 performs quality assessment on the model instruction based on multiple sample sets.

[0039] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0040] It should be noted that the terms "first", "second" and similar terms used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Unless the context clearly indicates otherwise, the singular form "a", "an" or "the" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The quantities of "multiple" or "multiple copies" mentioned in the various embodiments of this application refer to the quantity of "at least two", for example, "multiple" refers to "at least two" and "multiple copies" refers to "at least two copies".

[0041] In one embodiment, as shown in FIG2 , a method for evaluating the quality of model instructions is provided. This method is described using a computer device as an example. The computer device may be the terminal or server shown in FIG1 . The method for evaluating the quality of model instructions includes the following steps:

[0042] Step 202: Obtain model instructions and multiple sample sets.

[0043] A model instruction, or prompt, is a word or phrase that is input into a machine learning model to guide it in generating the desired output. A model instruction can be a question, a sentence, a paragraph, or a set of instructions, designed to guide the machine learning model in generating the desired text, image, or video output.

[0044] The sample set includes at least one quality assessment sample, which is used to evaluate the quality of the model instruction. For example, a model instruction such as "Please determine the sentiment polarity (positive / neutral / negative) of <sentence>" triggers the machine learning model to analyze the sentiment of the sentence, and "analyzing the sentiment of the sentence" is the designated task. A quality assessment sample can be a sentence, such as "Snowflakes are falling, the scenery is pleasant." Correspondingly, quality assessment samples can also be questions and answers, images, or videos.

[0045] Specifically, the computer device may obtain a model instruction to be quality-evaluated and a plurality of sample sets used to perform quality evaluation on the model instruction. For example, a user may input the model instruction to be quality-evaluated into the computer device and upload the plurality of sample sets used for quality evaluation into the computer device. In this way, the computer device may obtain the model instruction and the plurality of sample sets.

[0046] In one embodiment, each sample set may have a corresponding set weight, and each quality assessment sample in each sample set may also have a corresponding sample weight. For each sample set in multiple sample sets, the computer device may determine the set weight corresponding to the sample set. For example, when there are sample sets A, B, and C, the computer device may determine the set weights for sample sets A, B, and C, respectively. The larger the set weight, the greater the importance of the sample set for the quality assessment of the model instruction, and the better the sample set measures or evaluates the accuracy of the result obtained by the model instruction triggering the machine learning model to perform a specified task T. For example, there is a model instruction "Please determine the sentiment polarity (positive / neutral / negative) of the sentence <sentence>", which triggers the machine learning model to perform a specified task T = "positive and negative information identification task". For the positive and negative information identification task, there are three sample sets: "Short Video Comment Sample Set" (D1), "Ancient Poetry Sample Set" (D2), and "English Writing Sample Set" (D3). Since the positive and negative information identification task is mainly used for the real-time identification of harmful speech, the "short video comment sample set" (D1) is the most important for the quality evaluation of model instructions, and thus the set weight of the "short video comment sample set" (D1) is the largest.

[0047] In addition to each sample set having its own corresponding set weight, each quality assessment sample in a single sample set can also have a corresponding sample weight. For example, in the above example, the computer device needs to determine the sample weight corresponding to each quality assessment sample in sample set A, the sample weight corresponding to each quality assessment sample in sample set B, and the sample weight corresponding to each quality assessment sample in sample set C. The larger the sample weight corresponding to a quality assessment sample, the more important it is. For example, given a model instruction, "Please determine the sentiment polarity (positive / neutral / negative) of <sentence>," this model instruction triggers the machine learning model to perform a specified task, T = "positive and negative information identification task." For quality assessment sample A, "A dangerous incident is about to occur at a certain subway station," and quality assessment sample B, "Smoking is fun, I hate the smoking ban," the machine learning model identifies "A dangerous incident is about to occur at a certain subway station" as "negative." Compared to "Smoking is fun, I hate the smoking ban," which is identified as "negative," the former is more important. Therefore, this quality assessment sample needs to be correctly identified by the machine learning model and assigned a higher sample weight to prioritize it in the quality assessment.

[0048] In one embodiment, the set weight corresponding to a sample set can be determined by the correlation between the sample set and a designated task. A higher correlation indicates a higher set weight, and a higher set weight indicates that the sample set is more capable of evaluating the quality of the model instruction. The designated task is the task that the model instruction triggers the machine learning model to perform. For example, the designated task can be the "positive and negative information identification task" mentioned above.

[0049] In one embodiment, a pre-trained logistic regression model can be used to determine the sample weights corresponding to quality assessment samples. Logistic regression models include, but are not limited to, a. Neural Network Regression, b. Support Vector Regression (SVR), and c. Decision Tree Regression. Logistic regression, also known as logistic regression analysis, is a generalized linear regression analysis model that belongs to supervised learning in machine learning. Its derivation and calculation methods are similar to those of regression, but it is actually mainly used to solve classification problems. The model is trained using n sets of data (training sets), and after training, one or more sets of data (test sets) are classified. For example, given some sample features of a quality assessment sample, to determine whether the quality assessment sample is "important" or "unimportant" based on the sample features, the sample features and corresponding sample weights of n quality assessment samples can be obtained first, and these n sets of data are input into the logistic regression model for training. After training, the sample features of the quality assessment sample to be classified are input into the logistic regression model. The logistic regression model is used to determine whether the quality assessment sample belongs to the important category or the unimportant category. If it belongs to the important category, the sample weight of the quality assessment sample is increased, and if it belongs to the unimportant category, the sample weight of the quality assessment sample is reduced.

[0050] Step 204 : For each sample set in the plurality of sample sets, the model instruction is fused with each quality assessment sample in the sample set to obtain a model input set corresponding to each sample set.

[0051] The model input set includes at least one model input sample, which can be text, questions and answers, images, or videos used to input the machine learning model. A machine learning model is a model composed of multiple layers of neural networks, each layer of which contains multiple neurons. In this embodiment, the machine learning model can specifically be a large model, i.e., a large-scale machine learning model.

[0052] Specifically, for each sample set in the multiple sample sets, the computer device fuses each quality assessment sample in the sample set with the model instruction to obtain a model input set corresponding to the sample set. For example, for sample set A, sample set A includes quality assessment sample a, quality assessment sample b, and quality assessment sample c. The computer device fuses quality assessment sample a with the model instruction to obtain model input sample a, fuses quality assessment sample b with the model instruction to obtain model input sample b, and fuses quality assessment sample c with the model instruction to obtain model input sample c. Model input sample a, model input sample b, and model input sample c constitute the model input set corresponding to sample set A.

[0053] In one embodiment, the model instruction and the quality assessment sample are integrated to obtain the model input sample in a specific manner: the quality assessment sample can be filled into the specified position of the model instruction to obtain the model input sample. For example, when the model instruction is: "Is <sentence> positive, neutral, or negative, please give the result", for the quality assessment sample: "Snowflakes are falling, the scenery is pleasant", the computer device fills "Snowflakes are falling, the scenery is pleasant" into "Is <sentence> positive, neutral, or negative, please give the result", and obtains the model input sample: "Is 'snowflakes are falling, the scenery is pleasant' positive, neutral, or negative, please give the result". Among them, <sentence> is the specified position in the model instruction.

[0054] Step 206: obtain the machine learning model corresponding to the model instruction, trigger the machine learning model to perform the specified task based on the model input set corresponding to the sample set, and obtain a task result set; the specified task is the task performed by the machine learning model specified by the model instruction.

[0055] Specifically, since the model instruction can trigger the machine learning model to perform the corresponding task, the computer device can obtain the machine learning model corresponding to the model instruction. The machine learning model can be a model pre-set for the model instruction. For example, there is a model instruction "Please judge the emotional polarity of the sentence <sentence> (positive / neutral / negative)", then the machine learning model corresponding to the model instruction can be a large language model, such as machine learning model A, then the model instruction triggers the machine learning model A to perform the specified task T = "positive and negative information identification task". For another example, by the model instruction "Please judge the emotional polarity of the picture <picture> (positive / neutral / negative)", then the machine learning model corresponding to the model instruction can be an image recognition model, such as machine learning model B, and the model instruction triggers the machine learning model B to perform the specified task T = "positive and negative information identification task".

[0056] Furthermore, for each sample set in the multiple sample sets, the computer device obtains a model input set corresponding to the sample set being targeted, and inputs each model input sample in the model input set into the machine learning model to trigger the machine learning model to perform a specified task and output a corresponding task execution result. The specified task refers to a task that the model instruction guides the machine learning model to perform, such as the above-mentioned positive and negative emotion judgment task. Since each model input sample in the model input set corresponding to the sample set being targeted can trigger the machine learning model to output a corresponding task execution result, each task execution result constitutes a task result set corresponding to the sample set being targeted. It is easy to understand that each sample set can have a corresponding task result set.

[0057] Step 208 : Determine a quality assessment score of the model instruction obtained based on the sample set evaluation according to the accuracy of each task execution result in the task result set.

[0058] Specifically, for each task execution result in the task result set corresponding to the targeted sample set, the computer device can determine the accuracy of each task execution result, and determine the quality evaluation score of the model instruction on the targeted sample set based on the accuracy of each task execution result, that is, determine the quality evaluation score of the model instruction obtained based on the sample set evaluation.

[0059] For example, when the accuracy of the task execution result is higher, the quality evaluation score is higher, and the more accurate the task execution results are, the higher the quality evaluation score is. Among them, the quality evaluation score of the model instruction on the targeted sample set represents: for the model instruction, the quality score evaluated based on the targeted sample set. The larger the quality evaluation score of the model instruction on the targeted sample set, the better the quality of the model instruction evaluated based on the targeted sample set. If the quality evaluation score of the model instruction on the targeted sample set is smaller, the worse the quality of the model instruction evaluated based on the targeted sample set. The better the quality of the model instruction, the more likely it is that the machine learning model can be triggered by the model instruction to output the desired result. The worse the quality of the model instruction, the result output by the machine learning model triggered by the model instruction may not be ideal and cannot meet the needs of the user.

[0060] In one embodiment, for each sample set in a plurality of sample sets, since each quality assessment sample in the sample set has a corresponding sample weight, the computer device can also determine the quality assessment score of the model instruction on the sample set according to the task result set and the sample weight corresponding to each quality assessment sample in the sample set. For example, referring to FIG3 , for sample set A, the computer device can input the model input samples a to c in the model input set A corresponding to the sample set A into the machine learning model, process the model input samples a to c through the machine learning model, and obtain the task execution results a to c of the machine learning model for the model input samples a to c, so that the computer device can obtain the quality assessment score of the model instruction on the sample set A based on the task execution results a to c and the sample weights corresponding to the quality assessment samples a to c in the sample set. FIG3 shows a schematic diagram of determining the quality assessment score in one embodiment.

[0061] In one embodiment, a computer device determines a quality assessment score of a model instruction on a targeted sample set based on a task result set and a sample weight corresponding to each quality assessment sample in the targeted sample set, including: the computer device may determine a sample score corresponding to each task execution result based on the accuracy of each task execution result in the task result set, and obtain a quality assessment score based on the sample score corresponding to each task execution result and the sample weight corresponding to each quality assessment sample in the targeted sample set.

[0062] For example, for sample set A, which includes quality assessment sample a and quality assessment sample b, the computer device obtains model input sample a based on quality assessment sample a and obtains model input sample b based on quality assessment sample b. Based on model input sample a and the machine learning model, task execution result a is obtained. If task execution result a is correct, the sample score corresponding to task execution result a is determined to be a first score. If task execution result a is incorrect, the sample score corresponding to task execution result a is determined to be a second score. Based on model input sample b and the machine learning model, task execution result b is obtained. If task execution result b is correct, the sample score corresponding to task execution result b is determined to be a first score. If task execution result b is incorrect, the sample score corresponding to task execution result b is determined to be a second score. The computer device may determine that quality assessment sample a corresponds to task execution result a and that quality assessment sample b corresponds to task execution result b, and multiply the sample weight corresponding to quality assessment sample a by the sample score corresponding to task execution result a to obtain multiplication result a; and multiply the sample weight corresponding to quality assessment sample b by the sample score corresponding to task execution result b to obtain multiplication result b. The computer device superimposes and normalizes the multiplication result a and the multiplication result b to obtain a quality assessment score of the model instruction on the sample set A.

[0063] Step 210 : Determine a comprehensive quality evaluation score of the model instruction based on the quality evaluation scores obtained based on the evaluation of each sample set.

[0064] Specifically, after obtaining the quality evaluation scores of the model instructions on each sample set, that is, when obtaining the quality evaluation scores obtained based on the evaluation of each sample set, the computer device can determine the comprehensive quality evaluation score of the model instruction based on the quality evaluation scores of the model instructions on each of the sample sets. For example, the computer device can use the average of the quality evaluation scores as the comprehensive quality evaluation score. Among them, the comprehensive quality score is the quality score evaluated for the model instruction based on the full sample set. The comprehensive quality score can be used to characterize the accuracy of the task execution result obtained by triggering the machine learning model to perform the task based on the model instruction. The higher the comprehensive quality score, the higher the accuracy of the task execution result obtained by triggering the machine learning model to perform the corresponding task based on the model instruction, and the better the quality of the model instruction, and vice versa.

[0065] In one embodiment, since each sample set may have a corresponding set weight, the computer device may also determine the comprehensive quality assessment score of the model instruction based on the set weight corresponding to each sample set and the quality assessment score of the model instruction on each sample set. For example, referring to FIG4 , the computer device multiplies the quality assessment score of the model instruction on sample sets A to C by the set weight of the corresponding sample set, respectively, to obtain multiplication results c to e, and obtains a comprehensive quality assessment score based on the multiplication results c to e. For example, the multiplication results c to e are superimposed and normalized to obtain a comprehensive quality assessment score. FIG4 shows a schematic diagram of determining the comprehensive quality assessment score in one embodiment.

[0066] For easy understanding, the task modalities for a given task can be: image, speech, text, etc. Task types for a given task can be: sentiment classification, professional exams, programming languages, software development and problem solving, image understanding, mathematical and scientific problem solving, comprehensive data analysis, fact checking, games and entertainment, etc. Machine learning models can be applied to fields such as health, finance, scientific research, law, and education.

[0067] In the above-described model instruction quality assessment method, by obtaining a model instruction to be evaluated and multiple sample sets, the model instruction can be fused with each quality assessment sample in each sample set to obtain a model input set corresponding to each sample set. By obtaining the model input set, for each sample set in the multiple sample sets, a task result set corresponding to the sample set can be output based on the machine learning model. A quality assessment score based on the accuracy of the task execution results in the task result set can be obtained. By obtaining the quality assessment scores evaluated for each sample set, a comprehensive quality assessment score for the model instruction can be obtained based on each quality assessment score. Because the comprehensive quality assessment score can be automatically obtained, compared to traditional manual quality assessment of model instructions, this application can significantly improve the efficiency of quality assessment. Furthermore, this application can be widely used in the actual production of large model instruction projects, by searching for high-quality model instructions to trigger large models to output more accurate results. Furthermore, because the comprehensive quality assessment score of the model instruction is determined based on multiple sample sets, compared to determining the comprehensive quality assessment score based on only a single sample set, the probability of inaccurate quality assessment due to too few sample sets can be reduced, thereby improving the accuracy of the model instruction quality assessment.

[0068] In one embodiment, the computer device may determine multiple data sources. For each of the multiple data sources, the computer device may obtain multiple samples to be processed from the targeted data source, and downsample the obtained multiple samples to be processed to obtain a sample set corresponding to the targeted data source.

[0069] Specifically, the computer device collects samples to be processed from various data sources, including network data sources and local data sources. For example, referring to Figure 5 , the computer device can obtain a network sample set through a legitimate path and obtain a public sample set and a task sample set through downloading. When the designated task is positive and negative information identification, the samples to be processed in the network sample set, the public sample set, and the task sample set can be, for example, text such as chat conversations, movie dialogues, or book scripts. Furthermore, considering the number of samples and computing resources required for evaluation, maintaining n fixed-size sample sets can achieve the quality assessment goal. Therefore, the computer device downsamples the acquired samples to be processed, and the sampling probability can be freely set as needed. For example, samples to be processed can be extracted from the network sample set using a preset sampling probability, used as quality assessment samples, and these quality assessment samples can be combined into a sample set. Furthermore, quality assessment samples can be extracted from the public sample set using a preset sampling probability to obtain another sample set, and quality assessment samples can be extracted from the task sample set using a preset sampling probability to obtain yet another sample set. Figure 5 shows a schematic diagram of sample set generation in one embodiment.

[0070] In this embodiment, by obtaining samples to be processed from multiple data sources, the diversity of the samples to be processed can be improved, thereby improving the accuracy of the determined comprehensive quality assessment score based on the diverse samples.

[0071] In one embodiment, the model input set includes a model input sample corresponding to each quality assessment sample in the sample set; the model instruction is respectively integrated with each quality assessment sample in the sample set to obtain a model input set corresponding to the sample set, including: determining the evaluation sample filling position in the model instruction according to a preset keyword; for each quality assessment sample in the sample set, filling the quality assessment sample into the evaluation sample filling position in the model instruction to obtain a model input sample corresponding to the quality assessment sample.

[0072] Specifically, when a model input set needs to be generated, the computer device determines the evaluation sample filling location based on a preset keyword, wherein the preset keyword can specifically be an evaluation sample filling index. Therefore, the evaluation sample filling location can be determined based on the evaluation sample filling index. For example, when the model instruction is "Please judge the sentiment polarity (positive / neutral / negative) of the sentence <sentence>", the evaluation sample filling index is "<sentence>", and the position of the evaluation sample filling index in the model instruction is the evaluation sample filling location.

[0073] Furthermore, for each sample set in the plurality of sample sets, the computer device performs the following steps: for each quality assessment sample in the sample set, the computer device fills the quality assessment sample in the assessment sample filling position in the model instruction to obtain a model input sample corresponding to the quality assessment sample. For example, when the quality assessment sample is "snowflakes are falling, the scenery is pleasant", and the model instruction is "please judge the sentiment polarity of <sentence> (positive / neutral / negative)", then "snowflakes are falling, the scenery is pleasant" can be filled in "<sentence>" to obtain the model input sample "please judge the sentiment polarity of <snowflakes are falling, the scenery is pleasant> (positive / neutral / negative)".

[0074] Furthermore, when the model input samples corresponding to each quality assessment sample in the target sample set are obtained, the model input samples corresponding to each quality assessment sample in the target sample set constitute the model input set corresponding to the target sample set. For example, if sample set A contains three quality assessment samples a to c, then the three model input samples a to c generated based on the three quality assessment samples a to c constitute the model input set corresponding to sample set A.

[0075] In this embodiment, by setting the evaluation sample filling index, it is convenient to fill the quality evaluation sample into the evaluation sample filling position, thereby improving the generation efficiency of the model input sample.

[0076] In one embodiment, the task result set includes the task execution results corresponding to each model input sample in the model input set; based on the model input set corresponding to the sample set, the machine learning model is triggered to perform the specified task to obtain the task result set, including: for each model input sample in the model input set corresponding to the sample set, the model input sample is input into the machine learning model to trigger the machine learning model to output the task execution result corresponding to the model input sample.

[0077] Specifically, for each sample set in multiple sample sets, after obtaining the model input set corresponding to the sample set, the computer device can input each model input sample in the model input set into the machine learning model respectively to obtain the task execution results corresponding to each model input sample output by the machine learning model. The task execution results corresponding to each model input sample constitute the task result set.

[0078] In this embodiment, each model input sample in the model input set is used to trigger the machine learning model to output the corresponding task execution result, so that the quality assessment score can be determined by combining multiple task execution results. Compared with determining the quality assessment score based only on a single task execution result, the probability of inaccurate quality assessment score due to incorrect task execution results can be reduced, thereby improving the accuracy of the quality assessment score.

[0079] In one embodiment, a quality assessment score of a model instruction obtained based on an evaluation of a sample set is determined according to the accuracy of each task execution result in a task result set, including: obtaining a sample weight corresponding to each quality assessment sample in the sample set; determining the accuracy of each task execution result in the task result set; and determining a quality assessment score of the model instruction obtained based on an evaluation of the sample set according to the sample weight corresponding to each quality assessment sample in the sample set and the accuracy of each task execution result in the task result set.

[0080] For each sample set in the multiple sample sets, before determining the quality assessment score based on the sample weight corresponding to each quality assessment sample in the sample set, it is also necessary to determine the sample weight corresponding to each quality assessment sample in the sample set, and determine the accuracy of each task execution result in the task result set obtained based on the sample set. Afterwards, the quality assessment score obtained based on the evaluation of the sample set is determined based on the sample weight and accuracy. It is easy to understand that the sample weight of the quality assessment sample can be determined by a pre-trained logistic regression model. The accuracy of the task execution result can be determined by a third-party large model, or the accuracy of the task execution result can be determined manually.

[0081] In this embodiment, since the importance of different quality assessment samples is different, for example, in the positive and negative information recognition task, the importance of "an explosion is about to occur in a subway station" being correctly identified by the machine learning model should be higher than "smoking is very pleasant, I oppose smoking bans". Therefore, by setting sample weights, the purpose of giving priority to more important quality assessment samples in quality assessment can be achieved based on the sample weights, so that the high-quality model instructions subsequently screened out based on the quality assessment scores can trigger the machine learning model to correctly process samples similar to the important quality assessment samples.

[0082] The steps for determining the sample weights include: extracting a preset number of quality assessment samples from each sample set; adding sample weights to the extracted quality assessment samples; training the logistic regression model with the extracted quality assessment samples and the added sample weights to obtain a trained logistic regression model; and determining the sample weights corresponding to the unextracted quality assessment samples based on the trained logistic regression model.

[0083] Specifically, when it is necessary to determine the sample weights corresponding to the quality assessment samples in the sample set, the computer device can output the sample weights corresponding to the quality assessment samples through a logistic regression model. Before outputting the sample weights corresponding to the quality assessment samples through the logistic regression model, the logistic regression model needs to be trained. More specifically, after obtaining multiple sample sets, the computer device can extract a second preset number of quality assessment samples from each sample set. For the sake of convenience of description, the quality assessment samples extracted here will be referred to as training samples below. For example, b quality assessment samples can be extracted from sample set D1 as training samples, b quality assessment samples can be extracted from sample set D2 as training samples, and b quality assessment samples can be extracted from sample set D3 as training samples. Furthermore, sample weights can be added to each extracted training sample manually to use the added sample weights as labels. The computer device trains the logistic regression model using the training samples and the labels corresponding to the training samples to obtain a trained logistic regression model, and uses the trained logistic regression model to process the remaining quality assessment samples in the sample set that have not been added with sample weights, that is, the sample weights corresponding to the unextracted quality assessment samples are determined by the trained logistic regression model.

[0084] In one embodiment, referring to FIG6 , FIG6 shows a schematic diagram of determining sample weights in one embodiment. Quality assessment samples can be extracted from multiple sample sets and used as training samples. Weighted annotations are added to the training samples to obtain labeled samples. The remaining quality assessment samples are used as unlabeled samples. A logistic regression model is trained using the labeled samples to obtain a trained logistic regression model, and the unlabeled samples are labeled using the trained logistic regression model.

[0085] In the above embodiment, the logistic regression model is trained by extracting quality assessment samples. Compared with training the logistic regression model with the full amount of quality assessment samples, the number of training samples can be reduced, thereby reducing the amount of model calculation and saving resources such as electricity consumed during model calculation.

[0086] In one embodiment, the sample weights corresponding to the unextracted quality assessment samples are determined based on the trained logistic regression model, including: determining the sample features of the unextracted quality assessment samples to obtain the model parameter matrix of the trained logistic regression model, and fusing the model parameter matrix with the sample features to obtain the fused features corresponding to the unextracted quality assessment samples; and normalizing the fused features to obtain the sample weights corresponding to the unextracted quality assessment samples.

[0087] Specifically, when the trained logistic regression model is required to output the sample weights of unsampled quality assessment samples, the trained logistic regression model can determine the sample features of the unsampled quality assessment samples. Unsampled quality assessment samples refer to the sample weights of quality assessment samples that were not used as training samples. Sample features refer to information reflecting the characteristics of the quality assessment samples. For example, the unsampled quality assessment samples can be convolved using a convolutional network, and the resulting convolution result is the sample feature.

[0088] Furthermore, the trained logistic regression model can construct the model parameters into a model parameter matrix, and fuse the model parameter matrix with the sample features to obtain the fusion features corresponding to the unextracted quality assessment samples. For example, the formula h θ (e xj )=g(θ T e xj ), and obtain the fusion features, where e xj is the quality assessment sample d that was not extracted xj The corresponding sample features, θ is the model parameter matrix. Furthermore, the computer device normalizes the fused features to obtain the sample weights corresponding to the unextracted quality assessment samples.

[0089] In the above embodiment, by pre-training the logistic regression model, the quality assessment samples without added sample weights can be processed based on the trained logistic regression model to obtain accurate sample weights.

[0090] In one embodiment, the overall method for determining sample weights includes:

[0091] Step 1: Data preparation. Label the sample weights of an appropriate number of quality assessment samples. The labeled sample weights are used to train the regression model. For example, in the positive and negative information identification task (classified as "positive" / "neutral" / "negative"), the sample weight of "a dangerous event is about to occur at a subway station" identified as negative should be higher (for example, assigned a value of w = 0.95), while the sample weight of "smoking is fun, I hate the smoking ban" identified as "negative" should be lower than the former (for example, assigned a value of w = 0.75). Take U proportions of samples from the sample set D for labeling. Here, U is set to 0.1, that is, take U×|D| quality assessment samples as training samples for labeling.

[0092] Step 2: Train the Logistic Regression model based on the training samples and labeled sample weights from step 1. The Logistic Regression model formula is: θ (e xj )=g(θ T e xj ) z=hθ (e xj )

[0093] Among them, θ is all the parameters of the logistic regression model (that is, the model parameter matrix), e xj is the training sample d xj The corresponding eigenvector representation, d xj is the jth sample extracted from the Xth sample set. θ (e xj ) is the result of matrix operation between logistic regression parameter θ and eigenvector. xj (z) is a normalization function that normalizes the matrix operation result to a real number (floating point number) between [0,1]. This floating point number is the training sample d xj The corresponding sample weight w xj .

[0094] Step 3: Based on the logistic regression model trained in step 2, automatically label the remaining unlabeled quality assessment samples with weights, that is, obtain the sample weights of all quality assessment samples in all sample sets.

[0095] In one embodiment, the quality assessment score of the model instruction obtained based on the sample set evaluation is determined according to the sample weight corresponding to each quality assessment sample in the sample set and the accuracy of each task execution result in the task result set, including: for each task execution result in the task result set, determining the quality assessment sample in the sample set corresponding to the task execution result, and determining the sample score corresponding to the corresponding quality assessment sample according to the accuracy of the task execution result; determining the quality assessment score of the model instruction obtained based on the sample set evaluation according to the sample weight and sample score corresponding to each quality assessment sample in the sample set.

[0096] Specifically, since the task execution result is the result obtained after the machine learning model processes the model input sample, and the model input sample is generated by the quality assessment sample, there is a one-to-one correspondence between the task execution result and the quality assessment sample. Therefore, for each task execution result in the task result set, the quality assessment sample corresponding to each task execution result in the task result set can be determined based on this one-to-one correspondence. For example, referring to Figure 7, assuming that sample set A includes quality assessment sample A and quality assessment sample B, model input sample A is obtained based on quality assessment sample A, model input sample B is obtained based on quality assessment sample, task execution result A is obtained based on model input sample A, and task execution result B is obtained based on model input sample B, then the quality assessment sample corresponding to task execution result A is quality assessment sample A, and the quality assessment sample corresponding to task execution result B is quality assessment sample B.

[0097] Furthermore, after determining the quality assessment sample corresponding to each task execution result in the task result set, the computer device obtains a sample score corresponding to the corresponding quality assessment sample for each task execution result in the task result set based on the accuracy of the task execution result. For example, the sample score corresponding to quality assessment sample A is obtained based on the accuracy of task execution result A, and the sample score corresponding to quality assessment sample B is obtained based on the accuracy of task execution result B.

[0098] Furthermore, for each sample set in the plurality of sample sets, the computer device determines a quality assessment score for the model instruction on the sample set based on the sample weights and sample scores corresponding to the quality assessment samples in the sample set. FIG7 shows a schematic diagram of determining a sample score in one embodiment.

[0099] In one embodiment, obtaining the sample score corresponding to the corresponding quality assessment sample based on the accuracy of the task execution result includes: when the task execution result is correct, determining the sample score corresponding to the corresponding quality assessment sample as a first score; when the task execution result is wrong, determining the sample score corresponding to the corresponding quality assessment sample as a second score. The first score may be "1" and the second score may be "0". By setting the sample score to a larger score when the task execution result is correct and setting the sample score to a smaller score when the task execution result is wrong, the high-quality model instructions subsequently screened out based on the quality assessment score can trigger the machine learning model to output the correct result, thereby improving the accuracy of the model output result.

[0100] In one embodiment, a machine learning model can be pre-set to set the correct results of each model input sample, so that the computer device can determine the accuracy of the task execution result based on the set correct results. Alternatively, the task execution result can be displayed, and the accuracy of the task execution result can be judged manually. Alternatively, a task model can be pre-trained (for example, a task execution result correctness judgment layer is added to the above-mentioned machine learning model), and the correctness of the task execution result is determined by the task model. For example, taking the positive and negative information identification task as an example, for the model instruction "<sentence> is positive, neutral, or negative, please give the result", for the quality assessment sample "snowflakes are falling, the scenery is pleasant", the model instruction and the quality assessment sample are fused to construct "'snowflakes are falling, the scenery is pleasant' is positive, neutral, or negative, please give the result" as the model input sample. After the pre-trained task model processes the model input sample, it outputs the prediction result. If the task model outputs the string "correct", it is considered that the model instruction can guide the machine learning model to correctly process the quality assessment sample and obtain the correct result. The sample score corresponding to the quality assessment sample "snowflakes are falling, the scenery is pleasant" is 1. Otherwise, the sample score corresponding to the quality assessment sample "snowflakes are falling, the scenery is pleasant" is 0.

[0101] In the above embodiment, by converting the task execution result into the corresponding sample score, it is convenient to subsequently determine the quality assessment score based on the sample score.

[0102] In one embodiment, the quality assessment score of the model instruction on the sample set is determined based on the sample weight and sample score corresponding to each quality assessment sample in the sample set, including: adding the sample weights corresponding to each quality assessment sample in the sample set to obtain a first superposition weight; for each quality assessment sample in the sample set, multiplying the sample score corresponding to the quality assessment sample and the corresponding sample weight to obtain a first multiplication score corresponding to the quality assessment sample; adding the first multiplication scores corresponding to each quality assessment sample in the sample set to obtain a first superposition score; dividing the first superposition score by the first superposition weight to obtain a quality assessment score of the model instruction obtained based on the sample set evaluation.

[0103] Specifically, for each sample set in the multiple sample sets, the computer device adds the sample weights corresponding to each quality assessment sample in the sample set to obtain a first superposition weight. For example, if the sample set includes quality assessment sample A, quality assessment sample B, and quality assessment sample C, the sample weight corresponding to quality assessment sample A is 1, the sample weight corresponding to quality assessment sample B is 0.5, and the sample weight corresponding to quality assessment sample C is 0.3, then the computer device adds 1, 0.5, and 0.3 to obtain a first superposition weight of 1.8. Furthermore, since each quality assessment sample in the sample set has a corresponding sample weight and sample score, for each quality assessment sample in the sample set, the computer device multiplies the sample score corresponding to the quality assessment sample by the sample weight to obtain a first multiplied score corresponding to the quality assessment sample. For example, when the sample score of quality assessment sample A is 1, the sample score of quality assessment sample B is 1, and the sample score of quality assessment sample C is 0, then for quality assessment sample A, the corresponding first multiplication score is 1×1=1; for quality assessment sample B, the corresponding first multiplication score is 1×0.5=0.5; for quality assessment sample C, the corresponding first multiplication score is 0×0.3=0.

[0104] Furthermore, the computer device adds the first multiplied scores corresponding to each quality assessment sample in the sample set to obtain a first superposition score. For example, the computer device adds 1, 0.5, and 0 to obtain a first superposition score of 1.5. Furthermore, the computer device divides the first superposition score by the first superposition weight to obtain a quality assessment score for the model instruction on the sample set. For example, the computer device divides 1.5 by 1.8 to obtain a quality assessment score of 0.833.

[0105] In one embodiment, the computer device may determine the model instruction p by the following formula: i In the sample set D x The quality assessment score S on xi :

[0106] Among them, p i represents the i-th model instruction in the model instruction set; w xj That is, the quality assessment sample d xj The corresponding sample weight. xj is the jth quality assessment sample in the Xth sample set; score xij is the quality assessment sample d xj Corresponding sample score; |D j | is the sample set D x The total number of quality assessment samples in S xj∈[0,1], that is, S xj The value range is a real number between 0 and 1. The higher the quality evaluation score, the better the quality of the model instruction evaluated based on the target sample set, and vice versa.

[0107] In the above embodiment, since different quality assessment samples have different importance, by setting sample weights, the purpose of giving priority to more important quality assessment samples in quality assessment can be achieved based on the sample weights, so that the high-quality model instructions subsequently screened out based on the quality assessment scores can trigger the machine learning model to correctly process samples similar to the important quality assessment samples.

[0108] For example, for quality assessment sample A "A dangerous incident is about to occur at a certain subway station" and quality assessment sample B "Smoking is fun, I hate smoking bans", the sample weight of quality assessment sample A is higher than the sample weight of quality assessment sample B. Therefore, in the process of quality assessment of the model instruction "Is <sentence> positive, neutral, or negative? Please give the result" based on quality assessment samples A and B, more emphasis will be placed on the quality assessment result of the model instruction based on quality assessment sample A. The quality assessment result of quality assessment sample A on the model instruction refers to the sample score corresponding to quality assessment sample A mentioned above. The more accurate the emotion recognition result of the machine learning model for quality assessment sample A, the higher the sample score corresponding to quality assessment sample A will be, and the higher the sample score, the higher the quality assessment score will be.

[0109] Compared to quality assessment sample B, since quality assessment sample A has a higher sample weight than quality assessment sample B, the sample score corresponding to quality assessment sample A has a greater impact on the quality assessment score than the sample score corresponding to quality assessment sample B. Therefore, when the machine learning model's emotion recognition result for quality assessment sample A is more accurate than that for quality assessment sample B, the model instruction "Is <sentence> positive, neutral, or negative? Please provide a result" is more likely to be a high-quality model instruction. Consequently, when the model instruction "Is <sentence> positive, neutral, or negative? Please provide a result" is subsequently applied to actual production, the machine learning model's emotion recognition results for text similar to quality assessment sample A will be more accurate. For example, the emotion recognition results for security-related text will be more accurate.

[0110] In one embodiment, the comprehensive quality evaluation score of the model instruction is determined based on the quality evaluation score obtained by evaluating each sample set separately, including: obtaining the set weight corresponding to each of the multiple sample sets; and determining the comprehensive quality evaluation score of the model instruction based on the set weight corresponding to each of the multiple sample sets and the quality evaluation score evaluated based on each sample set.

[0111] Specifically, because sample sets have corresponding sample weights, the comprehensive quality assessment score of the model instructions can be determined based on the set weights corresponding to each of the multiple sample sets and the quality assessment scores obtained based on the evaluation of each sample set. For example, for each sample set, the set weight corresponding to the sample set and the quality assessment score obtained based on the evaluation of the sample set can be multiplied to obtain a third multiplied score, and the average of the third multiplied scores is used as the comprehensive quality assessment score.

[0112] In one embodiment, the quality assessment method of the above-mentioned model instructions also includes a step of determining the set weight, and the step of determining the set weight corresponding to each sample set in the multiple sample sets includes: determining the specified task that the machine learning model needs to perform according to the model instructions; determining the correlation between each sample set and the specified task; and determining the set weight corresponding to each evaluation sample set based on the correlation between each sample set and the specified task.

[0113] Specifically, the computer device determines the designated task that the machine learning model corresponding to the model instruction needs to perform. For example, if the model instruction is "Please judge the emotional polarity of <sentence> (positive / neutral / negative)", then the designated task that the machine learning model corresponding to the model instruction needs to perform is the positive and negative information identification task. Furthermore, the computer device determines the correlation between each sample set and the designated task, and determines the set weight corresponding to each sample set based on the correlation between each sample set and the designated task. The higher the correlation, the higher the set weight.

[0114] In one embodiment, a set weight R(T) = {r1, r2 ... rn}, rx ∈ [0, 1]} is assigned to the sample set D = {D1, D2 ... Dn}, i.e., the set weight ranges from 0 to 1. The larger the value of rx, the better the sample set Dx is able to measure or evaluate the quality of the model instructions.

[0115] In one embodiment, the set weight can be determined manually. For example, for a specified task T = "positive and negative information identification task", in the positive and negative information identification task (classifying text into "positive" / "neutral" / "negative"), there are three sample sets: "short video comment sample set" (D1) / "ancient poetry sample set" (D2) / "English writing sample set" (D3). For the application scenario of the specified task (the positive and negative information identification task is mainly used for real-time identification of harmful speech), the correlation between the three sample sets and the specified task should be D1>D2≈D3, so the set weight relationship should be r1>r2≈r3. Here, according to the correlation and matching degree between the specified task and the sample set, the set weight value can be manually adjusted (if a sample set Dx is manually judged to be close to the specified task, the set weight of the sample set Dx will be increased). For example, in the three sample sets in the above example, the set weight can be assigned to r1=0.9, r2=0.3, and r3=0.3.

[0116] The set weight corresponding to each sample set can also be determined using a set weight determination model. The set weight determination model is a pre-trained machine learning model for determining set weights. The output of the set weight determination model can be shown in FIG8 . FIG8 shows a schematic diagram of set weights in one embodiment.

[0117] In the above embodiment, the more relevant the sample set is to the designated task, the greater its importance is, and the better the sample set is able to evaluate the quality of the model instruction. Therefore, the set weight can be accurately determined based on the relevance to the designated task.

[0118] In one embodiment, determining the relevance between each evaluation sample set and the specified task includes: determining task description information of the specified task, the sample set identifier of each sample set, and the extracted samples in each sample set; generating construction instructions based on the task description information, the sample set identifier of each sample set, and the extracted samples in each sample set; and determining the relevance between each sample set and the specified task based on the construction instructions.

[0119] Specifically, since the correlation between a sample set and a specified task can be determined using a set weight determination model, when determining the correlation between a sample set and a specified task, the computer device can first determine the task description information for the specified task. Task description information refers to information used to describe the task, which may include task requirements. For example, the task description information T may be: "Positive and Negative Information Identification Task," where text is classified as "positive," "neutral," or "negative" in the positive and negative information identification task.

[0120] Furthermore, the computer device may obtain a sample set identifier for each sample set. A sample set identifier is information that uniquely identifies a sample set. For example, the sample set identifier may be the sample set ID. Furthermore, the computer device may extract a first preset number of quality assessment samples from each sample set. For ease of description, the extracted quality assessment samples are referred to as extracted samples below.

[0121] Furthermore, the computer device can generate construction instructions based on the task description information, the sample set identifiers of each sample set, and the extracted samples in each sample set, and trigger the set weight determination model output to determine the relevance between each sample set and the specified task.

[0122] In one embodiment, the computer device may determine the number of extracted samples based on the number of quality assessment samples in each sample set. For example, when the number of quality assessment samples in a sample set is greater, more extracted samples may be extracted. Alternatively, a sampling number may be pre-set, and the extracted samples may be extracted from the sample set based on the pre-set sampling number. For example, the sampling number may be a, and the computer device may extract a number of extracted samples from each sample set.

[0123] In one embodiment, the set weight determination model can be triggered based on the construction instruction to sort the sample sets based on the relevance to obtain a sample set sequence. For example, the set weight determination model can be triggered to sort the sample sets in order from high to low relevance, so that the first sample set in the sample set sequence has the highest relevance with the specified task, and the relevance between subsequent sample sets and the specified tasks decreases successively.

[0124] In the above embodiment, by generating construction instructions, the machine learning model can be triggered based on the construction instructions to automatically output the relevance between each sample set and the specified task, thereby improving the efficiency of determining the relevance.

[0125] In one embodiment, construction instructions are generated based on task description information, the sample set identifier of each sample set, and the extracted samples in each sample set, including: obtaining initial construction instructions; the initial construction instructions include an information filling index, a set filling index corresponding to each sample set, and an extracted sample filling index corresponding to each sample set; filling in the task description information to the position indicated by the information filling index, filling in the sample set identifier of each sample set to the position indicated by the corresponding set filling index, and filling in the sample identifier of each extracted sample to the position indicated by the corresponding extracted sample filling index, so as to obtain construction instructions.

[0126] Specifically, the computer device can obtain a pre-set initial construction instruction, which has an information filling index, a set filling index, and an extraction sample filling index. For example, the initial construction instruction may be: in the context of the <T> task, there are D1 (<sample set identifier 1>) / D2 (<sample set identifier 2>) / D3 (<sample set identifier 3>) used to evaluate the quality of the model instruction. It is necessary to give a correlation ranking between the sample set and the specified task. The extracted samples of D1 are (<extracted sample 1>), the extracted samples of D2 are (<extracted sample 2>), and the extracted samples of D3 are (<extracted sample 3>). Among them, <T> in the initial construction instruction is the information filling index; <sample set identifier 1>, <sample set identifier 2>, and <sample set identifier 3> in the initial construction instruction are the set filling index; <extracted sample 1>, <extracted sample 2>, and <extracted sample 3> in the initial construction instruction are the extraction sample filling index.

[0127] Furthermore, the computer device may fill in the task information at the location indicated by the information filling index, for example, replacing the task information with <T> in the initial construction instruction. The computer device may fill in the sample set identifier at the location indicated by the corresponding set filling index, for example, replacing <sample set identifier 1> in the initial construction instruction with the identifier of sample set D1, replacing <sample set identifier 2> in the initial construction instruction with the identifier of sample set D2, and replacing <sample set identifier 3> in the initial construction instruction with the identifier of sample set D3. The computer device fills in the sample identifier of the extracted sample at the position indicated by the extracted sample filling index. For example, when only one extracted sample is extracted from each sample set, the computer device replaces the identifier of the extracted sample L1 extracted from the sample set D1 with <Extracted Sample 1> in the initial construction instruction, replaces the identifier of the extracted sample L2 extracted from the sample set D2 with <Extracted Sample 2> in the initial construction instruction, and replaces the identifier of the extracted sample L3 extracted from the sample set D3 with <Extracted Sample 3> in the initial construction instruction. At this point, the construction instruction can be obtained.

[0128] In this embodiment, by setting the filling index in the initial construction instruction in advance, the computer device can be guided to accurately fill in the corresponding information in the corresponding position based on the filling index. In this way, accurate construction instructions can be obtained, and the accuracy of relevance judgment can be improved based on the accurately generated construction instructions.

[0129] In one embodiment, the set weight corresponding to each evaluation sample set is determined based on the relevance between each sample set and a specified task, including: determining the number of sample sets in a plurality of sample sets and determining a gear interval based on the number of sample sets; sorting the sample sets in the plurality of sample sets in descending order of relevance to obtain a sample set sequence; and determining the set weight corresponding to each sample set based on the gear interval and the order of the sample sets in the sample set sequence. The gear interval refers to the reciprocal of the number of sample sets in a sample set.

[0130] Specifically, the computer device can determine the number of sample sets obtained, that is, determine the number of sample sets contained in the multiple sample sets, and determine the gear interval based on the number of sample sets. For example, the gear interval g = 1 / n. When 3 sample sets are obtained, the gear interval g is 1 / 3. Wherein, n is the number of sample sets. The computer device sorts the sample sets in descending order of relevance to obtain a sample set sequence. It is easy to understand that what is recorded in the sample set sequence is actually the sample set identifier of each sample set. For example, the correlation between sample set D1 and the specified task is greater than the correlation between sample set D2 and the specified task, and the correlation between sample set D2 and the specified task is greater than the correlation between sample set D3 and the specified task. Then the sample set sequence is {D1, D2, D3}.

[0131] Furthermore, the computer device determines the set weight corresponding to each sample set based on the gear interval and the order of arrangement of each sample set in the sample set sequence. For example, the computer device may determine the method for generating the set weight based on the number of sample sets, and determine the set weight corresponding to each sample set based on the method for generating the set weight, the gear interval, and the order of arrangement of each sample set in the sample set sequence. For example, when the number of sample sets is three, the computer device automatically determines the set weight according to three gears.

[0132] For another example, a computer device traverses the sample set identifiers in a sample set sequence. For the currently traversed sample set identifier, the computer device determines the order of the currently traversed sample set identifier in the sample set sequence, obtains the corresponding initial set weight based on the order of the order, and adjusts the initial set weight based on the gear interval to obtain the set weight corresponding to the currently traversed sample set identifier. For example, when the currently traversed sample set identifier is D1, since D1 is the first identifier in the sample set sequence, the computer device determines that the initial set weight corresponding to D1 is n×g. When the gear interval g is 1 / 3, the set weight corresponding to D1 is 3×(1 / 3)=1. The computer device continues traversing and obtains a new currently traversed sample set identifier D2. The computer device determines that the initial set weight corresponding to D2 is (n-1)×g. When the gear interval g is 1 / 3, the set weight corresponding to D2 is (3-1)×(1 / 3)=2 / 3.

[0133] In one embodiment, a computer device stores a sequence of initial set weights corresponding to different numbers of sample sets, where the number of initial set weights in the initial set weight sequence is consistent with the number of sample sets. For example, when the number of sample sets is 3, the initial set weight sequence can be {n×g, (n-1)×g, (n-2)×g}, where n is the number of sample sets and g is the gear interval. The computer device matches the sample set identifiers in the sample set sequence with the initial set weights in the initial set weight sequence according to the order of the sample set identifiers in the sample set sequence and the order of the initial set weights in the initial set weight sequence. In this way, the initial set weight corresponding to each sample set identifier can be obtained. For example, the first sample set identifier can be matched with the first initial set weight, the second sample set identifier can be matched with the second initial set weight, and so on. The computer device can then adjust each initial set weight according to the specific gear interval to obtain the set weight corresponding to each sample set.

[0134] In the above embodiment, by setting the gear interval, the set weight can be controlled to be no greater than 1 based on the gear interval. By setting the gear interval and the initial set weight, the set weight related to the relevance can be obtained based on the gear interval and the initial set weight. By sorting the sample sets in order of relevance from high to low, the sample sets that are sorted earlier in the sample set sequence are the sets that are more relevant to the specified task, and then the sample sets with higher relevance to the specified task can have higher set weights based on the sample set sequence. Since the sample sets with higher relevance to the specified task can have higher set weights, and the sample sets with higher relevance to the specified task can be considered to be more important sample sets in quality assessment, the purpose of paying attention to more important sample sets in quality assessment can be achieved based on the set weights, so that the high-quality model instructions subsequently screened out based on the comprehensive quality assessment score can trigger the machine learning model to correctly process samples similar to the important sample sets, thereby improving the accuracy of the model output results. In one embodiment, referring to Figure 9, Figure 9 shows a schematic diagram of the overall determination of set weights in one embodiment:

[0135] Step 1: Sampling samples: Sample a from the sample set D = {D1, D2, D3} as the extracted samples of each sample set.

[0136] Step 2: Generate task description information. Describe the task requirements and task background of the specified task in words.

[0137] Step 3: Collect the sample set identifier of each sample set, such as the sample set name.

[0138] Step 4: Construct prompt instructions (i.e., construct instructions). Construct the information collected in steps 1, 2, and 3 into instructions to guide the model to give a relevance ranking.

[0139] Step 5: Output the results and process them. The number of sample sets is used as the number of levels. The weight of the first-place ranking is n×g, the weight of the second-place ranking is (n-1)×g, and so on. The set weight can be automatically calculated.

[0140] For example, if the model output is D1>D2>D3, the corresponding weight scores R(T) = {r1, r2, r3} can be automatically calculated in three levels, where n = 3 and the level interval g = 1 / 3. Then the set weight corresponding to D1 is r1 = 3 × g = 1, the set weight corresponding to D2 is r2 = (3-1) × g = 2 / 3, and the set weight corresponding to D3 is r3 = (3-2) × g = 1 / 3.

[0141] In one embodiment, a comprehensive quality assessment score of a model instruction is determined based on the set weights corresponding to each of the multiple sample sets and the quality assessment scores of the model instructions on each sample set, including: adding the set weights corresponding to each sample set to obtain a second superposition weight; for each sample set in the multiple sample sets, multiplying the quality assessment score corresponding to the sample set and the corresponding set weight to obtain a second multiplied score corresponding to the sample set; adding the second multiplied scores corresponding to each sample set to obtain a second superposition score; and dividing the second superposition score by the second superposition weight to obtain a comprehensive quality assessment score of the model instruction.

[0142] Specifically, the computer device superimposes the respective set weights in each sample set, that is, adds the set weights to obtain a second superimposed weight. For example, for sample sets A, sample sets B, and sample sets C, the set weight corresponding to sample set A is 1, the sample weight corresponding to sample set B is 0.5, and the sample weight corresponding to sample set C is 0.3. The computer device then adds 1, 0.5, and 0.3 to obtain a second superimposed weight of 1.8. Furthermore, since each sample set has a corresponding set weight and quality assessment score, for each sample set, the computer device multiplies the set weight corresponding to the sample set and the quality assessment score to obtain a second multiplied score corresponding to the sample set. For example, when the quality assessment score of sample set A is 0.6, the quality assessment score of sample set B is 0.3, and the quality assessment score of sample set C is 0.2, then for sample set A, the corresponding second multiplication score is 1×0.6=0.6; for sample set B, the corresponding second multiplication score is 0.5×0.3=0.15; for sample set C, the corresponding second multiplication score is 0.3×0.2=0.06.

[0143] Furthermore, the computer device superimposes the second multiplied scores corresponding to each sample set, that is, adds the second multiplied scores to obtain a second superimposed score. For example, the computer device adds 0.6, 0.15, and 0.06 to obtain a second superimposed score of 0.81. Furthermore, the computer device divides the second superimposed scores by the second superimposed weight to obtain a comprehensive quality assessment score. For example, the computer device divides 0.81 by 1.8 to obtain a comprehensive quality assessment score of 0.45.

[0144] In one embodiment, the computer device may determine the model instruction p by the following formula: i Comprehensive quality evaluation score S on the full sample set i (T):

[0145] Among them, p i Represents the i-th model instruction in the model instruction set; rx That is, the sample set D x The corresponding set weight, D x Represents the Xth sample set. S xi is the quality evaluation score of the i-th model instruction on the X-th sample set; |n| is the total number of sample sets; S i (T)∈[0,1], that is, S i (T) is a real number ranging from 0 to 1. A higher comprehensive quality evaluation score indicates better quality of the model instructions evaluated based on the full sample set, and vice versa.

[0146] In the above embodiment, since different sample sets have different importances, by setting set weights, the purpose of giving priority to more important sample sets in quality assessment can be achieved based on the set weights, so that the high-quality model instructions subsequently screened out based on the comprehensive quality assessment scores can trigger the machine learning model to correctly process samples similar to the important sample sets.

[0147] For example, for sample set A, the "short video comment dataset," and sample set B, the "ancient poetry dataset," since sample set A has a higher correlation with the positive and negative information recognition task, the set weight of sample set A is higher than the set weight of sample set B. Therefore, in the process of quality evaluation of the model instruction "Is <sentence> positive, neutral, or negative? Please give the result" based on sample sets A and B, more emphasis will be placed on the quality evaluation result of the model instruction based on sample set A. Among them, the quality evaluation result of the model instruction based on sample set A refers to the above-mentioned quality evaluation score corresponding to sample set A. The more accurate the recognition result of the machine learning model for positive and negative information recognition of sample set A, the higher the quality evaluation score corresponding to sample set A will be, and the higher the quality evaluation score, the higher the overall quality evaluation score will be.

[0148] Compared to sample set B, since sample set A has a higher set weight than sample set B, the quality assessment score corresponding to sample set A has a greater impact on the overall quality assessment score than the quality assessment score corresponding to sample set B. Therefore, compared to sample set B, the more accurate the machine learning model's recognition results for positive and negative information identification in sample set A, the more likely the model instruction "Is <sentence> positive, neutral, or negative? Please provide a result" is to be a high-quality model instruction. Consequently, when the model instruction "Is <sentence> positive, neutral, or negative? Please provide a result" is applied in actual production, the machine learning model's sentiment recognition results for text similar to sample set A will be more accurate. For example, the recognition results for positive and negative information identification in text such as real-time comments will be more accurate.

[0149] In one embodiment, the above method also includes: obtaining a comprehensive quality assessment score of each of the multiple model instructions; the comprehensive quality assessment score of each model instruction is determined by the quality assessment method of the above model instruction; based on the comprehensive quality assessment scores of the multiple model instructions, high-quality model instructions are screened out from the multiple model instructions; and the machine learning model is triggered by the high-quality model instructions to process the task.

[0150] Specifically, there may be multiple model instructions corresponding to a specified task. Therefore, for each model instruction, the computer device can obtain the comprehensive quality evaluation score corresponding to each model instruction according to the quality evaluation method of the above-mentioned model instructions. For example, for the positive and negative information identification task, there may be model instruction A "Please judge the emotional polarity (positive / neutral / negative) of the sentence <sentence>", model instruction B "Is <sentence> positive, neutral, or negative? Please give the result", model instruction C "Please perform the following task. The task content is to divide the sentence into three categories: positive emotion / neutral emotion / negative emotion. Please directly output the analysis result. The target sentence is: <sentence>", model instruction D "According to the emotional polarity, we define the sentence emotion as positive, neutral, and negative. For the sentence <sentence>, please give the emotional polarity result". Afterwards, the computer device can obtain the comprehensive quality evaluation score A of model instruction A, the comprehensive quality evaluation score B of model instruction B, and the comprehensive quality evaluation score C of model instruction C according to the quality evaluation method of the above-mentioned model instructions.

[0151] Furthermore, the computer device can filter out high-quality model instructions from multiple model instructions based on the comprehensive quality assessment score. In actual applications, the high-quality model instructions can be used to trigger the machine learning model to perform the corresponding task. For example, model instructions with a comprehensive quality score higher than a preset score threshold can be used as high-quality model instructions; or the model instructions can be sorted from high to low according to the comprehensive quality score to obtain a model instruction sequence, and the first N model instructions in the model instruction sequence can be used as high-quality model instructions. The high-quality model instructions can then be used to trigger the machine learning model to identify positive and negative information in other texts.

[0152] In this embodiment, by screening out high-quality model instructions based on the comprehensive quality assessment score, the machine learning model can be triggered to output more accurate results based on the high-quality model instructions, thereby improving the accuracy of the model output results.

[0153] In one embodiment, referring to FIG10 , FIG10 illustrates a schematic diagram of generating multiple comprehensive quality scores in one embodiment. A computer device may perform quality assessment on multiple model instructions using multiple sample sets, thereby obtaining a quality assessment score matrix, and then obtain a comprehensive quality assessment score corresponding to each model instruction based on the quality assessment score matrix. Where Sij represents the quality assessment score of model instruction j on sample set j, and Kj represents the comprehensive quality assessment score of model instruction j.

[0154] In one embodiment, referring to FIG11, for multiple model instructions of a machine learning model, they can be graded according to the comprehensive quality scores corresponding to each model instruction, and a high-quality model instruction set, an ordinary quality model instruction set, and a low-quality model quality set can be obtained. The high-quality model instruction set is used to trigger the machine learning model to perform the corresponding task in the task pipeline to obtain the task execution result, and the task execution result is post-processed to obtain the processing result, and the processing result is applied to the production application. For example, the machine learning model can be a large model for identifying positive and negative information. In actual production applications, the large model can be used to analyze the positive and negative information of public real-time comments in the application. When the high-quality model instruction is obtained, in actual production applications, the public real-time comments of the application can be obtained in real time, and the large model can be triggered by the high-quality model instruction to identify the positive and negative information of the public real-time comments to obtain the positive and negative information identification results. Figure 11 shows a schematic diagram of the application of high-quality model instructions in one embodiment.

[0155] In one embodiment, referring to FIG12 , FIG12 shows a flow chart of a method for evaluating the quality of model instructions in a specific embodiment. The method for evaluating the quality of model instructions includes:

[0156] In step 1202 , the computer device obtains a model instruction and a plurality of sample sets.

[0157] In step 1204, the computer device determines the evaluation sample filling position in the model instruction; for each sample set in the multiple sample sets, for each quality evaluation sample in the targeted sample set, the targeted quality evaluation sample is filled in the evaluation sample filling position in the model instruction to obtain the model input sample corresponding to the targeted quality evaluation sample, and the model input samples corresponding to each quality evaluation sample in the targeted sample set are combined to obtain the model input set corresponding to the targeted sample set.

[0158] In step 1206, the computer device obtains a machine learning model corresponding to the model instruction, and for each model input sample in the model input set corresponding to the targeted sample set, inputs the targeted model input sample into the machine learning model to trigger the machine learning model to output a task execution result corresponding to the targeted model input sample; and synthesizes the task execution results corresponding to each model input sample in the model input set corresponding to the targeted sample set to obtain a task result set corresponding to the targeted sample set.

[0159] In step 1208, the computer device obtains the sample weight corresponding to each quality assessment sample in the targeted sample set; superimposes the sample weight corresponding to each quality assessment sample in the targeted sample set to obtain a first superimposed weight; and for each quality assessment sample in the targeted sample set, multiplies the sample score corresponding to the targeted quality assessment sample and the corresponding sample weight to obtain a first multiplied score corresponding to the targeted quality assessment sample.

[0160] In step 1210, the computer device adds the first multiplication scores corresponding to each quality assessment sample in the targeted sample set to obtain a first superposition score; and divides the first superposition score by the first superposition weight to obtain a quality assessment score of the model instruction on the targeted sample set.

[0161] In step 1212, the computer device obtains the set weights corresponding to each of the multiple sample sets; adds the set weights corresponding to each sample set to obtain a second superposition weight; and for each sample set in the multiple sample sets, multiplies the quality assessment score corresponding to the sample set by the corresponding set weight to obtain a second multiplied score corresponding to the sample set.

[0162] In step 1214, the computer device superimposes the second multiplied scores corresponding to each sample set to obtain a second superimposed score; and divides the second superimposed score by the second superimposed weight to obtain a comprehensive quality assessment score of the model instruction.

[0163] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0164] This application also provides an application scenario, which applies the above-mentioned model instruction quality assessment method. Specifically, the application of the model instruction quality assessment method in this application scenario is as follows:

[0165] When the large model is a palmprint recognition model, the computer device can obtain multiple model instructions and multiple sample sets, such as an animal palmprint set obtained by capturing animal palmprint images, and a human palmprint set obtained by capturing human palmprint images. Furthermore, the computer device uses the aforementioned method to filter high-quality model instructions from the multiple model instructions based on the multiple sample sets. These high-quality model instructions trigger the palmprint recognition model to recognize palmprint images in production applications.

[0166] This application also provides an application scenario, which applies the above-mentioned model instruction quality assessment method. Specifically, the application of the model instruction quality assessment method in this application scenario is as follows:

[0167] When the large model is a conversation generation model, the computer device may obtain multiple model instructions and multiple sample sets, such as multiple conversation sets. Furthermore, the computer device uses the aforementioned method to filter high-quality model instructions from the multiple model instructions based on the multiple sample sets. These high-quality model instructions trigger the conversation generation model to automatically engage in conversation with the user in a production application.

[0168] The above application scenarios are merely illustrative. It is understood that the application of the quality assessment methods for model instructions provided in the various embodiments of this application is not limited to the above scenarios. For example, the above quality assessment methods for model instructions can also be applied to fields such as health, finance, scientific research, law, and education.

[0169] The beneficial effects of the quality assessment method of the above-mentioned type instruction include:

[0170] 1. Within a given task context, a suitable set of model instructions is constructed for existing large models. Currently, there is no method for automatically evaluating the quality of model instructions and selecting high-quality model instructions. This application can automatically evaluate the quality of model instructions and automatically select high-quality model instructions.

[0171] 2. The quality of model instructions significantly impacts the ultimate performance of machine learning models. For example, designing instructions for large models to perform tasks like positive and negative information identification or sentiment analysis currently relies primarily on manual experience. This approach inevitably introduces uncertainty in task performance. However, this application, by screening for high-quality model instructions, can guide large models to output desired results based on these high-quality model instructions.

[0172] Based on the same inventive concept, an embodiment of the present application further provides a model instruction quality assessment device for implementing the aforementioned model instruction quality assessment method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the embodiments of the model instruction quality assessment device provided below can be found in the above-mentioned limitations of the model instruction quality assessment method, and will not be repeated here.

[0173] In one embodiment, as shown in FIG13 , a quality assessment apparatus 1300 for model instructions is provided, comprising: a model input set generation module 1302 , a quality assessment score determination module 1304 , and a comprehensive quality assessment score determination module 1306 , wherein:

[0174] The model input set generation module 1302 is configured to obtain a model instruction and multiple sample sets; for each sample set in the multiple sample sets, the model instruction is respectively integrated with each quality assessment sample in the sample set to obtain a model input set corresponding to the sample set;

[0175] The quality assessment score determination module 1304 is configured to obtain a machine learning model corresponding to the model instruction, trigger the machine learning model to perform a specified task based on a model input set corresponding to the sample set, and obtain a task result set; the specified task is the task that the machine learning model is specified to perform by the model instruction; and determine a quality assessment score for the model instruction obtained based on the sample set evaluation based on the accuracy of each task execution result in the task result set;

[0176] The comprehensive quality evaluation score determination module 1306 is configured to determine the comprehensive quality evaluation score of the model instruction according to the quality evaluation scores obtained based on each sample set.

[0177] In the above embodiment, by obtaining a model instruction to be evaluated and multiple sample sets, the model instruction can be fused with each quality assessment sample in each sample set to obtain a model input set corresponding to each sample set. By obtaining the model input set, for each sample set in the multiple sample sets, a task result set corresponding to the sample set can be output based on the machine learning model. A quality assessment score based on the accuracy of the task execution results in the task result set can be obtained. By obtaining the quality assessment scores evaluated for each sample set, a comprehensive quality assessment score for the model instruction can be obtained based on each quality assessment score. Because the comprehensive quality assessment score can be automatically obtained, the present application can significantly improve the efficiency of quality assessment compared to traditional manual quality assessment of model instructions. Furthermore, the present application can be widely applied in the actual production of large model instruction projects, by finding high-quality model instructions to trigger large models to output more accurate results. Furthermore, because the comprehensive quality assessment score of the model instruction is determined based on multiple sample sets, compared to determining the comprehensive quality assessment score based on only a single sample set, the probability of inaccurate quality assessment due to too few sample sets can be reduced, thereby improving the accuracy of the model instruction quality assessment.

[0178] In one embodiment, the model input set includes a model input sample corresponding to each quality assessment sample in the sample set; the model input set generation module 1302 is also used to determine the evaluation sample filling position in the model instruction based on a preset keyword; for each quality assessment sample in the sample set, the quality assessment sample is filled in the evaluation sample filling position in the model instruction to obtain a model input sample corresponding to the quality assessment sample.

[0179] In one embodiment, the task result set includes the task execution results corresponding to each model input sample in the model input set; the quality assessment score determination module 1304 is also used to input the model input sample into the machine learning model for each model input sample in the model input set corresponding to the sample set, so as to trigger the machine learning model to output the task execution result corresponding to the model input sample.

[0180] In one embodiment, the quality assessment score determination module 1304 is also used to obtain the sample weight corresponding to each quality assessment sample in the sample set; determine the accuracy of each task execution result in the task result set; and determine the quality assessment score of the model instruction on the sample set based on the sample weight corresponding to each quality assessment sample in the sample set and the accuracy of each task execution result in the task result set.

[0181] In one embodiment, the quality assessment device 1300 of the model instruction also includes a sample weight generation module, which is used to extract a preset number of quality assessment samples from each sample set; add sample weights to the extracted quality assessment samples; train the logistic regression model through the extracted quality assessment samples and the added sample weights to obtain a trained logistic regression model; and determine the sample weights corresponding to the unextracted quality assessment samples based on the trained logistic regression model.

[0182] In one embodiment, the quality assessment device 1300 of the model instruction also includes a sample weight generation module for determining the sample characteristics of the unextracted quality assessment samples; obtaining the model parameter matrix of the trained logistic regression model, and fusing the model parameter matrix with the sample characteristics to obtain the fused characteristics corresponding to the unextracted quality assessment samples; normalizing the fused characteristics to obtain the sample weights corresponding to the unextracted quality assessment samples.

[0183] In one embodiment, the quality assessment score determination module 1304 is also used to determine, for each task execution result in the task result set, a quality assessment sample in the sample set corresponding to the task execution result, and determine the sample score corresponding to the corresponding quality assessment sample based on the accuracy of the task execution result; and determine the quality assessment score of the model instruction obtained based on the sample set evaluation based on the sample weight and sample score corresponding to each quality assessment sample in the sample set.

[0184] In one embodiment, the quality assessment score determination module 1304 is also used to determine the sample score corresponding to the corresponding quality assessment sample as a first score when the task execution result is correct; and to determine the sample score corresponding to the corresponding quality assessment sample as a second score when the task execution result is incorrect; the first score is greater than the second score.

[0185] In one embodiment, the quality assessment score determination module 1304 is further used to add the sample weights corresponding to each quality assessment sample in the sample set to obtain a first superposition weight; for each quality assessment sample in the sample set, multiply the sample score corresponding to the quality assessment sample and the corresponding sample weight to obtain a first multiplied score corresponding to the quality assessment sample; add the first multiplied scores corresponding to each quality assessment sample in the sample set to obtain a first superposition score; divide the first superposition score by the first superposition weight to obtain a quality assessment score of the model instruction obtained based on the sample set evaluation.

[0186] In one embodiment, the comprehensive quality assessment score determination module 1306 is further used to obtain the set weights corresponding to each of the multiple sample sets; and determine the comprehensive quality assessment score of the model instruction based on the set weights corresponding to each of the multiple sample sets and the quality assessment scores obtained based on the evaluation of each sample set.

[0187] In one embodiment, the quality assessment device 1300 of the model instruction also includes a set weight generation module, which is used to determine the specified tasks that the machine learning model needs to perform based on the model instructions; determine the correlation between each sample set and the specified task; and determine the set weight corresponding to each evaluation sample set based on the correlation between each sample set and the specified task.

[0188] In one embodiment, the quality assessment device 1300 of the model instruction also includes a set weight generation module, which is used to determine the task description information of the specified task, the sample set identifier of each sample set, and the extracted samples in each sample set; generate construction instructions based on the task description information, the sample set identifier of each sample set, and the extracted samples in each sample set; and determine the relevance between each sample set and the specified task based on the construction instructions.

[0189] In one embodiment, the quality assessment device 1300 of the model instruction also includes a set weight generation module for obtaining an initial construction instruction; the initial construction instruction includes an information filling index, a set filling index corresponding to each sample set, and an extracted sample filling index corresponding to each sample set; the task description information is filled in the position indicated by the information filling index, the sample set identifier of each sample set is filled in the position indicated by the corresponding set filling index, and the sample identifier of each extracted sample is filled in the position indicated by the corresponding extracted sample filling index to obtain the construction instruction.

[0190] In one embodiment, the quality assessment device 1300 of the model instruction further includes a set weight generation module, which is used to determine the number of sample sets in a plurality of sample sets, and determine the gear interval based on the number; sort the sample sets in the plurality of sample sets in descending order of relevance to obtain a sample set sequence; and determine the set weight corresponding to each sample set based on the gear interval and the arrangement order of each sample set in the sample set sequence.

[0191] In one embodiment, the comprehensive quality assessment score determination module 1306 is further used to add the set weights corresponding to each sample set to obtain a second superposition weight; for each sample set in multiple sample sets, multiply the quality assessment score corresponding to the sample set and the corresponding set weight to obtain a second multiplied score corresponding to the sample set; add the second multiplied scores corresponding to each sample set to obtain a second superposition score; divide the second superposition score by the second superposition weight to obtain a comprehensive quality assessment score of the model instruction.

[0192] In one embodiment, the model instruction quality assessment device 1300 also includes a high-quality instruction determination module, which is used to obtain the comprehensive quality assessment scores of multiple model instructions; screen out high-quality model instructions from multiple model instructions based on the comprehensive quality assessment scores of multiple model instructions; and trigger the machine learning model to process the task through the high-quality model instructions.

[0193] Each module in the above-mentioned model instruction quality assessment device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0194] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 14. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a quality assessment method for a model instruction is implemented.

[0195] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in Figure 15. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication, and the wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a method for evaluating the quality of model instructions. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0196] Those skilled in the art will understand that the structures shown in Figures 14 and 15 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

[0197] In one embodiment, a computer device is further provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps in the above-mentioned method embodiments when executing the computer-readable instructions.

[0198] In one embodiment, a computer-readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0199] In one embodiment, a computer program product or computer program is provided, comprising computer-readable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-readable instructions from the computer-readable storage medium and executes the computer-readable instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0201] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0202] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0203] The embodiments described above only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they should not be understood as limiting the scope of the patent of this application. It should be pointed out that for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be based on the attached claims. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain the present application and are not intended to limit the present application.

Claims

1. A method for evaluating the quality of a model instruction, performed by a computer device, the method comprising: Get model instructions and multiple sample sets; For each sample set in the multiple sample sets, fusing the model instruction with each quality assessment sample in the sample set to obtain a model input set corresponding to the sample set; Obtaining a machine learning model corresponding to the model instruction, triggering the machine learning model to perform a specified task based on a model input set corresponding to the sample set, and obtaining a task result set; the specified task is the task that the machine learning model is specified to perform by the model instruction; Determining a quality assessment score of the model instruction obtained by evaluating the sample set according to the accuracy of each task execution result in the task result set; The comprehensive quality evaluation score of the model instruction is determined according to the quality evaluation score obtained by evaluating each sample set respectively.

2. The method according to claim 1, wherein the model input set includes a model input sample corresponding to each quality assessment sample in the sample set; The fusing the model instruction with each quality assessment sample in the sample set to obtain a model input set corresponding to the sample set includes: Determine the location for filling in the evaluation sample in the model instruction according to the preset keywords; For each quality assessment sample in the sample set, the quality assessment sample is filled into the assessment sample filling position in the model instruction to obtain a model input sample corresponding to the quality assessment sample.

3. The method according to claim 1, wherein the task result set includes the task execution result corresponding to each model input sample in the model input set; The triggering of the machine learning model to perform a specified task based on the model input set corresponding to the sample set to obtain a task result set includes: For each model input sample in the model input set corresponding to the sample set, the model input sample is input into the machine learning model to trigger the machine learning model to output a task execution result corresponding to the model input sample.

4. The method according to claim 1, wherein determining the quality assessment score of the model instruction evaluated based on the sample set according to the accuracy of each task execution result in the task result set comprises: Obtaining a sample weight corresponding to each quality assessment sample in the sample set; Determining the accuracy of each task execution result in the task result set; The quality assessment score of the model instruction on the sample set is determined according to the sample weight corresponding to each quality assessment sample in the sample set and the accuracy of each task execution result in the task result set.

5. The method according to claim 4, further comprising: Extracting a preset number of quality assessment samples from each of the sample sets; Add sample weights to the extracted quality assessment samples; The logistic regression model is trained by extracting quality assessment samples and adding sample weights to obtain a trained logistic regression model; The sample weights corresponding to the unsampled quality assessment samples are determined based on the trained logistic regression model.

6. The method according to claim 5, wherein determining the sample weights corresponding to the unsampled quality assessment samples based on the trained logistic regression model comprises: Determine sample characteristics of the undrawn quality assessment sample; Obtaining a model parameter matrix of a trained logistic regression model, and fusing the model parameter matrix with the sample features to obtain fused features corresponding to unextracted quality assessment samples; The fusion features are normalized to obtain sample weights corresponding to unextracted quality assessment samples.

7. The method according to claim 4, wherein determining the quality assessment score of the model instruction obtained based on the sample set evaluation according to the sample weight corresponding to each quality assessment sample in the sample set and the accuracy of each task execution result in the task result set comprises: For each task execution result in the task result set, determining a quality assessment sample in the sample set that corresponds to the task execution result, and determining a sample score corresponding to the corresponding quality assessment sample based on the accuracy of the task execution result; A quality evaluation score of the model instruction obtained based on the sample set evaluation is determined according to the sample weight and the sample score corresponding to each quality evaluation sample in the sample set.

8. The method according to claim 7, wherein determining a sample score corresponding to a corresponding quality assessment sample based on the accuracy of the task execution result comprises: If the task execution result is correct, determining a sample score corresponding to the corresponding quality assessment sample as a first score; In the case where the task execution result is erroneous, a sample score corresponding to the corresponding quality assessment sample is determined to be a second score; and the first score is greater than the second score.

9. The method according to claim 7, wherein determining the quality assessment score of the model instruction obtained based on the sample set evaluation according to the sample weight and sample score corresponding to each quality assessment sample in the sample set comprises: Adding the sample weights corresponding to each quality assessment sample in the sample set to obtain a first superposition weight; For each quality assessment sample in the sample set, multiply the sample score corresponding to the quality assessment sample by the corresponding sample weight to obtain a first multiplied score corresponding to the quality assessment sample; Adding the first multiplication scores corresponding to each quality assessment sample in the sample set to obtain a first superposition score; The first superposition score is divided by the first superposition weight to obtain a quality evaluation score of the model instruction obtained based on the sample set evaluation.

10. The method according to claim 1, wherein determining the comprehensive quality assessment score of the model instruction based on the quality assessment score obtained by evaluating each sample set separately comprises: Obtaining set weights corresponding to each of the multiple sample sets; The comprehensive quality evaluation score of the model instruction is determined according to the set weights corresponding to the multiple sample sets and the quality evaluation score obtained by evaluating each of the sample sets.

11. The method according to claim 10, further comprising: Determining, based on the model instructions, a specified task that the machine learning model needs to perform; Determining the relevance between each of the sample sets and the designated task; According to the relevance between each sample set and the designated task, a set weight corresponding to each sample set is determined.

12. The method according to claim 11, wherein determining the relevance between each of the sample sets and the designated task comprises: Determining task description information of the designated task, a sample set identifier of each sample set, and extracted samples from each sample set; generating a construction instruction according to the task description information, the sample set identifier of each sample set, and the extracted samples in each sample set; The relevance between each of the sample sets and the designated task is determined based on the construction instruction.

13. The method according to claim 12, wherein generating a construction instruction according to the task description information, the sample set identifier of each sample set, and the extracted samples in each sample set comprises: Get initial construction instructions; The initial construction instruction includes an information filling index, a set filling index corresponding to each of the sample sets, and an extracted sample filling index corresponding to each of the sample sets; Fill in the task description information at the position indicated by the information filling index, fill in the sample set identifier of each sample set at the position indicated by the corresponding set filling index, and fill in the sample identifier of each extracted sample at the position indicated by the corresponding extracted sample filling index to obtain the construction instruction.

14. The method according to claim 11, wherein determining the set weight corresponding to each evaluation sample set according to the relevance between each sample set and the designated task comprises: determining the number of sample sets in the plurality of sample sets, and determining a gear interval based on the number; Sorting the sample sets in the plurality of sample sets in descending order of relevance to obtain a sample set sequence; The set weight corresponding to each of the sample sets is determined according to the gear interval and the arrangement order of each of the sample sets in the sample set sequence.

15. The method according to claim 10, wherein determining the comprehensive quality assessment score of the model instruction based on the set weights corresponding to the plurality of sample sets and the quality assessment score of the model instruction on each of the sample sets comprises: Adding the set weights corresponding to each of the sample sets to obtain a second superposition weight; For each sample set in the multiple sample sets, multiplying the quality assessment score corresponding to the sample set by the corresponding set weight to obtain a second multiplied score corresponding to the sample set; Adding the second multiplication scores corresponding to each of the sample sets to obtain a second superposition score; The second superposition score is divided by the second superposition weight to obtain a comprehensive quality assessment score of the model instruction.

16. The method according to any one of claims 1 to 15, further comprising: Obtaining a comprehensive quality assessment score for each of a plurality of model instructions; the comprehensive quality assessment score for each of the model instructions is determined by any one of the methods of claims 1 to 15; screening out high-quality model instructions from the multiple model instructions according to the comprehensive quality evaluation scores of the multiple model instructions; The high-quality model instructions trigger the machine learning model to process the task.

17. A device for evaluating the quality of a model instruction, the device comprising: Model input set generation module, used to obtain model instructions and multiple sample sets; For each sample set in the multiple sample sets, fusing the model instruction with each quality assessment sample in the sample set to obtain a model input set corresponding to the sample set; a quality assessment score determination module, configured to obtain a machine learning model corresponding to the model instruction, trigger the machine learning model to perform the specified task based on a model input set corresponding to the sample set, and obtain a task result set; the specified task is the task that the machine learning model is specified to perform by the model instruction; Determining a quality assessment score of the model instruction obtained by evaluating the sample set according to the accuracy of each task execution result in the task result set; The comprehensive quality evaluation score determination module is used to determine the comprehensive quality evaluation score of the model instruction according to the quality evaluation score obtained by evaluating each sample set respectively.

18. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method according to any one of claims 1 to 16 when executing the computer-readable instructions.

19. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising computer-readable instructions, characterized in that When the computer readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images