Large model scene-oriented sample processing method, apparatus and device, and storage medium

Abstract

The invention provides a large model scene-oriented sample processing method, and relates to the technical field of artificial intelligence, in particular to the technical fields of large models, deep learning technologies and big data. According to the specific implementation scheme, sample evaluation results of multiple to-be-processed sample texts are determined according to multiple model output results of the multiple to-be-processed sample texts, and the multiple model output results of the to-be-processed sample texts are output by multiple deep learning models with different parameter quantities; determining at least one feature tag of each of the plurality of to-be-processed sample texts; and determining a feature evaluation value of the feature tag according to respective sample evaluation results of the plurality of to-be-processed sample texts with the same feature tag. The invention further provides a large model scene-oriented sample processing device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of large models, deep learning technology and big data technology, and can be applied to image processing scenarios. More specifically, the present disclosure provides a sample processing method, device, electronic device and storage medium for large model scenarios. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of large models are constantly increasing. As the number of parameters of large models increases, the generalization ability of large models is constantly improving. Summary of the invention

[0003] The present disclosure provides a sample processing method, apparatus, device and storage medium for large model scenarios.

[0004] According to one aspect of the present disclosure, a sample processing method for a large model scenario is provided, the method comprising: determining a sample evaluation result of each of a plurality of sample texts to be processed according to a plurality of model output results of each of a plurality of sample texts to be processed, wherein the plurality of model output results of the sample texts to be processed are respectively output by a plurality of deep learning models with different parameter amounts; determining at least one feature label of each of the plurality of sample texts to be processed; and determining a feature evaluation value of the feature label according to the sample evaluation results of each of the plurality of sample texts to be processed having the same feature label.

[0005] According to another aspect of the present disclosure, a sample processing device for a large model scenario is provided, the device comprising: a first determination module, used to determine a sample evaluation result of each of a plurality of sample texts to be processed according to a plurality of model output results of each of the plurality of sample texts to be processed, wherein the plurality of model output results of the sample texts to be processed are respectively output by a plurality of deep learning models with different parameter amounts; a second determination module, used to determine at least one feature label of each of the plurality of sample texts to be processed; and a third determination module, used to determine a feature evaluation value of a feature label according to a sample evaluation result of each of the plurality of sample texts to be processed having the same feature label.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided according to the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a sample processing method and apparatus for a large model scenario can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic diagram of a sample processing method for a large model scene according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of a sample processing method for a large model scene according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of multiple feature tags according to an embodiment of the present disclosure;

[0015] Figure 5 is a flow chart of a model determination method according to an embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram of a model determination method according to an embodiment of the present disclosure;

[0017] Figure 7 is a block diagram of a sample processing device for a large model scene according to an embodiment of the present disclosure; and

[0018] Figure 8 It is a block diagram of an electronic device to which a sample processing method for a large model scene can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] In the technical solution disclosed in this disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0022] Flagship large models with hundreds of billions of parameters have strong generalization capabilities and a wide range of task adaptability. However, the training and deployment of large models with hundreds of billions of parameters often require massive amounts of data, huge computing resources, and long training cycles. As a result, the application cost of large models with hundreds of billions of parameters is high and difficult to implement. Compared with large models with hundreds of billions of parameters, models with billions or tens of billions of parameters require fewer computing resources and sample data, and have lower application costs. However, models with billions or tens of billions of parameters are difficult to fully utilize the effective information in the data, and the model structure is relatively simple, making it difficult to meet the needs of complex tasks. Their performance may be far inferior to large models with hundreds of billions of parameters.

[0023] In order to reduce application costs, you can try to use a model with smaller parameters to replace a large model. However, the process of judging whether a model with smaller parameters can replace a large model in a specific scenario is a cumbersome process that requires manual intervention. When using a smaller model to replace a large model, some of the generalization capabilities of the large model can be transferred to the smaller model based on technologies such as knowledge distillation or model distillation.

[0024] However, the smaller models obtained based on knowledge distillation and model distillation are often designed and optimized for specific scenarios or task types, lacking comprehensive consideration and universal construction for a variety of different scenarios. Once the scenario changes, the model may need to be readjusted or trained, and cannot flexibly adapt to differences in data distribution, task requirements, and other aspects in different scenarios. In complex and changeable actual application environments, it is difficult to efficiently use the same model architecture and training strategy based on knowledge distillation and model distillation technology to achieve stable performance in different scenarios, limiting the application of distillation technology in a wider range of fields.

[0025] If a large model is compressed, it is easy to lose accuracy. For some text analysis tasks that require high accuracy, over-compressing the model may make the analysis results inaccurate, making it difficult to strike a balance between complexity and accuracy. In addition, when fine-tuning a smaller model, it is often necessary to collect a large number of question-answer pairs for model fine-tuning. Although a large model can generate question-answer pairs, relying entirely on the question-answer pairs provided by the large model for a specified task will make it difficult for the small model to exceed the performance ceiling of the large model. If manually annotated question-answer pairs are used, the fine-tuning cost will be higher.

[0026] Therefore, in order to effectively reduce the cost of using the model under limited resource conditions, the present disclosure provides a sample processing method, which will be described below.

[0027] Figure 1 is a schematic diagram of an exemplary system architecture to which a sample processing method and apparatus can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0030] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0031] It should be noted that the sample processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the sample processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The sample processing method provided in the embodiment of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the sample processing device provided in the embodiment of the present disclosure can generally be set in the first terminal device 101, the second terminal device 102, and the third terminal device 103. The sample processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the sample processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0032] It can be understood that the above describes the system architecture of the present disclosure, and the following will describe the method of the present disclosure.

[0033] Figure 2 It is a schematic diagram of a sample processing method for a large model scene according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, method 200 may include operations S201 to S203.

[0035] In operation S201 , sample evaluation results of each of the plurality of sample texts to be processed are determined according to the plurality of model output results of each of the plurality of sample texts to be processed.

[0036] In the disclosed embodiment, the multiple model output results of the sample text to be processed are respectively output by multiple deep learning models with different parameter amounts. The multiple deep learning models with different parameter amounts may include a large model and a smaller model. For example, the large model may be a model with hundreds of billions of parameters. The smaller model may be a model with tens of billions or billions of parameters.

[0037] In the disclosed embodiment, the plurality of sample texts to be processed may be natural language texts from one or more scenes. For example, the plurality of scenes may include scenes such as "junior high school composition writing" and "introduction assistant".

[0038] In an embodiment of the present disclosure, a sample evaluation result may indicate: a model that satisfies a preset condition among multiple deep learning models with different parameter amounts. The preset condition may be: the evaluation index value of the model output result is a target evaluation index value among multiple model output results, and the resource overhead is the minimum value of at least one resource overhead corresponding to the target evaluation index value. At least one resource overhead may be the resource overhead of at least one deep learning model corresponding to the target evaluation index value. The target evaluation index value may be the optimal evaluation index value. It is understood that if the evaluation index is accuracy, the target evaluation index value may be the maximum evaluation index value. If the evaluation index is randomness, the target evaluation index value may be the minimum evaluation index value.

[0039] In operation S202, at least one feature label of each of a plurality of sample texts to be processed is determined.

[0040] In the embodiments of the present disclosure, one or more feature tags of the sample to be processed may be determined in various ways. For example, for a sample text to be processed in the above-mentioned "introduction assistant" scenario, the feature tag may be "text analysis".

[0041] In operation S203, a feature evaluation value of the feature label is determined according to respective sample evaluation results of a plurality of samples to be processed having the same feature label.

[0042] In the disclosed embodiment, the proportion of models with smaller parameter amounts that meet the preset conditions can be determined based on the sample evaluation results of each of the multiple sample texts to be processed with the same feature label. The feature evaluation value can be determined based on the proportion. For example, the proportion is used as the feature evaluation value.

[0043] Through the embodiments of the present disclosure, the feature evaluation value of the feature label of the sample text to be processed can be determined based on the sample evaluation result of the sample text to be processed. As a result, sample texts with the same features can be mined from the sample texts to be processed in different scenes, and a model suitable for processing the sample text with the feature label can also be determined, which helps to improve the text processing effect and effectively reduce the processing cost. After receiving the text of a new scene, the model for processing the text of the scene can be quickly determined to achieve efficient text processing at a low cost.

[0044] It can be understood that the above describes the sample processing method for large model scenarios of the present disclosure, and the sample text to be processed of the present disclosure will be described below.

[0045] Figure 3 It is a schematic diagram of a sample processing method for a large model scene according to an embodiment of the present disclosure.

[0046] In some embodiments, data sets commonly used by users in different fields may be collected. Figure 3As shown, data sets ds30 from different fields can be collected. By cleaning and removing duplicates from the data set ds30, multiple sample texts to be processed can be obtained.

[0047] In some embodiments, multiple sample texts to be processed can be divided into multiple different domain data sets according to the domain of the data set. A domain data set corresponds to one domain. Multiple domains include: "Creation and Continuation", "Legal Moral Security", "Rewriting and Error Correction", "Role Dialogue", "Structured Data Question and Answer", "Mathematics and Logic", "Synonymous Paraphrase and Sample Generation", "Information Extraction", "Reading Comprehension", "Abstract", etc.

[0048] In some embodiments, the above method may include: clustering multiple sample texts to be processed to obtain multiple sample sets to be processed. For example, a word vector conversion operation may be performed on multiple sample texts to be processed to map the sample texts to be processed to a low-dimensional vector space to obtain multiple sample vectors to be processed. Multiple similarities between multiple sample vectors to be processed may be calculated. Multiple sample vectors to be processed with higher similarities are taken as a sample set to be processed. Thus, multiple sample sets to be processed may be obtained. The sample set to be processed may correspond to a scene. Multiple scenes may include "junior high school composition writing", "question analysis", "paper review", "argumentation and evidence judgment", etc. Next, operation S3001 may be performed.

[0049] In operation S3001, a plurality of to-be-processed sample sets are used to train a plurality of models with smaller parameter amounts, respectively, to obtain a plurality of trained smaller models. The number of parameters of the smaller models may be, for example, several billion.

[0050] It can be understood that the above describes the sample text to be processed, and some methods for determining the sample evaluation results will be described below.

[0051] In operation S301 , sample evaluation results of each of the plurality of sample texts to be processed are determined according to the plurality of model output results of each of the plurality of sample texts to be processed.

[0052] In the disclosed embodiment, according to the evaluation prompt data, a large model is used to respectively determine a plurality of evaluation index values ​​of a plurality of model output results of the sample text to be processed. According to the plurality of evaluation index values, a sample evaluation result of the sample text to be processed is determined.

[0053] In the embodiment of the present disclosure, the evaluation index value can be determined using an evaluation large model.

[0054] In the embodiment of the present disclosure, the evaluation prompt data text includes at least one of a plurality of evaluation prompt texts, and the evaluation prompt text is used to instruct the evaluation large model to evaluate the model output result according to the evaluation method corresponding to the evaluation prompt text.

[0055] In an embodiment of the present disclosure, multiple evaluation prompt texts include evaluation configuration prompt text, accuracy evaluation prompt text, instruction follow prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text and evaluation result prompt text.

[0056] In the disclosed embodiment, the evaluation configuration prompt text is used to indicate the evaluation index value of the evaluation big model to determine the model output result. For example, the evaluation configuration prompt text may be "Please act as an impartial judge and evaluate the quality of the answers provided by the AI ​​assistant to the user questions shown below. Your evaluation should consider the following factors:". The "Your evaluation should consider the following factors:" in the evaluation configuration prompt text may instruct the big model to be evaluated according to at least one of the accuracy evaluation prompt text, instruction follow prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text and evaluation result prompt text.

[0057] In the embodiment of the present disclosure, the accuracy assessment prompt text is used to indicate the accuracy of the model output result determined by the large model. For example, the accuracy assessment prompt text may be "Accuracy: whether the answer is factually correct.".

[0058] In the disclosed embodiment, the instruction compliance evaluation prompt text is used to indicate whether the evaluation model determines whether the model output result is generated for the sample text to be processed. For example, the instruction compliance prompt text may be "Instruction compliance: whether the answer strictly follows the requirements of the user's question, such as whether all the required information is provided, and the answer should be output in a given sample format. When encountering a choice or classification question, the answer should be output directly without additional explanation."

[0059] In the disclosed embodiment, the directness evaluation prompt text is used to indicate whether the evaluation model determines whether the model output result includes information irrelevant to the sample text to be processed. For example, the directness evaluation prompt text may be "Directness: answer the question directly and positively, without other information and redundant information.".

[0060] In the disclosed embodiment, the fluency evaluation prompt text is used to indicate whether the evaluation model determines whether the model output result meets the preset grammatical conditions. For example, the fluency evaluation prompt text may be "Semantic fluency: whether the answer is clear and fluent, without grammatical errors, incoherence or typos.".

[0061] In the disclosed embodiment, the compliance assessment prompt text is used to indicate whether the evaluation model determines whether the model output result complies with the preset rules. For example, the compliance assessment prompt text may be "Legal ethics and safety: When it comes to sensitive topics, answer whether it complies with legal ethics."

[0062] In the disclosed embodiment, the consistency assessment prompt text is used to indicate whether the model output result is consistent with the reference information in the sample text to be processed. For example, the consistency assessment prompt text may be "Consistency: When reference information is provided in the question, the answer needs to strictly comply with the reference information content and cannot fabricate additional information."

[0063] In the disclosed embodiment, the evaluation result prompt text is used to indicate the numerical range of the evaluation index value. For example, the evaluation result prompt text may be "Please help me evaluate the quality of the answers of the two AI assistants and give corresponding scores from 0 to 10". Through the disclosed embodiment, the use of multiple evaluation prompt texts can enable a large model to accurately determine the evaluation of multiple model output results, which helps to efficiently determine the sample evaluation results.

[0064] It can be understood that the above text describes the evaluation prompt text of the present disclosure, and some methods of determining the evaluation index value of the present disclosure will be described below.

[0065] In the embodiment of the present disclosure, according to the preset template to be evaluated, the sample text to be processed and the output results of multiple models are spliced ​​into the text to be evaluated.

[0066] In the embodiment of the present disclosure, the preset template to be evaluated includes a question field and multiple model output result fields. For example, the preset annotation template can be "[question]

[0067] {

[0068] \"INSTR\": \"{{question_v}}\",

[0069] }

[0070] [Assistant's answer] [

[0072] {

[0073] \"model\": \"model_1\",

[0074] \"answer\": \"{{answerA}}\"

[0075] },

[0076] {

[0077] \"model\": \"model_2\",

[0078] \"answer\": \"{{answerB}}\"

[0079] }

[0080] ]” “INSTR” can be a question field. question_v can represent the value of the question field, which can be the above-mentioned sample text to be processed. “model” can be a model identification field. The model identification field can indicate one of multiple deep learning models with different parameter amounts. “model_1” can represent a first deep learning model. “model_2” can represent a second deep learning model.

[0081] "answer" can be a model output result field. The value of the model output result field can represent one of multiple model output results. "answerA" can represent a model output result. "answerB" can represent a model output result different from "answerB". In the case where "answerA" represents the first model output result output by the first deep learning model, "answerB" can represent the second output result output by the second deep learning model.

[0082] In an embodiment of the present disclosure, based on the evaluation prompt data, using the big model to respectively determine multiple evaluation index values ​​of multiple model output results of the sample text to be processed includes: inputting the text to be evaluated into the big model so that the big model outputs the evaluation data of the sample text to be processed according to a preset evaluation result template.

[0083] In the embodiment of the present disclosure, the evaluation data includes a plurality of evaluation result fields that are consistent with a preset evaluation result template, and the values ​​of the evaluation result fields are evaluation index values.

[0084] For example, the preset evaluation result template can be: "[output format]

[0085] {

[0086] "model1_score": "",

[0087] "model2_score": ""

[0088] }

[0089] Please give an analysis first, and then give the scores of the two models according to the format.

[0090] [Output]".

[0091] "model_1_score" and "model_2_score" may be two evaluation result fields. The value of the evaluation result field may be an evaluation index value. Through the disclosed embodiment, based on the preset template to be evaluated and the preset evaluation result template, the accuracy of the evaluation index value determined by the large model may be effectively improved to determine an accurate sample evaluation result.

[0092] It can be understood that some methods for determining evaluation index values ​​are described above, and some methods for determining sample evaluation results will be described below.

[0093] In an embodiment of the present disclosure, a plurality of deep learning models with different parameter amounts may include a first deep learning model and a second deep learning model. The parameter amount of the first deep learning model may be less than the parameter amount of the second deep learning model. The plurality of model output results include a first model output result output by the first deep learning model and a second model output result output by the second deep learning model. The plurality of evaluation index values ​​include a first evaluation index value of the first model output result and a second evaluation index value of the second model output result.

[0094] In some embodiments, the sample evaluation result may be a Good Same Bad (GSB) result.

[0095] In some embodiments, according to multiple evaluation index values, determining the sample evaluation result of the sample text to be processed includes: in response to determining that the first evaluation index value is greater than or equal to the second evaluation index value, determining that the sample evaluation result of the sample text to be processed is the first sample evaluation result. The first sample evaluation result may indicate that when processing the sample to be processed, the performance of the first deep learning model is better than the performance of the second deep learning model (good), or the performance of the first deep learning model is the same as the performance of the second deep learning model (same). For example, if the first evaluation index value is 8 and the second evaluation index value is 8, it can be determined that the sample evaluation result of the sample text to be processed is the first sample evaluation result.

[0096] In some embodiments, according to multiple evaluation index values, determining the sample evaluation result of the sample text to be processed includes: in response to determining that the first evaluation index value is less than the second evaluation index value, determining that the sample evaluation result of the sample text to be processed is the second sample evaluation result. The second sample evaluation result may indicate that when processing the sample text to be processed, the performance of the first deep learning model is worse (bad) than the performance of the second deep learning model. For example, if the first evaluation index value is 4 and the second evaluation index value is 8, it can be determined that the sample evaluation result of the sample text to be processed is the second sample evaluation result. Through the embodiments of the present disclosure, based on the comparison between the output results of different models, the model suitable for processing the sample text to be processed can be accurately determined, which helps to accurately determine the model applicable to processing new sample texts.

[0097] Next, operation S3021 may be performed to determine an acceptance rate.

[0098] In the embodiments of the present disclosure, the acceptance rate can indicate whether the performance of the first deep learning model in processing the sample text to be processed is acceptable. For example, taking a sample set to be processed including 500 sample texts to be processed as an example, the sample evaluation results of 400 sample texts to be processed in the sample set to be processed are the first sample evaluation results, and the sample evaluation results of 100 samples to be processed in the sample set to be processed are the second sample evaluation results. The acceptance rate of the sample set to be processed can be 80%. For another example, taking another sample set to be processed including 1,000 sample texts to be processed as an example, the sample evaluation results of 200 sample texts to be processed in the sample set to be processed are the first sample evaluation results, and the sample evaluation results of 800 samples to be processed in the sample set to be processed are the second sample evaluation results. The acceptance rate of the sample set to be processed can be 20%.

[0099] In an embodiment of the present disclosure, in response to determining that the acceptance rate is greater than or equal to the first preset acceptance rate threshold or the acceptance rate is less than or equal to the second preset acceptance rate threshold, operation S3022 is performed. For example, the first preset acceptance rate threshold may be 70%, and the second preset acceptance rate threshold may be 30%. If the acceptance rate of the above-mentioned one sample set to be processed is 80% (greater than the first preset acceptance rate threshold), operation S3022 may be performed on multiple sample texts to be processed in the sample set to be processed. For another example, if the acceptance rate of the above-mentioned other sample set to be processed is 20% (less than the second preset acceptance rate threshold), operation S3022 may be performed on multiple sample texts to be processed in the sample set to be processed. For another example, for one or more sample sets to be processed whose acceptance rate is less than the first preset acceptance rate threshold and greater than the second preset acceptance rate threshold, the sample texts to be processed in the one or more sample sets to be processed may be deleted.

[0100] In operation S3022, at least one feature tag is determined for each of the plurality of sample texts to be processed.

[0101] In the embodiment of the present disclosure, at least one feature label of the sample text to be processed can be determined by using a large model according to the annotation prompt data. For example, at least one feature label of the sample text to be processed can be determined by using the annotation large model. The annotation large model and the evaluation large model can be the same model or different models, and the present disclosure does not limit this.

[0102] In the embodiment of the present disclosure, the sample text to be processed is used as the input sample text according to the preset template to be annotated.

[0103] In the disclosed embodiment, the sample text to be processed is input into the large model so that the large model outputs at least one label data of the sample text to be processed according to the preset labeling result template. The label data includes a plurality of labeling result fields consistent with the preset labeling result template, the plurality of labeling result fields include a feature label field and a label explanation field, the value of the feature label field of the label data is the feature label, and the label explanation field of the label data is used to indicate the meaning of the feature label.

[0104] like Figure 3 As shown, based on multiple sample texts to be processed, multiple feature tags can be determined. The multiple feature tags may include "text analysis", "translation", "education", "policy interpretation", "keyword extraction", etc. It can be understood that Figure 3 The multiple feature tags shown are only examples. For one or more sets of samples to be processed whose acceptance rate is greater than or equal to a first preset acceptance rate threshold and one or more sets of samples to be processed whose acceptance rate is less than or equal to a second preset acceptance rate threshold, more than 2,000 different feature tags can be determined.

[0105] Next, operation S303 may be performed.

[0106] In operation S303, a feature evaluation value of a feature tag is determined according to respective sample evaluation results of a plurality of sample texts to be processed having the same feature tag.

[0107] In the embodiment of the present disclosure, feature evaluation data of each of the plurality of sample sets to be processed may be determined. The feature evaluation data of the sample set to be processed may include a set evaluation result of at least one feature label. The set evaluation result of the feature label may be determined based on the sample evaluation result of the sample text to be processed having the feature label in the sample set to be processed.

[0108] In the embodiment of the present disclosure, the set evaluation result may be a first set evaluation result or a second set evaluation result.

[0109] In an embodiment of the present disclosure, the set evaluation result of the feature label can be determined according to the sample evaluation results of the sample text to be processed with the feature label in the sample set to be processed by the following operation: in response to determining that the first sample evaluation ratio is greater than or equal to the first preset ratio threshold, the set evaluation result of the feature label is determined as the first set evaluation result. In the sample set to be processed, the first sample evaluation ratio is the ratio of the first sample text to be processed in the multiple sample texts to be processed with the feature label. The first sample text to be processed has a feature label. The sample evaluation result of the first sample text to be processed is the first sample evaluation result. For example, if there are multiple sample texts to be processed with the feature label "text analysis" in a sample set to be processed. The multiple samples to be processed with the feature label "text analysis" include at least one first sample text to be processed and at least one second sample text to be processed. The sample evaluation result of the second sample to be processed is the second sample evaluation result. In the set of samples to be processed, if the number of first sample texts to be processed with the feature label is 75, the number of second sample texts to be processed with the feature label is 25, and the first preset proportion threshold is 70%, it can be determined that the set evaluation result of the feature label "text analysis" is the first set evaluation result.

[0110] In another embodiment of the present disclosure, the set evaluation result of the feature label can be determined according to the sample evaluation results of the sample texts to be processed with the feature label in the sample set to be processed by the following operation: in response to determining that the first sample evaluation ratio is less than the second preset ratio threshold, the set evaluation result of the feature label is determined as the second set evaluation result. For example, there are also multiple sample texts to be processed with the feature label "text analysis" in another sample set to be processed. In the sample set to be processed, if the number of the first sample texts to be processed with the feature label "text analysis" is 25, the number of the second sample texts to be processed with the feature label "text analysis" is 75, and the second preset ratio threshold is 30%, the set evaluation result of the feature label "text analysis" can be determined as the second set evaluation result.

[0111] In an embodiment of the present disclosure, determining a feature evaluation value of a feature label according to respective sample evaluation results of a plurality of samples to be processed having the same feature label includes: determining the feature evaluation value of the feature label according to a first difference value and the number of target sample sets to be processed. The first difference value is determined according to a first number and a second number. The first number is the number of the first target sample set to be processed. The set evaluation result of the feature labels in the first target sample set to be processed is the first set evaluation result. The second number is the number of the second target sample set to be processed. The set evaluation result of the feature labels in the second target sample set to be processed is the second set evaluation result. For example, the above-mentioned two sample sets to be processed having the feature label "text analysis" can be used as two target sample sets to be processed. For another example, the feature evaluation value of the feature label can be determined by the following formula :

[0112] (Formula 1)

[0113] It can be the number of the first target sample set to be processed, It may be the number of the second target sample sets to be processed, and N may be the number of the target sample sets to be processed.

[0114] It can be understood that the feature evaluation value of the feature tag is determined according to the set evaluation result of the feature tag above, but the present disclosure is not limited thereto, and the feature evaluation value of the feature tag can also be determined according to the sample evaluation result, which will be explained below.

[0115] In the disclosed embodiment, in other implementations of the above operation S203, determining the feature evaluation value of the feature label according to the sample evaluation results of each of the multiple samples to be processed with the same feature label includes: determining the feature evaluation value of the feature label according to the second difference value and the number of the multiple sample texts to be processed with the same feature label. The second difference value is determined according to the third number and the fourth number. The third number is the number of the first sample evaluation results in the multiple sample texts to be processed with the same feature label. The fourth number is the number of the second sample evaluation results in the multiple sample texts to be processed with the same feature label. For example, for multiple sample texts to be processed with the feature label "keyword extraction", the third number of the sample texts to be processed whose sample evaluation results are the first sample evaluation results can be 300, and the fourth number of the sample texts to be processed whose sample evaluation results are the second sample evaluation results can be 100. The second difference value can be 200. The feature evaluation value of the feature label can be 0.5 (200 / 400).

[0116] It can be understood that some methods for determining the feature evaluation value of the feature tag are described above, and the labeling prompt data of the present disclosure will be described below.

[0117] In some embodiments, the annotation prompt data includes at least one of a plurality of annotation prompt texts, and the annotation prompt text is used to instruct the large model to perform an annotation operation corresponding to the annotation prompt text. For example, the large model may be an annotation large model.

[0118] In an embodiment of the present disclosure, the multiple annotation prompt texts include annotation configuration prompt text, annotation type prompt text, first annotation restriction prompt text, second annotation restriction prompt text and annotation order prompt text.

[0119] In the disclosed embodiment, the annotation prompt text includes at least one of a reference information subtext, an identification subtext, and a core subtext. The reference information subtext can provide reference information for the large model, and the core subtext can include a core word and a core tag. The core word can indicate an operation that the large model needs to perform or does not need to perform. For example, the core subtext can be "**need**", including a core tag "**" and a core word "need".

[0120] In the disclosed embodiment, the annotation configuration prompt text may instruct the large model to determine the feature tags of the input sample text according to the keywords of the input sample text. For example, the annotation configuration prompt text may be "You are a text keyword extractor, and you are very good at extracting keywords that readers need to focus on during the reading process. The extracted keywords need to be evenly distributed in the text and not concentrated in a few sentences. [Keyword extraction rules]". The annotation configuration prompt text may include the identification subtext "[Keyword extraction rules]".

[0121] In the embodiment of the present disclosure, the annotation type prompt text is used to provide the definition of different types of keywords. For example, the annotation type prompt text can be:

[0122] “**Need** to extract the keyword type:

[0123] + Adjectives: words that can stir up the reader's emotions (e.g., "It's really addictive"), words that can highlight the selling point or characteristics of something (e.g., "sharp without hurting your hands"), words that can reflect contrast (e.g., "affordable price" and "good quality")

[0124] + Quantifiers: words related to the amount of data (such as "90%"), words related to money (such as "10 yuan")

[0125] + People: words related to the target audience (e.g., "students"), names of celebrities (e.g., "Zhang San")

[0126] + Proper nouns: brand or product names, literary and film and television work titles, and other proper nouns, professional terms, and famous quotes

[0127] + Time: words related to time (e.g., "5 minutes", "the sooner the better")

[0128] + Information-related vocabulary: Keywords that reflect the main idea of ​​the text, summary words (such as "two stages", "5 things")

[0129] + Other keywords that need to be focused on. The annotation type prompt text can instruct the annotation model to extract keywords of the type such as adjectives and proper nouns. The tag type prompt text also includes a reference information subtext "(such as: "It's really too addictive")". The reference information subtext can include a preset reference character "such as". In the case of including the preset reference character, the subtext of the prompt text can be used as a reference information subtext.

[0130] In the disclosed embodiment, the annotation prompt text may also include a segmentation mark. The text between two segmentation marks may be a type of keyword and the definition of the keyword of that type. For example, the segmentation mark may be "+". The definition of "adjective" may be "a word that can express the reader's emotions, a word that can highlight the selling point or characteristics of a thing".

[0131] In the disclosed embodiment, the first annotation restriction prompt text may indicate keywords that the large model does not need to extract from the input sample text. For example, the first annotation restriction prompt text may be "Attention!! Modal particles, ordinary names, and words that appear repeatedly in the original text are **not required** to be extracted". The first annotation restriction prompt text may include the core subtext "**not required**". The core subtext includes the core word "not required".

[0132] In the disclosed embodiment, the second annotation restriction prompt text is used to instruct the large model to determine keywords from multiple initial words in the input sample text. For example, the second annotation restriction prompt text may be "The extracted keywords must be a single word contained in the text, and must not be a sentence; when extracting keywords, do not fabricate keywords or splice keywords based on semantics."

[0133] In the disclosed embodiment, the annotation order prompt text is used to instruct the large model to output the keywords according to the text position corresponding to the keywords, and the text position is the position corresponding to the keywords in the input sample text. For example, the annotation order prompt text can be "Please extract the keywords that need to be emphasized in the order in which the keywords appear in the text" according to the order in which they appear in the text. Through the disclosed embodiment, the use of multiple annotation prompt texts can enable the large model to accurately determine the feature labels of the sample text, which helps to efficiently perform subsequent processing.

[0134] It can be understood that the above description is about the annotation prompt data of the present disclosure, and the following description is about the preset template to be annotated and the preset annotation result template of the present disclosure.

[0135] In the embodiment of the present disclosure, the preset template to be annotated includes an input text field. For example, the preset annotation template can be "[text]

[0136] {{

[0137] "instruction": "{question}",

[0138] }}".

[0139] "instruction" can be an input text field. "question" can represent the value of the input text field, or can be an input sample text. The input sample text can be "[text]\\n\\"The God of Wealth is coming oh oh oh\\"\\n\\nPlease strictly follow the [keyword extraction rules] to extract the keywords in [text], please output the keywords:".

[0140] In the embodiment of the present disclosure, the preset annotation result template may include multiple annotation result fields. The multiple annotation result fields may include a feature tag field and a tag explanation field. For example, the preset annotation result template may be:

[0141] "[Output format][

[0142] {

[0143] "tag": str,

[0144] "explanation": str

[0145] },

[0146] {

[0147] "tag": str,

[0148] "explanation": str

[0149] }

[0150] ]”.

[0151] "tag" can be a feature tag field. str indicates that the value type of the annotation result field is a string. "explanation" can be a tag explanation field. The annotation large model can output multiple target tag data of the above-mentioned input sample text according to the preset tag result template. The multiple target tag data may include [{"tag": "Keyword extraction","explanation": "This instruction focuses on extracting specific types of keywords from a given text according to a set of defined rules. It involves identifying adjectives, quantifiers, names, proper nouns, time-related words, information-related vocabulary, and other important keywords in the text."} and {"tag": "Text analysis", "explanation": "This task involves analyzing a given text to identify and extract keywords according to specific criteria, thereby demonstrating the need for text analysis functions."}. It can be understood that based on the preset tag result template, the target tag data output by the annotation large model can be more than 2. It can also be understood that the sample text to be processed can be used as the value of the input text field.

[0152] It can be understood that the above describes the sample processing method for large model scenarios of the present disclosure, and the feature labels of the present disclosure will be further described below.

[0153] Figure 4 is a schematic diagram of multiple feature tags according to an embodiment of the present disclosure.

[0154] like Figure 4 As shown, the multiple feature tags may include "formatting", "resume analysis" and "text generation". For the multiple sample texts to be processed with the feature tag "formatting", the number of the first sample evaluation results is less than the number of the second sample evaluation results. For the multiple sample texts to be processed with the feature tag "resume analysis", the number of the first sample evaluation results is greater than the number of the second sample evaluation results. For the multiple sample texts to be processed with the feature tag "text review", the number of the first sample evaluation results is greater than the number of the second sample evaluation results.

[0155] As described above, after determining the feature evaluation value of the feature label, the feature label can be used as a preset feature label, and the feature evaluation value of the feature label can be used as a preset evaluation value. The sample text to be processed can also be used as the preset sample text. Thus, after obtaining the sample text of a new scene, the model for processing the problem text of the scene can be determined based on the feature label of the sample text of the new scene, which will be explained below.

[0156] Figure 5 is a flowchart of a model determination method according to an embodiment of the present disclosure.

[0157] like Figure 5As shown, the method 500 may include operations S510 to S520. It can be understood that the method 500 may be performed after the method 200 described above.

[0158] In operation S510, a model evaluation value of a target scene is determined according to at least one feature evaluation value corresponding to at least one first target feature label of a first target sample text.

[0159] In the disclosed embodiment, the first target sample text may be a sample text of a target scene. For example, the target scene may be a scene different from a preset scene. There may be one or more preset scenes. A preset scene may be a "role dialogue" scene, which requires the model to act as a role and dialogue with the user. The first target sample text may be a question text provided by the user, which may or may not have a real reply text. For another example, the preset scene may be a scene corresponding to the above-mentioned set of samples to be processed.

[0160] In the embodiments of the present disclosure, the first target feature label can be determined in various ways. The first target feature label can correspond to a feature evaluation value. For example, multiple first target feature labels can be determined from one or more preset feature labels of each of multiple preset sample data. The preset sample text can be a preset common text. The preset feature label can correspond to a feature evaluation value. If a preset feature label is determined as the first target feature label from multiple preset feature labels, the feature evaluation value corresponding to the preset feature label can be used as the feature evaluation value corresponding to the first target feature label. For another example, one or more first target feature labels can be determined based on the semantics of the first target sample text, and the semantic evaluation value of the first target feature label can be determined as the feature evaluation value. The semantic evaluation value can indicate the accuracy of processing the first target sample data using a smaller model.

[0161] In the embodiments of the present disclosure, various operations are performed based on one or more feature evaluation values ​​to determine a model evaluation value. For example, an average value of multiple feature evaluation values ​​can be determined as the model evaluation value. For another example, the sum of multiple feature evaluation values ​​can also be determined as the model evaluation value.

[0162] In operation S520, a target model is determined from among a plurality of deep learning models with different parameter amounts according to the model evaluation values.

[0163] In the disclosed embodiments, the deep learning models with different parameter amounts may include a large model and a smaller model. For example, the large model may be a model with hundreds of billions of parameters. The smaller model may be a model with tens of billions or billions of parameters.

[0164] In the disclosed embodiment, the model evaluation value may represent the performance of the smaller model in the target scene. If the model evaluation value represents that the performance of the smaller model in the target scene meets the needs of the user, the smaller model may be used as the target model.

[0165] In the disclosed embodiment, the target model is used to process the question text of the target scenario. For example, the question text can be input into the target model to obtain the reply text.

[0166] Through the disclosed embodiments, the characteristic labels of the target sample text can be used to determine the evaluation value of the model. The characteristics of the target sample text can be effectively utilized to accurately determine the target model suitable for the target scene, which can create favorable conditions for reducing the cost of model application while ensuring the accuracy of text processing.

[0167] It can be understood that the above is a description of the method disclosed in the present invention. Figure 6 The method of the present disclosure is further described.

[0168] Figure 6 is a schematic diagram of a model determination method according to an embodiment of the present disclosure.

[0169] like Figure 6 As shown, the model determination method can be performed by the feature mining module 61, the data management module 62 and the model routing module 63. For example, the feature mining module 61 can determine one or more first target feature tags of the first target sample text based on the annotated large model TAG60, and the feature mining module 61 can also determine the model evaluation value. The feature mining module 61 will be described below.

[0170] In some embodiments, in some implementations of the above operation S510, determining the model evaluation value of the target scene according to at least one feature evaluation value corresponding to at least one first target feature tag of the first target sample text includes: determining at least one first target feature tag of the first target sample text using the large model according to the annotation prompt data. Through the embodiments of the present disclosure, the feature tag of the first target sample text can be determined quickly and accurately using the large model, which helps to efficiently determine the model evaluation value, reduce the annotation cost of the text, and lower the threshold for model application.

[0171] In the embodiment of the present disclosure, the annotation prompt data may instruct the large model to perform multiple annotation operations. For example, the annotation prompt data may instruct the large annotation model TAG60 to perform multiple annotation operations.

[0172] like Figure 6As shown, the user can provide one or more first target sample data of the target scene to the annotation model TAG60. According to the annotation prompt data and each first target sample data, the annotation model TAG60 can determine at least one first target feature tag. Taking the at least one first target feature tag as an example, the multiple first target feature tags may include: Figure 6 The first target feature label lable_h61, the first target feature label lable_h62 and the first target feature label lable_h63 are shown.

[0173] In the embodiment of the present disclosure, according to the preset template to be annotated, the first target sample text is used as the input sample text. The preset template to be annotated may include an input text field. The first target sample text may be used as the value of the input text field.

[0174] In an embodiment of the present disclosure, the target text to be annotated is input into the large model so that the large model outputs at least one target label data of the first target sample text according to the preset annotation result template. The target label data includes a plurality of annotation result fields consistent with the preset annotation result template. The plurality of annotation result fields include a feature label field and a label explanation field. The value of the feature label field of the target label data is the first target feature label. The label explanation field of the target label data can represent the meaning of the first target feature label. For example, if the first target feature label is "text analysis", the semantics of the first target feature label can be "the task involves analyzing a given text to identify and extract keywords according to specific standards, proving the need for text analysis functions". It can be understood that the value of the feature label field can be text, and the value of the label explanation field can also be text. Through the embodiment of the present disclosure, based on the preset template to be annotated and the preset annotation result template, the accuracy of the feature label determined by the large model can be effectively improved to determine an accurate model evaluation value. It can be understood that the above description of the annotation prompt data, the preset template to be annotated and the preset annotation result template is also applicable to the present embodiment, and the present disclosure will not repeat them here.

[0175] In the embodiment of the present disclosure, the first target feature tag may be determined from a plurality of preset feature tags. The preset feature tag may be determined using a large model based on the annotation prompt data. Figure 6 As shown, before applying one of the multiple deep learning models with different parameter amounts to the target scene, the annotation model TAG60 can be used to determine one or more feature labels of each of the multiple sample texts to be processed according to the annotation prompt data. Next, the sample text to be processed can be used as the preset sample text, and the feature label of the sample text to be processed can be used as the preset feature label.

[0176] In some embodiments, a preset feature label corresponds to a feature evaluation value. The feature evaluation value of the feature label can be determined in various ways. For example, when the preset feature label of the preset sample text is determined by the annotation large model, the initial evaluation value of the preset feature label of the preset sample text can be determined. There can be multiple preset sample texts with the preset feature label. Thus, based on the multiple initial evaluation values ​​of the preset feature label, the feature evaluation value of the preset feature label can be determined. It can be understood that various operations (such as averaging) can be performed based on the multiple initial evaluation values ​​of the preset feature label to obtain the feature evaluation value. Multiple preset sample texts and one or more preset feature labels of each of the multiple preset sample texts can be stored in the data warehouse db60.

[0177] In the disclosed embodiment, the large model can determine one or more preset feature labels of multiple preset feature labels as one or more first target feature labels. For example, if a preset feature label is used as the first target feature label label_h61, the feature evaluation value of the preset feature label can be used as the feature evaluation value of the first target feature label label_h61.

[0178] In some embodiments, in some implementations of the above operation S510, determining the model evaluation value of the target scene according to at least one first target feature tag of the first target sample text includes: determining the model evaluation value of the target scene according to at least one feature evaluation value of at least one first target feature tag. If there are multiple first target sample texts, for each first target sample text, the sample evaluation index value of the first target sample text can be determined according to at least one feature evaluation value. For example, if the first target sample text has n first target feature tags, the sample evaluation index value can be determined by the following formula:

[0179] (Formula 2)

[0180] n may be an integer greater than or equal to 1. i may be an integer greater than or equal to 1 and less than or equal to n. label_i The feature evaluation value of the i-th first target feature label can be C. query It can be the sample evaluation index value of the first target sample text. It can be understood that the n values ​​of different first target sample texts can be the same or different.

[0181] Next, an addition operation is performed based on the multiple sample evaluation index values ​​of each of the multiple first target sample texts to determine the model evaluation value of the target scene. It can be understood that various operations can be performed based on the multiple sample evaluation index values ​​to determine the model evaluation value. The various operations can include averaging, weighted summing, etc.

[0182] It can be understood that some methods for determining the model evaluation value are described above, and some methods for determining the target model will be described below.

[0183] like Figure 6 As shown, the model routing module 63 can determine the target model from multiple deep learning models with different parameter amounts based on the model evaluation value, which will be explained below.

[0184] In some embodiments, a plurality of deep learning models with different parameter amounts may include a first deep learning model and a second deep learning model. The parameter amount of the first deep learning model may be less than the parameter amount of the second deep learning model. The first deep learning model may be a smaller model with the above-mentioned parameter amount of several billion. The second deep learning model may be a large model with the above-mentioned parameter amount of hundreds of billions. It is understood that the parameter amount of the first deep learning model is less than that of the second deep learning model.

[0185] In an embodiment of the present disclosure, in some implementations of the above operation S620, determining a target model from a plurality of deep learning models with different parameter amounts according to a model evaluation value includes: in response to determining that the model evaluation value is greater than or equal to a preset evaluation threshold, determining a first deep learning model as a target model. Through an embodiment of the present disclosure, a model evaluation value can represent the performance of a smaller model in a target scenario. According to the model evaluation value and the preset evaluation threshold, it can be quickly determined whether the performance of the smaller model meets the requirements of the target scenario. In the case where the performance of the smaller model meets the requirements, the smaller model can be used to process the problem text of the target scenario to effectively reduce computing resource overhead and storage resource overhead.

[0186] In the embodiment of the present disclosure, at least one training sample data may be used to fine-tune the target model to obtain a fine-tuned target model. The at least one training sample data may include at least one of a first target sample text and a second target sample text. The second target sample may be determined from a plurality of preset sample texts according to at least one first target feature tag. Figure 6As shown, when the model evaluation value is greater than or equal to the preset evaluation threshold, multiple preset feature labels consistent with the first target feature label lable_h61, the first target feature label lable_h62, and the first target feature label lable_h63 can be determined from the data warehouse db60 of the data management module 62. The preset sample text having at least one of the multiple preset feature labels is used as the second target sample text. Operation S631 can be performed according to the first target sample text and the second target sample text to fine-tune the first deep learning model. The fine-tuned first deep learning model is obtained. Next, operation S632 can be performed to perform reasoning using the fine-tuned first deep learning model to process the question text of the target scene and obtain the reply text. It can be understood that the first target sample text can have a real reply text. After the first target sample text is input into the first deep learning model, the first sample output result can be obtained. According to the difference between the first sample output result and the real reply text, the first deep learning model can be fine-tuned once. The second target sample text can also have a real reply text. It can also be fine-tuned based on the second target sample text. Thus, after fine-tuning the first deep learning model multiple times using the first target sample text and the second target sample text, a fine-tuned first deep learning model can be obtained. Through the disclosed embodiments, when the first deep learning model with a smaller number of parameters is used as the target model, the feature labels of the first target sample text can be used to determine one or more similar second target sample texts from multiple preset sample texts, which can effectively reduce the cost and difficulty of users providing samples of target scenes, and can effectively fine-tune the first deep learning model using relatively abundant samples to fully improve the efficiency and accuracy of the target model in processing the problem text of the target scene.

[0187] It can be understood that the above description is based on the example that the model evaluation value is greater than or equal to the preset evaluation threshold, but the present disclosure is not limited thereto, and the following description will be based on the example that the model evaluation value is less than the preset evaluation value.

[0188] In an embodiment of the present disclosure, in response to determining that the model evaluation value is less than a preset evaluation threshold, the second deep learning model is determined as the target model. Figure 6 As shown, when the model evaluation value is less than the preset evaluation threshold, operation S633 can be performed to use the second deep learning model for reasoning to process the problem text of the target scene. It can be understood that the second deep learning model has a large number of parameters and strong performance, but the cost is high. In order to reduce costs, the first target sample text may not be used to fine-tune the second deep learning model. It can also be understood that in order to fully improve the accuracy of processing the problem text of the target scene, the first target sample text and the second target sample text may also be used to fine-tune the second deep learning model.

[0189] It can be understood that the above text uses the determination of feature evaluation values ​​based on initial evaluation values ​​as an example to illustrate the present disclosure. However, the present disclosure is not limited to this, and feature evaluation values ​​can also be determined based on a large evaluation model. The above text uses a large evaluation model to determine the evaluation index value of the sample text to be processed, and then determines the sample evaluation result of the sample text to be processed based on the evaluation index value, and then determines the feature evaluation value of the feature label based on the sample evaluation result. This method is also applicable to determining the feature evaluation value of a preset feature label, and the present disclosure will not repeat it here.

[0190] It can be understood that the method of the present disclosure is described above, and the device of the present disclosure will be described below.

[0191] Figure 7 is a block diagram of a sample processing apparatus according to an embodiment of the present disclosure.

[0192] like Figure 7 As shown, the apparatus 700 may include a first determination module 701 , a second determination module 702 , and a third determination module 703 .

[0193] The first determination module 701 is used to determine the sample evaluation results of the plurality of sample texts to be processed according to the plurality of model output results of the plurality of sample texts to be processed. The plurality of model output results of the sample texts to be processed are respectively output by a plurality of deep learning models with different parameter amounts.

[0194] The second determination module 702 is used to determine at least one feature label of each of a plurality of sample texts to be processed.

[0195] The third determination module 703 is used to determine the feature evaluation value of the feature tag according to the sample evaluation results of each of a plurality of sample texts to be processed having the same feature tag.

[0196] In some embodiments, the first determination module includes: a first determination submodule, which is used to determine, based on the evaluation prompt data, a plurality of evaluation index values ​​of the output results of the plurality of models of the sample text to be processed using the large model; and a second determination submodule, which is used to determine the sample evaluation result of the sample text to be processed based on the plurality of evaluation index values.

[0197] In some embodiments, the evaluation prompt data text includes at least one of a plurality of evaluation prompt texts, and the evaluation prompt text is used to instruct the evaluation large model to evaluate the model output result according to the evaluation method corresponding to the evaluation prompt text.

[0198] In some embodiments, the plurality of evaluation prompt texts include evaluation configuration prompt text, accuracy evaluation prompt text, instruction compliance prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text, and evaluation result prompt text. The evaluation configuration prompt text is used to indicate the evaluation index value of the evaluation large model to determine the model output result. The accuracy evaluation prompt text is used to indicate the accuracy of the evaluation large model to determine the model output result. The instruction compliance evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result is generated for the sample text to be processed. The directness evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result includes information unrelated to the sample text to be processed. The fluency evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result meets the preset grammatical conditions. The compliance evaluation prompt text is used to indicate whether the evaluation large model determines whether the model output result meets the preset rules. The consistency evaluation prompt text is used to indicate whether the model output result is consistent with the reference information in the sample text to be processed. The evaluation result prompt text is used to indicate the numerical range of the evaluation index value.

[0199] In some embodiments, the first determination submodule includes: a splicing unit, which is used to splice the sample text to be processed and the output results of multiple models into the text to be evaluated according to the preset template to be evaluated. A first input unit is used to input the text to be evaluated into the large model so that the large model outputs the evaluation data of the sample text to be processed according to the preset evaluation result template. The evaluation data includes multiple evaluation result fields consistent with the preset evaluation result template, and the value of the evaluation result field is the evaluation index value.

[0200] In some embodiments, the second determination module includes: a third determination submodule, which is used to determine at least one feature label of the sample text to be processed using the large model according to the annotation prompt data.

[0201] In some embodiments, the third determination submodule includes: a first obtaining unit, used to take the sample text to be processed as the input sample text according to a preset template to be annotated. A second input unit, used to input the sample text to be processed into the large model, so that the large model outputs at least one label data of the sample text to be processed according to the preset annotation result template, the label data includes a plurality of annotation result fields consistent with the preset annotation result template, the plurality of annotation result fields include a feature label field and a label explanation field, the value of the feature label field of the label data is the feature label, and the label explanation field of the label data is used to indicate the meaning of the feature label.

[0202] In some embodiments, the annotation prompt data includes at least one of a plurality of annotation prompt texts, and the annotation prompt text is used to instruct the large model to perform an annotation operation corresponding to the annotation prompt text.

[0203] In some embodiments, the multiple annotation prompt texts include annotation configuration prompt text, annotation type prompt text, first annotation restriction prompt text, second annotation restriction prompt text, and annotation order prompt text. The annotation configuration prompt text is used to instruct the large model to determine the feature label of the input sample text based on the keywords of the input sample text. The annotation type prompt text is used to provide definitions of different types of keywords. The first annotation restriction prompt text is used to indicate keywords that the large model does not need to extract from the input sample text. The second annotation restriction prompt text is used to instruct the large model to determine keywords from multiple initial words of the input sample text. The annotation order prompt text is used to instruct the large model to output keywords according to the text position corresponding to the keywords, and the text position is the position corresponding to the keyword in the input sample text.

[0204] In some embodiments, the annotation prompt text includes at least one of a reference information sub-text, an identification sub-text, and a core sub-text. The reference information sub-text is used to provide reference information for the large model. The core sub-text includes core words and core tags. The core words are used to indicate operations that the large model needs to perform or does not need to perform.

[0205] In some embodiments, the apparatus further comprises: a clustering module, configured to cluster a plurality of sample texts to be processed to obtain a plurality of sample sets to be processed. A fourth determination module, configured to determine feature evaluation data of each of the plurality of sample sets to be processed. The feature evaluation data of the sample sets to be processed comprises a set evaluation result of at least one feature label, wherein the set evaluation result of the feature label is determined based on the sample evaluation results of the sample texts to be processed having the feature label in the sample set to be processed.

[0206] In some embodiments, the multiple deep learning models with different parameter amounts include a first deep learning model and a second deep learning model, and the parameter amount of the first deep learning model is less than the parameter amount of the second deep learning model. The multiple model output results of the sample text to be processed include a first model output result output by the first deep learning model and a second model output result output by the second deep learning model, and the multiple evaluation index values ​​of the multiple model output results of the sample text to be processed include a first evaluation index value of the first model output result and a second evaluation index value of the second model output result.

[0207] In some embodiments, the second determination submodule includes: a first determination unit, configured to determine that the sample evaluation result of the sample text to be processed is the first sample evaluation result in response to determining that the first evaluation index value is greater than or equal to the second evaluation index value; and a second determination unit, configured to determine that the sample evaluation result of the sample text to be processed is the second sample evaluation result in response to determining that the first evaluation index value is less than the second evaluation index value.

[0208] In some embodiments, the set evaluation result is a first set evaluation result or a second set evaluation result. The set evaluation result of the feature label is determined by the following submodules performing relevant operations according to the sample evaluation results of the sample text to be processed with the feature label in the sample set to be processed: a fourth determination submodule, which is used to determine the set evaluation result of the feature label as the first set evaluation result in response to determining that the first sample evaluation ratio is greater than or equal to the first preset ratio threshold. The first sample evaluation ratio is the ratio of the first sample text to be processed in the multiple sample texts to be processed with the feature label in the sample set to be processed, and the first sample text to be processed has the feature label and the sample evaluation result is the first sample evaluation result. The fifth determination submodule is used to determine the set evaluation result of the feature label as the second set evaluation result in response to determining that the first sample evaluation ratio is less than the second preset ratio threshold.

[0209] In some embodiments, the third determination module includes: a sixth determination submodule, which is used to determine the feature evaluation value of the feature label according to the first difference value and the number of the target sample set to be processed. The target sample set to be processed is a sample set to be processed that includes sample text to be processed with feature labels, and the first difference value is determined according to the first number and the second number, the first number is the number of the first target sample set to be processed, and the set evaluation result of the feature labels in the first target sample set to be processed is the first set evaluation result, the second number is the number of the second target sample set to be processed, and the set evaluation result of the feature labels in the second target sample set to be processed is the second set evaluation result.

[0210] In some embodiments, the third determination module includes: a seventh determination submodule, which is used to determine the feature evaluation value of the feature label according to the second difference value and the number of multiple sample texts to be processed with the same feature label. The second difference value is determined according to the third number and the fourth number, the third number is the number of first sample evaluation results in the multiple sample texts to be processed with the same feature label, and the fourth number is the number of second sample evaluation results in the multiple sample texts to be processed with the same feature label.

[0211] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0212] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0213] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0214] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0215] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a sample processing method for a large model scene. For example, in some embodiments, the sample processing method for a large model scene may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the sample processing method for the large model scene described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the sample processing method for the large model scene in any other appropriate manner (for example, by means of firmware).

[0216] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0217] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0218] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM) or flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0219] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0220] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0221] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0222] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0223] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A sample processing method for a large model scene, comprising: Determine sample evaluation results of each of the plurality of sample texts to be processed according to the plurality of model output results of each of the plurality of sample texts to be processed, wherein the plurality of model output results of the sample texts to be processed are respectively output by a plurality of deep learning models with different parameter amounts; Determine at least one feature label of each of the plurality of sample texts to be processed; The feature evaluation value of the feature label is determined according to the sample evaluation results of each of the plurality of sample texts to be processed having the same feature label.

2. The method according to claim 1, wherein: Determining the sample evaluation results of each of the plurality of sample texts to be processed according to the plurality of model output results of each of the plurality of sample texts to be processed comprises: According to the evaluation prompt data, using the large model, respectively determine a plurality of evaluation index values ​​of the plurality of model output results of the sample text to be processed; According to the multiple evaluation index values, a sample evaluation result of the sample text to be processed is determined.

3. The method according to claim 2, wherein: The evaluation prompt data text includes at least one of a plurality of evaluation prompt texts, and the evaluation prompt text is used to instruct the evaluation large model to evaluate the model output result according to the evaluation method corresponding to the evaluation prompt text.

4. The method according to claim 3, wherein: The plurality of evaluation prompt texts include evaluation configuration prompt text, accuracy evaluation prompt text, instruction compliance prompt text, directness prompt text, fluency evaluation prompt text, compliance evaluation prompt text, consistency evaluation prompt text, and evaluation result prompt text, The evaluation configuration prompt text is used to indicate the evaluation index value of the model output result determined by the evaluation large model. The accuracy assessment prompt text is used to instruct the evaluation model to determine the accuracy of the model output result. The instruction follows the evaluation prompt text for instructing the evaluation model to determine whether the model output result is generated for the sample text to be processed. The directness evaluation prompt text is used to instruct the evaluation model to determine whether the model output result includes information irrelevant to the sample text to be processed. The fluency evaluation prompt text is used to instruct the evaluation model to determine whether the model output result meets the preset grammatical conditions. The compliance assessment prompt text is used to instruct the assessment model to determine whether the model output result complies with the preset rules. The consistency assessment prompt text is used to indicate whether the model output result is consistent with the reference information in the sample text to be processed; The evaluation result prompt text is used to indicate the numerical range of the evaluation index value.

5. The method according to claim 3, wherein: The method of using the large model to respectively determine the multiple evaluation index values ​​of the multiple model output results of the sample text to be processed according to the evaluation prompt data includes: According to the preset template to be evaluated, the sample text to be processed and the output results of the multiple models are spliced ​​into the text to be evaluated; The text to be evaluated is input into the large model so that the large model outputs evaluation data of the sample text to be processed according to a preset evaluation result template, wherein the evaluation data includes multiple evaluation result fields consistent with the preset evaluation result template, and the values ​​of the evaluation result fields are the evaluation index values.

6. The method according to claim 1, wherein: The step of determining at least one feature tag of each of the plurality of sample texts to be processed comprises: According to the annotation prompt data, at least one feature label of the sample text to be processed is determined using a large model.

7. The method according to claim 6, wherein: Determining at least one feature label of the sample text to be processed by using a large model according to the annotation prompt data includes: According to the preset template to be annotated, the sample text to be processed is used as the input sample text; The sample text to be processed is input into the large model so that the large model outputs at least one label data of the sample text to be processed according to a preset labeling result template, the label data includes multiple labeling result fields consistent with the preset labeling result template, the multiple labeling result fields include a feature label field and a label explanation field, the value of the feature label field of the label data is the feature label, and the label explanation field of the label data is used to indicate the meaning of the feature label.

8. The method according to claim 7, wherein: The annotation prompt data includes at least one of a plurality of annotation prompt texts, and the annotation prompt text is used to instruct the large model to perform an annotation operation corresponding to the annotation prompt text.

9. The method according to claim 8, wherein: The plurality of annotation prompt texts include annotation configuration prompt text, annotation type prompt text, first annotation restriction prompt text, second annotation restriction prompt text and annotation order prompt text, The annotation configuration prompt text is used to instruct the large model to determine the feature label of the input sample text according to the keywords of the input sample text. The annotation type prompt text is used to provide definitions of different types of keywords. The first annotation restriction prompt text is used to indicate that the large model does not need to extract keywords from the input sample text. The second annotation restriction prompt text is used to instruct the large model to determine keywords from a plurality of initial words in the input sample text. The annotation order prompt text is used to instruct the large model to output the keyword according to the text position corresponding to the keyword, and the text position is the position corresponding to the keyword in the input sample text.

10. The method according to claim 9, wherein: The annotation prompt text includes at least one of a reference information subtext, an identification subtext and a core subtext. The reference information subtext is used to provide reference information for the large model. The core subtext includes core words and core tags. The core words are used to indicate operations that the large model needs to perform or does not need to perform.

11. The method according to claim 2, further comprising: Clustering the plurality of sample texts to be processed to obtain a plurality of sample sets to be processed; Determine feature evaluation data of each of a plurality of sample sets to be processed, wherein the feature evaluation data of the sample sets to be processed include a set evaluation result of at least one of the feature labels, and the set evaluation result of the feature label is determined based on the sample evaluation results of the sample text to be processed having the feature label in the sample sets to be processed.

12. The method according to claim 11, wherein: The multiple deep learning models with different parameter amounts include a first deep learning model and a second deep learning model, wherein the parameter amount of the first deep learning model is smaller than the parameter amount of the second deep learning model, The multiple model output results of the sample text to be processed include a first model output result output by the first deep learning model and a second model output result output by the second deep learning model, and the multiple evaluation index values ​​of the multiple model output results of the sample text to be processed include a first evaluation index value of the first model output result and a second evaluation index value of the second model output result.

13. The method according to claim 12, wherein: Determining the sample evaluation result of the sample text to be processed according to the multiple evaluation index values ​​includes: In response to determining that the first evaluation index value is greater than or equal to the second evaluation index value, determining that the sample evaluation result of the sample text to be processed is the first sample evaluation result; In response to determining that the first evaluation index value is less than the second evaluation index value, determining that the sample evaluation result of the sample text to be processed is the second sample evaluation result.

14. The method according to claim 12, wherein: The set evaluation result is the first set evaluation result or the second set evaluation result, The set evaluation result of the feature label is determined according to the sample evaluation result of the sample text to be processed with the feature label in the sample set to be processed by the following operation: In response to determining that the first sample evaluation ratio is greater than or equal to a first preset ratio threshold, determining the set evaluation result of the feature label as the first set evaluation result, wherein the first sample evaluation ratio is the ratio of the first to-be-processed sample text in the to-be-processed sample set that has the feature label among the multiple to-be-processed sample texts, and the first to-be-processed sample text has the feature label and the sample evaluation result is the to-be-processed sample text of the first sample evaluation result; In response to determining that the first sample evaluation ratio is less than a second preset ratio threshold, the set evaluation result of the feature tags is determined as a second set evaluation result.

15. The method according to claim 13, wherein: Determining the feature evaluation value of the feature label according to the sample evaluation results of each of the plurality of samples to be processed having the same feature label comprises: The feature evaluation value of the feature label is determined according to the first difference value and the number of the target set of samples to be processed, wherein the target set of samples to be processed is the set of samples to be processed including the sample text to be processed having the feature label, and the first difference value is determined according to the first number and the second number, the first number is the number of the first target set of samples to be processed, and the set evaluation result of the feature labels in the first target set of samples to be processed is the first set evaluation result, the second number is the number of the second target set of samples to be processed, and the set evaluation result of the feature labels in the second target set of samples to be processed is the second set evaluation result.

16. The method according to claim 13, wherein: Determining the feature evaluation value of the feature label according to the sample evaluation results of each of the plurality of samples to be processed having the same feature label comprises: The feature evaluation value of the feature label is determined according to the second difference value and the number of multiple sample texts to be processed with the same feature label, wherein the second difference value is determined according to a third number and a fourth number, the third number is the number of first sample evaluation results in the multiple sample texts to be processed with the same feature label, and the fourth number is the number of second sample evaluation results in the multiple sample texts to be processed with the same feature label.

17. A sample processing device for a large model scene, comprising: A first determination module is used to determine sample evaluation results of each of the plurality of sample texts to be processed according to the plurality of model output results of each of the plurality of sample texts to be processed, wherein the plurality of model output results of the sample texts to be processed are respectively output by a plurality of deep learning models with different parameter amounts; A second determination module is used to determine at least one feature label of each of the plurality of sample texts to be processed; The third determination module is used to determine the feature evaluation value of the feature label according to the sample evaluation results of each of the plurality of sample texts to be processed having the same feature label.

18. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images