Corpus quality evaluation method and device for multi-level task alignment, equipment and medium

Abstract

The invention discloses a corpus quality evaluation method and device for multilevel task alignment, equipment and a medium, and the method comprises the steps: sampling a to-be-evaluated corpus to obtain a target corpus sample set, and inputting each sample in the target corpus sample set into a target corpus quality evaluation model, obtaining a first single sample quality evaluation result of each sample in the target corpus sample set; wherein the target corpus quality evaluation model is obtained by performing model alignment training on a set corpus quality evaluation model; determining a sample set total quality evaluation result of the target corpus sample set according to a first evaluation rule; and determining a final corpus quality evaluation result of the to-be-evaluated corpus according to the first single sample quality evaluation result of each sample in the target corpus sample set and the total quality evaluation result of the sample set. According to the corpus quality evaluation method based on multilevel task alignment disclosed by the invention, the comprehensiveness and accuracy of corpus quality evaluation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-level task aligned corpus quality assessment method, device, equipment and medium. Background Art

[0002] High-quality corpus is the cornerstone of DataCentric-AI (data-centric artificial intelligence) and the key to building high-quality models. How to accurately evaluate and continuously optimize the quality of corpus and reduce bias and misleading in the model training process is a key challenge in model learning.

[0003] In the existing technology, the current corpus quality assessment field mainly uses two types of methods, namely indicator-based evaluation methods and model-based evaluation methods. Among them, indicator-based evaluation methods often have problems with high computational costs and poor generalization capabilities. Model evaluation methods include BERT model quality judgment, the use of multiple quality evaluators to judge corpus quality, MoDS instruction data quality evaluation and other methods, but in the implementation process, there are often problems such as the need for a large amount of data annotation for model training, the need for complex system integration and management, and over-reliance on automated evaluation.

[0004] Therefore, current technical means in the field of corpus quality assessment mostly focus on general intrinsic quality that is not related to specific tasks, while ignoring the relevant quality assessment of the model when it is oriented to specific tasks, resulting in inaccurate evaluation results. Summary of the invention

[0005] The present invention provides a multi-level task-aligned corpus quality assessment method, device, equipment and medium to achieve accurate assessment of the corpus quality in a corpus.

[0006] According to one aspect of the present invention, a multi-level task alignment corpus quality assessment method is provided, comprising:

[0007] A target corpus sample set is obtained by sampling from the corpus to be evaluated, and each sample in the target corpus sample set is input into a target corpus quality assessment model to obtain a first single sample quality assessment result of each sample in the target corpus sample set; wherein the target corpus quality assessment model is obtained by performing model alignment training on a set corpus quality assessment model;

[0008] Determine the overall quality evaluation result of the sample set of the target corpus sample set according to the first evaluation rule;

[0009] The final corpus quality evaluation result of the corpus to be evaluated is determined according to the first single sample quality evaluation result of each sample in the target corpus sample set and the overall quality evaluation result of the sample set.

[0010] Furthermore, the training method of the target corpus quality assessment model includes:

[0011] Obtain a model training corpus sample set, and determine a second single sample quality evaluation result of each sample in the model training corpus sample set according to a second evaluation rule;

[0012] The set corpus quality assessment model and the manual assessment results for the model training corpus sample set are obtained, and model alignment training is performed on the set corpus quality assessment model according to the second single sample quality assessment result and the manual assessment result to obtain the target corpus quality assessment model.

[0013] Further, determining a second single sample quality evaluation result of each sample in the model training corpus sample set according to a second evaluation rule includes:

[0014] For each sample in the model training corpus sample set, respectively identifying the sample field and task type corresponding to each sample;

[0015] According to the sample domain and task type corresponding to each sample, the domain expression quality evaluation result and the task context quality evaluation result of the corresponding sample are determined in combination with the second evaluation rule;

[0016] A weighted sum of the domain expression quality assessment result and the task context quality assessment result is used as the second single sample quality assessment result of the corresponding sample.

[0017] Further, according to the sample domain and task type corresponding to each sample, the domain expression quality evaluation result and the task context quality evaluation result of the corresponding sample are determined in combination with the second evaluation rule, including:

[0018] According to the sample fields corresponding to each sample and in combination with the second evaluation rule, respectively determine the term expression quality evaluation result, question / instruction expression completeness evaluation result, answer expression completeness evaluation result, expression structure evaluation result and expression style evaluation result of the corresponding sample;

[0019] Determine the domain expression quality assessment result of the corresponding sample according to the term expression quality assessment result, the question / instruction expression completeness assessment result, the answer expression completeness assessment result, the expression structure assessment result and the expression style assessment result;

[0020] According to the task type corresponding to each sample, combined with the second evaluation rule, respectively determine the background knowledge completeness evaluation result, the example relevance evaluation result and the logic completeness evaluation result of the corresponding sample;

[0021] The task context quality assessment result of the corresponding sample is determined according to the background knowledge completeness assessment result, the example suitability assessment result and the logic completeness assessment result.

[0022] Further, determining the overall quality evaluation result of the sample set of the target corpus sample set according to the first evaluation rule includes:

[0023] For each sample in the target corpus sample set, sample domain identification and task type identification are performed to determine the domain category and task type of the corresponding sample;

[0024] Determine the domain coverage evaluation result and the task coverage evaluation result of the target corpus sample set respectively according to the domain category and task type of each sample and in combination with the first evaluation rule;

[0025] Determining, according to the first evaluation rule, an expression form diversity evaluation result and a context combination diversity evaluation result of the target corpus sample set;

[0026] The sample set overall quality assessment result of the target corpus sample set is determined according to the domain coverage assessment result, the task coverage assessment result, the expression form diversity assessment result and the context combination diversity assessment result.

[0027] Further, determining the final corpus quality evaluation result of the corpus to be evaluated according to the first single sample quality evaluation result of each sample in the target corpus sample set and the overall quality evaluation result of the sample set includes:

[0028] Determine the average of the first single sample quality evaluation results of each sample in the target corpus sample set as the average single sample quality evaluation result of the target corpus sample set;

[0029] The average single sample quality assessment result and the sample set overall quality assessment result are weightedly summed to obtain the final corpus quality assessment result of the corpus to be assessed.

[0030] According to another aspect of the present invention, a corpus quality assessment device for multi-level task alignment is provided, comprising:

[0031] A first single sample quality assessment result determination module is used to sample a target corpus sample set from the corpus to be assessed, input each sample in the target corpus sample set into a target corpus quality assessment model, and obtain a first single sample quality assessment result of each sample in the target corpus sample set; wherein the target corpus quality assessment model is obtained by performing model alignment training on a set corpus quality assessment model;

[0032] A sample set overall quality evaluation result determination module, used to determine the sample set overall quality evaluation result of the target corpus sample set according to the first evaluation rule;

[0033] The final corpus quality evaluation result determination module is used to determine the final corpus quality evaluation result of the corpus to be evaluated according to the first single sample quality evaluation result of each sample in the target corpus sample set and the overall quality evaluation result of the sample set.

[0034] Optionally, the device further includes a model training module, which is used to:

[0035] Obtain a model training corpus sample set, and determine a second single sample quality evaluation result of each sample in the model training corpus sample set according to a second evaluation rule;

[0036] The set corpus quality assessment model and the manual assessment results for the model training corpus sample set are obtained, and model alignment training is performed on the set corpus quality assessment model according to the second single sample quality assessment result and the manual assessment result to obtain the target corpus quality assessment model.

[0037] Optionally, the model training module is also used to:

[0038] For each sample in the model training corpus sample set, respectively identifying the sample field and task type corresponding to each sample;

[0039] According to the sample domain and task type corresponding to each sample, the domain expression quality evaluation result and the task context quality evaluation result of the corresponding sample are determined in combination with the second evaluation rule;

[0040] A weighted sum of the domain expression quality assessment result and the task context quality assessment result is used as the second single sample quality assessment result of the corresponding sample.

[0041] Optionally, the model training module is also used to:

[0042] According to the sample fields corresponding to each sample and in combination with the second evaluation rule, respectively determine the term expression quality evaluation result, question / instruction expression completeness evaluation result, answer expression completeness evaluation result, expression structure evaluation result and expression style evaluation result of the corresponding sample;

[0043] Determine the domain expression quality assessment result of the corresponding sample according to the term expression quality assessment result, the question / instruction expression completeness assessment result, the answer expression completeness assessment result, the expression structure assessment result and the expression style assessment result;

[0044] According to the task type corresponding to each sample, combined with the second evaluation rule, respectively determine the background knowledge completeness evaluation result, the example relevance evaluation result and the logic completeness evaluation result of the corresponding sample;

[0045] The task context quality assessment result of the corresponding sample is determined according to the background knowledge completeness assessment result, the example suitability assessment result and the logic completeness assessment result.

[0046] Optionally, the sample set overall quality assessment result determination module is further used to:

[0047] For each sample in the target corpus sample set, sample domain identification and task type identification are performed to determine the domain category and task type of the corresponding sample;

[0048] Determine the domain coverage evaluation result and the task coverage evaluation result of the target corpus sample set respectively according to the domain category and task type of each sample and in combination with the first evaluation rule;

[0049] Determining, according to the first evaluation rule, an expression form diversity evaluation result and a context combination diversity evaluation result of the target corpus sample set;

[0050] The sample set overall quality assessment result of the target corpus sample set is determined according to the domain coverage assessment result, the task coverage assessment result, the expression form diversity assessment result and the context combination diversity assessment result.

[0051] Optionally, the final corpus quality assessment result determination module is further used to:

[0052] Determine the average of the first single sample quality evaluation results of each sample in the target corpus sample set as the average single sample quality evaluation result of the target corpus sample set;

[0053] The average single sample quality assessment result and the sample set overall quality assessment result are weightedly summed to obtain the final corpus quality assessment result of the corpus to be assessed.

[0054] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0055] at least one processor; and

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the corpus quality assessment method for multi-level task alignment described in any embodiment of the present invention.

[0058] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the corpus quality assessment method for multi-level task alignment described in any embodiment of the present invention when executed.

[0059] According to another aspect of the present invention, a computer program product is provided, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the multi-level task aligned corpus quality assessment method described in any embodiment of the present invention are implemented.

[0060] The corpus quality assessment method of multi-level task alignment disclosed by the present invention first samples a target corpus sample set from a corpus to be assessed, inputs each sample in the target corpus sample set into a target corpus quality assessment model, and obtains the first single sample quality assessment result of each sample in the target corpus sample set; wherein the target corpus quality assessment model is obtained by setting a corpus quality assessment model for model alignment training; then the overall quality assessment result of the sample set of the target corpus sample set is determined according to the first assessment rule; finally, the final corpus quality assessment result of the corpus to be assessed is determined according to the first single sample quality assessment result of each sample in the target corpus sample set and the overall quality assessment result of the sample set. The corpus quality assessment method of multi-level task alignment disclosed by the present invention not only evaluates the corpus quality of each sample in the sample set, but also considers the overall quality of the sample set. When performing single sample evaluation, a corpus quality assessment model obtained through model alignment training is used, which can deepen the model's understanding of professional field tasks, thereby making the final corpus quality assessment more comprehensive and accurate.

[0061] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a flowchart of a corpus quality assessment method for multi-level task alignment provided according to Embodiment 1 of the present invention;

[0064] Figure 2 is a flowchart of a corpus quality assessment method for multi-level task alignment according to Embodiment 2 of the present invention;

[0065] Figure 3 is a schematic diagram of the structure of a corpus quality assessment device for multi-level task alignment provided according to Embodiment 3 of the present invention;

[0066] Figure 4 It is a structural schematic diagram of an electronic device for implementing the multi-level task alignment corpus quality assessment method of the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0068] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0069] Embodiment 1

[0070] Figure 1 This is a flowchart of a multi-level task-aligned corpus quality assessment method provided in the first embodiment of the present invention. This embodiment is applicable to the situation of assessing the quality of corpus in a corpus. The method can be performed by a multi-level task-aligned corpus quality assessment device. The multi-level task-aligned corpus quality assessment device can be implemented in the form of hardware and / or software. The multi-level task-aligned corpus quality assessment device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0071] S110 , sampling a target corpus sample set from the corpus to be evaluated, inputting each sample in the target corpus sample set into a target corpus quality evaluation model, and obtaining a first single sample quality evaluation result of each sample in the target corpus sample set.

[0072] Among them, the corpus to be evaluated is the corpus database that currently needs to be evaluated for corpus quality, the target corpus sample set is a collection of multiple corpus samples randomly sampled from the corpus to be evaluated, and the first single sample quality assessment result is the corpus quality assessment result of each sample obtained by the target corpus quality assessment model.

[0073] In this embodiment, the target corpus quality assessment model is obtained by performing model alignment training on the set corpus quality assessment model. The set corpus quality assessment model can be an RM model (Reward Model), which can be used to score and sort the corpus and evaluate whether the corpus conforms to human thinking habits. In order to improve the performance and efficiency of the set corpus quality assessment model for corpus quality assessment, the initial set corpus quality assessment model can be optimized by performing model alignment training to obtain the target corpus quality assessment model. Among them, model alignment training is a technology that aims to adjust and optimize large-scale pre-trained language models to better understand human language and intentions, while avoiding potential risks, such as generating text that does not meet human expectations or being used for malicious behavior. During training, the model can be trained using the pytorch+deepspeed framework, and the model can be fine-tuned using the pre-trained model GPT. When the fine-tuning loss drops to a level that cannot be reduced or reaches a certain threshold, the RM model training is completed.

[0074] Preferably, the training method of the target corpus quality assessment model can be: obtaining a model training corpus sample set, and determining a second single sample quality assessment result of each sample in the model training corpus sample set according to a second evaluation rule; obtaining a set corpus quality assessment model and manual evaluation results for the model training corpus sample set, and performing model alignment training on the set corpus quality assessment model according to the second single sample quality assessment result and the manual evaluation result to obtain a target corpus quality assessment model.

[0075] The second evaluation rule is a rule for performing quality evaluation on a single corpus sample, and the second single sample quality evaluation result is a corpus quality evaluation result of each sample in the model training corpus sample set obtained according to the second evaluation rule.

[0076] Specifically, in order to perform model alignment training on the set corpus quality assessment model, the RLHF (Reinforcement Learning from Human Feedback) alignment learning algorithm can be used. The RLHF algorithm is a reinforcement learning technology based on human feedback, which can optimize the model to better meet human intentions and expectations. When performing model alignment training, you can first obtain a model training corpus sample set, where the model training corpus sample set can be obtained by sampling the training corpus database. Then the preset second evaluation rule evaluates each sample in the model training sample set to obtain the second single sample quality assessment result of each sample. In addition, based on the obtained model training corpus sample set, manual corpus quality assessment can also be performed through manual scoring to obtain a manual evaluation result. Finally, the rule-based second single sample quality assessment result is aligned and integrated with the manual-based manual evaluation result.

[0077] Optionally, the method for determining the second single sample quality evaluation result of each sample in the model training corpus sample set according to the second evaluation rule may be: for each sample in the model training corpus sample set, identifying the sample domain and task type corresponding to each sample respectively; determining the domain expression quality evaluation result and task context quality evaluation result of the corresponding sample according to the sample domain and task type corresponding to each sample in combination with the second evaluation rule; and taking the weighted sum of the domain expression quality evaluation result and the task context quality evaluation result as the second single sample quality evaluation result of the corresponding sample.

[0078] Specifically, the model training corpus sample set can be obtained by random sampling from the set corpus, and Sample represents the corpus sample in the model training corpus sample set. The form of Sample can be expressed as [Query|Instruction, BG_KG, Example, CoT, Answer] to facilitate the detection of subsequent indicators. Among them, Query|Instruction refers to the question or instruction that the user wants the model to answer; BG_KG refers to the domain knowledge and background information related to the instruction; Example refers to a set of data or scenario description related to the instruction, which can be a specific case, factual statement, or example of a previous problem solving; CoT refers to the thinking chain, which is the logical reasoning step when solving a problem; Answer is the solution to the query, which can be a conclusion based on background knowledge, examples, and thinking chains. Query|Instruction and Answer are required, and the other three items can be missing. The following are 3 specific Sample examples:

[0079] Table 1 Examples of three corpus samples

[0080]

[0081]

[0082] For each sample Sample in the model training corpus sample set, the sample domain D_cls and task type Task_type corresponding to each Sample can be identified. The identification method may include but is not limited to a rule-based pattern matching method, a model-based event recognition method, a graph-based event detection method, etc., which are not limited here. The second evaluation rule sets the corpus quality evaluation method for a single sample. The evaluation is divided into two aspects: domain expression quality (Domain Representation Quality, DRQ) evaluation and task context quality (Task Context Quality, TCQ) evaluation. DRQ and TCQ represent the domain expression quality evaluation results and task context quality evaluation results, respectively, and the two can be calculated based on more and lower-level evaluation indicators. After obtaining the DRQ and TCQ of each sample, the second single sample quality evaluation result of the corresponding sample can be calculated according to the weighted sum or other calculation rules, and recorded as Single_score * Taking the calculation rule of weighted sum as an example, Single_score * =DRQ×α1+TCQ×β1, where α1 and β1 are the weight coefficients of DRQ and TCQ respectively. Single_score * The calculation rules are not unique and can be adjusted according to user needs, and the embodiments of the present invention do not limit them. Taking the three samples in Table 1 as an example, identify the sample fields D_cls and task types Task_type corresponding to each sample: According to the keywords such as "diabetes" and "dietary habits" in Sample1, determine the field category D_cls1 of Sample1 = endocrine system diseases, and the task type Task_type1 = provide personalized suggestions; According to the keywords such as "hypertension" and "daily monitoring" in Sample2, determine the field category D_cls2 of Sample2 = cardiovascular diseases, and the task type Task_type2 = health monitoring guidance and providing suggestions; According to the keywords such as "fatigue" and "anemia" in Sample3, determine the field category D_cls3 of Sample3 = blood diseases, and the task type Task_type3 = symptom analysis and providing suggestions.

[0083] Furthermore, the method of determining the domain expression quality evaluation results and task context quality evaluation results of the corresponding samples according to the sample domain and task type corresponding to each sample in combination with the second evaluation rule may be: according to the sample domain corresponding to each sample and in combination with the second evaluation rule, respectively determine the terminology expression quality evaluation results, question / instruction expression completeness evaluation results, answer expression completeness evaluation results, expression structure evaluation results and expression style evaluation results of the corresponding samples; determine the domain expression quality evaluation results of the corresponding samples according to the terminology expression quality evaluation results, question / instruction expression completeness evaluation results, answer expression completeness evaluation results, expression structure evaluation results and expression style evaluation results; according to the task type corresponding to each sample and in combination with the second evaluation rule, respectively determine the background knowledge completeness evaluation results, example relevance evaluation results and logical completeness evaluation results of the corresponding samples; determine the task context quality evaluation results of the corresponding samples according to the background knowledge completeness evaluation results, example relevance evaluation results and logical completeness evaluation results.

[0084] Specifically, according to the second evaluation rule, the domain expression quality evaluation result DRQ of each sample is obtained by evaluating the five dimensions of the sample's term expression quality, question / instruction expression completeness, answer expression completeness, expression structure and expression style. The specific evaluation process is as follows:

[0085] (1) Terminology quality evaluation. Terminology quality evaluation gives a corpus score based on the accuracy, relevance, and professionalism of the term in a specific field. The purpose is to ensure that the terminology used in the corpus is consistent with the field standard. The evaluation goes through three evaluation modules: term extraction module, term matching module, and quality calculation module. The final output is the terminology quality evaluation result TERM_score of the corpus sample.

[0086] 1) Term extraction: The corpus sample is used as the input of the term extraction module. The term extraction module extracts terms from the text based on the content of the sample, and the output result is a term word set:

[0087] key_word={keyword1, keyword2,...}.

[0088] The specific implementation method can be a rule-based regularization method, a vocabulary matching method, a model-based extraction method, etc. Taking the three samples in Table 1 as an example, the term word set key_word1 of Sample1 = {diabetes, chronic metabolic diseases, hyperglycemia, carbohydrates, dietary fiber, blood sugar level}; the term word set key_word2 of Sample2 = {hypertension, cardiovascular disease, blood pressure measurement, blood pressure value}; the term word set key_word3 of Sample3 = {anemia, red blood cells, hemoglobin, vitamins}.

[0089] 2) Term matching: The term word set key_word is used as the input of the term matching module, and the matching domain term library is determined according to the sample domain D_cls of each sample identified. The term matching module calculates the similarity based on the matching of key_word and the domain term library, and the output result is the similarity similarity, which includes the similarity value of each term in key_word. The specific implementation method can be such as the semantic similarity model of WordNet, the calculation method of cosine similarity, etc. Taking the three samples in Table 1 as an example, the similarity similarity of the key_word corresponding to each sample is calculated, and the similarity similarity1 of Sample1 is similarity1={0.9, 0.8, 0.9, 0.7, 0.6, 0.8}, the similarity similarity2 of Sample2 is similarity2={0.9, 0.8, 0.9, 0.7, 0.8}, and the similarity similarity3 of Sample3 is similarity3={0.9, 0.8, 0.9, 0.6}.

[0090] 3) Quality calculation: The similarity of the term word set key_word is used as the input of the quality calculation module, and the quality calculation module calculates according to the set term expression quality rules. The specific rules can be equal division calculation or other configurable rules. If the equal division rule is used, the calculation expression of the term expression quality evaluation result is as follows:

[0091] TERM_score=AVG(similarity(keyword1)+similarity(keyword2)+...).

[0092] Taking the three samples in Table 1 as an example, the term expression quality evaluation results of the three samples are:

[0093] TERM_score1=10×(0.9+0.8+0.9+0.7+0.6+0.8) / 6=7.8,

[0094] TERM_score2=10×(0.9+0.8+0.9+0.7+0.8) / 5=8.2,

[0095] TERM_score3=10×(0.9+0.8+0.9+0.6) / 4=8.0.

[0096] (2) Question / instruction expression integrity evaluation. This evaluation goes through three modules: logical structure extraction module, knowledge representation matching module, and logical integrity calculation module, and finally outputs the question / instruction expression integrity evaluation result QueryLC_score.

[0097] 1) Logical structure extraction: The query / instruction of the sample is used as the input of the logical structure extraction module. The logical structure extraction module performs semantic relationship analysis on the query / instruction, and the output result is a logical structure representation in the form of a triple, logicForm = {(entity 1, relationship 1, entity 2), (entity 3, relationship 2, entity 4) ...}. The specific implementation method can be word segmentation and part-of-speech tagging, named entity recognition NER technology, attribute and attribute value extraction and other technologies. Taking the three samples in Table 1 as an example, the query|instruction logical structure representation of sample 1 is logicForm1 = {(I am diagnosed with diabetes)}, the query|instruction logical structure representation of sample 2 is logicForm2 = {(hypertensive patients, how to conduct daily blood pressure monitoring)}, and the query|instruction logical structure representation of sample 3 is logicForm3 = {(I feel tired), (fatigue may be a symptom of anemia)}.

[0098] 2) Knowledge representation matching: According to the sample domain D_cls of each identified sample, the matching domain knowledge graph DomainKG is determined. The knowledge representation matching module matches the entities, relations, and attributes in logicForm with the entities, relations, and attributes in DomainKG. The matching method can use similarity calculation, and the subgraph set composed of all matching nodes and relations is recorded as:

[0099] QlogicGraph={G1, G2, ....., Gn}.

[0100] Taking the three samples in Table 1 as an example, the entities and relationships in each logicForm are matched with the corresponding domain knowledge graph DomainKG, and similarity calculation is used to match entities and relationships to obtain the QlogicGraph corresponding to each Sample:

[0101] QlogicGraph1={G1, G2, G3,,G4, G5, G6},

[0102] QlogicGraph2={G1, G2, G3, G4, G5},

[0103] QlogicGraph3={G1, G2, G3}.

[0104] 3) Logical integrity calculation: The logical integrity calculation module calculates according to the set instruction expression integrity rules, and uses the calculated logical integrity score as the question / instruction expression integrity evaluation result QueryLC_score. The specific rules can be based on matching calculation, path calculation, subgraph distribution calculation or other configurable rules. If the calculation rule of subgraph segmentation status is used, the expression is as follows:

[0105] QueryLC_score=(100×Factor) / count(QLogicGraph).

[0106] That is, the more subgraphs there are, the lower the score is, where Factor is the score adjustment factor, which can be set based on experience.

[0107] Taking the three samples in Table 1 as an example, if Factor=0.4 is set, the question / instruction expression completeness evaluation result QueryLC_score corresponding to each sample can be calculated:

[0108] QueryLC_score1=(100×0.4) / 6=6.7,

[0109] QueryLC_score2=(100×0.4) / 5=8,

[0110] QueryLC_score3=(100×0.4) / 3=13.3.

[0111] (3) Answer expression completeness evaluation. This step mainly evaluates the logic and completeness of the sample Answer part in terms of logic and content compared with the standard answer. The evaluation goes through three modules: logical structure extraction module, knowledge representation matching module, and answer expression completeness calculation module, and finally outputs the answer expression completeness evaluation result AnswerLC_score. The logical structure extraction and knowledge representation matching processing steps for the answer are the same as the question / instruction expression completeness evaluation method, and finally outputs the subgraph set AlogicGraph = {G1, G2, ..., Gn}. The reason why the subsequent steps are different is that the answer needs to consider the situation of multiple solutions to one question. Therefore, the answer expression completeness calculation module evaluates AlogicGraph from two aspects: answer coverage and answer completeness:

[0112] 1) Answer coverage evaluation: For each sample, AlogicGraph is compared with the standard answer in the domain knowledge graph DomainKG, and the answer coverage AcoverDegree is obtained based on the ratio of the number of solutions covered by AlogicGraph to the number of standard answer solutions.

[0113] 2) Answer completeness evaluation: Extract the connected components of each subgraph in AlogicGraph, and obtain the answer completeness AcompleteDegree based on the ratio of the total number of nodes in the connected components to the total number of nodes in the subgraph.

[0114] The answer expression completeness calculation module calculates the answer expression completeness score based on AcoverDegree and AcompleteDegree. The specific rules can be weighted calculation or other configurable rules, and the output result is the answer expression completeness evaluation result AnswerLC_Score. For example, using the weighted calculation rule, AnswerLC_Score = AcoverDegree × 10 × α2 + AcompleteDegree × 10 × β2, where α2 and β2 are the weight coefficients of AcoverDegree and AcompleteDegree respectively.

[0115] Take the three samples in Table 1 as an example, and get the AlogicGraph corresponding to each sample:

[0116] AlogicGraph1 = {G1, G2},

[0117] AlogicGraph2={G1, G2, G3, G4, G5},

[0118] AlogicGraph3={G1, G2, G3, G4}.

[0119] Evaluate the answer coverage of each sample:

[0120] Sample1 contains 3 answers, and there are 10 standard answers in the DomainKG database. We can get AcoverDegree1 of Sample1 = 3 / 10 = 0.3. Similarly, we can get AcoverDegree2 of Sample2 = 2 / 11 = 0.18, and AcoverDegre3e of Sample3 = 2 / 18 = 0.11.

[0121] Then evaluate the completeness of the answer for each sample:

[0122] The total number of connected component nodes of Sample1 is 15, and the total number of nodes is 42. We can obtain AcompleteDegree1 of Sample1 = 0.36. Similarly, we can obtain AcompleteDegree2 of Sample2 = 0.18, and AcompleteDegree3 of Sample3 = 10 / 59 = 0.17.

[0123] The weighted calculation rule is used to calculate the answer expression completeness evaluation result AnswerLC_Score of each Sample. Assuming α2 and β2 are 40% and 60% respectively, it can be obtained that AnswerLC_score1 of Sample1 is 3.36, AnswerLC_score2 of Sample2 is 11.8, and AnswerLC_score3 of Sample3 is 1.46.

[0124] (4) Expression structure evaluation. The expression structure can evaluate the organizational quality of the text, including clarity, complexity, etc. The evaluation goes through four sub-modules: information density evaluation module, sentence structure evaluation module, vocabulary diversity evaluation module, and expression structure rating module, and finally outputs the expression structure evaluation result Stru_Grade.

[0125] 1) The information density assessment module takes Sample as input, evaluates the keyword information density based on the ratio of keyword frequency to text length, and outputs the keyword information density index info_density of Sample (values ​​range from 0 to 1). The specific implementation method can be word segmentation tools, vocabulary matching methods, and keyword extraction algorithms such as TextRank.

[0126] Take the three samples in Table 1 as an example. The keywords of Sample 1 are: diabetes (2), carbohydrate (1), sugar (1), dietary fiber (1), blood sugar level (1), and the text length is 100 words. Then the info_density1 of Sample 1 is 6 / 100 = 0.06. Similarly, the info_density2 of Sample 2 is 0.075, and the info_density3 of Sample 3 is 0.11.

[0127] 2) The sentence structure evaluation module takes Sample as input, analyzes the complexity of the sentence structure based on the sentence type, sentence length, and nesting degree of Sample, and outputs the sentence structure complexity index sent_complexity of Sample (with a value between 0 and 1). The specific implementation method can be a dependency tree, a syntactic analysis tool, etc.

[0128] Taking the three samples in Table 1 as an example, the complex sentences (such as sentences with nested structures) of Sample 1 account for 20%, then sent_complexity1=0.2, the complex sentences of Sample 2 account for 30%, then sent_complexity2=0.3, and the complex sentences of Sample 3 account for 25%, then sent_complexity3=0.25.

[0129] 3) The lexical diversity assessment module takes Sample as input, and counts the lexical diversity based on the lexical types and semantic diversity in Sample. The output result is the lexical diversity index word_diversity of Sample (with a value between 0 and 1). The specific implementation method can be type-token ratio, entropy measurement and other methods.

[0130] Taking the three samples in Table 1 as an example, Sample 1 uses 50 vocabulary types and the text length is 100 words, then word_diversity1=50 / 100=0.5. Similarly, word_diversity2=0.5 and word_diversity3=0.55.

[0131] 4) The expression structure rating module takes the info_density, sent_complexity and word_diversity of each sample as input, rates the Sample based on the set expression structure rating rules, and obtains the expression structure evaluation result Stru_Grade. The specific rules can be the indicator evaluation method or other configurable rules to calculate the expression structure score Structure_score of each sample, and then determine the corresponding expression structure grade = simple / general / complex according to the Structure_score. The expression structure evaluation results Stru_Grade corresponding to the three grades are Stru_Grade = 1 / 2 / 3 respectively. If the indicator evaluation method rule is used to calculate the expression structure score, Structure_score = info_density×α3+sent_complexity×β3+word_diversity×γ3, where α3, β3 and γ3 are the weight coefficients of info_density, sent_complexity and word_diversity respectively. Determine the expression structure evaluation result Stru_Grade according to the table shown in the following table.

[0132] Table 2 Comparison of expression structure grades and expression structure evaluation results

[0133] Structure_score Expression structure level Stru_Grade (0，0.3] Simple 1 (0.3，0.6] generally 2 (0.6，1.0] complex 3

[0134] Taking the three samples in Table 1 as an example, assuming that the weight coefficients α3=0.1, β3=0.3, and γ3=0.6, the Structure_score1 of Sample1 is 0.06×0.1+0.2×0.3+0.5×0.6=0.366, and Stru_Grade1=2; the Structure_score2 of Sample2 is 0.3975, and Stru_Grade2=2; the Structure_score3 of Sample3 is 0.416, and Stru_Grade3=2.

[0135] (5) Expression style evaluation. The style of written expression in the corpus is analyzed. The evaluation is performed by outputting the style category through the expression style classification model. The expression style classification model is trained through labeled style corpus and can adopt a traditional machine learning model or a deep learning model. In this step, Sample is used as the input of the expression style classification module. The expression style classification module classifies the expression style of Sample based on the expression style classification model. The output result is the language style style_Type of Sample and the expression style evaluation result Style_Grade. The corresponding relationship between language style and expression style evaluation result is shown in the following table.

[0136] Table 3 Comparison of language style and expression style evaluation results

[0137] Style_Type Style_Grade Spoken Language 3 formal 2 major 1

[0138] Taking the three samples in Table 1 as an example, Style_Type1 of Sample1=professional, Style_Grade1=1; Style_Type2 of Sample2=formal, Style_Grade2=2; Style_Type3 of Sample3=professional, Style_Grade3=1.

[0139] Based on the above evaluation, the domain expression quality evaluation result DRQ of the corresponding sample is determined according to the term expression quality evaluation result TERM_score, the question / instruction expression completeness evaluation result QueryLC_score, the answer expression completeness evaluation result AnswerLC_score, the expression structure evaluation result Stru_Grade and the expression style evaluation result Style_Grade. The DRQ calculation module calculates the total score according to the set domain expression quality rules. The specific rules can be average calculation or other configurable rules. The domain expression quality evaluation result is recorded as DRQ. If the average rule is used, DRQ = AVG (TERM_Score + Query_score + Answer_score + 100 / Stru_Grade + 100 / Style_Grade). The above rules can be adjusted according to the specific needs of the user to meet the needs of training different large models, and the embodiments of the present invention are not limited.

[0140] Taking the three samples in Table 1 as an example, the DRQ of the samples is calculated using the equal division rule, then DRQ1 of Sample 1 = AVG (7.8 + 6.7 + 3.36 + 100 / 2 + 100 / 1) = 33.572; similarly, DRQ2 of Sample 2 = 23.6; DRQ3 of Sample 3 = 34.552.

[0141] Task context refers to the information description that helps solve the task. As part of the prompt, it significantly affects the ability of the large model to solve specific tasks. According to the second evaluation rule, the task context quality evaluation result TCQ of each sample can be obtained by evaluating the three dimensions of background knowledge completeness, example relevance and logical completeness of the sample. The specific evaluation process is as follows:

[0142] (1) Background knowledge completeness evaluation. This evaluation can measure the completeness and consistency of the background knowledge provided by the sample in supporting specific domain tasks. The evaluation goes through five modules: keyword extraction module, relationship identification module, semantic network richness analysis module, consistency verification module and background knowledge completeness score calculation module, and finally outputs the background knowledge completeness evaluation result BK_score of the corpus sample.

[0143] 1) Keyword extraction: Sample's BG_KG is used as the input of the keyword extraction module. The keyword extraction module identifies and extracts key terms and concepts from the text. The output result is a term set key_concepts = {concept1, concept2, ...}. The specific implementation method can be rule-based regular expressions, dictionary matching, machine learning-based extraction technology, etc.

[0144] Taking the three samples in Table 1 as examples, the term set key_concepts1 extracted from BG_KG of Sample1 is key_concepts1 = {diabetes, chronic metabolic diseases, hyperglycemia, carbohydrates}; the term set key_concepts2 extracted from BG_KG of Sample2 is key_concepts2 = {hypertension, cardiovascular disease, blood pressure monitoring}; the term set key_concepts3 extracted from BG_KG of Sample3 is key_concepts3 = {anemia, red blood cells, hemoglobin, fatigue, weakness, dizziness, iron deficiency}.

[0145] 2) Relationship identification: key_concepts is used as the input of the relationship identification module. The relationship identification module determines the connection between concepts by analyzing the semantic relationship in the text, and the output result is a preliminary semantic network: semantic_network = {(concept1, relationship1, concept2), (concept3, relationship2, concept4), ...}. The specific implementation method can be dependency syntax analysis, semantic role labeling, relationship inference based on knowledge graph, etc.

[0146] Taking the three samples in Table 1 as examples, the semantic network semantic_network1 of Sample1 = {(diabetes is a chronic metabolic disease), (high blood sugar requires strict dietary control)}; the semantic network semantic_network2 of Sample2 = {(hypertension is a cardiovascular disease), (blood pressure monitoring is used to evaluate treatment effects)}; the semantic network semantic_network3 of Sample3 = {(anemia is caused by low red blood cell or hemoglobin content), (fatigue is a symptom of anemia)}.

[0147] 3) Semantic network richness analysis: semantic_network is used as the input of the semantic network richness analysis module. The semantic network richness analysis module scores the semantic richness (ranging from 0 to 10) based on the characteristics of node connectivity, node clustering coefficient, relationship diversity, etc. in the network, and the output result is recorded as richness.

[0148] Taking the three samples in Table 1 as an example, the richness score of Sample 1 is richness1=7; similarly, the richness score of Sample 2 is richness2=6, and the richness score of Sample 3 is richness3=8.

[0149] 4) Consistency verification: semantic_network is used as the input of the consistency verification module. The steps of this module are the same as the question / instruction expression completeness evaluation in the domain expression quality evaluation result DRQ, and the final output is the consistency score between the background knowledge and the industry knowledge base.

[0150] Taking the three samples in Table 1 as an example, the semantic network is matched with the industry knowledge base, and the consistency is evaluated. The consistency score of Sample1 is consistency1=9. Similarly, the consistency score of Sample2 is consistency2=8, and the consistency score of Sample3 is consistency3=9.

[0151] 5) Background knowledge completeness score calculation: The richness score and consistency score are used as inputs of the background knowledge completeness score calculation module. The background knowledge completeness score calculation module calculates the score according to the set background knowledge completeness calculation rules. The specific rules can be average calculation or other configurable rules, etc. The output result is the background knowledge completeness evaluation result BK_score. If the average rule is used, BK_score = AVG (richness + consistency).

[0152] Taking the three samples in Table 1 as an example, BK_score1 of Sample 1=(7+9) / 2=8, BK_score2 of Sample 2=7, and BK_score3 of Sample 3=8.5.

[0153] (2) Example relevance evaluation. The purpose of this evaluation is to ensure that the examples in the sample are closely related to the query / instruction and can properly support the understanding and answering of the question or instruction. This evaluation goes through three steps: example extraction, instruction matching, and comprehensive scoring, and finally outputs the example relevance evaluation result ES_score.

[0154] 1) Example extraction: Take the Example of Sample as the input of the example extraction module. The example extraction module parses the content of Example, extracts key information and context, and outputs the example feature set Example_features = {feature1, feature2, ...}. The specific implementation method can be a rule-based regular method, a vocabulary matching method, etc.

[0155] Taking the three samples in Table 1 as examples, Example_features1 of Sample1 = {reduce sugar intake, reduce refined carbohydrate intake, increase dietary fiber, and control blood sugar levels}, Example_features2 of Sample2 = {measure blood pressure every morning and evening, record blood pressure changes, and report to the doctor}, and Example_features3 of Sample3 = {iron deficiency leads to anemia, fatigue and dizziness, supplement iron and vitamins, and symptoms improve}.

[0156] 2) Instruction matching: Example_features is used as the input of the instruction matching module. The instruction matching module calculates the similarity based on the matching between Example_features and the corresponding Sample's Query / Instruction. The output result is recorded as a similarity score set Esimilarity_scores = {score1, score2, ...}. The specific implementation method can be keyword co-occurrence analysis, vector similarity calculation in semantic embedding space, etc.

[0157] Taking the three samples in Table 1 as an example, the similarity scores corresponding to the features in Example_features1 of Sample1 are: reducing sugar intake - 0.9, reducing refined carbohydrate intake - 0.8, increasing dietary fiber - 0.7, and controlling blood sugar levels - 0.9, so Sample1's Esimilarity_scores1 = {0.9, 0.8, 0.7, 0.9}. Similarly, Sample2's Esimilarity_scores2 = {0.8, 0.7, 0.8}, and Sample3's Esimilarity_scores3 = {0.9, 0.8, 0.7, 0.6}.

[0158] 3) Comprehensive scoring: Esimilarity_scores is used as the input of the comprehensive scoring module, which calculates the score according to the set example fit evaluation rules. The specific rules can be average score calculation or other configurable rules, etc. The output result is recorded as ES_score, that is, the example fit evaluation result. If the average score rule is used, ES_score = AVG (Esimilarity_scores).

[0159] Taking the three samples in Table 1 as an example, ES_score1 of Sample 1 = (0.9 + 0.8 + 0.7 + 0.9) / 4 = 0.8. Similarly, ES_score2 of Sample 2 = 0.767; ES_score3 of Sample 3 = 0.75.

[0160] (3) Logical integrity assessment. This assessment can ensure that the logical chain of the Sample is consistent and complete with the logic of the industry knowledge base. The specific steps of this assessment include logical element extraction, logical matching, logical integrity analysis and comprehensive scoring, and finally output the logical integrity assessment result LI_score.

[0161] 1) Logical element extraction: Sample's CoT is used as the input of the logical element extraction module. The logical element extraction module identifies and extracts key logical elements from CoT, such as assumptions, premises, conclusions, etc. The output result is a logical element set logic_elements = {element1, element2, ...}. The specific implementation method can be a rule-based regularization method, a keyword extraction algorithm, etc.

[0162] Taking the three samples in Table 1 as an example, logic_elements1 of Sample1 = {If you are diagnosed with diabetes, you need to change your eating habits, reduce your intake of sugar and refined carbohydrates, and increase your intake of dietary fiber}; logic_elements2 of Sample2 = {Patients with hypertension need daily blood pressure monitoring, measure their blood pressure at a fixed time every day, and contact a doctor if their blood pressure remains abnormal}; logic_elements3 of Sample3 = {Feeling tired may be a symptom of anemia, which is caused by iron deficiency. Supplementing with iron and vitamins can improve symptoms}.

[0163] 2) Logical matching: logic_elements is used as the input of the logical matching module. The logical matching module calculates the similarity based on the matching between logic_elements and the logical elements in the industry knowledge base, and the output result is the similarity score set Csimilarity_scores. The specific implementation method can be dependency syntax analysis, vector matching in semantic embedding space, etc.

[0164] Taking the three samples in Table 1 as an example, Csimilarity_scores1 of Sample 1 = {0.9, 0.8, 0.9, 0.7}; Csimilarity_scores2 of Sample 2 = {0.8, 0.9, 0.7}; Csimilarity_scores3 of Sample 3 = {0.8, 0.9, 0.6}.

[0165] 3) Logical integrity analysis: CoT is used as the input of the logical integrity analysis module. The logical integrity analysis module scores whether there is a logical break in CoT and the logical integrity, and the output result is the integrity score completeness_score.

[0166] Taking the three samples in Table 1 as an example, since the logic chain of Sample 1 is complete and unbroken, completeness_score1 of Sample 1 is 8. Similarly, completeness_score2 of Sample 2 is 7, and completeness_score3 of Sample 3 is 8.

[0167] 4) Comprehensive scoring: Csimilarity_scores and completeness_score are used as inputs of the comprehensive scoring module, which is calculated according to the set logic completeness scoring rules. The rules can be weighted calculations or other configurable rules, and the output result is recorded as LI_score, i.e., the logic completeness evaluation result. If the weighted calculation rule is used, LI_score = AVG (Csimilarity_scores) × α4 + completeness_score × β4, where α4 and β4 are the weight coefficients of Csimilarity_scores and completeness_score, respectively.

[0168] Taking the three samples in Table 1 as an example, assuming that the weight coefficients α4=0.5, β4=0.5, the LI_score1 of Sample1=(0.9+0.8+0.9+0.7) / 4×0.5+8×0.5=0.8×0.5+4=4+2=6, the LI_score2 of Sample2=5.75, and the LI_score3 of Sample3=5.6665.

[0169] Based on the above evaluation, the task context quality evaluation result TCQ of the corresponding sample is determined according to the background knowledge completeness evaluation result BK_score, the example relevance evaluation result ES_score and the logic integrity evaluation result LI_score. The TCQ calculation module obtains the total score according to the pre-set task context quality rules. The specific rules can be average calculation or other configurable rules, and the output result is recorded as TCQ. If the average rule is used, TCQ = AVG (BK_score + ES_score + LI_score).

[0170] Taking the three samples in Table 1 as an example, TCQ1 of Sample 1 = AVG (8 + 0.8 + 6) = 4.93, and similarly, TCQ2 of Sample 2 = 4.506, and TCQ3 of Sample 3 = 4.972. The second single sample quality evaluation result Single_score of each sample is calculated using the weighted sum calculation rule. * , Single_score of Sample1 *1=DRQ1×α1+TCQ1×β1=AVG(33.572+4.93)=19.251, Single_score of Sample2 * 2=14.053, Single_score of Sample3 * 3=39.524.

[0171] S120: Determine an overall quality evaluation result of the target corpus sample set according to the first evaluation rule.

[0172] The first evaluation rule is a rule for overall quality evaluation of the corpus sample set.

[0173] In this embodiment, after obtaining the first single sample quality evaluation result of each sample in the target corpus sample set using the target corpus quality evaluation model, the target corpus sample set can be evaluated for overall corpus quality according to the first evaluation rule. When evaluating the overall sample set quality of the target corpus sample set, the richness and differences of different samples in the target corpus sample set in terms of content, form, style, etc. can be considered. Corpus sample sets with high diversity quality can better represent the diversity in the real world and help train more robust and generalized artificial intelligence models.

[0174] Optionally, the first evaluation rule may specify a scoring method for the corpus sample set in dimensions such as domain coverage, task coverage, expression diversity, and context combination diversity, and finally obtain an overall quality evaluation result of the sample set based on the scores of each dimension by weighted summation or the like.

[0175] S130: Determine a final corpus quality assessment result of the corpus to be assessed based on a first single sample quality assessment result of each sample in the target corpus sample set and an overall quality assessment result of the sample set.

[0176] In this embodiment, according to sampling from the corpus to be evaluated to obtain a target corpus sample set, each sample in the target corpus sample set can be evaluated using the target corpus quality evaluation model to obtain the first single sample quality evaluation result of each sample; at the same time, using the first evaluation rule, the target corpus sample set can be evaluated overall to obtain the overall quality evaluation result of the sample set. According to the first single sample quality evaluation result and the overall quality evaluation result of the sample set, combined with the set calculation rules, a final score can be calculated as the final corpus quality evaluation result of the corpus to be evaluated.

[0177] Optionally, the calculation rule of the final corpus quality assessment result can be an average calculation or a weighted calculation or other configurable rules, which are not limited here. The first single sample quality assessment result is recorded as Single_score, the overall quality assessment result of the sample set is recorded as Multi_score, and the final corpus quality assessment result is recorded as Final_score. If the weighted calculation rule is used, Final_score = AVG (Single_score) × α + Multi_score × β, where α and β are the weight coefficients of Single_score and Multi_score respectively.

[0178] The corpus quality assessment method for multi-level task alignment provided by the embodiment of the present invention first samples a target corpus sample set from the corpus to be evaluated, inputs each sample in the target corpus sample set into the target corpus quality assessment model, and obtains the first single sample quality assessment result of each sample in the target corpus sample set; wherein the target corpus quality assessment model is obtained by setting the corpus quality assessment model for model alignment training; then the overall quality assessment result of the sample set of the target corpus sample set is determined according to the first assessment rule; finally, the final corpus quality assessment result of the corpus to be evaluated is determined according to the first single sample quality assessment result of each sample in the target corpus sample set and the overall quality assessment result of the sample set. The corpus quality assessment method for multi-level task alignment disclosed by the present invention not only evaluates the corpus quality of each sample in the sample set, but also considers the overall quality of the sample set. When performing single sample evaluation, the corpus quality assessment model obtained through model alignment training is adopted, which can deepen the model's understanding of professional field tasks, thereby making the final corpus quality assessment more comprehensive and accurate.

[0179] Embodiment 2

[0180] Figure 2 Flow chart of a multi-level task alignment corpus quality assessment method provided in Embodiment 2 of the present invention. This embodiment is a refinement of the above embodiment. Figure 2 As shown, the method includes:

[0181] S210: sampling a target corpus sample set from the corpus to be evaluated, inputting each sample in the target corpus sample set into a target corpus quality evaluation model, and obtaining a first single sample quality evaluation result of each sample in the target corpus sample set.

[0182] Among them, the target corpus quality assessment model is obtained by performing model alignment training on the set corpus quality assessment model.

[0183] In this embodiment, the target corpus sample set is a collection of multiple corpus samples randomly sampled from the corpus to be evaluated. The set corpus quality assessment model can be an RM model (Reward Model), which can be used to score and sort the corpus and evaluate whether the corpus conforms to human thinking habits. In order to improve the performance and efficiency of the set corpus quality assessment model for corpus quality assessment, the initial set corpus quality assessment model can be optimized by model alignment training to obtain the target corpus quality assessment model. Among them, model alignment training is a technology that aims to better understand human language and intentions by adjusting and optimizing large-scale pre-trained language models, while avoiding potential risks, such as generating text that does not meet human expectations or being used for malicious behavior. During training, the model can be trained using the pytorch+deepspeed framework, and the model can be fine-tuned using the pre-trained model GPT. When the fine-tuning loss drops to a level that cannot be reduced or reaches a certain threshold, the RM model training is completed. Based on the trained set corpus quality assessment model, each sample in the target corpus sample set is input into the model, and the first single sample quality assessment result of each sample in the target corpus sample set can be obtained.

[0184] Specifically, a target corpus sample set is obtained by random sampling from the corpus to be evaluated. Table 4 is an example of a target corpus sample set, which contains 3 corpus samples. The format of each sample Sample can be expressed as [Query|Instruction, BG_KG, Example, CoT, Answer]. Among them, Query|Instruction refers to the question or instruction that the user wants the model to answer; BG_KG refers to the domain knowledge and background information related to the instruction; Example refers to a set of data or scenario description related to the instruction, which can be a specific case, factual statement or example of a previous problem solving; CoT refers to the thinking chain, which is the logical reasoning step when solving a problem; Answer is the solution to the query, which can be a conclusion based on background knowledge, examples and thinking chain.

[0185] Table 4 Example of target corpus sample set

[0186]

[0187]

[0188]

[0189] S220: For each sample in the target corpus sample set, sample domain identification and task type identification are performed to determine the domain category and task type of the corresponding sample.

[0190] In this embodiment, for each sample in the target corpus sample set, the domain category and task type corresponding to the sample can be determined respectively.

[0191] Specifically, when performing sample domain identification, the corpus sample Sample can be used as the input of the event identification module. The event identification module analyzes the sample domain and task category based on the content of Sample, and the output result is the domain category D_cls and the task type Task_type. Optionally, the specific implementation method of the event identification module can be a rule-based pattern matching method, a model-based event identification method, a graph-based event detection method, etc., which is not limited here.

[0192] Taking the target corpus sample set in Table 4 as an example, after sample domain identification and task type identification, the domain category and task type of each sample can be obtained as follows: Sample 1: D_cls1 = chronic disease, Task_type1 = cause analysis, treatment suggestions; Sample 2: D_cls2 = musculoskeletal system diseases, Task_type2 = concept explanation, treatment suggestions; Sample 3: D_cls3 = ENT diseases, Task_type3 = symptom analysis, treatment suggestions.

[0193] S230: Determine the domain coverage evaluation result and the task coverage evaluation result of the target corpus sample set respectively according to the domain category and task type of each sample and in combination with the first evaluation rule.

[0194] The first evaluation rule is a rule for overall quality evaluation of the corpus sample set.

[0195] In this embodiment, when the overall corpus quality evaluation of the target corpus sample set is performed, the diversity quality of the target corpus sample set is mainly considered. The diversity quality of the sample set refers to the richness and difference of different samples in a corpus in terms of content, form, style, etc. Corpus with high diversity quality is more representative of the diversity in the real world and helps to train more robust and generalized artificial intelligence models. The sample set diversity evaluation evaluates the sample set quality diversity from four aspects: domain coverage, task coverage, expression diversity, and context combination diversity. The output result is the sample set diversity score, that is, the overall quality evaluation result of the sample set.

[0196] Specifically, when evaluating the domain coverage of the target corpus sample set, the purpose of the evaluation is to measure the coverage breadth and diversity of the target corpus sample set in a specific field. According to the domain category D_cls of each sample, the total number of samples quantity and the number of domain categories species in the target corpus sample set can be counted, and the domain coverage score DC_score is calculated according to the set rules as the domain coverage evaluation result. Optionally, the calculation rule of the domain coverage evaluation result can be: DC_score = 10×species / quantity.

[0197] Taking the target corpus sample set provided in Table 4 as an example, the number of domain categories species=3 (chronic diseases, musculoskeletal system diseases, and ENT diseases) and the number of samples quantity=3 are obtained, and the domain coverage DC_score=10×3 / 3=10.

[0198] Furthermore, when evaluating the task coverage of the target corpus sample set, the purpose of the evaluation is to measure the coverage breadth and diversity of the Sample set in a specific task type. According to the task type Task_type of each sample, the number of task types task_count in the target corpus sample set can be counted, and the total number of all possible task types total_tasks in a specific field can be counted, and the task coverage score TC_score is calculated according to the set rules as the task coverage evaluation result. Optionally, the calculation rule of the task coverage evaluation result can be: TC_score = 100 × task_count / total_tasks.

[0199] Taking the target corpus sample set provided in Table 4 as an example, the total number of task types in the sample is counted as task_count=4, and the total number of possible tasks in the domain knowledge base is counted as total_tasks=48, then TC_score=100×4 / 48=6.25.

[0200] S240: Determine, according to the first evaluation rule, an expression form diversity evaluation result and a context combination diversity evaluation result of the target corpus sample set.

[0201] In this embodiment, in order to perform an overall corpus quality assessment on the target corpus sample set, after calculating the domain coverage assessment results and task coverage assessment results of the target corpus sample set, it is also necessary to calculate the expression diversity assessment results and the context combination diversity assessment results.

[0202] Specifically, the diversity of expression forms measures the diversity of language usage in the sample set, which can help the evaluator understand the diversity of the expression forms of the sample set. The steps for evaluating the diversity of expression forms of the target corpus sample set are as follows:

[0203] (1) Classification and distance calculation: First, classify the samples according to their task type Task_type, and evaluate the expression structure and expression style of each sample to obtain the expression structure level and expression style level of each sample. For samples of the same task type, count the proportion of each expression structure level Stru_proportion = x1:y1:z1 (representing the number ratio of samples at the three levels of "simple, general, and complex" respectively) and the proportion of each expression style level Style_proportion = x2:y2:z2 (representing the number ratio of samples at the three levels of "oral, formal, and professional" respectively), and then calculate the distance between these two ratios and 1:1:1 according to the set rules. Optionally, the specific calculation rule can be the Euclidean distance method or other configurable rules, which are not limited here. The distance calculation results of each type are recorded as Distance = {(distance11,distance12), (distance21,distance22),...}, where distance [seq][type] [seq] represents the task type sequence of the sub-dataset, [type] is the distance type, 1 represents the expression structure distance, and 2 represents the expression style distance. If the distance is calculated using the Euclidean distance rule, first divide x, y, and z by the smallest number among x, y, and z, and then use the formula calculate.

[0204] Taking the target corpus sample set in Table 4 as an example, there are 4 task types in 3 samples, namely: cause analysis, treatment advice, concept explanation and symptom analysis. The cause analysis type consists of Sample1, the treatment advice type consists of Sample1, Sample2 and Sample3, the concept explanation consists of Sample2, and the symptom analysis consists of Sample3, so only the expression form diversity of the treatment advice type needs to be analyzed. If the expression structure level of the 3 samples is the same, all at level 2, then Stru_proportion = 1:1:1, and the expression structure diversity is 0; if the expression style levels of the 3 samples are 1, 2, and 1 respectively, the number of level 1 samples is 2, and the number of level 2 samples is 1, then Style_proportion = 2:1:0.

[0205] (2) Classification scoring: Calculate the expression diversity score of each category according to the set rules. The specific rules can be average calculation or other configurable rules. If the average calculation rule is used, the score of each type i =AVG(10 / distance i1 +10 / diatance i2), let SED_score = {score1, score2, ...}, and calculate the expression diversity evaluation result ED_score of the target corpus sample set according to the set rules. If the average calculation rule is used, ED_score = AVG (SED_score).

[0206] Furthermore, when evaluating the context combination diversity of the target corpus sample set, the number of context form types CCD_count contained in the target corpus sample set can be counted, and the context combination diversity evaluation result CCD_score can be obtained. The specific implementation method can be a text analysis tool, a statistical analysis method, etc. Preferably, if a statistical method is used, the score can be assigned according to the following table.

[0207] Table 5 Context combination diversity scoring table

[0208] CCD_count CCD_score 0 0 1 3 2 6 3 10

[0209] S250, determining the overall quality assessment result of the sample set of the target corpus sample set according to the domain coverage assessment result, the task coverage assessment result, the expression diversity assessment result and the context combination diversity assessment result.

[0210] In this embodiment, after obtaining the domain coverage evaluation result DC_score, the task coverage evaluation result TC_score, the expression diversity evaluation result ED_score and the context combination diversity evaluation result CCD_score, the sample set overall quality evaluation result Multi_score of the slogan sample set can be further calculated. The specific implementation method can be equal division calculation or weighted calculation or other configurable rules.

[0211] Optionally, if the equal division rule is used, Multi_score=AVG(DC_score+TC_score+ED_score+CCD_score). The above rule is not unique, and the user can adjust the rule according to the needs of the actual task, which is not limited here.

[0212] S260: Determine the average of the first single sample quality assessment results of each sample in the target corpus sample set as the average single sample quality assessment result of the target corpus sample set, and perform weighted summation on the average single sample quality assessment result and the overall quality assessment result of the sample set to obtain the final corpus quality assessment result of the corpus to be evaluated.

[0213] In this embodiment, the target corpus sample set is sampled from the corpus to be evaluated, the first single sample quality evaluation result of each sample in the target corpus sample set is obtained using the target corpus quality evaluation model, and the overall sample set quality evaluation result of the target corpus sample set is obtained using the first evaluation rule. To calculate the final corpus quality evaluation result of the corpus to be evaluated, the average of the first single sample quality evaluation result of each sample in the target corpus sample set can be calculated first, and then weighted summed with the overall quality evaluation result of the sample set.

[0214] Specifically, the quality assessment result of the first single sample is recorded as Single_score, the overall quality assessment result of the sample set is recorded as Multi_score, and the final corpus quality assessment result is recorded as Final_score. If the weighted calculation rule is used, Final_score = AVG (Single_score) × α + Multi_score × β, where α and β are the weight coefficients of Single_score and Multi_score, respectively.

[0215] The multi-level task-aligned corpus quality assessment method provided in the embodiment of the present invention not only evaluates the corpus quality of each sample in the sample set, but also considers the overall quality of the sample set. When performing single sample evaluation, a corpus quality assessment model obtained through model alignment training is used to deepen the model's understanding of professional field tasks. When performing overall quality assessment of the sample set, it is evaluated from multiple dimensions according to the set evaluation rules, thereby making the final corpus quality assessment more comprehensive and accurate.

[0216] Embodiment 3

[0217] Figure 3 A schematic diagram of the structure of a corpus quality assessment device for multi-level task alignment provided in the third embodiment of the present invention is shown in FIG. Figure 3 As shown, the device includes: a first single sample quality assessment result determination module 310, a sample set overall quality assessment result determination module 320 and a final corpus quality assessment result determination module 330.

[0218] The first single sample quality assessment result determination module 310 is used to sample a target corpus sample set from the corpus to be assessed, input each sample in the target corpus sample set into a target corpus quality assessment model, and obtain the first single sample quality assessment result of each sample in the target corpus sample set; wherein the target corpus quality assessment model is obtained by performing model alignment training on a set corpus quality assessment model.

[0219] The sample set overall quality evaluation result determination module 320 is used to determine the sample set overall quality evaluation result of the target corpus sample set according to the first evaluation rule.

[0220] The final corpus quality evaluation result determination module 330 is used to determine the final corpus quality evaluation result of the corpus to be evaluated according to the first single sample quality evaluation result of each sample in the target corpus sample set and the overall quality evaluation result of the sample set.

[0221] Optionally, the device further includes a model training module 340, which is used to:

[0222] A model training corpus sample set is obtained, and a second single sample quality assessment result of each sample in the model training corpus sample set is determined according to a second evaluation rule; a set corpus quality assessment model and a manual evaluation result for the model training corpus sample set are obtained, and model alignment training is performed on the set corpus quality assessment model according to the second single sample quality assessment result and the manual evaluation result to obtain a target corpus quality assessment model.

[0223] Optionally, the model training module 340 is further used to:

[0224] For each sample in the model training corpus sample set, the sample domain and task type corresponding to each sample are identified respectively; according to the sample domain and task type corresponding to each sample, the domain expression quality evaluation result and the task context quality evaluation result of the corresponding sample are determined in combination with the second evaluation rule; the weighted sum of the domain expression quality evaluation result and the task context quality evaluation result is taken as the second single sample quality evaluation result of the corresponding sample.

[0225] Optionally, the model training module 340 is further used to:

[0226] According to the sample domain corresponding to each sample and in combination with the second evaluation rule, the terminology expression quality evaluation results, question / instruction expression completeness evaluation results, answer expression completeness evaluation results, expression structure evaluation results and expression style evaluation results of the corresponding sample are determined respectively; the domain expression quality evaluation results of the corresponding sample are determined according to the terminology expression quality evaluation results, question / instruction expression completeness evaluation results, answer expression completeness evaluation results, expression structure evaluation results and expression style evaluation results; according to the task type corresponding to each sample and in combination with the second evaluation rule, the background knowledge completeness evaluation results, example relevance evaluation results and logical completeness evaluation results of the corresponding sample are determined respectively; the task context quality evaluation results of the corresponding sample are determined according to the background knowledge completeness evaluation results, example relevance evaluation results and logical completeness evaluation results.

[0227] Optionally, the sample set overall quality assessment result determination module 320 is further used to:

[0228] For each sample in the target corpus sample set, sample domain identification and task type identification are performed respectively to determine the domain category and task type of the corresponding sample; according to the domain category and task type of each sample and in combination with the first evaluation rule, the domain coverage evaluation results and task coverage evaluation results of the target corpus sample set are determined respectively; according to the first evaluation rule, the expression diversity evaluation results and context combination diversity evaluation results of the target corpus sample set are determined; according to the domain coverage evaluation results, the task coverage evaluation results, the expression diversity evaluation results and the context combination diversity evaluation results, the overall sample set quality evaluation result of the target corpus sample set is determined.

[0229] Optionally, the final corpus quality assessment result determination module 330 is further used to:

[0230] The average value of the first single sample quality assessment result of each sample in the target corpus sample set is determined as the average single sample quality assessment result of the target corpus sample set; the average single sample quality assessment result and the overall quality assessment result of the sample set are weighted summed to obtain the final corpus quality assessment result of the corpus to be evaluated.

[0231] The multi-level task-aligned corpus quality assessment device provided in the embodiment of the present invention can execute the multi-level task-aligned corpus quality assessment method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0232] Embodiment 4

[0233] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0234] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0235] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0236] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a corpus quality assessment method for multi-level task alignment.

[0237] In some embodiments, the corpus quality assessment method for multi-level task alignment may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the corpus quality assessment for multi-level task alignment described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the corpus quality assessment method for multi-level task alignment in any other appropriate manner (e.g., by means of firmware).

[0238] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0239] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0240] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0241] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0242] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0243] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0244] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0245] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-level task alignment corpus quality assessment method, characterized in that: include: A target corpus sample set is obtained by sampling from the corpus to be evaluated, and each sample in the target corpus sample set is input into a target corpus quality assessment model to obtain a first single sample quality assessment result of each sample in the target corpus sample set; wherein the target corpus quality assessment model is obtained by performing model alignment training on a set corpus quality assessment model; Determine the overall quality evaluation result of the sample set of the target corpus sample set according to the first evaluation rule; The final corpus quality evaluation result of the corpus to be evaluated is determined according to the first single sample quality evaluation result of each sample in the target corpus sample set and the overall quality evaluation result of the sample set.

2. The method according to claim 1, characterized in that The training method of the target corpus quality assessment model includes: Obtain a model training corpus sample set, and determine a second single sample quality evaluation result of each sample in the model training corpus sample set according to a second evaluation rule; The set corpus quality assessment model and the manual assessment results for the model training corpus sample set are obtained, and model alignment training is performed on the set corpus quality assessment model according to the second single sample quality assessment result and the manual assessment result to obtain the target corpus quality assessment model.

3. The method according to claim 2, characterized in that Determining a second single sample quality evaluation result of each sample in the model training corpus sample set according to the second evaluation rule includes: For each sample in the model training corpus sample set, respectively identifying the sample field and task type corresponding to each sample; According to the sample domain and task type corresponding to each sample, the domain expression quality evaluation result and the task context quality evaluation result of the corresponding sample are determined in combination with the second evaluation rule; A weighted sum of the domain expression quality assessment result and the task context quality assessment result is used as the second single sample quality assessment result of the corresponding sample.

4. The method according to claim 3, characterized in that: According to the sample domain and task type corresponding to each sample, the domain expression quality evaluation result and the task context quality evaluation result of the corresponding sample are determined in combination with the second evaluation rule, including: According to the sample fields corresponding to each sample and in combination with the second evaluation rule, respectively determine the term expression quality evaluation result, question / instruction expression completeness evaluation result, answer expression completeness evaluation result, expression structure evaluation result and expression style evaluation result of the corresponding sample; Determine the domain expression quality assessment result of the corresponding sample according to the term expression quality assessment result, the question / instruction expression completeness assessment result, the answer expression completeness assessment result, the expression structure assessment result and the expression style assessment result; According to the task type corresponding to each sample, combined with the second evaluation rule, respectively determine the background knowledge completeness evaluation result, the example relevance evaluation result and the logic completeness evaluation result of the corresponding sample; The task context quality assessment result of the corresponding sample is determined according to the background knowledge completeness assessment result, the example suitability assessment result and the logic completeness assessment result.

5. The method according to claim 1, characterized in that Determining the overall quality evaluation result of the sample set of the target corpus sample set according to the first evaluation rule includes: For each sample in the target corpus sample set, sample domain identification and task type identification are performed to determine the domain category and task type of the corresponding sample; Determine the domain coverage evaluation result and the task coverage evaluation result of the target corpus sample set respectively according to the domain category and task type of each sample and in combination with the first evaluation rule; Determining, according to the first evaluation rule, an expression form diversity evaluation result and a context combination diversity evaluation result of the target corpus sample set; The sample set overall quality assessment result of the target corpus sample set is determined according to the domain coverage assessment result, the task coverage assessment result, the expression form diversity assessment result and the context combination diversity assessment result.

6. The method according to claim 1, characterized in that Determining a final corpus quality assessment result of the corpus to be assessed according to a first single sample quality assessment result of each sample in the target corpus sample set and an overall quality assessment result of the sample set includes: Determine the average of the first single sample quality evaluation results of each sample in the target corpus sample set as the average single sample quality evaluation result of the target corpus sample set; The average single sample quality assessment result and the sample set overall quality assessment result are weightedly summed to obtain the final corpus quality assessment result of the corpus to be assessed.

7. A multi-level task alignment corpus quality assessment device, characterized in that: include: A first single sample quality assessment result determination module is used to sample a target corpus sample set from the corpus to be assessed, input each sample in the target corpus sample set into a target corpus quality assessment model, and obtain a first single sample quality assessment result of each sample in the target corpus sample set; wherein the target corpus quality assessment model is obtained by performing model alignment training on a set corpus quality assessment model; A sample set overall quality evaluation result determination module, used to determine the sample set overall quality evaluation result of the target corpus sample set according to the first evaluation rule; The final corpus quality evaluation result determination module is used to determine the final corpus quality evaluation result of the corpus to be evaluated according to the first single sample quality evaluation result of each sample in the target corpus sample set and the overall quality evaluation result of the sample set.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the corpus quality assessment method for multi-level task alignment according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the corpus quality assessment method for multi-level task alignment according to any one of claims 1 to 6 when executed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the multi-level task aligned corpus quality assessment method according to any one of claims 1 to 6 are implemented.

Images