A method and device for determining the quality of question and answer data labeling for large language model training

Abstract

The present disclosure relates to the technical field of computers, and discloses a method and device for determining the quality of question and answer data labeling for large language model training. The method comprises: obtaining a first labeling result of to-be-labeled data; determining a first statistical analysis result of a first scoring value corresponding to the first labeling result; performing vectorization processing on the first labeling result to obtain a vector set after vectorization; determining a second labeling result and a second statistical analysis result of a second scoring value corresponding to the second labeling result according to the vector set; and determining a target quality evaluation type for quality evaluation of the first labeling result according to the first statistical analysis result and the second statistical analysis result. The present disclosure solves the defect that the quality of data labeling cannot be guaranteed by manual labeling.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and specifically to a method and apparatus for determining the quality of question-answer data annotations used in training large language models. Background Technology

[0002] With the rapid development of computer artificial intelligence, a high-performing machine learning model requires a large amount of high-quality labeled data to support it. Therefore, ensuring the high quality of labeled data has become a key aspect of the model training process.

[0003] Related technologies typically employ methods to ensure the quality of labeled data, such as self-inspection, cross-checking, voting, and client acceptance. However, these methods all have limitations. For example, over-reliance on the work attitude and knowledge of the data labelers can lead to excessively high labor costs and very low labeling efficiency.

[0004] Therefore, the quality of data annotation cannot be effectively guaranteed by manual annotation. There is an urgent need for a way to quickly and accurately assess the quality of data annotation. Summary of the Invention

[0005] In view of this, this disclosure provides a method and apparatus for determining the quality of question-answer data annotations used in training large language models, in order to solve the problem that the quality of data annotation cannot be effectively guaranteed by manual annotation, and there is an urgent need for a way to quickly and accurately evaluate the quality of data annotation.

[0006] Firstly, this disclosure provides a method for determining the quality of question-answering data annotations used in training large language models, the method comprising:

[0007] Obtain the first annotation result of the data to be annotated;

[0008] The first statistical analysis result determines the first score value corresponding to the first annotation result, wherein the first statistical analysis result is the evaluation parameter obtained under the first quality assessment method;

[0009] The first annotation result is vectorized to obtain a set of vectorized vectors.

[0010] The second statistical analysis result is determined based on the vector set, which includes the second annotation result and the second score corresponding to the second annotation result. The second statistical analysis result is the evaluation parameter obtained under the second quality assessment method.

[0011] Based on the results of the first and second statistical analyses, the target quality assessment type for evaluating the quality of the first annotation results is determined. The target quality assessment type includes the first quality assessment method and the second quality assessment method.

[0012] In this embodiment, the process involves: obtaining a first annotation result for the data to be annotated; determining a first statistical analysis result for the first score corresponding to the first annotation result; vectorizing the first annotation result to obtain a vector set; determining a second annotation result and a second statistical analysis result for the second score corresponding to the second annotation result based on the vector set; and determining a target quality assessment type for evaluating the quality of the first annotation result based on the first and second statistical analysis results. This embodiment combines a first quality assessment method and a second quality assessment method to evaluate the quality of quantitatively annotated data and the work quality of the annotators. Based on the first statistical analysis result obtained from the first quality assessment method and the second statistical analysis result obtained from the second quality assessment method, a target quality assessment model for evaluating the data annotation quality is determined. This allows for more targeted screening of data with potential quality risks when facing data annotation indicators, better control of annotation quality, and addresses the shortcomings of manual annotation methods in guaranteeing data annotation quality.

[0013] In one optional implementation, a second statistical analysis result, which determines the second annotation result and the second score corresponding to the second annotation result based on the vector set, includes:

[0014] Get the similarity between any predetermined number of vectors within a vector set;

[0015] Cluster the similarity to determine the groupings after clustering;

[0016] The second annotation result is determined based on the number of groups;

[0017] Obtain the second score for the second annotation result;

[0018] The second statistical analysis results are determined based on the second score.

[0019] In this embodiment of the disclosure, the clustered groups are obtained based on the similarity between vectors within the vector set, and the second labeling result is determined based on the grouping situation. Then, the second statistical analysis result is obtained. This makes it easier for manual evaluation of the quality of the previous data labeling, while reducing the workload of sorting and labeling similar results.

[0020] In one optional implementation, determining the second annotation result based on the number of groups includes:

[0021] When the number of groups is a preset number, obtain the second annotation result for the groups;

[0022] If the number of groups exceeds the preset number, target records are randomly selected from each group, and the second annotation result of the group is determined based on the target records.

[0023] In this embodiment of the disclosure, the method for obtaining the second annotation result is determined based on the grouping result, which can save time and obtain the second annotation result in a targeted manner.

[0024] In one optional implementation, when the number of groups exceeds a preset number, obtaining a second score for the second annotation result includes:

[0025] The second annotation result of the target record is scored to obtain the corresponding third score;

[0026] The fourth score is determined based on the third score, excluding the target record.

[0027] The second score for each group is determined based on the third and fourth score values.

[0028] In this embodiment of the disclosure, by combining the score of the target record with the score of other records besides the target record, the score corresponding to each clustered group can be determined more accurately, which facilitates the analysis of subsequent annotation quality.

[0029] In one optional implementation, determining a fourth score for records other than the target record based on the third score includes:

[0030] Calculate the average of the third score;

[0031] Set the fourth score to the average value.

[0032] In one optional implementation, based on the results of the first statistical analysis and the second statistical analysis, a target quality assessment type for evaluating the quality of the first annotation result is determined, including:

[0033] Obtain the numerical comparison results between the first and second statistical analysis results;

[0034] Based on the numerical comparison results, the target quality assessment type for the first annotation result is determined.

[0035] In one optional implementation, the first statistical analysis result includes a first mean or a first variance, and the second statistical analysis result includes a second mean or a second variance. Based on the numerical comparison results, a target quality assessment type for evaluating the quality of the first annotation result is determined, including:

[0036] Compare the values ​​of the first mean and the second mean;

[0037] If the first mean is greater than the second mean, the first quality assessment method will be used as the target quality assessment type for assessing the quality of the first annotation result; otherwise, the second quality assessment method will be used as the target quality assessment type for assessing the quality of the first annotation result.

[0038] or,

[0039] Compare the magnitudes of the first and second variances;

[0040] If the first variance is greater than the second variance, the second quality assessment method shall be used as the target quality assessment type for assessing the quality of the first annotation result; otherwise, the first quality assessment method shall be used as the target quality assessment type for assessing the quality of the first annotation result.

[0041] In this embodiment of the disclosure, a first quality assessment method and a second quality assessment method are introduced to evaluate the quality of the quantitatively labeled data and the work quality of the labelers, which is both accurate and efficient.

[0042] Secondly, this disclosure provides an apparatus for determining the quality of question-answering data annotations for training large language models, the apparatus comprising:

[0043] The acquisition module is used to obtain the first annotation result of the data to be labeled;

[0044] The first determining module is used to determine the first statistical analysis result of the first score corresponding to the first annotation result, wherein the first statistical analysis result is the evaluation parameter obtained under the first quality assessment method;

[0045] The vectorization module is used to vectorize the first annotation result to obtain a set of vectorized vectors.

[0046] The second determining module is used to determine the second annotation result and the second statistical analysis result of the second score corresponding to the second annotation result based on the vector set, wherein the second statistical analysis result is the evaluation parameter obtained under the second quality assessment method;

[0047] The third determining module is used to determine the target quality assessment type for quality assessment of the first annotation result based on the first statistical analysis result and the second statistical analysis result. The target quality assessment type includes the first quality assessment method and the second quality assessment method.

[0048] Thirdly, this disclosure provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the method for determining the quality of question-answer data annotation for training a large language model as described in the first aspect or any corresponding embodiment.

[0049] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to execute the method for determining the quality of question-answer data annotations for training a large language model, as described in the first aspect or any corresponding embodiment above. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating a method for determining the quality of question-answer data annotations for training a large language model, according to some embodiments of this disclosure.

[0052] Figure 2 This is a schematic diagram of the overall process of data annotation according to some embodiments of this disclosure;

[0053] Figure 3 This is a structural block diagram of an apparatus for determining the quality of question-answer data annotations for training a large language model, according to some embodiments of the present disclosure.

[0054] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0056] Q&A: Questions and Answers, an application of a large language model, can be presented in the form of dialogue. Currently, the quality of Q&A dialogue-based labeled data is typically assessed using self-checking, cross-checking, voting, and client acceptance. However, these methods have limitations. For example, self-checking and cross-checking rely heavily on the capabilities and knowledge of the data labelers, resulting in high labor costs. Voting for data quality labeling has a narrow application scope, limited to classification tasks. Relying on client acceptance to ensure labeling quality requires quantifying the accuracy of the labelers' work and data quality during the inspection / acceptance phase. However, quantifying accuracy relies excessively on manual labeling, which is often affected by external factors and the labelers' own professional abilities. Therefore, manual labeling cannot guarantee the quality of data labeling.

[0057] To address the aforementioned issues, according to embodiments of this disclosure, a method for determining the quality of question-answer data annotations for training large language models is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0058] This embodiment provides a method for determining the quality of question-answer data annotations used in training large language models. Figure 1 This is a flowchart of a method for determining the quality of question-answering data annotations for training a large language model according to embodiments of this disclosure, such as... Figure 1 As shown, this method can be applied to the server side, and the method process can include the following steps:

[0059] Step S101: Obtain the first annotation result of the data to be annotated.

[0060] Optionally, in this embodiment of the disclosure, some data to be labeled is obtained according to the labeling specifications, and then these data to be labeled are labeled manually or by machine. This labeling stage can adopt an independent labeling mode, where the task is completed independently by either manual or machine, thereby obtaining the first labeling result L(Q)={A1, A2, A3, ..., An}.

[0061] Step S102: Determine the first statistical analysis result of the first score corresponding to the first annotation result, wherein the first statistical analysis result is the evaluation parameter obtained under the first quality assessment method.

[0062] Optionally, scoring and evaluation criteria for the annotation results can be set according to the current annotation standards, and the scores of the annotation results can be determined based on these criteria. The evaluation criteria here include the fluency, semantic completeness, accuracy, and security of the annotation results. These are used as scoring criteria to output the first score value corresponding to the first annotation result.

[0063] Furthermore, in this embodiment of the disclosure, a first quality assessment method (e.g., machine evaluation of annotation quality score) is used to score the first annotation result. For example, a state-of-the-art (SOTA) model and some pre-defined prompts or questions are used to sort and score L(Q) = {A1, A2, A3, ..., An}, resulting in a first machine score S(Q) = {s1, s2, s3, ..., sn}.

[0064] For example: Question:

[0065] What is artificial intelligence?

[0066] Answer list (e.g., the first labeled result L(Q)):

[0067] 1. Artificial intelligence is a branch of computer science that aims to simulate, replicate, and improve human intelligence. It enables machines to learn like humans.

[0068] 2. Artificial intelligence is a new technology that studies and develops theories, methods, techniques, and application systems for simulating, extending, and expanding human intelligence.

[0069] 3. Artificial intelligence is a new technology that studies and develops theories, methods, techniques, and application systems for simulating, enhancing, or extending human intelligence.

[0070] 4. Artificial intelligence is a new technology that studies and develops theories, methods, techniques, and application systems for simulating, extending, and expanding human intelligence.

[0071] The SOTA model outputs the first score S(Q):

[0072] 1.9 points

[0073] 2.8 points

[0074] 3.10 points

[0075] 4.8 points

[0076] That is, S(Q) = {9, 8, 10, 8}.

[0077] Then, the corresponding first statistical analysis result is obtained based on S(Q).

[0078] It is understandable that the first statistical analysis result here is determined by the first score S(Q), and S(Q) is obtained by scoring the first annotation result by the first quality assessment method. Therefore, the first statistical analysis result can be used as a parameter for the annotation assessment of the first quality assessment method.

[0079] Step S103: The first annotation result is vectorized to obtain a set of vectorized vectors.

[0080] Optionally, embodiments of this disclosure may use the SOTA model to vectorize the first annotation result, obtain the vectorized representation of each annotation result, and obtain the vectorized vector set E(Q) = {e1, e2, e3, ..., en}.

[0081] Step S104: Determine the second annotation result and the second statistical analysis result of the second score corresponding to the second annotation result based on the vector set, wherein the second statistical analysis result is the evaluation parameter obtained under the second quality assessment method.

[0082] Optionally, this embodiment of the disclosure also involves a second quality assessment method (e.g., manual assessment of annotation quality scores) when evaluating the quality of the annotation results. In this case, it is necessary to determine the second annotation result after annotating each vector based on the vector set determined in the above embodiment, and then manually score the second annotation result to obtain the corresponding second score value, and obtain the corresponding second statistical analysis result based on the second score value.

[0083] It is understandable that the second statistical analysis result here is determined by the second scoring value, which is obtained by scoring the second annotation result by the second quality assessment method. Therefore, the second statistical analysis result can be used as a parameter for the annotation assessment of the second quality assessment method.

[0084] Step S105: Based on the first statistical analysis results and the second statistical analysis results, determine the target quality assessment type for the quality assessment of the first annotation result, wherein the target quality assessment type includes the first quality assessment method and the second quality assessment method.

[0085] Optionally, after obtaining the first statistical analysis result and the second statistical analysis result, the target quality assessment type that can be used to assess the quality of the first annotation result can be determined based on the first statistical analysis result and the second statistical analysis result. The target quality assessment type can be the first quality assessment method or the second quality assessment method. That is, the embodiment of this disclosure aims to select the quality assessment type that yields the best data annotation quality assessment result for the first annotation result.

[0086] In this embodiment, the process involves: obtaining a first annotation result for the data to be annotated; determining a first statistical analysis result for the first score corresponding to the first annotation result; vectorizing the first annotation result to obtain a vector set; determining a second annotation result and a second statistical analysis result for the second score corresponding to the second annotation result based on the vector set; and determining a target quality assessment type for evaluating the quality of the first annotation result based on the first and second statistical analysis results. This embodiment combines a first quality assessment method and a second quality assessment method to evaluate the quality of quantitatively annotated data and the work quality of the annotators. Based on the first statistical analysis result obtained from the first quality assessment method and the second statistical analysis result obtained from the second quality assessment method, a target quality assessment model for evaluating the data annotation quality is determined. This allows for more targeted screening of data with potential quality risks when facing data annotation indicators, better control of annotation quality, and addresses the shortcomings of manual annotation methods in guaranteeing data annotation quality.

[0087] In some optional implementations, a second statistical analysis result, which determines the second annotation result and the second score corresponding to the second annotation result based on the vector set, includes:

[0088] Get the similarity between any predetermined number of vectors within a vector set;

[0089] Cluster the similarity to determine the groupings after clustering;

[0090] The second annotation result is determined based on the number of groups;

[0091] Obtain the second score for the second annotation result;

[0092] The second statistical analysis results are determined based on the second score.

[0093] Optionally, in this embodiment of the disclosure, algorithms such as cosine similarity, Euclidean distance, and Pearson correlation coefficient can be used to calculate the similarity between any preset number of vectors (e.g., any pairwise vectors) in the vector set, such as calculating the similarity of [e(i), e(j)].

[0094] Then, these vectors are clustered based on their similarity. At this point, those with a similarity greater than a certain threshold (such as 0.8) can be grouped into one class. The grouping after clustering can be represented by Kn(Q). The result of clustering is reflected in the first labeling result and can be represented as K(Q)={[A1, A3],[A2],[A4, An, ….,Ak]}.

[0095] As can be seen from K(Q) above, the first identification results A1 and A3 are divided into one category, the first identification result A2 is divided into another category, and the first identification results A4, An, …., Ak are divided into another category. At this time, K(Q) has three groups. Then, in this embodiment of the present disclosure, the second annotation result after grouping is determined according to the number of groups, and then a manual score is performed on the second annotation result to obtain the second score value. Finally, the corresponding second statistical analysis result is determined based on the second score value.

[0096] In this embodiment of the disclosure, the clustered groups are obtained based on the similarity between vectors within the vector set, and the second labeling result is determined based on the grouping situation. Then, the second statistical analysis result is obtained. This makes it easier for manual evaluation of the quality of the previous data labeling, while reducing the workload of sorting and labeling similar results.

[0097] In some optional implementations, the second annotation result is determined based on the number of groups, including:

[0098] When the number of groups is a preset number, obtain the second annotation result for the groups;

[0099] If the number of groups exceeds the preset number, target records are randomly selected from each group, and the second annotation result of the group is determined based on the target records.

[0100] Optionally, the more groups there are, the larger Kn(Q) is, the less correlated the first annotation results are. Therefore, when the number of groups is a preset number (e.g., one), the first annotation results are considered relatively reliable. In order to reduce the amount of annotation, sorting and annotation are no longer performed, and the second annotation results of the manually annotated group are directly obtained.

[0101] If the number of groups exceeds the preset number, it means there is more than one group. In this case, target records need to be randomly selected from each group, and these target records are labeled to obtain the labeling results for these target records. These labeling results can be used as the second labeling results for the grouping. It should be noted that the target records do not have to be the same record, as long as they are randomly selected from each group; and the number of target records can be one or more.

[0102] In this embodiment of the disclosure, the method for obtaining the second annotation result is determined based on the grouping result, which can save time and obtain the second annotation result in a targeted manner.

[0103] In some optional implementations, when the number of groups exceeds a preset number, a second score is obtained for the second annotation result, including:

[0104] The target records are labeled and scored to obtain the corresponding third score value;

[0105] The fourth score is determined based on the third score, excluding the target record.

[0106] The second score for each group is determined based on the third and fourth score values.

[0107] Optionally, the second annotation result of each selected target record is scored to obtain a corresponding third score. The scoring criteria can also be based on fluency, semantic completeness, accuracy, security, etc.

[0108] Since this embodiment selects any target record from each group for scoring, the purpose of this is to reduce the amount of annotation. However, the third score of the target record cannot be used to characterize the second score of the entire group, as this is not accurate enough. Therefore, this embodiment also adopts a method of determining the fourth score of other records besides the target record based on the third score, and combining the third score and the fourth score to determine the second score of each group.

[0109] It should be noted that when determining the second score for each group based on the third and fourth scores, the second score can be obtained by adding the third and fourth scores together. Alternatively, each third score can be assigned a first weight, or each fourth score can be assigned a second weight, and then the second score can be calculated as: third score × first weight + fourth score = second score; third score + fourth score × second weight = second score; or third score × first weight + fourth score × second weight = second score, etc. These calculation methods can all be flexibly set.

[0110] It is important to understand that the corresponding second score should be the same for each group.

[0111] In this embodiment of the disclosure, by combining the score of the target record with the score of other records besides the target record, the score corresponding to each clustered group can be determined more accurately, which facilitates the analysis of subsequent annotation quality.

[0112] In some optional implementations, determining a fourth score for records other than the target record based on the third score includes:

[0113] Calculate the average of the third score;

[0114] Set the fourth score to the average value.

[0115] Optionally, in this embodiment of the disclosure, when determining the fourth score for records other than the target record within each group, it may be obtained based on the third score. For example, if there are multiple target records, multiple third scores can be obtained, and then these third scores can be averaged to obtain the average value. The fourth score can then be set as this average value.

[0116] In some optional implementations, based on the results of the first statistical analysis and the second statistical analysis, a target quality assessment type for evaluating the quality of the first annotation result is determined, including:

[0117] Obtain the numerical comparison results between the first and second statistical analysis results;

[0118] Based on the numerical comparison results, the target quality assessment type for the first annotation result is determined.

[0119] Optionally, in this embodiment of the disclosure, the data magnitudes are compared based on a first statistical analysis result obtained from the first scoring value and a second statistical analysis result obtained from the second scoring value to obtain a numerical comparison result. Then, based on the data comparison result, the target quality assessment type is determined. It is understood that this target quality assessment type is a quality assessment type that is superior to the labeled quality assessment.

[0120] In some optional implementations, the first statistical analysis result includes a first mean or a first variance, and the second statistical analysis result includes a second mean or a second variance. Based on the numerical comparison results, a target quality assessment type for evaluating the quality of the first annotation result is determined, including:

[0121] Compare the values ​​of the first mean and the second mean;

[0122] If the first mean is greater than the second mean, the first quality assessment method will be used as the target quality assessment type for assessing the quality of the first annotation result; otherwise, the second quality assessment method will be used as the target quality assessment type for assessing the quality of the first annotation result.

[0123] or,

[0124] Compare the magnitudes of the first and second variances;

[0125] If the first variance is greater than the second variance, the second quality assessment method shall be used as the target quality assessment type for assessing the quality of the first annotation result; otherwise, the first quality assessment method shall be used as the target quality assessment type for assessing the quality of the first annotation result.

[0126] Optionally, regardless of whether it is the first mean in the first statistical analysis result or the second mean in the second statistical analysis result, as long as the mean is low, it means that the quality risk of the labeled result L(Q) is higher. Therefore, when determining the target quality assessment type, the method of comparing the numerical values ​​of the first mean and the second mean can be adopted: if the first mean is greater than the second mean, the first quality assessment method is taken as the target quality assessment type; otherwise, the second quality assessment method is taken as the target quality assessment type.

[0127] In addition, regardless of whether it is the first variance in the first statistical analysis result or the second variance in the second statistical analysis result, as long as the variance is large, it means that there is a risk in vector similarity. At this time, the quality risk of the labeling result L(Q) is also higher. Therefore, when determining the target quality assessment type, we can compare the numerical values ​​of the first variance and the second variance: if the first variance is greater than the second variance, the second quality assessment method is used as the target quality assessment type; otherwise, the first quality assessment method is used as the target quality assessment type.

[0128] In one application scenario: taking the first mean Avg-M(Q) and the second mean Avg-H(Q) as examples, a larger mean indicates higher annotation quality. Assume there are m records Q that need annotation:

[0129] The first quality assessment method evaluates and labels the quality score: Quality-M=(Avg-M(Q1+ Avg-M(Q2)+……+Avg-M(Qm)) / m;

[0130] The second quality assessment method evaluates and marks the quality score: Quality-H=(Avg-H(Q1+ Avg-H(Q2)+……+Avg-H(Qm)) / m.

[0131] The above indicators can also be used to quantify the work quality of annotators. Assuming an annotator annotates f results, the indicator calculation is as follows:

[0132] The first quality assessment method evaluates the work quality score of the marked personnel: Mark-Quality-M=(Avg-M(Q1+ Avg-M(Q2)+……+Avg-M(Qf)) / f;

[0133] The second quality assessment method evaluates the work quality score of the marked personnel: Mark-Quality-H=(Avg-H(Q1+ Avg-H(Q2)+……+Avg-H(Qf)) / f.

[0134] In this embodiment of the disclosure, a first quality assessment method and a second quality assessment method are introduced to evaluate the quality of the quantitatively labeled data and the work quality of the labelers, which is both accurate and efficient.

[0135] In some alternative implementations, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the overall data annotation process according to some embodiments of this disclosure. The specific process is as follows:

[0136] Obtain question Q; manually annotate the list of answers to question Q to obtain manual annotations A1... An, then vectorize the manual annotations within the SOTA model to obtain a vector set E(Q) = {e1, e2, e3, ..., en}, and simultaneously sort and score the manual annotations L(Q) = {A1, A2, A3, ..., An} in the SOTA model to obtain the machine's first score S(Q) = {s1, s2, s3, ..., sn}.

[0137] Calculate the similarity sij between A1 and An based on e1 to en. Then, perform clustering based on the similarity and group those with similarity greater than the threshold into the same class. The clustering result can be represented as: C1: A1, A3; C2: A2; C3: A4, An, ….,Ak; Cn:…….

[0138] If there is only one cluster after clustering, the initial manual annotation result is considered reliable, no sorting annotation is required, scores are manually set, and annotation ends. If there is not one cluster after clustering, n records are randomly selected from each cluster result Ck for annotation. A score is manually assigned to each record. At this time, the same score is assigned to the same cluster group, and then annotation ends.

[0139] This embodiment also provides a device for determining the quality of question-answer data annotations for training large language models. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0140] This embodiment provides a device for determining the quality of question-answer data annotations used in training large language models, such as... Figure 3 As shown, it includes:

[0141] Module 301 is used to obtain the first annotation result of the data to be annotated;

[0142] The first determining module 302 is used to determine the first statistical analysis result of the first score corresponding to the first annotation result, wherein the first statistical analysis result is the evaluation parameter obtained under the first quality assessment method;

[0143] The vectorization module 303 is used to perform vectorization processing on the first annotation result to obtain a set of vectorized vectors.

[0144] The second determining module 304 is used to determine the second annotation result and the second statistical analysis result of the second scoring value corresponding to the second annotation result based on the vector set, wherein the second statistical analysis result is the evaluation parameter obtained under the second quality assessment method;

[0145] The third determining module 305 is used to determine the target quality assessment type for quality assessment of the first annotation result based on the first statistical analysis result and the second statistical analysis result. The target quality assessment type includes the first quality assessment method and the second quality assessment method.

[0146] In some alternative implementations, the second determining module 304 includes:

[0147] The first acquisition unit is used to acquire the similarity between any preset number of vectors in the vector set;

[0148] Clustering units are used to cluster similarity and determine the groups after clustering;

[0149] The first determining unit is used to determine the second annotation result based on the number of groups;

[0150] The second acquisition unit is used to acquire the second score value of the second annotation result;

[0151] The second determining unit is used to determine the second statistical analysis result based on the second score.

[0152] In some optional implementations, the first determining unit includes:

[0153] The acquisition submodule is used to obtain the second annotation result for the groups when the number of groups is a preset number;

[0154] The first determination submodule is used to randomly select target records from each group when the number of groups exceeds the preset number, and determine the second labeling result of the group based on the target records.

[0155] In some optional implementations, when the number of groups exceeds a preset number, the second acquisition unit includes:

[0156] The submodule is used to annotate and score the target record, and obtain the corresponding third score value;

[0157] The second determination submodule is used to determine the fourth score for records other than the target record based on the third score;

[0158] The third determination submodule is used to determine the second score for each group based on the third and fourth score values.

[0159] In some optional implementations, the third determining submodule includes:

[0160] The calculation subunit is used to calculate the average value of the third score;

[0161] Set a sub-unit to set the fourth score as the average value.

[0162] In some alternative implementations, the third determining module 305 includes:

[0163] The third acquisition unit is used to acquire the numerical comparison results between the first statistical analysis result and the second statistical analysis result;

[0164] The third determining unit is used to determine the target quality assessment type for the quality assessment of the first annotation result based on the numerical comparison results.

[0165] In some optional implementations, the first statistical analysis result includes a first mean or a first variance, the second statistical analysis result includes a second mean or a second variance, and the third determining unit includes:

[0166] The first comparison submodule is used to compare the numerical values ​​of the first mean and the second mean.

[0167] The first setting submodule is used to use the first quality assessment method as the target quality assessment type for the first annotation result when the first mean is greater than the second mean; otherwise, it uses the second quality assessment method as the target quality assessment type for the first annotation result.

[0168] or,

[0169] The second comparison submodule is used to compare the numerical values ​​of the first variance and the second variance;

[0170] The second setting submodule is used to use the second quality assessment method as the target quality assessment type for the quality assessment of the first annotation result if the first variance is greater than the second variance; otherwise, it uses the first quality assessment method as the target quality assessment type for the quality assessment of the first annotation result.

[0171] In this embodiment, the device for determining the quality of question-answer data annotations for training large language models is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0172] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0173] This disclosure also provides a computer device having the above-described features. Figure 3 The apparatus shown is for determining the quality of question-answer data annotations used in training large language models.

[0174] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this disclosure, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0175] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0176] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0177] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0178] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0179] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0180] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0181] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for determining the quality of question and answer data labeling for large language model training, characterized in that, The method comprises: obtaining a first labeling result of to-be-labeled data, wherein the first labeling result is obtained by using artificial labeling or machine labeling on the obtained to-be-labeled question and answer data; determining a first statistical analysis result of a first scoring value corresponding to the first labeling result, wherein the first statistical analysis result is an evaluation parameter obtained under a first quality evaluation mode; performing vectorization processing on the first labeling result to obtain a vector set after vectorization; determining a second labeling result and a second statistical analysis result of a second scoring value corresponding to the second labeling result according to the vector set, wherein the second statistical analysis result is an evaluation parameter obtained under a second quality evaluation mode; determining a target quality evaluation type for quality evaluation of the first labeling result according to the first statistical analysis result and the second statistical analysis result, wherein the target quality evaluation type comprises the first quality evaluation mode and the second quality evaluation mode.

2. The method of claim 1, wherein, The determination of the second labeling result and the second statistical analysis result of the second scoring value corresponding to the second labeling result according to the vector set comprises: obtaining a similarity between any preset number of vectors in the vector set; performing clustering on the similarity to determine grouped groups after clustering; determining the second labeling result according to the number of groups; obtaining the second scoring value of the second labeling result; determining the second statistical analysis result according to the second scoring value.

3. The method of claim 2, wherein, The determination of the second labeling result according to the number of groups comprises: in a case where the number of groups is a preset number, obtaining the second labeling result of the groups; in a case where the number of groups exceeds the preset number, randomly selecting a target record from each group, and determining the second labeling result of the group according to the target record.

4. The method of claim 3, wherein, In the case where the number of groups exceeds the preset number, the obtaining of the second scoring value of the second labeling result comprises: performing labeling scoring on the target record to obtain a corresponding third scoring value; determining a fourth scoring value of other records except the target record according to the third scoring value; determining the second scoring value of each group according to the third scoring value and the fourth scoring value.

5. The method of claim 4, wherein, The determination of the fourth scoring value of other records except the target record according to the third scoring value comprises: averaging the third scoring value; setting the fourth scoring value as the average value.

6. The method of claim 1, wherein, The determination of the target quality evaluation type for quality evaluation of the first labeling result according to the first statistical analysis result and the second statistical analysis result comprises: obtaining a numerical comparison result between the first statistical analysis result and the second statistical analysis result; determining the target quality evaluation type for quality evaluation of the first labeling result according to the numerical comparison result.

7. The method of claim 6, wherein, The first statistical analysis result includes a first mean or a first variance, the second statistical analysis result includes a second mean or a second variance, and the target quality evaluation type for quality evaluation of the first annotation result is determined according to the numerical comparison result, including: Comparing the numerical values of the first mean and the second mean; In the case where the first mean is greater than the second mean, the first quality evaluation mode is used as the target quality evaluation type for quality evaluation of the first annotation result, otherwise the second quality evaluation mode is used as the target quality evaluation type for quality evaluation of the first annotation result; Or, Comparing the numerical values of the first variance and the second variance; In the case where the first variance is greater than the second variance, the second quality evaluation mode is used as the target quality evaluation type for quality evaluation of the first annotation result, otherwise the first quality evaluation mode is used as the target quality evaluation type for quality evaluation of the first annotation result.

8. A device for determining the quality of question and answer data labeling for large language model training, characterized in that, The device includes: An acquisition module configured to acquire a first annotation result of to-be-annotated data, wherein the first annotation result is obtained by using artificial annotation or machine annotation on the acquired to-be-annotated question and answer data; A first determination module configured to determine a first statistical analysis result of a first scoring value corresponding to the first annotation result, wherein the first statistical analysis result is an evaluation parameter obtained in a first quality evaluation mode; A vectorization module configured to perform vectorization processing on the first annotation result to obtain a vector set after vectorization; A second determination module configured to determine a second annotation result and a second statistical analysis result of a second scoring value corresponding to the second annotation result according to the vector set, wherein the second statistical analysis result is an evaluation parameter obtained in a second quality evaluation mode; A third determination module configured to determine a target quality evaluation type for quality evaluation of the first annotation result according to the first statistical analysis result and the second statistical analysis result, wherein the target quality evaluation type includes the first quality evaluation mode and the second quality evaluation mode.

9. A computer device, comprising: including: A memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the method for determining the quality of question and answer data annotation for large language model training according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the method for determining the quality of question and answer data annotation for large language model training according to any one of claims 1 to 7.

Images