Training data quality inspection method and device, equipment, storage medium and program product

Abstract

The invention discloses a training data quality inspection method and a related device, and relates to the technical field of data processing, and the method comprises the steps: marking instruction data to obtain a label set of the instruction data, the label set comprising labels of multiple granularities, labels of multiple visual angles and labels of at least two marking dimensions in labels associated with multiple contexts; clustering the labels in each label set to obtain a plurality of label clusters, and performing quality evaluation from at least two evaluation dimensions of semantic accuracy, logic consistency, field correlation and user demand matching degree corresponding to each label cluster to obtain a comprehensive quality score based on the at least two evaluation dimensions corresponding to each label cluster; determining a diversity score of the instruction data set according to the distribution uniformity of the plurality of instruction data clusters, the distribution condition of the instruction length and the label distribution condition; the complexity score of the instruction data set is determined at least according to the density of the instruction data cluster and the density of the semantic association matrix, and the accuracy of training data quality inspection is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a training data quality inspection method, apparatus, device, storage medium, and program product. Background Art

[0002] The performance of large models is highly dependent on high-quality, domain-appropriate training data. Therefore, it is necessary to perform quality control on the training data of large models, especially on the quality control of supervised fine-tuning (SFT) training data.

[0003] Supervised fine-tuning training data consists of instruction data and corresponding labeled answers. Quality inspection of supervised fine-tuning training data involves both instruction data and answers. Instruction data quality inspection involves analyzing its quality, diversity, and complexity, while answer quality inspection involves verifying the correctness of the answers. While answer quality inspection can be performed using existing large models, instruction data quality inspection relies primarily on manual review, resulting in poor quality inspection results. Summary of the Invention

[0004] In view of the above problems, this application provides a training data quality inspection method, device, equipment, storage medium and program product to improve the accuracy of training data quality inspection. The specific solution is as follows:

[0005] The first aspect of the present application provides a training data quality inspection method, comprising:

[0006] Label each instruction data in the instruction data set to be quality inspected with multiple labeling dimensions to obtain a label set corresponding to each instruction data; the multiple labeling dimensions include at least two of the following labeling dimensions: multi-granularity labeling, multi-perspective labeling, and context-related labeling;

[0007] For each label set, cluster the labels in the label set to obtain multiple label clusters;

[0008] For each tag cluster, perform a quality assessment on the tag cluster based on at least two of the four evaluation dimensions: semantic accuracy, logical consistency, domain relevance, and user demand matching, to obtain a comprehensive quality score for the tag cluster in the at least two evaluation dimensions;

[0009] Clustering the instruction data set according to the semantic similarity between each instruction data and each label in all label sets to obtain multiple instruction data clusters;

[0010] Determining a diversity score of the instruction data set based on distribution uniformity of the multiple instruction data clusters, distribution of instruction lengths of the instruction data set, and label distribution of all label sets;

[0011] The complexity score of the instruction data set is determined at least according to the density of each instruction data cluster and the density of a semantic association matrix; the semantic association matrix represents the semantic similarity between each instruction data in the instruction data set and each label.

[0012] In a possible implementation, for each tag set, the tags in the tag set are clustered to obtain multiple tag clusters, including:

[0013] Perform semantic understanding on each tag set to obtain a semantic understanding result corresponding to the tag set; the semantic understanding result includes: the semantic meaning of each tag in the tag set, the semantic association between the tags, and the potential hierarchical structure;

[0014] The tags in the tag set are clustered based on the semantic understanding result to obtain multiple tag clusters.

[0015] In a possible implementation, clustering the tags in the tag set based on the semantic understanding result includes:

[0016] Performing initial clustering on the tags in the tag set to obtain multiple initial tag clusters; the tags in each initial tag cluster are tags with similar semantics or with a hierarchical relationship;

[0017] The tags in the tag set are treated as nodes in a heterogeneous graph, and the graph neural network algorithm is used to update the vector representation of each tag. The initial vector representation of each tag is determined based on the semantic meaning of the tag.

[0018] The labels in the label set are clustered based on the updated vector representation of each label to obtain multiple optimized label clusters.

[0019] In a possible implementation, the semantic understanding result further includes: the semantic importance of each tag;

[0020] The initial clustering of the tags in the tag set includes:

[0021] For each tag, the weight of the tag is obtained by adding the frequency of occurrence of the tag in all tag sets, the semantic importance of the tag, and the user evaluation corresponding to the tag.

[0022] Based on the weight of each tag in the tag set, the tag set is filtered and merged to obtain an optimized tag set;

[0023] Initial clustering is performed on the tags in the optimized tag set.

[0024] In a possible implementation, for each tag cluster, performing a quality assessment on the tag cluster from at least two assessment dimensions to obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions includes:

[0025] Using the large model, quality assessment is performed on the tag cluster in the at least two assessment dimensions respectively to obtain a score for the tag cluster in each of the at least two assessment dimensions;

[0026] The scores of the tag cluster in the at least two evaluation dimensions are weighted and summed to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions.

[0027] In a possible implementation, for each tag cluster, performing a quality assessment on the tag cluster from at least two assessment dimensions to obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions includes:

[0028] Using the large model, quality assessment is performed on the tag cluster in the at least two assessment dimensions respectively to obtain a score for the tag cluster in each of the at least two assessment dimensions;

[0029] A weighted sum of the scores of the tag cluster in the at least two evaluation dimensions is performed to obtain a first comprehensive quality score of the tag cluster in the at least two evaluation dimensions;

[0030] Obtaining a second comprehensive quality score for the tag cluster; the second comprehensive quality score is obtained based on a weighted sum of manual scores of the tag cluster in the at least two evaluation dimensions;

[0031] The first comprehensive quality score and the second comprehensive quality score are weightedly summed to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions.

[0032] In a possible implementation, the distribution of instruction lengths includes distribution entropy of instruction lengths; the label distribution includes distribution entropy of labels;

[0033] Determining the diversity score of the instruction data set according to the distribution uniformity of the multiple instruction data clusters, the distribution of instruction lengths of the instruction data set, and the label distribution of all label sets includes:

[0034] The diversity score is obtained by weighted summing the distribution uniformity of the multiple instruction data clusters, the distribution entropy of the instruction length, and the distribution entropy of the label.

[0035] In a possible implementation, determining the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix includes:

[0036] The complexity score of the instruction data set is obtained by performing a weighted summation of the average value of the density of each instruction data cluster and the density of the semantic association matrix.

[0037] In a possible implementation, clustering the instruction data set according to the semantic similarity between each instruction data and each tag in all tag sets includes: performing hierarchical clustering on the instruction data set according to the semantic similarity between each instruction data and each tag;

[0038] The determining of the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix includes:

[0039] The average value of the density of each instruction data cluster is weighted and summed with the depth of the instruction data cluster to obtain the fusion feature of the instruction data cluster;

[0040] The fusion feature of the instruction data cluster and the density of the semantic association matrix are weightedly summed to obtain a complexity score of the instruction data set.

[0041] In a possible implementation, the method further includes:

[0042] Calculating the gradient of the comprehensive quality score with respect to the weight coefficient to be updated based on the comprehensive quality score and user feedback data; the weight coefficient to be updated is at least part of the weight coefficient used in the quality inspection process;

[0043] With the goal of increasing the comprehensive quality score, updating the weight to be updated based on the gradient;

[0044] The instruction data set to be quality-checked is quality-checked again based on the updated weights.

[0045] A second aspect of the present application provides a training data quality inspection device, comprising:

[0046] A labeling module is used to label each instruction data in the instruction data set to be inspected with multiple labeling dimensions to obtain a label set corresponding to each instruction data; the multiple labeling dimensions include: multi-granularity labeling, multi-perspective labeling, and context-related labeling;

[0047] A tag clustering module is used to cluster the tags in each tag set to obtain multiple tag clusters;

[0048] A quality assessment module is used to evaluate the quality of each tag cluster based on four evaluation dimensions: semantic accuracy, logical consistency, domain relevance, and user demand matching, and obtain a comprehensive quality score for the tag cluster in the four evaluation dimensions;

[0049] An instruction data clustering module is used to cluster the instruction data set according to the semantic similarity between each instruction data and each label in all label sets to obtain multiple instruction data clusters;

[0050] a diversity assessment module, configured to determine a diversity score of the instruction data set based on a distribution uniformity of the plurality of instruction data clusters, a distribution of instruction lengths of the instruction data set, and a label distribution of all label sets;

[0051] A complexity evaluation module is used to determine the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of a semantic association matrix; the semantic association matrix represents the semantic similarity between each instruction data in the instruction data set and each label.

[0052] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the training data quality inspection method of the first aspect or any implementation of the first aspect.

[0053] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0054] The memory is used to store computer programs;

[0055] The processor is used to execute the computer program so that the electronic device can implement the training data quality inspection method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0056] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the training data quality inspection method of the above-mentioned first aspect or any implementation of the first aspect.

[0057] By means of the above technical solution, the training data quality inspection method, apparatus, equipment, storage medium and program product provided by the present application respectively perform label annotating of multiple annotation dimensions on each instruction data in the instruction data set to be quality inspected, and obtain a label set corresponding to each instruction data; the multiple annotation dimensions include at least two of the following annotation dimensions: multi-granularity annotation, multi-perspective annotation, and context-related annotation; corresponding to each label set, the labels in the label set are clustered to obtain multiple label clusters; corresponding to each label cluster, the label cluster is quality evaluated from at least two evaluation dimensions of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance and user demand matching, and a comprehensive quality score of the label cluster in the above at least two evaluation dimensions is obtained; the instruction data set is clustered according to the semantic similarity between each instruction data and each label, and multiple instruction data clusters are obtained; the diversity score of the instruction data set is determined according to the distribution uniformity of the multiple instruction data clusters, the distribution of the instruction length of the instruction data set and the label distribution of all label sets; the complexity score of the instruction data set is determined at least according to the density of each instruction data cluster and the density of the semantic association matrix; the semantic association matrix represents the semantic similarity between each instruction data and each label in the instruction data set.

[0058] The present application labels each instruction data from multiple labeling dimensions to obtain a label set for each instruction data, which includes labels from at least two of the following labeling dimensions: labels of multiple granularities, labels from multiple perspectives, and labels associated with multiple contexts; clusters the labels in each label set to obtain multiple label clusters corresponding to the label set, and performs quality assessment on each label cluster from at least two of the four assessment dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching to obtain a comprehensive quality score based on at least two assessment dimensions for different label clusters corresponding to each instruction data, thereby improving the accuracy of the quality assessment. In addition, the diversity score of the instruction data set is determined based on the distribution uniformity of the multiple instruction data clusters obtained by clustering the instruction data, the distribution of the instruction lengths of the instruction data set, and the label distribution of all label sets; the complexity score of the instruction data set is determined based on at least the density of each instruction data cluster and the density of the semantic association matrix, thereby improving the accuracy of the instruction diversity and complexity assessment. In summary, the present application improves the accuracy of the quality inspection of the instruction data in the training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0060] Figure 1A flowchart for implementing the training data quality inspection method provided in this application;

[0061] Figure 2 A flowchart for implementing clustering of tags in the i-th tag set provided by this application;

[0062] Figure 3 A flowchart for implementing clustering of tags in the i-th tag set based on the semantic understanding results corresponding to the i-th tag set provided in this application;

[0063] Figure 4 A flowchart for implementing the initial clustering of the i-th tag set provided in this application;

[0064] Figure 5 A flowchart for implementing quality assessment of each tag cluster provided in this application based on at least two of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching;

[0065] Figure 6 Another implementation flowchart for performing quality assessment on each tag cluster provided in this application based on at least two of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching;

[0066] Figure 7 A schematic diagram of the structure of the training data quality inspection device provided in this application;

[0067] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0068] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0069] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0070] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0071] The command data in supervised fine-tuning training data refers to the various commands (also known as questions) users ask the AI ​​assistant. The answers in supervised fine-tuning training data refer to the responses to the commands. When fine-tuning the large model, the command data is input into the pre-trained large model, and the pre-trained large model outputs the processed results. With the goal of making the processed results approach the answers corresponding to the command data, the parameters of the pre-trained large model are fine-tuned. The pre-trained large model is trained using general unsupervised data.

[0072] When fine-tuning a large model in a target domain through supervised training, the training data is typically data from the target domain. Therefore, to ensure the performance of the large model in the target domain, the quality of the supervised training data must be high. High-quality supervised training data is premised on high-quality instruction data, as only high-quality instruction data can guarantee accurate answers. Therefore, how to quality-check the instruction data in supervised fine-tuning training data has become a pressing technical challenge.

[0073] The target area can be the financial field or other fields, such as education, medical field, industrial field, office field, etc.

[0074] Currently, the quality inspection of instruction data is mainly done by quality inspectors based on their experience, which has poor quality inspection results and low quality inspection accuracy, and cannot guarantee the quality of supervised fine-tuning training data. In order to improve the quality inspection accuracy of training data, this application solution is proposed.

[0075] like Figure 1 As shown, a flowchart of an implementation of the training data quality inspection method provided in an embodiment of the present application may include:

[0076] Step S101: Label each instruction data in the instruction data set to be quality inspected with multiple labeling dimensions to obtain a label set corresponding to each instruction data; the above multiple labeling dimensions include at least two of the following three labeling dimensions: multi-granularity labeling, multi-perspective labeling, and context-related labeling.

[0077] The instruction data in the instruction data set to be quality-checked is the instruction data in the supervised fine-tuning training data set to be quality-checked.

[0078] Multi-granularity means mining instruction intent at different levels from coarse to fine granularity.

[0079] Multi-perspective refers to comprehensive annotation from different angles such as user needs, task objectives, domain characteristics, etc.

[0080] Contextual association considers the contextual relationship between instructions in a specific conversation or task process, and marks intent labels that are closely related to the context.

[0081] By labeling multiple annotation dimensions, the annotation results can more comprehensively and accurately reflect the rich semantics and actual application scenarios of the instruction data.

[0082] Optionally, the specific dimensions to be labeled can be determined based on the actual needs of the target field.

[0083] An existing large model (a large model that has been trained and does not require supervised fine-tuning) can be used to annotate each instruction data with labels of multiple annotation dimensions. As an example, for the i-th (i=1, 2, 3, ..., I; I is the number of instruction data in the instruction data set) instruction data, the i-th instruction data can be added to the instruction data slot of the annotation prompt word template to obtain the annotation prompt word corresponding to the i-th instruction data; the annotation prompt word template also includes an annotation task instruction, which indicates that the instruction data in the instruction data slot should be annotated with labels of multiple annotation dimensions from multiple annotation dimensions. The annotation prompt word corresponding to the i-th instruction data is input into the existing large model to obtain a label set corresponding to the i-th instruction data, which includes labels of multiple granularities, labels of multiple perspectives, and labels of at least two annotation dimensions among multiple context-related labels. Assume that the label set corresponding to the i-th instruction data includes labels of k granularities (the k labels form a hierarchical label system from coarse granularity to fine granularity), labels of m perspectives, and There are k+m+ context-related labels in the label set corresponding to the i-th instruction data. tags.

[0084] As shown in Table 1, it is an example of the marking prompt words provided in the embodiment of the present application.

[0085] Table 1

[0086]

[0087] In the example shown in Table 1, "{instruction}" is the instruction data slot, which is used to fill the instruction data to be annotated. The i-th instruction data can be used to replace "instruction" in the annotation prompt word template to obtain the annotation prompt word corresponding to the i-th instruction data.

[0088] Step S102: corresponding to each tag set, clustering the tags in the tag set to obtain multiple tag clusters.

[0089] Optionally, a rule-based clustering method may be used to cluster the labels in the label set corresponding to the i-th instruction data (denoted as the i-th label set). Other clustering algorithms may also be used to cluster the labels in the i-th label set. Clustering algorithms may include, but are not limited to, any of the following: a K-means clustering algorithm, a hierarchical clustering algorithm, and the like.

[0090] Step S103: For each tag cluster, perform a quality assessment on the tag cluster from at least two of the four assessment dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching, and obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions.

[0091] Alternatively, an existing large model can be used to perform a comprehensive quality assessment on the i-th tag cluster based on at least two of the four evaluation dimensions: semantic accuracy, logical consistency, domain relevance, and user need matching, to obtain a comprehensive quality score for the i-th tag cluster. That is, the large model directly outputs the comprehensive quality score of the i-th tag cluster based on at least two evaluation dimensions.

[0092] Optionally, the specific dimensions to be evaluated can be determined based on the actual needs of the target area.

[0093] Step S104: clustering the instruction data set according to the semantic similarity between each instruction data and each label to obtain multiple instruction data clusters.

[0094] The pre-trained word vector model can be used to convert instruction data and labels into vector representations, and the semantic similarity between them can be calculated based on the vector representation of the instruction data and the vector representation of the label.

[0095] The instruction data set may be clustered using any of the following clustering algorithms, but not limited to: K-means clustering algorithm, hierarchical clustering algorithm, etc.

[0096] This application can use the semantic similarity between the i-th instruction data and each label in all label sets as the target vector representation of the i-th instruction data, and then use the target vector representation of each instruction data in the instruction data set to cluster the instruction data in the instruction data set using a clustering algorithm to obtain multiple instruction data clusters.

[0097] Step S105: determining a diversity score of the instruction data set according to the distribution uniformity of the multiple instruction data clusters, the distribution of instruction lengths of the instruction data set, and the label distribution of all label sets.

[0098] In other words, the diversity score of instruction data is obtained by comprehensively considering the distribution uniformity of multiple instruction data clusters, the distribution of instruction lengths of the instruction data set, and the label distribution of all label sets.

[0099] Statistical feature analysis can be performed on the instruction data set and each label set to determine the distribution of instruction lengths and label distribution.

[0100] Optionally, the distribution uniformity of the multiple instruction data clusters can be represented by the standard deviation of the sizes of the instruction data clusters. The size of each instruction data cluster can be the number of instruction data contained in the instruction data cluster. The smaller the standard deviation, the more uniform the cluster sizes.

[0101] Optionally, the distribution uniformity of the multiple instruction data clusters can be represented by the variance of the sizes of the instruction data clusters. The smaller the variance, the more uniform the cluster sizes.

[0102] Optionally, the distribution uniformity of the multiple instruction data clusters can be represented by the ratio of the size of the largest instruction cluster to the size of the smallest instruction cluster. The closer the ratio is to 1, the more uniform the cluster sizes are.

[0103] Optionally, the distribution uniformity of multiple instruction data clusters can be represented by the variance of the instruction data at the cluster center of each instruction data cluster. The smaller the variance, the more uniform the position distribution of the clusters.

[0104] Step S106: Determine the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix.

[0105] The semantic association matrix represents the semantic similarity between each instruction data in the instruction dataset and each label in all label sets.

[0106] That is, the complexity score of the instruction data is obtained by comprehensively considering at least the density of each instruction data cluster and the density of the semantic association matrix.

[0107] Optionally, the density of the instruction data cluster may be the intra-cluster distance, which refers to the average distance between instruction data within the cluster. The smaller the intra-cluster distance, the higher the cluster density; the larger the intra-cluster distance, the lower the cluster density.

[0108] Optionally, the density of the instruction data cluster can be the cluster radius. The cluster radius refers to the maximum distance between all instruction data in the cluster and the instruction data at the cluster center. The smaller the cluster radius, the higher the cluster density; the larger the cluster radius, the lower the cluster density.

[0109] The density of the semantic association matrix may be the ratio of non-zero elements in the semantic association matrix.

[0110] It should be noted that this application does not specifically limit the execution order of step S105 and step S106. Step S105 can be executed first and then step S106, or step S106 can be executed first and then step S105, or both steps can be executed simultaneously.

[0111] The training data quality inspection method provided in the embodiment of the present application labels each instruction data from multiple annotation dimensions to obtain a label set for each instruction data, which includes labels of at least two of the following annotation dimensions: labels of multiple granularities, labels of multiple perspectives, and labels associated with multiple contexts; clusters the labels in each label set to obtain multiple label clusters corresponding to the label set, and performs quality assessment on each label cluster from at least two of the four assessment dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching to obtain a comprehensive quality score based on at least two assessment dimensions for different label clusters corresponding to each instruction data, thereby improving the accuracy of the quality assessment; in addition, the diversity score of the instruction data set is determined based on the distribution uniformity of the multiple instruction data clusters obtained by clustering the instruction data, the distribution of the instruction lengths of the instruction data set, and the label distribution of all label sets; the complexity score of the instruction data set is determined based on at least the density of each instruction data cluster and the density of the semantic association matrix, thereby improving the accuracy of the instruction diversity and complexity assessment. In summary, the present application improves the accuracy of instruction data quality inspection in training data.

[0112] In an optional embodiment, a flowchart for implementing clustering of tags in the i-th tag set is as follows: Figure 2 As shown, this may include:

[0113] Step S201: semantically understand the i-th tag set to obtain a semantic understanding result corresponding to the tag set. The semantic understanding result includes: the semantic meaning of each tag in the tag set, the semantic association between tags, and the potential hierarchical structure.

[0114] The semantic associations between tags may include, but are not limited to, synonymy, near-synonymy, and hyponymy.

[0115] The potential hierarchical structure can be to organize the i-th tag set into a hierarchical structure based on the hierarchical relationship between tags.

[0116] The existing large model can be used to perform semantic understanding on the i-th tag set. In addition to the semantic meaning of the tags, the semantic associations between tags, and the potential hierarchical structure, the semantic understanding results can also include other information, such as the semantic importance of each tag. As an example, the i-th tag set can be added to the tag slot of the semantic understanding prompt word template to obtain the semantic understanding prompt word corresponding to the i-th tag set; the semantic understanding prompt word template also includes semantic understanding task instructions, which indicate the semantic analysis and hierarchical classification of the tag set in the tag slot. The semantic understanding prompt word corresponding to the i-th tag set is input into the existing large model to obtain the semantic understanding result corresponding to the i-th tag set.

[0117] As shown in Table 2, this is an example of the semantic understanding prompt words provided in the embodiment of the present application.

[0118] Table 2

[0119]

[0120] In the example shown in Table 2, "{tags}" is a tag slot used to fill the tag set. The i-th tag set can be used to replace "tags" in the semantic understanding prompt word template to obtain the semantic understanding prompt word corresponding to the i-th tag set.

[0121] Optionally, before semantic understanding is performed on the i-th tag set, the i-th tag set may be preprocessed, for example, duplicate tags may be removed.

[0122] In order to help relevant personnel better understand the quality inspection process, statistical analysis can be performed on the labels in each pre-processed label set to determine the frequency and distribution of each label and output it.

[0123] Step S202: clustering (also called aggregation) the tags in the tag set based on the semantic understanding result to obtain multiple tag clusters.

[0124] Based on the semantic understanding results corresponding to the i-th label set, the labels in the i-th label set are clustered to obtain the i-th label set and corresponding multiple label clusters.

[0125] In an optional embodiment, a flowchart for implementing clustering of tags in the i-th tag set based on the semantic understanding result corresponding to the i-th tag set is shown as follows: Figure 3 As shown, this may include:

[0126] Step S301: performing initial clustering on the tags in the i-th tag set to obtain multiple initial tag clusters corresponding to the i-th tag set; the tags in each initial tag cluster are tags with similar semantics or with a hierarchical relationship.

[0127] Here, we perform an initial clustering of the labels in the i-th label set based on rules. Semantically similar labels are those that are semantically synonymous or nearly so. For example, if we consider a hierarchical classification where "apple" and "orange" are grouped into the "fruit" category, and "fruit" is grouped into the "food" category, we can group the four labels "apple," "orange," "fruit," and "food" into one initial label cluster.

[0128] Step S302: Treat the tags in the i-th tag set as nodes in a heterogeneous graph and use a graph neural network algorithm to update the vector representation of each tag. The initial vector representation of each tag is determined based on the semantic meaning of the tag. Specifically, the initial vector representation of each tag can be obtained by using a large model (e.g., a large language model) to perform semantic understanding on the tag to obtain the semantic meaning of the tag. The large model then converts the semantic meaning of the tag into a dense vector (i.e., an embedding feature) as the initial vector representation of the tag.

[0129] In a heterogeneous graph, labels belonging to the same initial label cluster have edges. By passing messages between nodes, we can use a graph neural network algorithm to update the vector representation of each label. The specific implementation process can be found in existing implementations and will not be detailed here.

[0130] Step S303: clustering the labels in the i-th label set based on the updated vector representation of each label to obtain a plurality of optimized label clusters.

[0131] The tags in the i-th tag set may be clustered using any of the following clustering algorithms, but not limited to: K-means clustering algorithm, hierarchical clustering algorithm, etc.

[0132] In an optional embodiment, when the semantic understanding result corresponding to the i-th tag set includes the semantic importance of each tag in the i-th tag set, a flowchart for implementing the initial clustering of the i-th tag set is as follows: Figure 4 As shown, this may include:

[0133] Step S401: For each tag, the frequency of occurrence of the tag in all tag sets (i.e., I tag sets), the semantic importance of the tag, and the user evaluation corresponding to the tag are weighted and summed to obtain the weight of the tag.

[0134] Assume that there are n labels in the i-th label set and the j-th label is t j , j = 1, 2, 3, ..., n. Then the label t j The weight w j It can be:

[0135] .

[0136] Among them, f j For label t j The frequency of occurrence in the I tag set, that is, the frequency of occurrence of tag t in the I tag set j The ratio of the number of occurrences to the total number of tags in the tag set I; s j For label t j The semantic importance of u j For label t j Corresponding user evaluation, for example, the label t provided by the user j Frequency of use, etc. 、 and is the weight coefficient; The weight of each tag can be adjusted dynamically. For details on how to adjust it, please refer to the following examples.

[0137] Step S402: Based on the weights of the individual tags in the i-th tag set, the i-th tag set is screened and merged to remove tags with weights less than the target weight and tags with lower weights among semantically similar tags, thereby obtaining an optimized tag set.

[0138] By removing tags with weights less than the target weight and tags with lower weights among semantically similar tags, we can reduce redundancy while retaining key information, forming a refined i-th tag set, so that the final tag aggregation result is semantically clear and accurate.

[0139] Step S403: Initially cluster the tags in the optimized tag set. By dynamically adjusting weights and combining graph neural networks, the final tag set is made more refined, semantically clear, and in line with actual application scenarios and user needs.

[0140] In an optional embodiment, for each tag cluster, a quality assessment is performed on the tag cluster from at least two of the four assessment dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching. Figure 5 As shown, this may include:

[0141] Step S501: Use the large model to perform quality assessment on the tag cluster in at least two of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching, and obtain the score of the tag cluster in each of the at least two evaluation dimensions.

[0142] The large model here is an existing large model, that is, a large model that has been trained and does not require supervised fine-tuning.

[0143] For the rth (r=1, 2, 3, ..., R; R is the number of multiple tag clusters corresponding to the i-th tag set) tag cluster, the rth tag cluster can be added to the tag slot of the quality assessment prompt word template to obtain the quality assessment prompt word corresponding to the rth tag cluster; the quality assessment prompt word template also includes a quality assessment task instruction, which instructs to give the tag cluster in the tag slot a score in at least two of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching;

[0144] The quality assessment prompt words corresponding to the r-th tag cluster are input into the large model to obtain the scores of at least two evaluation dimensions of the r-th tag cluster, including the scores in semantic accuracy, logical consistency, domain relevance, and user demand matching.

[0145] As shown in Table 3, it is an example of quality assessment prompt words provided in the embodiment of the present application.

[0146] Table 3

[0147]

[0148] In the example shown in Table 3, "{tags}" is a tag slot used to fill the rth tag cluster. The rth tag cluster can be used to replace "tags" in the quality assessment prompt word template to obtain the quality assessment prompt word corresponding to the rth tag cluster.

[0149] Step S502: Weighted sum of the scores of the tag cluster in the at least two evaluation dimensions is performed to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions. In other words, this application uses the comprehensive quality scores of each tag cluster corresponding to the i-th instruction data to represent the comprehensive quality score of the i-th instruction data.

[0150] Optionally, the comprehensive quality scores of the tag clusters corresponding to the i-th instruction data may be averaged to obtain the comprehensive quality score of the i-th instruction data.

[0151] When the weighted sum of the scores of the tag cluster in the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching is taken, the formula can be expressed as:

[0152] .

[0153] Among them, corresponding to the r-th label cluster, G represents the comprehensive quality score of the r-th label cluster obtained based on the large model in the above four evaluation dimensions; SA G represents the score of the rth label cluster obtained by the large model evaluation in terms of semantic accuracy; LC GDR represents the score of the rth label cluster obtained by the large model evaluation in terms of logical consistency; G represents the domain relevance score of the rth label cluster obtained by the large model evaluation; UM G It represents the score of the rth tag cluster obtained by the large model evaluation in terms of user demand matching; 、 、 、 are weight coefficients respectively.

[0154] Figure 5 In the embodiment shown, only the large model is used to evaluate the four evaluation dimensions. In an optional embodiment, the rth tag cluster can be evaluated in at least two evaluation dimensions in combination with manual evaluation to obtain a comprehensive quality score of the rth tag cluster in the at least two evaluation dimensions. Based on this, the present application provides another implementation flow chart for performing quality evaluation on each tag cluster from at least two of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching. Figure 6 As shown, this may include:

[0155] Step S601: Use the large model to perform quality assessment on the tag cluster in at least two of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching, and obtain the score of the tag cluster in each of the at least two evaluation dimensions.

[0156] For the rth tag cluster, the process of using the large model to evaluate the quality of at least two of the four evaluation dimensions of semantic accuracy, logical consistency, domain relevance and user demand matching can be found in Figure 5 The embodiments shown are not described in detail here.

[0157] Step S602: performing weighted summation of the scores of the tag cluster in the above two evaluation dimensions to obtain a first comprehensive quality score of the tag cluster in the above at least two evaluation dimensions.

[0158] For the specific implementation process, please refer to Figure 5 The embodiments shown are not described in detail here.

[0159] Step S603: Obtain a second comprehensive quality score for the tag cluster. The second comprehensive quality score is obtained based on a weighted sum of manual scores for the tag cluster in the at least two evaluation dimensions.

[0160] When the second comprehensive quality score is obtained based on the weighted sum of the manual scores of the tag cluster in the above four evaluation dimensions, it can be expressed as follows:

[0161] .

[0162] Among them, corresponding to the rth label cluster, H represents the comprehensive quality score of the rth label cluster in the above four evaluation dimensions based on manual evaluation; SA H represents the score of the rth label cluster obtained by manual evaluation in terms of semantic accuracy; LC H DR represents the score of the rth label cluster obtained by manual evaluation in terms of logical consistency; H represents the domain relevance score of the rth label cluster obtained by manual evaluation; UM H represents the score of the rth tag cluster obtained by manual evaluation in terms of user demand matching; 、 、 、 are weight coefficients respectively.

[0163] The weight coefficients used to calculate the overall quality score, whether based on manual evaluation or large-scale model evaluation, can be adjusted based on the actual evaluation objectives and the priority of the evaluation dimensions to ensure that the overall quality score accurately reflects the overall quality of the label. The higher the priority of the evaluation dimension, the larger the corresponding weight coefficient.

[0164] Optionally, the weight coefficient used in calculating the comprehensive quality score may be a constant, which may be determined based on the actual evaluation objective, the priority of the evaluation dimension, etc. As an example, the weight coefficient may be determined based on a preset mapping relationship between the actual evaluation objective and the priority of the evaluation dimension and the weight coefficient.

[0165] Optionally, the weight coefficients used in calculating the comprehensive quality score can also be adjustable. Based on this, the initial values ​​of the weight coefficients used in calculating the comprehensive quality score can be determined based on the actual evaluation objectives, the priorities of the evaluation dimensions, and so on. As an example, the initial values ​​of the weight coefficients can be determined based on a preset mapping relationship between the actual evaluation objectives and the priorities of the evaluation dimensions and the weight coefficients. The weight coefficients can subsequently be updated based on a learning algorithm (e.g., a reinforcement learning algorithm) and / or other information (including but not limited to at least one of the following: historical comprehensive quality scores, user feedback data, etc.).

[0166] Step S604: performing weighted summation of the first comprehensive quality score and the second comprehensive quality score of the tag cluster to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions.

[0167] It can be expressed as:

[0168] .

[0169] Among them, corresponding to the rth label cluster, Q represents the comprehensive quality score of the rth label cluster in at least two evaluation dimensions. and 1- is the weight coefficient; The value range of is [0,1], and its value can be adjusted according to the actual application scenario and the degree of manual evaluation.

[0170] Human evaluation focuses on the rationality of labels in real-world application scenarios and the user experience, while large-scale model evaluation leverages its powerful semantic understanding and reasoning capabilities to conduct in-depth analysis of labels' semantic coherence and accuracy within the knowledge system. By combining human and large-scale model evaluation, the results of these two methods complement and correct each other, further improving the accuracy and reliability of the evaluation.

[0171] Whether it is manual evaluation or large-scale model evaluation, the evaluation results of different dimensions (at least two evaluation dimensions of semantic accuracy, logical consistency, domain relevance and user demand matching) are organically integrated to generate a comprehensive score that fully reflects the quality of the label.

[0172] Furthermore, after obtaining the comprehensive quality assessment results of each label cluster, the comprehensive quality scores of each label cluster of each instruction data can be statistically analyzed and visualized, so that quality inspectors can better understand and utilize these quality assessment data, providing stronger support for subsequent model optimization and application.

[0173] In an optional embodiment, the distribution of instruction lengths may include distribution entropy of instruction lengths; and the distribution of tags may include distribution entropy of tags.

[0174] The distribution entropy of instruction length can be determined by calculating the frequency of the length of each instruction data in the length of all instruction data; and calculating the distribution entropy of instruction length using the following formula:

[0175] .

[0176] Among them, E1 represents the distribution entropy of instruction length, reflecting the diversity of instruction length; p 1i Indicates the frequency of occurrence of the length of the i-th instruction data in the lengths of all instruction data.

[0177] The distribution entropy of a tag can be determined by calculating the frequency of each tag in the set of all tags. The distribution entropy of a tag can be calculated using the following formula:

[0178] .

[0179] Among them, E2 represents the distribution entropy of the label, reflecting the diversity of the label; p2j Indicates the frequency of the jth tag in the set of all tags.

[0180] Accordingly, one implementation method for determining the diversity score of the instruction data set based on the distribution uniformity of multiple instruction data clusters, the distribution of instruction lengths of the instruction data set, and the label distribution of all label sets may be:

[0181] The diversity score is obtained by weighted summing the distribution uniformity of multiple instruction data clusters, the distribution entropy of instruction length, and the distribution entropy of labels. It can be expressed as:

[0182] .

[0183] Among them, Div is the diversity score of the instruction dataset, H tag represents the distribution entropy of the label, H length represents the distribution entropy of instruction length, H cluster Indicates the distribution uniformity of multiple instruction data clusters. 、 and is the weight coefficient. .

[0184] Furthermore, after obtaining the diversity score of the instruction data set, data can be added or deleted from the instruction data set based on the size of each instruction data cluster to avoid the long-tail problem of the data distribution in the instruction data set. Optionally, if the size of the instruction data cluster is greater than a first threshold, a preset proportion of the instruction data cluster can be sampled to delete the preset proportion of instruction data; if the size of the instruction data cluster is less than a second threshold, data enhancement can be performed on the instruction data cluster to increase the size of the instruction data cluster. The second threshold is less than the first threshold. The modified data set is subsequently subjected to another quality inspection.

[0185] In an optional embodiment, one implementation of determining the complexity score of the instruction data set based at least on the density of each instruction data cluster and the density of the semantic association matrix may be:

[0186] The complexity score of the instruction data set is obtained by performing a weighted summation of the average value of the density of each instruction data cluster and the density of the semantic association matrix.

[0187] It can be expressed as:

[0188] .

[0189] Among them, Com1 represents the complexity score of the instruction dataset, D cluster-1 Indicates the average value of the density of each instruction data cluster; D semantic-1It represents the density of the semantic association matrix, reflecting the semantic complexity of instruction data. and is the weight coefficient, The density of instruction data clusters reflects the complexity of the data structure.

[0190] In an optional embodiment, when the instruction data set is hierarchically clustered according to the semantic similarity between each instruction data and each tag, another implementation of determining the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix may be:

[0191] The average value of the density of each instruction data cluster is weighted and summed with the depth of the instruction data cluster to obtain the fusion feature of the instruction data cluster.

[0192] The complexity score of the instruction data set is obtained by weighted summing the fusion features of the instruction data cluster and the density of the semantic association matrix.

[0193] It can be expressed as:

[0194] .

[0195] Among them, Com2 represents the complexity score of the instruction dataset, D cluster-2 Indicates the fusion characteristics of the instruction data cluster; D semantic-2 Represents the density of the semantic association matrix. and is the weight coefficient.

[0196] The diversity assessment and complexity assessment of this application combine the advantages of statistical analysis and semantic understanding, which can reveal the intrinsic characteristics of the instruction dataset more comprehensively and deeply, and provide more accurate guidance for model training and optimization.

[0197] In an optional embodiment, to further improve quality inspection accuracy, this application also provides a dynamic iterative optimization mechanism based on real-time feedback, building a complete quality inspection closed loop. This mechanism aims to continuously improve the quality of instruction data by adjusting parameters such as labeling rules and label aggregation weights through continuous monitoring of historical evaluation results, user behavior data, and the performance of the large model (referring to the large model constructed using training data supervised and fine-tuned after quality inspection of instruction data and quality inspection answers). Based on this, after completing the quality assessment and diversity and complexity analysis of the instruction data, the training data quality inspection method of this application may also include:

[0198] The gradient of the comprehensive quality score with respect to the weight coefficient to be updated is calculated based on the comprehensive quality score and user feedback data; the weight coefficient to be updated is at least part of the weight coefficient used in the quality inspection process.

[0199] User feedback data may include but is not limited to at least one of the following data: the frequency of use of user-provided labels in actual application scenarios in the field to which the instruction dataset to be quality-checked belongs, the accuracy log of the large model output after supervised fine-tuning, etc.

[0200] The initial value of the weight coefficient to be updated (i.e., the weight coefficient during the first quality inspection) can be an empirical value or an average value, that is, in the same weight calculation formula, all weight coefficients are the same.

[0201] Optionally, backpropagation or numerical differentiation can be used to calculate the gradient of the comprehensive quality score with respect to the updated weight coefficient. User feedback data can be used to assist in gradient calculation using heuristic rules (for example, increasing the weight of high-frequency tags in user feedback). The process of using heuristic rules to assist gradient calculation can be found in existing solutions and will not be detailed here.

[0202] The weight of the tag changes dynamically and can dynamically adapt to different application scenarios and user needs.

[0203] Optionally, when calculating the gradient of the weight coefficient to be updated in the comprehensive instruction score, reference can also be made to the dynamic adjustment suggestions for label weights given by existing large models when performing semantic understanding and / or the weight improvement suggestions given by existing large models when performing quality assessment. The dynamic adjustment suggestions for label weights given by existing large models when performing semantic understanding and / or the weight improvement suggestions given by existing large models when performing quality assessment can assist in gradient calculation through heuristic rules (for example, the weight of "technology" related labels should be increased in science and technology tasks, etc.).

[0204] The weight coefficients to be updated are updated based on the gradient with the goal of increasing the overall quality score. An increase in the overall instruction score means that the overall quality score obtained by quality checking the instruction dataset based on the updated weight coefficients is increased compared to before the weight coefficients were updated.

[0205] The weight coefficients can be updated by an update function based on the gradient direction and the learning rate λ. For example, for any weight coefficient, if the gradient of the weight coefficient is positive, the influence of the gradient on the weight coefficient is increased, and if the gradient of the weight coefficient is negative, the influence of the gradient on the weight coefficient is reduced.

[0206] The above quality inspection process can be optimized for multiple iterations. Assume that the weight coefficient set for the kth quality inspection is for: , then the weight coefficient set for the k+1th quality inspection is It can be:

[0207] .

[0208] in, Q is the comprehensive quality score parameter The gradient, is the learning rate.

[0209] The instruction data set to be quality checked is quality checked again based on the updated weight coefficient.

[0210] Each time the weight coefficient is updated, the updated weight coefficient set can be used to perform quality inspection iterations on the instruction data set to be inspected.

[0211] Furthermore, each time the weight coefficient is updated, the updated weight coefficient can be output so that quality inspectors can adjust the prompt word template (for example, the labeling prompt word template, the semantic understanding prompt word template, the quality assessment prompt word template, etc.) according to the updated weight coefficient.

[0212] Furthermore, when adjusting the standard prompt word template, quality inspectors can also adjust the annotation prompt word template based on the frequency of use of tags in actual application scenarios in user feedback data. For example, if users report that a certain tag is frequently used, relevant sub-tags can be added to the annotation prompt word template. For example, in the financial field, if users report that the tag "portfolio optimization" is frequently used, relevant sub-tags (such as "stock portfolio optimization" and "bond portfolio optimization") can be added to the annotation prompt word template.

[0213] When the instruction data to be quality-checked is instruction data in the financial field, the training data quality inspection method of this application can significantly improve the timeliness and domain adaptability of the financial large model SFT training data.

[0214] The closed-loop feedback mechanism of this application upgrades the quality inspection process from a static process to a dynamic optimization process, including the following aspects:

[0215] Weight coefficient adaptation: Gradient descent is used to optimize weight coefficients, making label clustering and quality assessment more suitable for actual scenarios;

[0216] Human-machine collaborative iteration: Combining user feedback data with large-scale model analysis to achieve two-way optimization of labeling rules;

[0217] Data ecosystem governance: Dynamically add and delete data based on diversity analysis to solve the long-tail problem of data distribution.

[0218] This dynamic optimization process not only continuously improves data quality but also dynamically adjusts strategies (e.g., weight coefficients, prompt word templates, etc.) based on user needs and domain changes, ensuring the optimal diversity and complexity of SFT training data. This ultimately creates a flywheel effect of "labeling-evaluation-optimization-relabeling," significantly improving the timeliness and domain adaptability of SFT training data for large models in target domains (e.g., financial models).

[0219] In summary, this application proposes an innovative annotation method that combines multi-granularity, multi-perspective and contextual association, which can comprehensively and accurately mine the deep semantics and practical application scenarios of instruction data. This technology mines instruction intent from different levels from coarse granularity to fine granularity, and comprehensively annotates from multiple angles such as user needs, task objectives, and domain characteristics. It also considers the contextual relationship of instructions in specific dialogues or task processes, and annotates intent labels that are closely related to the context, ensuring that the annotation results fully cover the semantics and application scenarios of instruction data. On this basis, an intelligent semantic aggregation technology that combines semantic understanding, hierarchical classification and dynamic weight adjustment is introduced. A large model is used to perform semantic understanding of labels, identify semantic associations and potential hierarchical structures between labels, aggregate labels into different hierarchical categories, and form a hierarchical label system. During the aggregation process, a dynamic weight adjustment mechanism is introduced to automatically adjust the weight of each label based on factors such as the frequency of occurrence of the label, semantic importance, and user feedback, reducing redundancy while retaining key information.

[0220] This application also constructs a multi-dimensional evaluation system that integrates manual and large model capabilities, and comprehensively considers the quality of labels from multiple perspectives such as semantic accuracy, logical consistency, domain relevance, and user demand matching. Manual evaluation focuses on the rationality of labels in actual application scenarios and user experience, while large model evaluation uses its powerful semantic understanding and reasoning capabilities to conduct in-depth analysis of the semantic coherence of labels, accuracy in the knowledge system, etc. At the same time, an innovative evaluation index calculation method is introduced to organically integrate the evaluation results of different dimensions to generate a comprehensive score that fully reflects the quality of labels. In addition, a dynamic feedback mechanism is designed to enable the results of manual evaluation and large model evaluation to complement and correct each other, continuously improving the accuracy and reliability of the evaluation. In order to address the limitations of traditional data set analysis methods in diversity and complexity assessment, the present invention proposes an innovative method that integrates statistical feature analysis, semantic association mining, and hierarchical clustering. First, statistical feature extraction is performed on the instructions and labels in the data set, and basic statistical information such as the distribution of labels and the length distribution of instructions is analyzed. Then, the semantic similarity between the instruction data and the label is calculated to construct a semantic association matrix. Next, a hierarchical clustering algorithm is used to partition the dataset into clusters based on semantic similarity and statistical features, forming a hierarchical data structure. The diversity and complexity of the dataset are quantitatively assessed by analyzing cluster distribution, depth, and density.

[0221] This application also provides a dynamic iterative mechanism based on real-time feedback, which aims to achieve continuous improvement in data quality by automatically or semi-automatically adjusting the labeling rules and label aggregation weight coefficients through continuous monitoring of historical evaluation results, user behavior data and model performance. The specific process of this mechanism includes feedback data integration, dynamic strategy adjustment and incremental data update. First, data such as quality assessment results, diversity complexity analysis and user feedback in actual applications are collected; then, reinforcement learning algorithms are used to optimize the parameters in the label prompt word template and label weight calculation formula based on the integrated feedback data; finally, low-quality and redundant data are eliminated or re-labeled, and scarce data is dynamically supplemented based on the diversity score, thereby forming a high-quality iterative version of the training set. By maximizing the objective function Q and gradually optimizing the parameter configuration, the shortcomings of the traditional quality inspection process that is static and lacks closed-loop optimization are overcome, and continuous improvement of data quality and continuous improvement of model performance are achieved.

[0222] Corresponding to the method embodiment, the present application also provides a training data quality inspection device. A structural diagram of the training data quality inspection device provided in the embodiment of the present application is shown as follows: Figure 7 As shown, this may include:

[0223] Labeling module 701, label clustering module 702, quality assessment module 703, instruction data clustering module 704, diversity assessment module 705 and complexity assessment module 706; wherein,

[0224] The labeling module 701 is used to label each instruction data in the instruction data set to be inspected with multiple labeling dimensions to obtain a label set corresponding to each instruction data; the multiple labeling dimensions include at least two of the following labeling dimensions: multi-granularity labeling, multi-perspective labeling, and context-related labeling;

[0225] The tag clustering module 702 is used to cluster the tags in each tag set to obtain multiple tag clusters;

[0226] The quality assessment module 703 is used to perform a quality assessment on each tag cluster based on at least two of the four assessment dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching, to obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions.

[0227] The instruction data clustering module 704 is used to cluster the instruction data set according to the semantic similarity between each instruction data and each tag in all tag sets to obtain multiple instruction data clusters;

[0228] The diversity evaluation module 705 is configured to determine a diversity score of the instruction data set based on the distribution uniformity of the multiple instruction data clusters, the distribution of instruction lengths of the instruction data set, and the label distribution of all label sets;

[0229] The complexity evaluation module 706 is used to determine the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix; the semantic association matrix represents the semantic similarity between each instruction data in the instruction data set and each label.

[0230] The training data quality inspection device provided in the embodiment of the present application labels each instruction data from multiple labeling dimensions to obtain a label set for each instruction data, which includes labels of at least two of the following labeling dimensions: labels of multiple granularities, labels of multiple perspectives, and labels associated with multiple contexts; clusters the labels in each label set to obtain multiple label clusters corresponding to the label set, and performs quality assessment on each label cluster from at least two of the four assessment dimensions of semantic accuracy, logical consistency, domain relevance, and user demand matching to obtain a comprehensive quality score based on at least two assessment dimensions for different label clusters corresponding to each instruction data, thereby improving the accuracy of the quality assessment; in addition, the diversity score of the instruction data set is determined based on the distribution uniformity of the multiple instruction data clusters obtained by clustering the instruction data, the distribution of the instruction lengths of the instruction data set, and the label distribution of all label sets; the complexity score of the instruction data set is determined based on at least the density of each instruction data cluster and the density of the semantic association matrix, thereby improving the accuracy of the instruction diversity and complexity assessment. In summary, the present application improves the accuracy of instruction data quality inspection in training data.

[0231] In an optional embodiment, the tag clustering module 702 clusters the tags in each tag set to obtain multiple tag clusters, and is used to:

[0232] Perform semantic understanding on each tag set to obtain a semantic understanding result corresponding to the tag set; the semantic understanding result includes: the semantic meaning of each tag in the tag set, the semantic association between the tags, and the potential hierarchical structure;

[0233] The tags in the tag set are clustered based on the semantic understanding result to obtain multiple tag clusters.

[0234] In an optional embodiment, when the tag clustering module 702 clusters the tags in the tag set based on the semantic understanding result, it is configured to:

[0235] Performing initial clustering on the tags in the tag set to obtain multiple initial tag clusters; the tags in each initial tag cluster are tags with similar semantics or with a hierarchical relationship;

[0236] The tags in the tag set are treated as nodes in a heterogeneous graph, and the graph neural network algorithm is used to update the vector representation of each tag. The initial vector representation of each tag is determined based on the semantic meaning of the tag.

[0237] The labels in the label set are clustered based on the updated vector representation of each label to obtain multiple optimized label clusters.

[0238] In an optional embodiment, the semantic understanding result further includes: the semantic importance of each tag;

[0239] When the tag clustering module 702 performs initial clustering on the tag set, it is used to:

[0240] For each tag, the weight of the tag is obtained by adding the frequency of occurrence of the tag in all tag sets, the semantic importance of the tag, and the user evaluation corresponding to the tag.

[0241] Based on the weight of each tag in the tag set, the tag set is filtered and merged to obtain an optimized tag set;

[0242] Initial clustering is performed on the optimized label set.

[0243] In an optional embodiment, the quality assessment module 703 performs quality assessment on each tag cluster from at least two assessment dimensions to obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions, and is used to:

[0244] Using the large model, quality assessment is performed on the tag cluster in the at least two assessment dimensions respectively to obtain a score for the tag cluster in each of the at least two assessment dimensions;

[0245] The scores of the tag cluster in the at least two evaluation dimensions are weighted and summed to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions.

[0246] In an optional embodiment, the quality assessment module 703 performs quality assessment on each tag cluster from at least two assessment dimensions to obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions, and is used to:

[0247] Using the large model, quality assessment is performed on the tag cluster in the at least two assessment dimensions respectively to obtain a score for the tag cluster in each of the at least two assessment dimensions;

[0248] A weighted sum of the scores of the tag cluster in the at least two evaluation dimensions is performed to obtain a first comprehensive quality score of the tag cluster in the at least two evaluation dimensions;

[0249] Obtaining a second comprehensive quality score for the tag cluster; the second comprehensive quality score is obtained based on a weighted sum of manual scores of the tag cluster in the at least two evaluation dimensions;

[0250] The first comprehensive quality score and the second comprehensive quality score are weightedly summed to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions.

[0251] In an optional embodiment, the distribution of instruction lengths includes distribution entropy of instruction lengths; the label distribution includes distribution entropy of labels;

[0252] When determining the diversity score of the instruction data set based on the distribution uniformity of the multiple instruction data clusters, the distribution of instruction lengths of the instruction data set, and the label distribution of all label sets, the diversity assessment module 705 is configured to:

[0253] The diversity score is obtained by weighted summing the distribution uniformity of the multiple instruction data clusters, the distribution entropy of the instruction length, and the distribution entropy of the label.

[0254] In an optional embodiment, when determining the complexity score of the instruction data set based at least on the density of each instruction data cluster and the density of the semantic association matrix, the complexity evaluation module 706 is configured to:

[0255] The complexity score of the instruction data set is obtained by performing a weighted summation of the average value of the density of each instruction data cluster and the density of the semantic association matrix.

[0256] In an optional embodiment, when clustering the instruction data set according to the semantic similarity between each instruction data and each tag, the instruction data clustering module 704 is configured to: perform hierarchical clustering on the instruction data set according to the semantic similarity between each instruction data and each tag;

[0257] The complexity evaluation module 706 is configured to determine the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix:

[0258] The average value of the density of each instruction data cluster is weighted and summed with the depth of the instruction data cluster to obtain the fusion feature of the instruction data cluster;

[0259] The fusion feature of the instruction data cluster and the density of the semantic association matrix are weightedly summed to obtain a complexity score of the instruction data set.

[0260] In an optional embodiment, the training data quality inspection device further includes an optimization module for:

[0261] Calculating the gradient of the comprehensive quality score with respect to the weight coefficient to be updated based on the comprehensive quality score and user feedback data; the weight coefficient to be updated is at least part of the weight coefficient used in the quality inspection process;

[0262] With the goal of increasing the comprehensive quality score, updating the weight to be updated based on the gradient;

[0263] The instruction data set to be quality-checked is quality-checked again based on the updated weights.

[0264] An electronic device is also provided in an embodiment of the present application. Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can be a terminal device (such as a car computer, a large-screen device, a smart home, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 8 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0265] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 802 or programs loaded from a storage device 808 into a random access memory (RAM) 803. When the electronic device is powered on, the RAM 803 also stores various programs and data required for the operation of the electronic device. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0266] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a memory card, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 8 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0267] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the training data quality inspection methods provided in the embodiments of the present application.

[0268] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any training data quality inspection method provided in the embodiment of the present application.

[0269] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0270] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0271] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0272] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0273] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0274] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training data quality inspection method, characterized in that: include: Label each instruction data in the instruction data set to be inspected with multiple labeling dimensions to obtain a label set corresponding to each instruction data; The multiple annotation dimensions include at least two of the following annotation dimensions: multi-granularity annotation, multi-perspective annotation, and context-related annotation; For each label set, cluster the labels in the label set to obtain multiple label clusters; For each tag cluster, perform a quality assessment on the tag cluster based on at least two of the four evaluation dimensions: semantic accuracy, logical consistency, domain relevance, and user demand matching, to obtain a comprehensive quality score for the tag cluster in the at least two evaluation dimensions; Clustering the instruction data set according to the semantic similarity between each instruction data and each label in all label sets to obtain multiple instruction data clusters; Determining a diversity score of the instruction data set based on distribution uniformity of the multiple instruction data clusters, distribution of instruction lengths of the instruction data set, and label distribution of all label sets; The complexity score of the instruction data set is determined at least according to the density of each instruction data cluster and the density of a semantic association matrix; the semantic association matrix represents the semantic similarity between each instruction data in the instruction data set and each label.

2. The method according to claim 1, characterized in that For each tag set, the tags in the tag set are clustered to obtain multiple tag clusters, including: Perform semantic understanding on each tag set to obtain a semantic understanding result corresponding to the tag set; the semantic understanding result includes: the semantic meaning of each tag in the tag set, the semantic association between the tags, and the potential hierarchical structure; The tags in the tag set are clustered based on the semantic understanding result to obtain multiple tag clusters.

3. The method according to claim 2, characterized in that Clustering the tags in the tag set based on the semantic understanding result includes: Performing initial clustering on the tags in the tag set to obtain multiple initial tag clusters; the tags in each initial tag cluster are tags with similar semantics or with a hierarchical relationship; The tags in the tag set are treated as nodes in a heterogeneous graph, and the graph neural network algorithm is used to update the vector representation of each tag. The initial vector representation of each tag is determined based on the semantic meaning of the tag. The labels in the label set are clustered based on the updated vector representation of each label to obtain multiple optimized label clusters.

4. The method according to claim 3, characterized in that The semantic understanding result also includes: the semantic importance of each tag; The initial clustering of the tags in the tag set includes: For each tag, the weight of the tag is obtained by adding the frequency of occurrence of the tag in all tag sets, the semantic importance of the tag, and the user evaluation corresponding to the tag. Based on the weight of each tag in the tag set, the tag set is filtered and merged to obtain an optimized tag set; Initial clustering is performed on the tags in the optimized tag set.

5. The method according to claim 1, wherein For each tag cluster, a quality assessment is performed on the tag cluster from at least two assessment dimensions to obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions, including: Using the large model, quality assessment is performed on the tag cluster in the at least two assessment dimensions respectively to obtain a score for the tag cluster in each of the at least two assessment dimensions; The scores of the tag cluster in the at least two evaluation dimensions are weighted and summed to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions.

6. The method according to claim 1, characterized in that For each tag cluster, a quality assessment is performed on the tag cluster from at least two assessment dimensions to obtain a comprehensive quality score of the tag cluster in the at least two assessment dimensions, including: Using the large model, quality assessment is performed on the tag cluster in the at least two assessment dimensions respectively to obtain a score for the tag cluster in each of the at least two assessment dimensions; A weighted sum of the scores of the tag cluster in the at least two evaluation dimensions is performed to obtain a first comprehensive quality score of the tag cluster in the at least two evaluation dimensions; Obtaining a second comprehensive quality score for the tag cluster; the second comprehensive quality score is obtained based on a weighted sum of manual scores of the tag cluster in the at least two evaluation dimensions; The first comprehensive quality score and the second comprehensive quality score are weightedly summed to obtain a comprehensive quality score of the tag cluster in the at least two evaluation dimensions.

7. The method according to claim 1, characterized in that The distribution of the instruction lengths includes the distribution entropy of the instruction lengths; the label distribution includes the distribution entropy of the labels; Determining the diversity score of the instruction data set according to the distribution uniformity of the multiple instruction data clusters, the distribution of instruction lengths of the instruction data set, and the label distribution of all label sets includes: The diversity score is obtained by weighted summing the distribution uniformity of the multiple instruction data clusters, the distribution entropy of the instruction length, and the distribution entropy of the label.

8. The method according to claim 1, characterized in that The determining of the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix includes: The complexity score of the instruction data set is obtained by performing a weighted summation of the average value of the density of each instruction data cluster and the density of the semantic association matrix.

9. The method according to claim 1, characterized in that The clustering of the instruction data set according to the semantic similarity between each instruction data and each tag in all tag sets includes: performing hierarchical clustering on the instruction data set according to the semantic similarity between each instruction data and each tag; The determining of the complexity score of the instruction data set based on at least the density of each instruction data cluster and the density of the semantic association matrix includes: The average value of the density of each instruction data cluster is weighted and summed with the depth of the instruction data cluster to obtain the fusion feature of the instruction data cluster; The fusion feature of the instruction data cluster and the density of the semantic association matrix are weightedly summed to obtain a complexity score of the instruction data set.

10. The method according to any one of claims 4 to 9, characterized in that: Also includes: Calculating the gradient of the comprehensive quality score with respect to the weight coefficient to be updated based on the comprehensive quality score and user feedback data; the weight coefficient to be updated is at least part of the weight coefficient used in the quality inspection process; With the goal of increasing the comprehensive quality score, updating the weight to be updated based on the gradient; The instruction data set to be quality-checked is quality-checked again based on the updated weights.

11. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the training data quality inspection method according to any one of claims 1 to 10.

12. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the training data quality inspection method according to any one of claims 1 to 10.

Images