A method and system for improving model training sample quality

Abstract

The application discloses a method and system for improving the quality of model training samples, and relates to the field of data processing. The method comprises the following steps: in response to a labeling operation on customer consultation text, obtaining a newly labeled sample, wherein the newly labeled sample comprises customer consultation text and an intention classification label assigned to the customer consultation text; calling a preset text embedding model to convert the customer consultation text into a corresponding text vector; comparing the text vector with a historical sample vector stored in a vector database and having been labeled; sequentially performing in-label similarity verification and cross-label discriminability verification; and processing the newly labeled sample according to the verification result: if the verification is passed, storing the newly labeled sample and the text vector thereof in the vector database; if any verification is not passed, generating a pre-warning prompt containing specific semantic conflict information and pausing the storage process, and then performing subsequent operations according to the pre-warning prompt by a labeler.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for improving the quality of model training samples. Background Technology

[0002] In the printing industry, the quality of training samples is crucial when training large language models to perform text classification tasks; low-quality samples will directly lead to a decline in model performance.

[0003] Currently, mainstream methods for improving sample quality in the industry have significant limitations. While manual review can reduce errors, it is costly, inefficient, and difficult to scale. Automated verification schemes based on preset rules rely on manually formulated rules, which are poorly adaptable to complex semantic scenarios and cannot effectively handle semantic-level annotation conflicts. On the other hand, comparison schemes using traditional word vectors or basic embedding models suffer from high false negative rates due to the limited semantic representation capabilities of the models, especially when processing long or technical texts, resulting in insufficient accuracy.

[0004] In summary, none of the existing solutions have been able to achieve high-precision, low-cost, and real-time label consistency verification with a large set of labels. The industry urgently needs a more effective solution for optimizing sample quality. Summary of the Invention

[0005] This application provides a method and system for improving the quality of model training samples to solve the above-mentioned problems.

[0006] On the one hand, this application provides a method for improving the quality of model training samples, the method comprising the following steps: Step S1: In response to the annotation operation of customer consultation text, obtain new annotated samples, the new annotated samples including customer consultation text and its assigned intent classification label; Step S2: Call a preset text embedding model to convert the customer consultation text into a corresponding text vector; Step S3: Compare the text vector with the annotated historical sample vectors stored in the vector database, and sequentially perform intra-label similarity verification and cross-label discrimination verification, wherein the intra-label similarity verification is used to prevent excessive aggregation of semantically similar samples under a single label, and the cross-label discrimination verification is achieved by calculating the semantic distance between the text vector and other label historical sample vectors and determining whether the semantic distance is greater than a preset threshold; Step S4: Process the new annotated samples according to the verification results: if the verification passes, store the new annotated samples and their text vectors in the vector database; if any verification fails, generate a warning prompt containing specific semantic conflict information and suspend the database entry process, and the annotator performs subsequent operations based on the warning prompt.

[0007] In one implementation of this application, the intra-label similarity verification specifically includes: retrieving all historical sample vectors with the same label as the current newly labeled sample from the vector database; calculating the semantic similarity between the current text vector and each of the historical sample vectors; counting the number of historical samples with a semantic similarity not lower than a first predetermined threshold; determining whether the number exceeds a preset upper limit; if it does not exceed the limit, the intra-label similarity verification passes; if it exceeds the limit, the verification fails, and a first warning message indicating semantic redundancy of samples within the label is generated.

[0008] In one implementation of this application, the cross-label discriminability verification is performed after the intra-label similarity verification passes, specifically including: retrieving historical sample vectors from the vector database whose labels are different from the currently newly labeled sample; calculating the semantic distance between the current text vector and the historical sample vectors of each of the different labels; determining whether there is at least one semantic distance that is not greater than a second predetermined threshold; if all semantic distances are greater than the second predetermined threshold, the cross-label discriminability verification passes; if there is at least one semantic distance that is not greater than the second predetermined threshold, the verification fails, and a second warning message indicating insufficient discriminability with samples under at least one other label is generated, the second warning message including the label and text content prompts of the relevant samples.

[0009] In one implementation of this application, the upper limit of the quantity is dynamically determined based on the number of historical samples under the current label, so as to ensure that the constraint strength on the diversity of the sample set can be adaptively adjusted with the data scale.

[0010] In one implementation of this application, when the cross-label discrimination verification fails, the method further includes: receiving a confirmation instruction input by the labeler for the second warning information; in response to the confirmation instruction being a forced submission, marking the newly labeled sample as pending review and storing it in the vector database, and recording the submission reason.

[0011] In one implementation of this application, the text embedding model is a general pre-trained language model that can map text into high-dimensional semantic vectors. The dimension of its output vector and the length of the input text support the requirement for accurate semantic representation of printed customer consultation texts.

[0012] In one implementation of this application, the vector database is configured with an index structure suitable for fast approximate nearest neighbor search of high-dimensional vectors, and cosine similarity is used as a measure of semantic relevance between vectors.

[0013] In one implementation of this application, in step S2, if the text embedding model fails to generate text vectors, a backup processing procedure is executed. The backup processing procedure includes: performing a limited number of automatic retries; if the retries still fail, the newly labeled sample is stored in the queue to be processed, and a system alarm is triggered.

[0014] In one implementation of this application, if the connection with the vector database is interrupted during the execution of step S3, a local caching and recovery process is initiated. The process includes: temporarily storing the relevant data of the current verification task locally; continuously monitoring the database connection status; and automatically restoring the task status from the local cache and continuing to perform verification after the connection is restored.

[0015] On the other hand, this application also provides a system for improving the quality of model training samples, applying the aforementioned method for improving the quality of model training samples. The system includes: a labeling unit, used to obtain new labeled samples in response to the labeling operation of customer consultation text, the new labeled samples including customer consultation text and its assigned intent classification label; a vector conversion unit, used to call a preset text embedding model to convert the customer consultation text into a corresponding text vector; a comparison and verification unit, used to compare the text vector with the labeled historical sample vectors stored in the vector database, and sequentially perform intra-label similarity verification and cross-label discrimination verification, wherein the intra-label similarity verification is used to prevent excessive aggregation of semantically similar samples under a single label, and the cross-label discrimination verification is achieved by calculating the semantic distance between the text vector and other label historical sample vectors and determining whether the semantic distance is greater than a preset threshold; and a result processing unit, used to process the new labeled samples according to the verification results: if the verification passes, the new labeled sample and its text vector are stored in the vector database; if any verification fails, a warning prompt containing specific semantic conflict information is generated and the database entry process is suspended, and the labeling personnel perform subsequent operations based on the warning prompt.

[0016] This application provides a method and system for improving the quality of model training samples, which has the following beneficial effects:

[0017] 1. Significantly improves annotation consistency. By comparing the semantic vectors of historical samples in real time, it automatically identifies and warns of potential annotation conflicts, effectively reducing label inconsistencies caused by subjective differences among annotators, and ensuring the annotation quality and standardization of the training set from the source.

[0018] 2. Optimize the internal structure and diversity of the sample set. Control the number of semantically overly similar samples within a single label, while ensuring sufficient discriminative power between samples with different labels. This will construct a high-quality dataset that is rich within classes and distinct between classes, providing a solid foundation for model learning.

[0019] 3. Significantly improves annotation efficiency and reduces costs. It enables real-time, automated quality verification during the annotation process, avoiding the high-cost, low-efficiency manual batch review in the later stages, significantly shortening the data preparation cycle, and reducing the overall cost from data annotation to model training.

[0020] 4. Ultimately, it improves the performance of downstream models. Using the high-quality sample set optimized by this method for model training can effectively improve the accuracy and generalization ability of the model on classification tasks, and reduce the performance degradation or confusion caused by training data noise. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 A flowchart illustrating a method for improving the quality of model training samples provided in this application embodiment;

[0023] Figure 2 A system composition diagram for improving the quality of model training samples provided in this application embodiment; Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a method and system for improving the quality of model training samples. The technical solutions proposed in this application will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 This is a flowchart illustrating a method for improving the quality of model training samples provided in an embodiment of this application. Figure 1 As shown, the method mainly includes the following steps:

[0027] Step S1: In response to the annotation operation on the customer inquiry text, obtain a new annotation sample, which includes the customer inquiry text and its assigned intent classification label.

[0028] Step S2: Invoke the preset text embedding model to convert the customer inquiry text into the corresponding text vector.

[0029] Specifically, the text embedding model is a general pre-trained language model that can map text into high-dimensional semantic vectors. The dimension of its output vector and the length of the input text support the need for accurate semantic representation of printed customer consultation texts.

[0030] If the text embedding model fails to generate text vectors, an alternative processing procedure is executed, which includes: performing a limited number of automatic retries; if the retries still fail, the newly labeled sample is stored in the processing queue and a system alarm is triggered.

[0031] In this application, the text embedding model adopts the gte_Qwen2-7B-instruct version, with an input length limit of 32k tokens and an output vector dimension of 4096. It is deployed on a GPU server (configuration: NVIDIA 4090 24G VRAM, CUDA10.1 environment).

[0032] Step S3: Compare the text vector with the labeled historical sample vectors stored in the vector database, and perform intra-label similarity verification and cross-label discrimination verification in sequence. The intra-label similarity verification is used to prevent excessive aggregation of semantically similar samples under a single label. The cross-label discrimination verification is achieved by calculating the semantic distance between the text vector and other historical sample vectors of other labels and determining whether the semantic distance is greater than a preset threshold.

[0033] It should be noted that the tag-based similarity verification specifically includes: retrieving all historical sample vectors with the same tags as the currently newly labeled sample from the vector database; calculating the semantic similarity between the current text vector and each of the historical sample vectors; counting the number of historical samples with semantic similarity not lower than a first predetermined threshold; determining whether the number exceeds a preset upper limit; if it does not exceed the limit, the tag-based similarity verification passes; if it exceeds the limit, the verification fails, and a first warning message indicating semantic redundancy of samples within the tag is generated.

[0034] Specifically, the cross-label discriminability verification is performed after the intra-label similarity verification passes, and includes: retrieving historical sample vectors from the vector database whose labels are different from the currently newly labeled sample; calculating the semantic distance between the current text vector and the historical sample vectors of each of the different labels; determining whether there is at least one semantic distance that is not greater than a second predetermined threshold; if all semantic distances are greater than the second predetermined threshold, the cross-label discriminability verification passes; if there is at least one semantic distance that is not greater than the second predetermined threshold, the verification fails, and a second warning message indicating insufficient discriminability with samples under at least one other label is generated, the second warning message including the label and text content prompts of the relevant samples.

[0035] Furthermore, the upper limit of the quantity is dynamically determined based on the number of historical samples under the current label, so as to ensure that the constraint strength on the diversity of the sample set can be adaptively adjusted with the data scale.

[0036] Furthermore, when the cross-label discrimination verification fails, the method further includes: receiving a confirmation instruction input by the labeler for the second warning information; in response to the confirmation instruction being a forced submission, marking the newly labeled sample as pending review and storing it in the vector database, and recording the submission reason.

[0037] The vector database is configured with an index structure suitable for fast approximate nearest neighbor search of high-dimensional vectors, and uses cosine similarity as a measure of semantic relevance between vectors. The vector database uses Milvus version 2.3.0, the index type is IVF_FLAT, and the distance is calculated using cosine similarity (value range [-1,1], the closer to 1, the more semantically similar).

[0038] The cosine similarity threshold, denoted as S (default S=0.85, adjustable according to task type), is the critical value for judging sample "similarity" (default S=0.85, adjustable according to task type). The distance threshold, denoted as D, is the critical value for judging insufficient differentiation between cross-label samples (cosine distance = 1 - cosine similarity, default D=0.2, i.e., an alarm is triggered when cosine similarity ≥ 0.8). The N value is the maximum number of similar samples allowed within a single label, default N=5, dynamically adjusted based on the total historical sample volume under the label according to "N = historical sample volume × 10%" (minimum not less than 3, maximum not more than 8).

[0039] If the connection to the vector database is interrupted, a local caching and recovery process is initiated. The process includes: temporarily storing the relevant data of the current verification task locally; continuously monitoring the database connection status; and automatically restoring the task status from the local cache and continuing to perform verification after the connection is restored.

[0040] Step S4: Process the newly labeled sample according to the verification results: If the verification passes, the new labeled sample and its text vector are stored in the vector database; if any verification fails, an early warning prompt containing specific semantic conflict information is generated and the database entry process is suspended, and the labeler performs subsequent operations based on the early warning prompt.

[0041] After completing the annotation of a new sample in the annotation tool and clicking "Submit", the system automatically calls the GTE model to embed the annotated text and performs vector comparison and verification in the Milvus database.

[0042] Validation of the number of similar samples within this category:

[0043] Step 1: Filter all historical sample vectors in the vector database by label L_new, and calculate the cosine similarity between V_new and each vector.

[0044] Step 2: Count the number of samples with similarity ≥ S, and denote it as M.

[0045] Step 3: Result Judgment and Processing: If M ≤ N: Validation passed, proceed to cross-label sample distance validation. If M > N: Validation failed, the annotation tool will display a pop-up message: "The current sample already has M similar samples (similar sample IDs: XXX, XXX...) under label [L_new], exceeding the upper limit of N. Please modify the sample content or confirm the label accuracy." The new sample is temporarily stored in the "Pending Review Area" and is not written to the vector database.

[0046] Cross-label sample distance verification:

[0047] Step 1: Filter out all historical sample vectors that are not labeled L_new from the vector database, and calculate the cosine distance between V_new and each vector (D_calc=1-cosine similarity).

[0048] Step 2: Result Judgment and Processing:

[0049] If all D_calc > D: the validation passes, the new sample (T_new, L_new, V_new, timestamp) is written to the vector database, and the annotation tool will prompt "Validation passed, sample saved";

[0050] If at least one sample has D_calc ≤ D: a verification warning will be issued, and the annotation tool will display a pop-up message: "The current sample has insufficient distinguishability from the sample under label [L_other] (text preview: XXX...) (distance = D_calc). Please check if the label is correct or modify the sample content." The annotator can choose to "modify label", "modify text" or "force submit" (force submission requires a reason, and the sample will be marked as "pending review" for subsequent manual review).

[0051] Error Handling: Model Call Failure: If the GTE model response times out (more than 5 seconds), the annotation tool will display "Vector generation failed, please retry." If it still fails after 3 retries, the sample will be stored in the "Pending Processing Area," and an email alert will be sent to the administrator. Database Connection Failure: When the Milvus connection is interrupted, new samples will be temporarily stored in the local cache. They will be automatically re-uploaded and verified after the connection is restored. The cache validity period is 24 hours.

[0052] The above describes a method for improving the quality of model training samples provided by embodiments of this application. Based on the same inventive concept, embodiments of this application also provide a system for improving the quality of model training samples. Figure 2 A system composition diagram for improving the quality of model training samples provided in this application embodiment is shown below. Figure 2 As shown, the system mainly includes: a labeling unit 201, used to obtain new labeled samples in response to the labeling operation of customer consultation text, the new labeled samples including customer consultation text and its assigned intent classification label; a vector conversion unit 202, used to call a preset text embedding model to convert the customer consultation text into a corresponding text vector; a comparison and verification unit 203, used to compare the text vector with the labeled historical sample vectors stored in the vector database, and sequentially perform intra-label similarity verification and cross-label discrimination verification, wherein the intra-label similarity verification is used to prevent excessive aggregation of samples with similar semantics under a single label, and the cross-label discrimination verification is achieved by calculating the semantic distance between the text vector and other label historical sample vectors and determining whether the semantic distance is greater than a preset threshold; and a result processing unit 204, used to process the new labeled samples according to the verification results: if the verification passes, the new labeled sample and its text vector are stored in the vector database; if any verification fails, a warning prompt containing specific semantic conflict information is generated and the database entry process is suspended, and the labeling personnel perform subsequent operations based on the warning prompt.

[0053] This application, through the above technical solution, has the following improvements compared to existing solutions:

[0054] First, the consistency of annotations is improved. By referencing labels from similar historical samples, the problem of rule inconsistency during the annotation process can be significantly reduced, thereby improving the quality of annotated samples. The consistency of annotation results from 10 annotators is significantly improved, and the overall accuracy of 5,000 samples across 62 categories is improved by 30%.

[0055] Secondly, regarding the sample quality optimization effect, firstly, intra-class richness: using "sample vector variance" as a measure, the average vector variance of the sample set processed by this scheme is 0.18, which is 50% higher than the "no-validation" scheme (0.12), ensuring semantic diversity of samples within a single label. Secondly, inter-class discrimination: the average cosine distance between cross-label samples is 0.45, which is 40.6% higher than the "no-validation" scheme (0.32) and 18.4% higher than the "rule-based validation" scheme (0.38), reducing the model classification confusion rate to below 5%.

[0056] Secondly, regarding efficiency and cost optimization, the verification efficiency is as follows: the average verification time per sample is 0.3 seconds, a 97% improvement compared to "manual review" (10 seconds per sample), supporting concurrent verification of 30 samples per second. Cost reduction is also significant: no additional review personnel are required, reducing the total annotation cost by 45%-60% compared to the "manual secondary review" approach; due to improved sample quality, the number of model training iterations is reduced by two (from five to three), resulting in a 30% reduction in training costs.

[0057] Finally, regarding the improvement in model performance, the average classification accuracy of the fine-tuned large language model (such as Qwen1.5-32B) trained with the optimized sample set of this scheme is 98.2%.

[0058] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0059] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0060] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for improving the quality of training samples for a model, applied to a print customer inquiry intent classification model, characterized in that, The method includes the following steps: Step S1: In response to the annotation operation on the customer inquiry text, obtain a new annotation sample, the new annotation sample including the customer inquiry text and its assigned intent classification label; Step S2: Invoke the preset text embedding model to convert the customer inquiry text into a corresponding text vector; Step S3: Compare the text vector with the labeled historical sample vectors stored in the vector database, performing intra-label similarity verification and cross-label discrimination verification sequentially. The intra-label similarity verification prevents excessive clustering of semantically similar samples under a single label. The cross-label discrimination verification calculates the semantic distance between the text vector and historical sample vectors of other labels and determines whether all semantic distances are greater than a preset threshold. The intra-label similarity verification specifically includes: Retrieve all historical sample vectors with the same label as the currently newly labeled sample from the vector database; Calculate the semantic similarity between the current text vector and each of the historical sample vectors; The number of historical samples with a statistical semantic similarity not lower than a first predetermined threshold; Determine whether the quantity exceeds a preset quantity limit; If the similarity is within the specified range, the similarity check within the label passes; if it exceeds the specified range, the check fails, and a first warning message indicating semantic redundancy of samples within the label is generated. The cross-tag discrimination verification is performed after the intra-tag similarity verification passes, and specifically includes: Retrieve historical sample vectors from the vector database whose labels differ from the currently labeled samples. Calculate the semantic distance between the current text vector and the historical sample vectors of each of the different labels; Determine whether there exists at least one semantic distance that is not greater than a second predetermined threshold; If all semantic distances are greater than the second predetermined threshold, the cross-label discriminability verification passes. If at least one semantic distance is not greater than the second predetermined threshold, the verification fails and a second warning message indicating insufficient distinguishability with at least one other sample under a different label is generated. The second warning message includes the label and text content prompts of the relevant sample. When the cross-label discrimination verification fails, the method further includes: Receive confirmation instructions from the annotator regarding the second warning information; In response to the confirmation instruction being a forced submission, the newly labeled sample is marked as pending review and stored in the vector database, along with a record of the submission reason; Step S4: Process the newly labeled sample according to the verification results: If the verification passes, the new labeled sample and its text vector are stored in the vector database; if any verification fails, an early warning prompt containing specific semantic conflict information is generated and the database entry process is suspended, and the labeler performs subsequent operations based on the early warning prompt.

2. The method for improving the quality of model training samples according to claim 1, characterized in that, The upper limit of the number is dynamically determined based on the number of historical samples under the current label, so as to ensure that the constraint on the diversity of the sample set can be adaptively adjusted with the data scale.

3. The method for improving the quality of model training samples according to claim 1, characterized in that, The text embedding model is a general pre-trained language model that can map text into high-dimensional semantic vectors. The dimension of its output vector and the length of the input text support the need for accurate semantic representation of printed customer consultation texts.

4. The method for improving the quality of model training samples according to claim 1, characterized in that, The vector database is configured with an index structure suitable for fast approximate nearest neighbor search of high-dimensional vectors, and uses cosine similarity as a measure of semantic relevance between vectors.

5. The method for improving the quality of model training samples according to claim 1, characterized in that, In step S2, if the text embedding model fails to generate text vectors, an alternative processing procedure is executed. The alternative processing procedure includes: performing a limited number of automatic retries; if the retries still fail, the newly labeled sample is stored in the processing queue and a system alarm is triggered.

6. The method for improving the quality of model training samples according to claim 1, characterized in that, If the connection with the vector database is interrupted during the execution of step S3, the local caching and recovery process is initiated. The process includes: temporarily storing the relevant data of the current verification task locally; continuously monitoring the database connection status; and automatically restoring the task status from the local cache and continuing to perform verification after the connection is restored.

7. A system for improving the quality of model training samples, employing the method for improving the quality of model training samples according to any one of claims 1-6, characterized in that, The system includes: The annotation unit is used to obtain a new annotation sample in response to the annotation operation on the customer inquiry text. The new annotation sample includes the customer inquiry text and its assigned intent classification label. The vector conversion unit is used to call a preset text embedding model to convert the customer consultation text into a corresponding text vector; The comparison and verification unit is used to compare the text vector with the labeled historical sample vectors stored in the vector database, and sequentially perform intra-label similarity verification and cross-label discrimination verification. The intra-label similarity verification is used to prevent excessive aggregation of semantically similar samples under a single label. The cross-label discrimination verification is achieved by calculating the semantic distance between the text vector and other historical sample vectors of other labels and determining whether the semantic distance is greater than a preset threshold. The result processing unit is used to process the newly labeled sample according to the verification result: if the verification passes, the new labeled sample and its text vector are stored in the vector database; if any verification fails, an early warning prompt containing specific semantic conflict information is generated and the database entry process is suspended, and the labeler performs subsequent operations based on the early warning prompt.

Images