High-quality intelligent data labeling method

Abstract

The application is suitable for the technical field of data labeling, and provides a high-quality intelligent data labeling method, comprising the following steps: vectorization and vector space construction: inputting text, performing seed labeling, and selecting representative samples; mapping the text into a d-dimensional semantic vector to obtain a vector space of the entire corpus; using part of the labeled samples as a prompt to drive the LLM to output rule features and determine rule feature vectors; constructing a hybrid feature space; constructing a similarity graph and performing label propagation; performing uncertainty mathematical reliability testing, embedding the reliability into the sample, determining vector evidence, rule evidence and LLM direct judgment evidence, performing evidence fusion, changing the evidence into final reliability, and sending the reliability into label propagation for decision-making. The application uses the LLM to output rule features and determine rule feature vectors, mines semantic and pattern information in the data, and fuses the two in the hybrid feature space, thereby representing the data from multiple angles and improving the labeling accuracy.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, specifically to a high-quality intelligent data annotation method. Background Technology

[0002] Data annotation can be categorized into traditional manual annotation and automated annotation. Traditional data annotation relies heavily on manual annotation, which is not only labor-intensive but also prone to inconsistencies and inaccuracies due to variations in the annotators' expertise and subjective understanding. Manual annotation is also slow; for large datasets, the annotation cycle becomes extremely lengthy, failing to meet the current demands for rapid processing and analysis of massive amounts of data. For example, manually annotating a long audio segment sentence by sentence is time-consuming and severely impacts project progress. Some automated annotation methods based on traditional machine learning or simple deep learning lack sufficient depth and accuracy in understanding the semantics of text and other data. For instance, in text classification tasks, classifying solely based on surface-level word features fails to capture the deeper semantic information of the text, leading to annotation results that do not match the actual semantic meaning. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a high-quality intelligent data annotation method to solve the problems existing in the background technology.

[0004] This invention is implemented as follows: a high-quality intelligent data annotation method, the method comprising the following steps:

[0005] Vectorization and vector space construction: Input text, perform seed labeling, select representative samples; map the text into d-dimensional semantic vectors, normalize the d-dimensional semantic vectors to obtain the vector space of the entire corpus, where d is a constant value;

[0006] Using partially labeled samples as prompts, LLM is driven to generate regular features and determine the regular feature vectors.

[0007] Construct a hybrid feature space based on semantic vectors and regular feature vectors;

[0008] Construct a similarity graph and propagate labels;

[0009] Reliability testing is performed using the principles of uncertainty mathematics. Reliability is embedded into the samples to identify vector evidence, rule-based evidence, and LLM direct judgment evidence. Evidence fusion is performed based on regularity, duality, subadditivity, and multiplicativity to transform the evidence into final reliability. The reliability is then sent to label propagation for decision-making.

[0010] As a further aspect of the present invention, the input text includes original sentences, paragraphs, or documents, and the input text is denoised, regularized, and segmented into sentences; when seeding is performed, it is initially labeled manually or using LLM.

[0011] As a further aspect of the present invention, the text is mapped to... d 3D semantic vector, for d The steps for normalizing a 3D semantic vector specifically include:

[0012] Use an encoder to map text into a d-dimensional vector: , The generated d-dimensional semantic vector; This represents a vectorization function that maps text to a vector space. This represents the z-th input text; Let d be a real vector space, representing It is a d-dimensional vector; d For vector dimensions;

[0013] Use L2 normalization to stabilize similarity: , Representing vectors After L2 normalization, the length is normalized to 1. Representing vectors The L2 norm; Representing vectors In the Components in each dimension.

[0014] As a further aspect of the present invention, the step of driving LLM to produce rule features specifically includes:

[0015] The LLM is driven to produce readable rules, which include keywords, patterns, syntactic clues and domain predicates;

[0016] Transform readable rules into rule features, which include binary / counting features and matching scores;

[0017] Let the regular feature vector be... , where m is the number of rule entries.

[0018] As a further aspect of the present invention, the step of constructing a hybrid feature space based on semantic vectors and regular feature vectors specifically includes:

[0019] The semantic vector is concatenated or weighted with the regular feature vector to obtain the fused feature vector: ,in, and All are greater than 0. The weights of the semantic vector channels are represented. Indicates the weight of the rule channel;

[0020] A hybrid feature space is obtained to preserve the global semantic structure, while embedding interpretable constraints to facilitate propagation and auditing.

[0021] As a further aspect of the present invention, the step of constructing the similarity graph specifically includes:

[0022] Determine the cosine similarity or Euclidean distance between two semantic vectors and use it as a metric.

[0023] Based on fusion feature vectors ， Using the metric, find the k nearest neighbors for each sample to construct a sparse graph. ， Where k is a positive integer hyperparameter used to control the sparsity of the adjacency set; n For the sample size, For the sample i fused feature vectors; G It is an undirected weighted graph. Let E represent the sample set, and E be the set of all edges, representing the connections between nodes.

[0024] (1) Nearest neighbor set and mask:

[0025] ;

[0026] ;

[0027] in, For nodes i The set of k-nearest neighbors, Two vectors and similarity, The k nodes with the highest similarity; This is an adjacency mask, used to indicate whether there is a nearest neighbor relationship between nodes. For the indicator function, if the node j At the node i k-nearest neighbor set If it is true, it is 1; otherwise, it is 0.

[0028] (2) Basic edge weights, choose one:

[0029] or ;

[0030] in, For nodes i and nodes j Basic edge weights; For cosine similarity, It ensures that the weights are not negative. p The power exponent is used to control the steepness of the similarity distribution; The distance is the square Euclidean distance. These are hyperparameters used to control the rate of similarity decay. It is an exponential function used to map the negative squared distance to a similarity value between (0,1].

[0031] (3) Rule-based consistency modulation:

[0032] ;

[0033] in, For nodes i and nodes j The final edge weights after modulation; For adjacency mask, For nodes i and nodes j Basic border rights, For regular consistency modulation functions, their numerical range is limited to... ,in To fine-tune the hyperparameters, when When the original weight is maintained, When the edge weight is increased, Time decay or edge weight cutoff;

[0034] (4) To obtain an undirected weighted graph, we can let

[0035] ;

[0036] Where W is the modulated weight matrix. This operation transposes the original weight matrix W with its transpose. Take the average value to ensure This transforms the graph structure into an undirected graph; T is the transpose operation.

[0037] (5) Row normalization yields the transition matrix:

[0038] ;

[0039] in, D For degree matrix, For the transition matrix, the elements on the diagonal It is a node i The sum of the weights of all outgoing edges If it appears Then let S ii= 1, S ij = 0, ; Indicates from the sample i Transfer label quality to samples j The weights are determined, and the resulting transition matrix S will be used in subsequent label propagation steps.

[0040] As a further aspect of the present invention, the step of tag propagation specifically includes:

[0041] Determine the number of categories C, and encode the known labeled seed samples as... n × C of matrix The iterative propagation then proceeds as follows: ,in, This is the row-normalized adjacency matrix. For the first t The distribution of labels on the wheel; α Let be the propagation damping coefficient; then when the spectral radius When the iteration converges, its closed-form solution is: , for the i take The maximum value can be retained as the confidence level; at the same time, the rule weight contribution of the transmission path can be recorded as an explanation of the reason.

[0042] As a further aspect of the present invention, the step of embedding confidence into samples to determine vector evidence, rule-based evidence, and LLM direct judgment evidence specifically includes:

[0043] For each sample i and category c Define events, Indicates sample i Category c And assign it a confidence level. This credibility is given and fused from three sources of evidence: vector evidence. Similar neighbors from the vector space; rule evidence , derived from rule features; LLM direct judgment evidence The LLM determines the category of the sample.

[0044] Define the computable form of three-way evidence: For vector evidence, embed the text into a d-dimensional vector: Perform L2 normalization: In semantic tasks, cosine similarity is: The Euclidean distance is: If unit vectors are used, from i Tagned neighbor nearest neighbor set The edge weights are obtained by aggregation. or Vector evidence was obtained: For rule-based evidence, let categories be defined. c The set of rules is Each rule r The weights obtained through learning Indicator of whether a sample triggers a rule ,but If the rule output is a confidence score, then directly... Converted to scores; for LLM direct-judgment evidence, LLM returns the softmax probability. After calibration, it is recorded as ,in, This is a probability calibration function used to normalize or temperature-scale the probabilities output by the LLM to correct the deviation between the predicted probabilities and the true distribution.

[0045] As a further aspect of the present invention, the step of evidence fusion includes: for each sample i and category c The reliability values ​​are determined by three types of evidence, denoted as follows: , ;in, For vector evidence reliability, For the reliability of rule-based evidence, To assess the reliability of direct evidence in LLM;

[0046] When fusing evidence based on regularity, duality, subadditivity, and multiplicative properties, the vector evidence, rule-based evidence, and LLM direct judgment evidence are independent of each other, simultaneously supporting... The reliability of intersection events is determined based on regularity, duality, subadditivity, and multiplicativity. .

[0047] As a further aspect of the present invention, when the confidence level is fed into the label propagation, a graph is constructed on the mixed feature space to obtain a sparse weight matrix. , practice of unification ;initialization For seed samples The distribution uses the confidence vector as a cohesive term for propagation: For unmarked points, evidence from the three sources is merged into... Added as a soft label prior Used to stabilize early diffusion; variables For the sample i Category c Prior reliability; unlabeled points can Enter directly No.i Actions are soft priors, or related to the initial... Linear mixing is used to stabilize early diffusion.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] This invention maps text to d By constructing a vector space using semantic vectors, the semantic similarity of data can be accurately represented, making the annotation process closer to the true semantics of the data. Simultaneously, LLM (Limited Least Meaning Model) is used to generate rule features and determine rule feature vectors, further mining semantic and pattern information in the data. These two are fused in a hybrid feature space, representing the data from multiple perspectives, thus significantly improving the accuracy of annotation. Through uncertain mathematical reliability testing, reliability is embedded in the samples, and evidence fusion is performed based on regularity, duality, subadditivity, and multiplicative properties, transforming the evidence into final reliability. This testing and fusion mechanism fully considers various uncertainties in the annotation process, comprehensively evaluating and correcting the annotation results, further improving the reliability and stability of the annotation results. Attached Figure Description

[0050] Figure 1 This is a flowchart of a high-quality intelligent data annotation method. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0053] like Figure 1 As shown in the figure, this embodiment of the invention provides a high-quality intelligent data annotation method, which includes the following steps:

[0054] S100, Vectorization and Vector Space Construction: Input text, perform seed labeling, and select representative samples: using vector-based diversity selection or the vote-k approach, prioritize representative samples with "broad coverage and no redundancy" for subsequent use; map the text to vector space using encoders such as SBERT / MPNet. d 3D semantic vector, for d Normalize the semantic vectors to obtain the vector space of the entire corpus. d It is a constant value;

[0055] S200, using partially labeled samples as prompts, drives LLM to generate regular features and determine the regular feature vector;

[0056] S300 constructs a hybrid feature space based on semantic vectors and regular feature vectors;

[0057] S400: Construct a similarity graph and propagate labels.

[0058] S500 uses the principles of uncertainty mathematics to test reliability: it embeds reliability into the sample, identifies vector evidence, rule evidence, and LLM direct judgment evidence, and performs evidence fusion based on regularity, duality, subadditivity, and multiplicativity to transform the evidence into final reliability. The reliability is then sent to label propagation for decision-making.

[0059] It should be noted that the annotation process can be summarized as follows: semantic similarity is carried by a dense vector space, interpretable rule features are extracted from a small number of labeled samples using a feature extraction method based on large language models (FeatLLM) and converted into vectors, the two are fused in a hybrid feature vector space, and then the label propagation results are calculated through similarity / nearest neighbor graph vector calculation. Validity and reliability tests are performed using uncertainty mathematics to determine the final result of automatic annotation, thus completing the automatic annotation process based on vector calculation, FeatLLM engineering, and uncertainty mathematics reliability testing.

[0060] In this embodiment of the invention, the input text includes original sentences, paragraphs or documents, and basic cleaning operations such as denoising, regularization and sentence segmentation are performed on the input text; when seed labeling, a small number of representative samples are selected from the population for manual or LLM initial labeling, or a class-by-class diffusion demonstration strategy can be used to select 3-10 high-confidence samples for each class.

[0061] In this embodiment of the invention, the text is mapped to d 3D semantic vector, for d The steps for normalizing a 3D semantic vector specifically include: mapping the text to a 3D semantic vector using an encoder. d Dimensional vector: , The generated d-dimensional semantic vector; This represents a vectorization function that maps text to a vector space. This represents the z-th input text; for d A 3D real vector space, representing It is a d-dimensional vector; d For vector dimensions; use L2 normalization to stabilize similarity: , Representing vectors After L2 normalization, the length is normalized to 1. Representing vectors L2 norm (Euclidean length); Representing vectors In the The components are distributed across each dimension. This yields the vector space of the entire corpus, where all subsequent similarity, nearest neighbor, and graph construction operations are performed.

[0062] In this embodiment of the invention, the step of driving LLM to generate rule features (FeatLLM rule features) specifically includes: using a small number of labeled samples as prompts, driving LLM to generate readable rules, wherein the readable rules include keywords, patterns, syntactic clues, and domain predicates; converting the readable rules into rule features, wherein the rule features include binary / counting features (whether rule keywords appear, the number of times they appear) and matching scores (matching confidence based on regularization / parser / sub-model); denoted as the rule feature vector as... , m This represents the number of rule entries.

[0063] In this embodiment of the invention, the step of constructing a hybrid feature space (vector + FeatLLM fusion) based on semantic vectors and regular feature vectors specifically includes: concatenating or weighting the semantic vectors and regular feature vectors to obtain a fused feature vector. ,in, and All are greater than 0. The weights of the semantic vector channels are represented. The weights of the channels in the FeatLLM rule are represented by a value that, in practice, can be chosen to avoid scale effects. A hybrid feature space is obtained to maintain the global semantic structure while embedding interpretable constraints for easy propagation and auditing.

[0064] In this embodiment of the invention, the step of constructing a similarity graph specifically includes: determining the cosine similarity or Euclidean distance between two semantic vectors. Cosine similarity: the similarity of the angle between two vectors.

[0065]

[0066] Where, u, These are two semantic vectors; "•" indicates the vector dot product. For vectors u The modulus ‖•‖2 is the L2 norm; when written later hour, , For the sample i, jThe fusion vector is calculated as follows: the numerator is the dot product (inner product) of the two vectors; the denominator is the product of the L2 norms of the two vectors. Cosine similarity calculation focuses only on the direction of the vectors, unaffected by their size (length). This makes it particularly suitable for handling "semantic vectors". The value range is [-1, 1], where "1" represents the exact same direction; "0" represents perpendicular (no correlation); and "-1" represents the exact opposite direction. Semantic similarity is usually measured in terms of its approximation to 1. Cosine similarity calculation is efficient, accelerating neighbor retrieval through approximate nearest neighbor algorithms (ANN / HNSW / FAISS), which is crucial for large-scale vector searches. It is particularly suitable for text similarity calculation, specifically for comparing the semantic similarity of documents and sentences.

[0067] Euclidean distance: a metric for the geometric linear distance between vectors.

[0068]

[0069] in, , They are vectors u, v In the k The components of the dimension, d For vector dimensions. This article uses the following notation: If it appears This refers to the squared Euclidean distance. The meanings of the other variables are the same as above. The formula mainly calculates the "straight-line distance" between two vectors in space. The range is [0, +∞), where "0" indicates that the two vectors are exactly the same; the smaller the value, the closer they are, that is, the higher the degree of similarity between the vectors.

[0070] The steps for constructing the similarity graph specifically include:

[0071] Determine the cosine similarity or Euclidean distance between two semantic vectors and use it as a metric.

[0072] Based on fusion feature vectors ， Using the metric, find the k nearest neighbors for each sample to construct a sparse graph. ， Where k is a positive integer hyperparameter used to control the sparsity of the adjacency set; n For the sample size, For the sample i fused feature vectors; G It is an undirected weighted graph. Let E represent the sample set, and E be the set of all edges, representing the connections between nodes.

[0073] (1) Nearest neighbor set and mask:

[0074] ;

[0075] ;

[0076] in, For nodes i The set of k-nearest neighbors, Two vectors and similarity, The k nodes with the highest similarity; This is an adjacency mask, used to indicate whether there is a nearest neighbor relationship between nodes. For the indicator function, if the node j At the node i k-nearest neighbor set If it is true, it is 1; otherwise, it is 0.

[0077] (2) Basic edge weights, choose one:

[0078] or ;

[0079] in, For nodes i and nodes j Basic edge weights; For cosine similarity, It ensures that the weights are not negative. p The power exponent is used to control the steepness of the similarity distribution; The distance is the square Euclidean distance. These are hyperparameters used to control the rate of similarity decay. It is an exponential function used to map the negative squared distance to a similarity value between (0,1].

[0080] (3) Rule-based consistency modulation:

[0081] ;

[0082] in, For nodes i and nodes j The final edge weights after modulation; For adjacency mask, For nodes i and nodes j Basic border rights, For regular consistency modulation functions, their numerical range is limited to... ,in To fine-tune the hyperparameters, when When the original weight is maintained, When the edge weight is increased, Time decay or edge weight cutoff;

[0083] (4) (Optional) To obtain an undirected weighted graph, we can let

[0084] ;

[0085] Where W is the modulated weight matrix. This operation transposes the original weight matrix W with its transpose. Take the average value to ensure This transforms the graph structure into an undirected graph; T is the transpose operation.

[0086] (5) Row normalization yields the transition matrix:

[0087] ;

[0088] Where D is the degree matrix, For the transition matrix, the elements on the diagonal It is a node i The sum of the weights of all outgoing edges If it appears D ii =0, then let S ii = 1, S ij = 0, ; Indicates from the sample i Transfer label quality to samples j The weights are determined, and the resulting transition matrix S will be used in subsequent label propagation steps.

[0089] In this embodiment of the invention, the step of tag propagation specifically includes: determining the number of categories C, and encoding the known labeled seed samples into n×C. matrix (If the sample is not labeled with a zero vector corresponding to the row, then the iterative propagation random walk / heat conduction form is as follows:) ,in, This is the row-normalized adjacency matrix. Let be the label distribution in round t; α be the propagation damping coefficient; then when the spectral radius When the iteration converges, its closed-form solution is: Take the i-th row The maximum value can be retained as the confidence level; at the same time, the rule weight contribution of the propagation path (from...) can be recorded. (Amplification / suppression), as an explanation.

[0090] In this embodiment of the invention, the step of embedding confidence into samples to determine vector evidence, rule evidence, and LLM direct judgment evidence specifically includes: for each sample i and category c Define events, Indicates sample i Category c And assign it a confidence level. This credibility is given and fused from three sources of evidence: vector evidence. Similar neighbors from the vector space; rule evidence Rule features derived from FeatLLM generated and quantized; LLM direct judgment evidence The LLM then determines the category of the sample. Next, it defines the computable form of the three types of evidence: for vector evidence (similarity → confidence), the text is embedded into a d-dimensional vector. Perform L2 normalization: In semantic tasks, cosine similarity is: The Euclidean distance is: If unit vectors are used, from i Labeled nearest neighbor set The edge weights are obtained by aggregation. or Vector evidence was obtained: For rule-based evidence (FeatLLM rules → reliability), let the categories be... c The set of rules is Each rule r The weights obtained through learning Indicator of whether a sample triggers a rule ,but If the rule output is a confidence score, then directly... Converted to scores; for LLM direct-judgment evidence (probability calibration → reliability), LLM returns the softmax probability. After calibration, it is recorded as ,in, This is a probability calibration function used to normalize or temperature-scale the probabilities output by the LLM to correct the deviation between the predicted probabilities and the true distribution.

[0091] In this embodiment of the invention, the step of performing evidence fusion includes: for each sample i and category c The reliability values ​​are determined by three types of evidence, denoted as follows: , ;in, For vector evidence reliability, For the reliability of rule-based evidence, To ensure the reliability of LLM direct judgment evidence; when fusing evidence based on regularity, duality, subadditivity, and multiplicativity, the vector evidence, rule-based evidence, and LLM direct judgment evidence are independent of each other, simultaneously supporting... Considering the intersection of events, under the four axioms of uncertainty theory: 1. Regularity: The confidence of all events in the set is 1. 2. Duality: 3. Subadditivity: 4. Productivity: If events are independent, Therefore, the reliability of valid evidence can be defined as the reliability of the cross-event: .

[0092] In this embodiment of the invention, when the confidence level is fed into the label propagation, a graph is constructed on the mixed feature space to obtain a sparse weight matrix. , practice of unification ;initialization For seed samples The distribution uses the confidence vector as a cohesive term for propagation: For unmarked points, evidence from the three sources is merged into... Added as a soft label prior (or linearly mixed with the propagation outcome), used to stabilize early diffusion; variables For the sample i Prior reliability belonging to category c (obtained from multi-source evidence fusion); unlabeled points can be... Enter directly No. i Actions are soft priors, or related to the initial... Linear mixing is used to stabilize early diffusion.

[0093] Finally, decision-making and human-machine closed-loop (quality control) will be implemented:

[0094] A. Acceptable outcome: and ,variable , These represent the categories with the highest scores (Top-1 / Top-2); Δ is the minimum confidence margin (0.1–0.2); threshold. The minimum reliability for the Top-1 class.

[0095] B. If any of the following conditions are triggered, the automatic annotation results will be marked and require manual review: a. High conflict: significant discrepancy among the three sources of evidence (low variance or mutual information); b. Propagation-rule conflict: the propagation results are significantly inconsistent with the rules; c. Low confidence: the Bel distribution is nearly uniform; d. Importantly, manually reviewed samples are added to the seed set, and the rules are updated incrementally, forming an active learning loop for the automatic annotation program.

[0096] The following describes an experiment using the tools involved in this invention for intelligent text data annotation: The core of this experiment is to use the complete technical framework proposed in this invention to perform intelligent annotation and quality inspection on 4153 novel synopsis texts in Channel A (hereinafter referred to as "A-channel") and Channel B (hereinafter referred to as "B-channel"). The experiment will focus on simulating a complete data annotation and quality inspection process and comparing its performance with multiple baseline models to verify the comprehensive effectiveness of this invention in improving annotation accuracy, achieving intelligent quality inspection, and reducing manual costs. The experimental dataset contains a severely imbalanced test set (1246 samples, of which Channel A accounts for 35% and Channel B accounts for 65%), designed to test the robustness of the model in real business scenarios. The entire process includes three key steps: 1. Vector space evidence: Capturing the macro-level semantics and overall style of the text through algorithms such as SVD and LDA. 2. Feature LLM evidence: Interpretable rule features (such as protagonist titles) discovered and extracted through a large language model are used to identify micro-level, clearly identifiable classification cues. 3. Reliability fusion and verification: Introducing the core of this invention—the uncertain mathematical reliability verification mechanism. This mechanism is based on strict mathematical axioms (regularity, duality, subadditivity, etc.) to mathematically fuse the two independent pieces of evidence, calculate the final comprehensive reliability, and quantify the degree of conflict between the two pieces of evidence.

[0097] This experiment aims to achieve the following four core objectives: (1) Validation: By comparing with the core indicators (accuracy, macro F1 score) of various baseline models, the superiority of the "dual evidence generation + reliability fusion" complete technical framework proposed in this invention is verified. (2) Evaluation of evidence complementarity: The complementary relationship between vector space evidence (macro semantics) and Feature LLM feature evidence (micro rules) is evaluated. The focus is on testing whether the uncertainty mathematical reliability test mechanism significantly enhances the identification ability of minority categories (A frequency) after fusing these two types of information, thereby proving its advantage in handling imbalanced data. (3) Quantification of intelligent quality inspection capability: The core is to quantify the accuracy of uncertainty mathematical reliability test as an automated quality inspection tool. The specific objectives are: A. To test whether it can effectively separate samples with "high confidence" (consistent conclusions of dual evidence and high reliability) and calculate the proportion of such samples and their prediction accuracy. B. To test whether it can accurately identify samples that "require manual review" (conflicting conclusions of dual evidence or low overall reliability) and analyze their composition to verify the accuracy of quality inspection. (4) Proof of application value: Experimental data proves that this method can build a reliable workflow of "automatic machine labeling + precise manual review", which can significantly improve the quality of the final data while effectively reducing the cost of manual review, thus proving its economic and practical value in real industrial scenarios.

[0098] The experiment will be conducted strictly according to the following six steps: (1) Data preparation: 4153 novel synopsis text data will be randomly divided into a training set (2,907 texts) and a test set (1,246 texts) in a 7:3 ratio. (2) Multi-model training: The reliability fusion model of this invention and all comparison models (including basic vector, extended vector, public model, and hybrid model) will be trained on the training set respectively. (3) Dual-source evidence generation: For each text in the test set, two independent pieces of evidence will be generated using the framework of this invention: A. Vector evidence: Deep semantic features of the text will be extracted using algorithms such as SVD and LDA, and the reliability of its belonging to "A frequency" will be calculated. B. Feature LLM evidence: Interpretability features will be extracted using keywords and rules (such as protagonist titles, specific plots, etc.) discovered by the large language model, and the reliability of its belonging to "A frequency" will be calculated. (4) Reliability fusion and prediction: The fusion axiom of uncertain mathematics will be applied to perform mathematical fusion of the above dual-source evidence to generate the final predicted label (A frequency / B frequency) and comprehensive confidence score for each test sample. This step will output two key results: A. Overall confidence score: representing the model's overall confidence in the final prediction. B. Conflict score: quantifying the degree of discrepancy between the vector space and the FeatLLM feature judgment conclusion. (5) Intelligent quality inspection stratification: based on the preset confidence threshold and evidence conflict threshold, all test samples are automatically divided into two categories: A. High confidence samples: high confidence and no obvious conflict between evidence, the system suggests direct adoption. B. Samples requiring manual review: low confidence or opposite conclusions from two evidence sources (e.g., vector evidence judges B frequency, FeatLLM evidence judges A frequency), marked as requiring manual intervention. (6) Performance evaluation and comparison: A. Statistically compare the overall accuracy, macro F1 score, A frequency accuracy and B frequency accuracy of all models on the complete test set. B. Separately calculate the proportion of the "high confidence samples" subset and the prediction accuracy. C. Analyze the composition of "samples requiring manual review" (proportion of low confidence, proportion of evidence conflict) to verify the accuracy of quality inspection.

[0099] Based on the technical principles of this invention, the following effects are expected to be achieved in this experiment: (1) Leading recognition of minority categories: Through the effective fusion of uncertain mathematical reliability test, the bias correction effect of FeatLLM features will be fully utilized, making the model's recognition accuracy in minority categories (A frequency) significantly higher than all comparative models, proving that it can effectively overcome the problem of data imbalance. (2) Significant effect of quality inspection stratification: It is expected that the uncertain mathematical reliability test can successfully identify about 50% of the test samples as "high confidence", and the prediction accuracy of this part of the samples will be 5-8 percentage points higher than the overall accuracy, reaching about 75%. (3) Accurately locating problem data: It is expected that the samples marked as "requiring manual review" will contain a large number of conflicting cases where the vector space and FeatLLM feature judgment are inconsistent. This proves that the mechanism not only judges "whether it is credible", but also explains "why it is not credible", providing a very valuable entry point for manual review. (4) The application value has been confirmed: The experimental results will jointly prove that the technical framework proposed in this invention is a theoretically rigorous and effective automated annotation + quality inspection scheme that can significantly improve the quality of the final adopted data while reducing the cost of manual review by about 50% in real business scenarios.

[0100] Based on the above, five comparative models were designed to verify the effectiveness of the present invention:

[0101] Table 1: Model Comparison Table

[0102]

[0103] Dataset Description: 1. Data Scale: 4,153 text data entries (including online novel titles, summaries, etc.). Of these, 436 entries (35%) are explicitly categorized as Category A; 810 entries (65%) are Category B. 2. Experimental Task: Classify the above data into Category A and Category B categories. 3. Dataset Split: a. Training set: 2,907 entries (70%); b. Test set: 1,246 entries (30%). Experimental Results and Demonstration of Advantages:

[0104] Table 2: Model Performance Table

[0105]

[0106] Key findings: (1) The extended vector model had the highest overall accuracy (76.16%), proving the effectiveness of the Feature LLM feature. (2) The confidence fusion model achieved the best performance in A-frequency identification (69.04%), solving the class imbalance problem. (3) The average confidence score of the confidence fusion model (0.786) was second only to the extended vector model, indicating that the prediction after fusion was more reliable.

[0107] Quality control proficiency verification: 1. Quality control sample distribution: (1) Total test samples: 1,246; (2) Low confidence samples: 272 (21.8%); (3) Conflicting evidence samples: 583 (46.8%); (4) Requires manual review: 622 (49.9%). The confidence calibration effect is as follows:

[0108] Table 3: Confidence Level Calibration Table

[0109]

[0110] Key advantage: Through reliability fusion, 50.1% of high-quality samples were successfully separated, with an accuracy of 75%, which is 7.66 percentage points higher than the overall accuracy.

[0111] Quality control capability verification: 1. Case 1: Complementarity of vector and FeatLLM: (1) Text: As a financial tycoon, Li Weimin saw a young and beautiful widow as soon as he opened his eyes...; (2) Vector evidence: A frequency = 0.600 (based on semantics); (3) FeatLLM evidence: A frequency = 0.091 (no A main feature detected); (4) Conflict degree: 0.509; (5) Fusion result: A frequency = 0.636; (6) Quality control mark: manual review required. 2. Case 2: Complementarity of vector and FeatLLM: (1) Text: Yunxuan, the sect leader of Tianyi Island, accidentally received a letter from his master...; (2) Vector evidence: A frequency = 0.000 (semantic judgment error); (3) FeatLLM evidence: A frequency = 0.714 (A main features such as "sect leader" and "divine doctor" were detected); (4) Conflict degree: 0.714; (5) Fusion result: A frequency = 0.714 (correct).

[0112] Comparison with existing technologies: The following table compares the methods with traditional vector methods and with simple combinatorial methods:

[0113] Table 4: Comparison of Traditional Vectors

[0114]

[0115] Table 5: Comparison of Simple Combination Methods

[0116]

[0117] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0118] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0119] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0120] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A high-quality intelligent data labeling method, characterized in that, The method includes the following steps: Vectorization and vector space construction: Input text, perform seed labeling, select representative samples; map the text into d-dimensional semantic vectors, normalize the d-dimensional semantic vectors to obtain the vector space of the entire corpus, where d is a constant value; Using partially labeled samples as prompts, LLM is driven to generate regular features and determine the regular feature vectors. Construct a hybrid feature space based on semantic vectors and regular feature vectors; Construct a similarity graph and propagate labels; Reliability testing is conducted using the principles of uncertainty mathematics: reliability is embedded into the sample to identify vector evidence, rule evidence, and LLM direct judgment evidence. Evidence fusion is performed based on regularity, duality, subadditivity, and multiplicativity to transform the evidence into final reliability. The reliability is then sent to label propagation for decision-making. The steps of mapping text to a d-dimensional semantic vector and normalizing the d-dimensional semantic vector specifically include: Mapping text to d-dimensional vectors with an encoder: , for the generated d-dimensional semantic vector; denotes a vectorization function that maps text to a vector space; denotes the z-th input text; is a d-dimensional real vector space, representing is a d-dimensional vector; d is the vector dimension; L2-normalize to stabilize similarity: , denotes a vector L2-normalized result, length normalized to 1; denotes a vector L2-norm of a vector denotes a vector component in the th dimension; The steps for driving LLM to produce rule features specifically include: The LLM is driven to produce readable rules, which include keywords, patterns, syntactic clues and domain predicates; Transform readable rules into rule features, which include binary / counting features and matching scores; The rule feature vector is , m is the number of rule entries; The steps for constructing a hybrid feature space based on semantic vectors and regular feature vectors specifically include: The semantic vector is concatenated or weightedly spliced with the rule feature vector to obtain a fusion feature vector: wherein, 0, represents the weight of the semantic vector channel, represents the weight of the rule channel; A hybrid feature space is obtained to preserve the global semantic structure while embedding interpretable constraints, which facilitates propagation and auditing. The steps for constructing the similarity graph specifically include: Determine the cosine similarity or Euclidean distance between two semantic vectors and use it as a metric. Fusion feature vector ， The k nearest neighbors of each sample are found using the metric, thereby constructing a sparse graph ， where k is a positive integer hyper-parameter, used to control the sparsity of the adjacency set; n for the number of samples, for the sample i fusion feature vector; G is an undirected weighted graph, denotes the sample set, E is the set of all edges, and denotes the connection between nodes; (1) Nearest neighbor set and mask: ； ； wherein, is a k-neighborhood of a node i , is a similarity of two vectors and , is a node with the top similarities; is an adjacency mask indicating whether there is a neighborhood relationship between nodes, is an indicator function that is 1 if node j is in the k-neighborhood of node i , otherwise. (2) Basic edge weights, choose one: ； wherein, is the base edge weight of node i and node j ; is the cosine similarity, is to ensure the weight is not negative, p is the power exponent, used to control the steepness of the similarity distribution; is the squared Euclidean distance, is the hyperparameter, used to control the speed of similarity decay, is the exponential function, used to map the negative distance square to a similarity value between (0, 1]; (3) Rule-based consistency modulation: ； wherein, is a node i and a node j modulated final edge weight; is a neighborhood mask, is a node i and a node j base edge weight of the node is a rule-consistency modulation function whose numerical range is limited to wherein is a fine-tuning hyperparameter that keeps the original weight when , amplifies the edge weight when , and attenuates or cuts off the edge weight when . (4) To obtain an undirected weighted graph, we can let ； where W is the modulated weight matrix, is the transpose of W, this step (4) modulates the weight matrix W with its transpose averaging, ensuring so that the graph structure becomes an undirected graph; T is the transpose operation; (5) Row normalization yields the transition matrix: ； wherein, D is a degree matrix, is a transition matrix, the elements on the diagonal are the nodes i the sum of the weights of all outgoing edges ; if the occurrence D ii = 0, set S ii = 1, S ij = 0, ; denotes the weight of passing the label quality from a sample i to a sample j , the resulting transition matrix S will be used for the subsequent label propagation step.

2. The high-quality intelligent data labeling method of claim 1, wherein, The input text includes original sentences, paragraphs, or documents. The input text will be denoised, regularized, and segmented into sentences. Seed labeling is performed manually or using LLM initial labeling.

3. The high-quality intelligent data labeling method of claim 1, wherein, The steps for disseminating tags specifically include: Determine the number of categories C, encode the seed samples with known labels as n x C matrix Then the iteration propagates as follows: where is the row-normalized adjacency matrix, is the label distribution of the t th iteration; α is the propagation damping coefficient; then the iteration converges when the spectral radius is less than 1, and its closed-form solution is: for the i th row The maximum value can be kept as the confidence; meanwhile, the rule weight contribution of the passing path can be recorded as the cause explanation.

4. The high-quality intelligent data annotation method according to claim 3, characterized in that, The steps of embedding credibility into samples to determine vector evidence, rule-based evidence, and LLM direct judgment evidence specifically include: Define an event for each sample i and category c. This indicates that sample i belongs to category c, and assigns it a confidence level. This credibility is given and fused from three sources of evidence: vector evidence. Similar neighbors from the vector space; rule evidence , derived from rule features; LLM direct judgment evidence The LLM determines the category of the sample. Define the computable form of three-way evidence: For vector evidence, embed the text into a d-dimensional vector: Perform L2 normalization: In semantic tasks, cosine similarity is: The Euclidean distance is: If unit vectors are used, from i Labeled nearest neighbor set The edge weights are obtained by aggregation. or Vector evidence was obtained: For rule-based evidence, let categories be defined. c The set of rules is Each rule r The weights obtained through learning Indicator of whether a sample triggers a rule ,but If the rule output is a confidence score, then directly... Converted to scores; for LLM direct-judgment evidence, LLM returns the softmax probability. After calibration, it is recorded as ;in, This is a probability calibration function used to normalize or temperature-scale the probabilities output by the LLM to correct the deviation between the predicted probabilities and the true distribution.

5. The high-quality intelligent data annotation method according to claim 4, characterized in that, The evidence fusion step includes: for each sample i and category c, determining the corresponding reliability value using the three types of evidence, denoted as . , ;in, Vector evidence reliability For the reliability of rule-based evidence, To assess the reliability of direct evidence in LLM; When fusing evidence based on regularity, duality, subadditivity, and multiplicative properties, the vector evidence, rule-based evidence, and LLM direct judgment evidence are independent of each other, simultaneously supporting... For the intersection of events, the reliability of the intersection events is determined based on regularity, duality, subadditivity, and multiplicativity: .

6. The high-quality intelligent data annotation method according to claim 5, characterized in that, When the confidence score is fed into the label propagation, a graph is constructed on the mixed feature space to obtain a sparse weight matrix. , practice of unification ;initialization For seed samples The distribution uses the confidence vector as a cohesive term for propagation: For unmarked points, evidence from the three sources is merged into... Added as a soft label prior Used to stabilize early diffusion; variables For the sample i Prior reliability belonging to category c; Unmarked points can Enter directly No. i Actions are soft priors, or related to the initial... Linear mixing is used to stabilize early diffusion.

Images