Data marking method and device, equipment and storage medium
By combining a large language model with a vector knowledge base and dynamically injecting classification and grading rules, the problem of efficient and accurate labeling of unstructured data is solved, achieving unified processing of structured and unstructured data and improving labeling efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-03-31
AI Technical Summary
Existing technologies suffer from low accuracy and reliance on manual annotation when processing unstructured data, making it difficult to achieve efficient and accurate data identification and labeling.
By employing a large language model combined with multi-layered prompt word templates and a vector knowledge base, and through data fragmentation, semantic encoding, and dynamic injection of classification and grading rules, unified and efficient labeling of unstructured and structured data is achieved.
It improves the accuracy and efficiency of data labeling, reduces manual intervention, and ensures the interpretability and consistency of labels.
Smart Images
Figure CN121765094A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to data labeling methods, apparatus, devices, and storage media. Background Technology
[0002] With regulators imposing clear requirements on data security, enterprises need to identify and label data according to relevant guidelines. However, multiple challenges remain in practice: First, in general scenarios, processing unstructured data is difficult due to its semantic ambiguity and diverse formats, making traditional rule-based matching methods inaccurate and hindering precise information extraction. Second, static or pre-set identification rules are prone to failure when dealing with dynamic data formats such as real-time data streams and processed synthetic data, further complicating data identification and making it difficult to improve overall labeling accuracy. Furthermore, the labeling of massive amounts of data still heavily relies on manual labor, which is not only inefficient and costly but also prone to errors due to subjective judgment differences or fatigue, affecting the reliability of subsequent data governance and compliance work.
[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0004] The main purpose of this application is to provide a data labeling method, apparatus, device, and storage medium, aiming to solve the technical problem of how to improve the efficiency and accuracy of data labeling.
[0005] To achieve the above objectives, this application proposes a data labeling method, which includes: Acquire unstructured and structured data; According to the preset data segmentation strategy, the structured data and the unstructured data are divided to obtain the corresponding target data segments; Based on the target data fragment, semantically relevant classification and grading rules are retrieved from the pre-constructed vector knowledge base; Based on the classification and grading rules, and combined with a preset large language model, the target data fragments are tagged using preset multi-layer prompt word templates to obtain corresponding data labels.
[0006] In one embodiment, the step of dividing the structured data and the unstructured data according to a preset data fragmentation strategy to obtain target data fragments includes: The structured data is grouped based on the relationships between fields to obtain structured data fragments; The data type characteristics of the unstructured data are obtained, and the unstructured data is divided according to the data type characteristics to obtain unstructured data fragments.
[0007] In one embodiment, prior to the step of retrieving semantically relevant classification and grading rules from a pre-built vector knowledge base based on the target data fragment, the method includes: Obtain the classification and grading rules, and convert the classification and grading rules into rule text; The rule text is vectorized using a preset semantic encoding model to generate corresponding rule vectors; A vector knowledge base is constructed based on the aforementioned rule vectors.
[0008] In one embodiment, after the step of constructing a vector knowledge base based on the rule vector, the method further includes: In response to the updated input of classification and grading rules, the updated input classification and grading rules are vectorized to generate the corresponding new rule vector; Add the newly added rule vector to the vector knowledge base; Based on the classification and grading rules of the updated input, the semantic encoding model is incrementally trained to maintain the semantic consistency of the rule vectors in the vector space.
[0009] In one embodiment, the step of tagging the target data fragment based on the classification and grading rules, combined with a preset large language model, and using a preset multi-layer prompt word template to obtain corresponding data tags includes: The classification and grading rules are injected into the preset multi-level prompt template to obtain the corresponding prompt words; The target data fragment and the prompt words are input into the preset large language model for reasoning, and the reasoning result is output. The reasoning results are analyzed to obtain the classification, grading information and confidence level corresponding to the target data fragment, which are used as the data label.
[0010] In one embodiment, the step of inputting the target data fragment and the prompt word into the preset large language model for reasoning and outputting the reasoning result includes: Determine the corresponding inference parameters based on the data type of the target data fragment; The target data fragment and the prompt word are used as model input, and the preset large language model is called to perform inference based on the inference parameters to obtain the inference result.
[0011] In one embodiment, after the step of parsing the inference result to obtain the classification, grading information and confidence level corresponding to the target data fragment as the data label, the method further includes: Determine whether the confidence level is greater than or equal to a preset threshold; If the confidence level is greater than or equal to the preset threshold, then the classification, grading information and corresponding confidence level in the data tag are associated with the target data segment and stored in the database; If the confidence level is less than the preset threshold, then in response to the feedback from manual review, mislabeled samples with confidence levels lower than the preset threshold are filtered out, the mislabeled samples are added to the fine-tuning dataset, and incremental training is performed on the preset large language model based on the fine-tuning dataset.
[0012] Furthermore, to achieve the above objectives, this application also proposes a data marking device, which includes: The acquisition module is used to acquire unstructured and structured data; The partitioning module is used to partition the structured data and the unstructured data according to a preset data partitioning strategy to obtain corresponding target data fragments; The retrieval module is used to retrieve semantically relevant classification and grading rules from a pre-built vector knowledge base based on the target data fragment. The tagging module is used to tag the target data fragments based on the classification and grading rules, combined with a preset large language model, and through a preset multi-layer prompt word template, so as to obtain the corresponding data tags.
[0013] In addition, to achieve the above objectives, this application also proposes a data marking device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data marking method as described above.
[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data tagging method described above.
[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data labeling method described above.
[0016] This application proposes a data labeling method, apparatus, device, and storage medium. The method includes: acquiring unstructured data and structured data; dividing the structured data and unstructured data according to a preset data segmentation strategy to obtain corresponding target data segments; retrieving semantically relevant classification and grading rules from a pre-constructed vector knowledge base based on the target data segments; and labeling the target data segments based on the classification and grading rules, combined with a preset large language model, using preset multi-layer prompt word templates to obtain corresponding data tags. This method combines the semantic understanding capabilities of the large language model, improves labeling accuracy through vector retrieval and dynamic rule injection, and enhances labeling efficiency by uniformly processing structured and unstructured data. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating the data labeling method of this application in Embodiment 1. Figure 2 This is a flowchart illustrating Embodiment 2 of the data labeling method for this application. Figure 3 This is a flowchart illustrating Embodiment 3 of the data labeling method for this application. Figure 4 The flowchart provided in Embodiment 4 of the data labeling method of this application is shown in the following diagram. Figure 5 A simplified flowchart illustrating the data labeling method provided in Embodiment 1 of this application; Figure 6 This is a schematic diagram of the module structure of the data marking device according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the data labeling method in this application embodiment.
[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0023] The main solution of this application embodiment is as follows: acquiring unstructured data and structured data; dividing the structured data and unstructured data according to a preset data segmentation strategy to obtain corresponding target data segments; retrieving semantically related classification and grading rules from a pre-built vector knowledge base based on the target data segments; and tagging the target data segments based on the classification and grading rules, combined with a preset large language model, using a preset multi-layer prompt word template to obtain corresponding data tags.
[0024] In this embodiment, for ease of description, the following description will focus on the data labeling system as the execution subject.
[0025] Existing data identification and labeling methods suffer from several drawbacks when processing unstructured data. This is because the data's features are often ambiguous, leading to inherently low accuracy in rule matching (e.g., sensitive information in medical record PDFs and customer call recordings, such as ID numbers and diagnostic results, is often hidden within natural language or visual elements and lacks a fixed format, making it difficult to accurately extract using traditional rules (e.g., regular expression matching)). Furthermore, dynamically changing data, such as real-time streams and processed synthetic data, can render existing identification rules ineffective, further complicating the identification process and hindering accuracy improvement. In addition, the labeling of massive amounts of data still relies heavily on manual work, which is not only inefficient but also prone to errors due to human factors.
[0026] This application provides a solution that leverages the core capabilities of large models in natural language understanding, multimodal processing, and context awareness, integrating large models, RAG, and prompt word technologies to achieve accurate identification of unstructured data and effectively improve the classification, grading, and labeling results.
[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following description uses a personal computer as an example to illustrate this embodiment and the subsequent embodiments.
[0028] Based on this, the embodiments of this application provide a data labeling method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data labeling method of this application.
[0029] In this embodiment, the data labeling method includes steps S10 to S40: Step S10: Obtain unstructured data and structured data; It should be noted that the structured data refers to data with a fixed format and clearly defined fields, such as relational database tables; Unstructured data refers to data without a fixed format or that is semi-structured, such as PDFs and Word documents.
[0030] In this embodiment, structured and unstructured data are first collected in batches through API interfaces, database connections, file access, etc., and then uniformly stored in a distributed file system for subsequent centralized processing and management.
[0031] Next, the collected data undergoes data cleaning. Specifically, structured data is cleaned by performing missing value imputation, field type validation, and format standardization to ensure data integrity and consistency. For unstructured data, the focus is on automated parsing and cleaning of text content. For example, for PDFs or scanned documents, OCR (Optical Character Recognition) combined with Natural Language Processing (NLP) technology is used to extract key semantic information, such as "Party A's ID number in the contract" and "the validity period of the confidentiality clause." Alternatively, regular expressions are used to filter redundant symbols, and NLP tools are used to remove stop words. Special format data (such as dates and encodings) are standardized and converted.
[0032] The aforementioned cleaning and standardization rules have been compiled into a reusable cleaning rule library, supporting automated and configurable data preprocessing workflows.
[0033] By following the steps above, the quality and consistency of the original data can be effectively improved, providing a reliable and standardized input foundation for subsequent vectorized retrieval and large-scale model labeling, thereby ensuring the accuracy and repeatability of the classification and grading results.
[0034] Step S20: According to the preset data segmentation strategy, the structured data and the unstructured data are divided to obtain the corresponding target data segments; Understandably, since large language models have an upper limit on the length of a single input, directly inputting an entire document or a whole data table into the model would result in truncation and loss of key information or exceed processing capacity. At the same time, if the semantic integrity of unstructured data and the field correlation of structured data are damaged, it will also affect the accuracy of subsequent labeling. Therefore, step S20 is executed to divide the original data into units that fit the input length of the model while ensuring semantic or structural integrity. This avoids information truncation or processing failure due to exceeding the input limit, thereby achieving efficient and accurate large model labeling.
[0035] In one feasible embodiment, step S20 may include steps S21-S22: Step S21: Group the structured data based on the relationships between fields to obtain structured data fragments; In this step, for structured data such as relational database tables, strongly related fields are grouped into a structured data fragment based on primary and foreign key constraints, business logic relationships (such as "User ID" and "ID Card Number" belonging to the same entity), or predefined grouping rules. For example, fields such as the name, mobile phone number, and ID card number of the same user are grouped into a single record unit, ensuring that each fragment semantically represents a complete data entity, which facilitates subsequent rule matching and tag generation.
[0036] Step S22: Obtain the data type characteristics of the unstructured data, and divide the unstructured data according to the data type characteristics to obtain unstructured data fragments.
[0037] In this step, the type of unstructured data (such as PDF contracts, Word reports, log files, etc.) is first identified, and its content organization features (such as paragraphs, chapters, and table areas) are extracted. For text data, it is segmented according to semantic boundaries, such as based on natural paragraphs, heading levels, or sentence coherence, dividing the content into text blocks of appropriate length (typically controlled within the range of 200 to 1000 characters to fit the context window of mainstream large models). For documents containing tables or forms, segmentation is prioritized according to table cells or field areas to preserve local structural information. The resulting unstructured data fragments satisfy the model input length limit while preserving the original semantic context to the greatest extent possible.
[0038] Through the above-mentioned fragmentation strategy, various types of data are transformed into target data fragments with complete semantics or structure and controllable length, providing high-quality and standardized input units for subsequent vector retrieval and large model labeling.
[0039] Step S30: Based on the target data fragment, retrieve semantically relevant classification and grading rules from the pre-constructed vector knowledge base; It should be noted that the classification and grading rules refer to the normative basis used to determine the sensitivity of data and processing requirements, including but not limited to industry data security standards, enterprise internal data standardization strategies, and definitions of sensitive fields preferred by business scenarios.
[0040] In this embodiment, the target data fragment is first converted into a query vector using a preset semantic encoding model (such as Sentence-BERT). Then, an approximate nearest neighbor search (ANN) is performed in the vector knowledge base to retrieve several classification and grading rules with the highest semantic similarity to the query vector. The vector knowledge base is constructed by vectorizing the aforementioned classification and grading rules using the same semantic encoding model, ensuring alignment between the query space and the knowledge base space.
[0041] Through the above steps, the system can automatically match the classification and grading rules most relevant to the content of the current data segment, providing accurate and interpretable external knowledge support for subsequent large model labeling, and avoiding rule bias or outdated risks caused by relying on the memory of internal model parameters.
[0042] Step S40: Based on the classification and grading rules and combined with the preset large language model, the target data segment is tagged using a preset multi-layer prompt word template to obtain the corresponding data label.
[0043] It should be noted that the multi-layered prompt template is a hierarchical structured prompt design, specifically including the following three layers: Base layer: Used to define task boundaries, such as defining the basic boundaries and output format of a labeling task, for example: "You are a data security expert. Please label the following data fragments with sensitivity levels. The output format is {'Classification':'XXX', 'Grade': 'XXX', 'Confidence': 0.X}". Intermediate layer: used for dynamically injecting semantically relevant classification and grading rules retrieved from the vector knowledge base; The instance layer provides typical annotation examples to guide model behavior; for example, data containing ID card numbers is marked as highly sensitive.
[0044] Understandably, large language models are susceptible to internal parameter memory or training data bias in open generation scenarios. Directly labeling the raw data may lead to outputs deviating from enterprise or industry standards, resulting in illusions or mislabeling. At the same time, single static prompts are difficult to adapt to diverse data types and rule combinations. Therefore, step S41 is executed to organically integrate externally introduced classification and grading rules through multi-layer prompt word templates and use them as context input to the large language model. This avoids label inconsistencies or compliance risks caused by the model deviating from the rules and acting freely, thereby achieving highly accurate, highly consistent, and interpretable data labeling.
[0045] In one feasible implementation, step S40 may include steps S41 to S43: Step S41: Inject the classification and grading rules into the preset multi-layer prompt template to obtain the corresponding prompt words; In this step, the classification and grading rules retrieved in step S30 are dynamically populated into the middle layer of the multi-layered prompt word template. Simultaneously, combined with preset task instructions (base layer) and typical annotation examples (instance layer), complete structured prompt words are generated. For example, if the rule "data containing ID card numbers is marked as highly sensitive" is retrieved, this rule is inserted into the middle layer of the prompt words, forming a contextual constraint to ensure that subsequent model inference strictly follows this security specification.
[0046] Step S42: Input the target data fragment and the prompt word into the preset large language model for reasoning, and output the reasoning result; It should be noted that the inference behavior of large language models is significantly affected by parameters such as temperature coefficient and maximum generation length: excessively high temperature can lead to strong output randomness and unstable labels, while excessively low temperature may inhibit the generalization ability to complex semantics. Furthermore, single-line inference is inefficient, making it difficult to meet the needs of batch labeling. Therefore, while ensuring labeling accuracy, inference parameters need to be dynamically configured according to data characteristics, and a batch processing mechanism should be adopted to balance processing speed and resource utilization.
[0047] In another feasible embodiment, step S42 may further include steps S421 to S422: Step S421: Determine the corresponding inference parameters based on the data type of the target data segment; In this step, the data type of the target data fragment is first identified. If it is structured data, its content format is fixed and the rules are clear, so a lower temperature coefficient (e.g., 0.1) is preferred to improve output determinism and processing efficiency. If it is unstructured data, its semantic expression is diverse and its context dependence is strong, so a relatively higher temperature coefficient (e.g., 0.3-0.5) is preferred to enhance the model's ability to identify ambiguous or implicit sensitive information.
[0048] Step S422: The target data fragment and the prompt word are used as model input, and the preset large language model is called to perform inference based on the inference parameters to obtain the inference result.
[0049] In this step, multiple target data fragments to be processed and their corresponding complete prompt words are organized in batches to form a batch input sequence; then, the inference interface (such as API or local inference engine) of the large language model is called with the temperature coefficient and other inference parameters (such as top_p, max_tokens, etc.) determined in step S421 to perform parallel inference; finally, the original output text returned by the large language model is received as the input source for subsequent parsing steps (such as step S43).
[0050] Step S43: Analyze the reasoning result to obtain the classification, grading information and confidence level corresponding to the target data segment, which are used as the data label.
[0051] In this step, the raw inference results returned by the large language model are subjected to structured parsing. Specifically, it is first determined whether the output conforms to the preset format specifications (such as JSON or key-value pair format); if it does, the classification category (such as "personal identification information" or "financial data"), sensitivity level (such as "public", "internal", or "highly sensitive") and confidence value (usually a floating-point number between 0.0 and 1.0) are extracted through regular expression matching, a JSON parser, or a dedicated parsing module; if the output format is abnormal, fields are missing, or the confidence level is lower than the preset threshold (e.g., 0.7), the sample is marked as a low-confidence labeling result, and a subsequent manual review or retry mechanism is triggered.
[0052] Finally, the successfully parsed structured information is encapsulated into standard data tags, which are then uniquely associated with the corresponding target data fragments for subsequent storage or feedback fine-tuning processes.
[0053] Through the above steps, external classification and grading rules are dynamically integrated into the reasoning process of the large language model, and parsable and verifiable data labels are generated based on structured prompts and parameter optimization, which effectively improves the accuracy, consistency and traceability of the labeling results.
[0054] The method described above acquires unstructured and structured data; according to a preset data segmentation strategy, the structured and unstructured data are divided to obtain corresponding target data segments; based on the target data segments, semantically relevant classification and grading rules are retrieved from a pre-constructed vector knowledge base; based on the classification and grading rules, combined with a preset large language model, the target data segments are labeled using preset multi-layer prompt word templates to obtain corresponding data tags. This method combines the semantic understanding capabilities of a large language model, improves labeling accuracy through vector retrieval and dynamic rule injection, and enhances labeling efficiency by uniformly processing structured and unstructured data.
[0055] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S30, the data labeling method further includes steps S301 to S303: Step S301: Obtain the classification and grading rules, and convert the classification and grading rules into rule text; In this step, data security-related classification rules are collected from multiple sources. These rules are then uniformly converted into structured natural language text (i.e., "rule text"), with each rule text including but not limited to key fields such as sensitivity level, feature description, and judgment criteria.
[0056] Step S302: The rule text is vectorized using a preset semantic encoding model to generate corresponding rule vectors; In this step, a pre-defined semantic encoding model is used to embed each rule text generated in step S301, mapping it into a fixed-dimensional rule vector. This semantic encoding model has been optimized during training to capture semantic similarity, making semantically similar rules (such as "ID number" and "personal identification") closer together in the vector space, thus supporting subsequent precise semantic-based retrieval.
[0057] Step S303: Construct a vector knowledge base based on the rule vector.
[0058] In this step, all rule vectors generated in step S302 and their associated original rule texts are stored in a vector database to construct an efficient retrieval vector knowledge base. This knowledge base supports approximate nearest neighbor search and can return the most relevant classification and grading rules based on the semantic features of the input data within a millisecond response time, providing real-time and accurate external knowledge support for large model labeling.
[0059] Through the methods described above, scattered industry standards and enterprise specifications are transformed into a structured, computable vector knowledge base, realizing the semantic representation and efficient retrieval of classification and grading rules, and providing accurate, dynamic, and scalable external knowledge support for data labeling based on large language models.
[0060] Based on the second embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the second embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After step S303, the data labeling method further includes steps A1 to A3: Step A1: In response to the update input of the classification and grading rules, the updated classification and grading rules are vectorized to generate the corresponding new rule vector; In this step, when the system receives newly added or revised classification and grading rules, it first converts them into structured rule text. Then, it calls the current semantic encoding model to embed the rule text, generating a corresponding new rule vector. This vector is initially generated based on existing model parameters and is used for preliminary inclusion in the knowledge base.
[0061] Step A2: Add the rule vector to the vector knowledge base; In this step, the rule vectors generated in step A1 and their associated metadata (including sensitivity level, feature description, rule source, version number, etc.) are written into the vector database, completing the real-time expansion of the vector knowledge base. To ensure retrieval efficiency, the system synchronously updates the index structure (such as HNSW graphs or IVF cluster centers) to ensure that newly added rules can be immediately matched by subsequent retrieval processes.
[0062] Step A3: Based on the classification and grading rules of the updated input, the semantic encoding model is incrementally trained to maintain the semantic consistency of the rule vectors in the vector space.
[0063] In this step, to avoid retrieval bias caused by semantic distribution shifts between new and old rules (e.g., the new rule "biometric information" and the old rule "facial features" are semantically similar but have large vector distances), the newly added rule text and its semantic labels (such as similarity annotations or manual alignment relationships with existing rules) are used as fine-tuning samples to perform incremental training (e.g., contrastive learning fine-tuning or adapter fine-tuning) on the semantic encoding model. After training, the model is updated to the latest version, and vector updates can be selectively performed on some or all rule vectors in the knowledge base, thereby ensuring semantic consistency and retrieval accuracy throughout the vector space.
[0064] The above-described methods enable online expansion and semantic alignment of classification and grading rules, ensuring that the vector knowledge base maintains high retrieval accuracy and rule coverage integrity throughout its dynamic evolution.
[0065] Based on the first embodiment of this application, in the fourth embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 After step S43, the data labeling method further includes steps B1 to B3: Step B1: Determine whether the confidence level is greater than or equal to a preset threshold; In this step, the confidence scores in the data labels generated in step S43 are compared with a preset confidence threshold (e.g., 0.7, which can be configured according to actual business needs) to determine the reliability of the current labeling results. This threshold is used to distinguish between high-confidence outputs and low-confidence samples that require manual review.
[0066] Step B2: If the confidence level is greater than or equal to the preset threshold, then the classification, grading information and corresponding confidence level in the data tag are associated with the target data segment and stored in the database; If the confidence level is greater than or equal to a preset threshold, a unique mapping relationship is established between the data tags (including category, sensitivity level, and confidence level) and the original target data fragment (e.g., associated through data ID or hash value), and the data is persistently stored in a relational database or data lake. This storage result can be used for subsequent data governance, audit traceability, or de-identification strategy implementation.
[0067] Step B3: If the confidence level is less than the preset threshold, in response to the feedback from manual review, the mislabeled samples with confidence levels lower than the preset threshold are filtered out, the mislabeled samples are added to the fine-tuning dataset, and incremental training is performed on the preset large language model based on the fine-tuning dataset.
[0068] If the confidence level is less than a preset threshold, the sample with that confidence level is pushed to a manual review interface (such as a visual annotation platform) for correction by relevant personnel. Samples confirmed by manual review as mislabeled (i.e., the model output is inconsistent with the manual annotation), along with their original input, contextual clues, and correct labels, are included in a preset fine-tuning dataset. Subsequently, based on this fine-tuning dataset, the preset large language model is incrementally trained using methods such as supervised fine-tuning or preference optimization to optimize its labeling accuracy in similar scenarios.
[0069] The above-described methods achieve a tiered processing of the labeling results based on their reliability. High-confidence samples are automatically stored in the database, ensuring efficient data governance; low-confidence mislabeled samples are manually corrected to drive incremental training of the large language model, forming a closed-loop feedback mechanism that effectively improves the long-term accuracy and adaptability of the labeling system.
[0070] For example, to help understand the implementation process of the data labeling method obtained by combining this embodiment with the first embodiment described above, please refer to... Figure 5 , Figure 5 A simplified flowchart of a data labeling method is provided, specifically: First, structured and unstructured data are collected in batches via API interfaces, database connections, or file access methods, and stored uniformly in a distributed file system. Then, the raw data is cleaned and standardized according to its data type. For structured data, perform operations such as field type validation, missing value filling, and format standardization; For unstructured data, OCR+NLP technology is used to extract text content, and regular expressions are used to filter noise, remove stop words, and standardize special fields such as dates / encoding.
[0071] This process ultimately generates a high-quality raw dataset, providing reliable input for subsequent sharding and retrieval.
[0072] Then, the classification and grading rules in industry standards and internal enterprise specifications are transformed into structured text entries (including fields such as sensitivity level, feature description, and judgment criteria), and vectorized using a preset semantic encoding model (such as BERT) to build a vector knowledge base. When a target data fragment is received, the system encodes it into a query vector, performs an approximate nearest neighbor search (ANN) in the vector knowledge base, retrieves the most semantically relevant classification and grading rules, and dynamically injects them into multi-layer prompt word templates as contextual support for large language model inference.
[0073] Next, based on the retrieved rules and pre-defined multi-layered prompt word templates (including task definition, rule injection, and example guidance), the large language model is invoked to infer the target data fragments and generate preliminary labels (including classification, grading, and confidence). Subsequently, the system parses and evaluates the output results: if the confidence score is not lower than a preset threshold (e.g., 0.7), the labels are automatically associated with the original data and stored in the database; otherwise, they are pushed to a manual review interface for correction. Confirmed mislabeled samples are included in the fine-tuning dataset for incremental training of the large language model, forming a closed-loop mechanism of "labeling—evaluation—feedback—optimization" to continuously improve the system's accuracy.
[0074] The above-described method acquires unstructured and structured data; according to a preset data segmentation strategy, the structured and unstructured data are divided to obtain corresponding target data fragments; based on the target data fragments, semantically relevant classification and grading rules are retrieved from a pre-built vector knowledge base; based on the classification and grading rules, combined with a preset large language model, the target data fragments are labeled using preset multi-layer prompt word templates to obtain corresponding data tags. This achieves unified, efficient, and accurate labeling of structured and unstructured data, significantly improving the automation level and compliance assurance capabilities of data classification and grading.
[0075] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data labeling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.
[0076] This application also provides a data labeling device, please refer to... Figure 6 The data marking device includes: Module 10 is used to acquire unstructured data and structured data; The partitioning module 20 is used to partition the structured data and the unstructured data according to a preset data partitioning strategy to obtain corresponding target data segments; The retrieval module 30 is used to retrieve semantically relevant classification and grading rules from a pre-built vector knowledge base based on the target data fragment. The tagging module 40 is used to tag the target data fragment based on the classification and grading rules, combined with a preset large language model, and through a preset multi-layer prompt word template to obtain the corresponding data tags.
[0077] The data marking apparatus provided in this application, employing the data marking method described in the above embodiments, can solve the technical problem of how to improve the efficiency and accuracy of data marking. Compared with the prior art, the beneficial effects of the data marking apparatus provided in this application are the same as those of the data marking method described in the above embodiments, and other technical features in the data marking apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0078] This application provides a data marking device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data marking method in the first embodiment described above.
[0079] The following is for reference. Figure 7 The diagram illustrates a structural schematic of a data marking device suitable for implementing embodiments of this application. The data marking device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The data marking device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0080] like Figure 7As shown, the data marking device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the data marking device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data marking devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.
[0081] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0082] The data marking device provided in this application, employing the data marking method described in the above embodiments, can solve the technical problem of how to improve the efficiency and accuracy of data marking. Compared with the prior art, the beneficial effects of the data marking device provided in this application are the same as those of the data marking method provided in the above embodiments, and other technical features of this data marking device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0083] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0084] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0085] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data marking method in the above embodiments.
[0086] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0087] The aforementioned computer-readable storage medium may be included in the data marking device; or it may exist independently and not be assembled into the data marking device.
[0088] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the data tagging device, the data tagging device performs the following actions: acquires unstructured data and structured data; divides the structured data and unstructured data according to a preset data segmentation strategy to obtain corresponding target data segments; retrieves semantically relevant classification and grading rules from a pre-built vector knowledge base based on the target data segments; and, based on the classification and grading rules and combined with a preset large language model, tags the target data segments using a preset multi-layer prompt word template to obtain corresponding data labels.
[0089] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0091] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0092] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data labeling method, thereby solving the technical problem of how to improve the efficiency and accuracy of data labeling. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data labeling method provided in the above embodiments, and will not be repeated here.
[0093] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data tagging method described above.
[0094] The computer program product provided in this application can solve the technical problem of how to improve the efficiency and accuracy of data labeling. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the data labeling method provided in the above embodiments, and will not be repeated here.
[0095] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A data labeling method, characterized in that, The data labeling method includes: Acquire unstructured and structured data; According to the preset data segmentation strategy, the structured data and the unstructured data are divided to obtain the corresponding target data segments; Based on the target data fragment, semantically relevant classification and grading rules are retrieved from the pre-constructed vector knowledge base; Based on the classification and grading rules, and combined with a preset large language model, the target data fragments are tagged using preset multi-layer prompt word templates to obtain corresponding data labels.
2. The data labeling method as described in claim 1, characterized in that, The step of dividing the structured data and the unstructured data according to a preset data fragmentation strategy to obtain target data fragments includes: The structured data is grouped based on the relationships between fields to obtain structured data fragments; The data type characteristics of the unstructured data are obtained, and the unstructured data is divided according to the data type characteristics to obtain unstructured data fragments.
3. The data labeling method as described in claim 1, characterized in that, Before the step of retrieving semantically relevant classification and grading rules from a pre-built vector knowledge base based on the target data fragment, the following steps are included: Obtain the classification and grading rules, and convert the classification and grading rules into rule text; The rule text is vectorized using a preset semantic encoding model to generate corresponding rule vectors; A vector knowledge base is constructed based on the aforementioned rule vectors.
4. The data labeling method as described in claim 3, characterized in that, Following the step of constructing a vector knowledge base based on the rule vectors, the method further includes: In response to the updated input of classification and grading rules, the updated input classification and grading rules are vectorized to generate the corresponding new rule vector; Add the newly added rule vector to the vector knowledge base; Based on the classification and grading rules of the updated input, the semantic encoding model is incrementally trained to maintain the semantic consistency of the rule vectors in the vector space.
5. The data labeling method as described in claim 1, characterized in that, The step of tagging the target data fragment based on the classification and grading rules, combined with a preset large language model, and using a preset multi-layer prompt word template to obtain the corresponding data labels includes: The classification and grading rules are injected into the preset multi-level prompt template to obtain the corresponding prompt words; The target data fragment and the prompt words are input into the preset large language model for reasoning, and the reasoning result is output. The reasoning results are analyzed to obtain the classification, grading information and confidence level corresponding to the target data fragment, which are used as the data label.
6. The data labeling method as described in claim 5, characterized in that, The step of inputting the target data fragment and the prompt word into the preset large language model for reasoning and outputting the reasoning result includes: Determine the corresponding inference parameters based on the data type of the target data fragment; The target data fragment and the prompt word are used as model input, and the preset large language model is called to perform inference based on the inference parameters to obtain the inference result.
7. The data labeling method as described in claim 5, characterized in that, After the step of parsing the inference result to obtain the classification, grading information and confidence level corresponding to the target data fragment as the data label, the method further includes: Determine whether the confidence level is greater than or equal to a preset threshold; If the confidence level is greater than or equal to the preset threshold, then the classification, grading information and corresponding confidence level in the data tag are associated with the target data segment and stored in the database; If the confidence level is less than the preset threshold, then in response to the feedback from manual review, mislabeled samples with confidence levels lower than the preset threshold are filtered out, the mislabeled samples are added to the fine-tuning dataset, and incremental training is performed on the preset large language model based on the fine-tuning dataset.
8. A data marking device, characterized in that, The data marking device includes: The acquisition module is used to acquire unstructured and structured data; The partitioning module is used to partition the structured data and the unstructured data according to a preset data partitioning strategy to obtain corresponding target data fragments; The retrieval module is used to retrieve semantically relevant classification and grading rules from a pre-built vector knowledge base based on the target data fragment. The tagging module is used to tag the target data fragments based on the classification and grading rules, combined with a preset large language model, and through a preset multi-layer prompt word template, so as to obtain the corresponding data tags.
9. A data marking device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data marking method as described in any one of claims 1 to 7.
10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data labeling method as described in any one of claims 1 to 7.