Data classification method and device, equipment, medium and product
By using multimodal vector representation and incremental update mechanism, the problem of low data classification efficiency in existing technologies is solved, and efficient classification that dynamically adapts to data evolution and changes in business rules is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-01
AI Technical Summary
Most existing technologies rely on static templates, which cannot adapt to data evolution and lack the application of statistical patterns in field example data, resulting in low data classification efficiency.
By combining the semantic features of field text with the statistical regularity features of example data, a unified multimodal vector representation is formed. Based on the uncertainty detection of classification results, manual/model annotation and incremental updates are triggered to achieve continuous evolution of classification templates and vector libraries.
It improves data classification efficiency by dynamically updating templates and vector libraries to adapt to changes in data content and business rules, thereby reducing classification error rate and computational resource consumption.
Smart Images
Figure CN121958433A_ABST
Abstract
Description
A data classification method, apparatus, equipment, medium, and product Technical Field
[0001] This application relates to the field of data classification management, and in particular to a data classification method, apparatus, equipment, medium and product. Background Technology
[0002] In today's digital age, data has become a key element in driving enterprise development and enhancing competitiveness. However, the disorderly accumulation of data assets also poses challenges to enterprises, making data classification and hierarchical management crucial.
[0003] Existing technical solutions mainly achieve classification and grading through field text semantic modeling and the utilization of structured context information. Alternatively, they may use regular expression rules for validation to assist in classification results.
[0004] However, most existing technologies rely on static templates, which cannot adapt to data evolution and lack the application of statistical patterns in field example data, resulting in low data classification efficiency. Summary of the Invention
[0005] This application provides a data classification method, apparatus, device, medium, and product to address the problem that existing technologies mostly rely on static templates, cannot adapt to data evolution, and lack the application of statistical patterns in field example data, resulting in low data classification efficiency.
[0006] Firstly, this application provides a data classification method, including:
[0007] Retrieve current database metadata;
[0008] Preprocess the current database metadata to obtain the first data;
[0009] The first data is fused using a multimodal fusion algorithm to obtain a fused vector.
[0010] Similarity filtering is performed based on the template vector set and the fused vector to obtain the classification result matrix; wherein, the template vector set is generated by the template tree;
[0011] The classification result matrix is determined using a category identification strategy to determine whether the new category is included.
[0012] If the classification result matrix contains newly added categories, then update the template tree, re-acquire the current database metadata, and perform data classification.
[0013] In one possible design, the current database metadata includes multiple fields;
[0014] The current database metadata also includes: table name, table description, field name, field description, and sample data;
[0015] Preprocess the current database metadata to obtain the first data, including:
[0016] Generate enhanced text for each field; the enhanced text for each field consists of the table name, table description, field name, field description, and sample data for each field.
[0017] In one possible design, a vector fusion process is performed between the first data and a multimodal fusion algorithm to obtain a fused vector, including:
[0018] Each enhanced text is encoded using a semantic vector model to obtain multiple text modality vectors;
[0019] Each example data segment undergoes structured processing to obtain multiple structured modality vectors. The structured processing includes extraction, statistical processing, rule-based feature processing, and numerical vectorization. Each field corresponds to a text modality vector and a structured modality vector.
[0020] By mapping all structural modal vectors to the same-dimensional space of text modal vectors through linear mapping, structural modal vectors of the same dimension are obtained;
[0021] The text modality vector corresponding to each field is fused with the structural modality vector of the same dimension using a multimodal fusion algorithm to generate multiple fused vectors.
[0022] In one possible design, the template vector set contains multiple path vectors;
[0023] The template tree contains multiple paths, each corresponding to a path vector;
[0024] Similarity filtering is performed based on the template vector set and the fused vector to obtain a classification result matrix, including:
[0025] Based on the cosine similarity algorithm, the cosine similarity between each fusion vector and each path vector is calculated; where each fusion vector corresponds to multiple cosine similarities.
[0026] Filter out the maximum cosine similarity corresponding to each fusion vector;
[0027] Each maximum cosine similarity is used as a classification result, and all classification results form a classification result matrix.
[0028] In one possible design, a category recognition strategy is used to determine whether the classification result matrix contains a new category, including:
[0029] Determine whether each classification result meets the uncertainty detection strategy based on the category recognition strategy;
[0030] If the classification result meets the uncertainty detection strategy, then label the classification result with low-confidence sample labels;
[0031] Based on the pre-set model, label the classification results with low-confidence sample labels with category labels;
[0032] Determine whether the category label is a newly added category based on the template vector set.
[0033] In one possible design, before retrieving the current database metadata, the following is also included:
[0034] The template tree is processed into a string to obtain the first template data;
[0035] The hash value of the first template data is calculated based on the hash algorithm.
[0036] The hash value is matched against the vector database to obtain the matching results;
[0037] If a matching result exists, load the matching result directly.
[0038] If no matching result is found, the template tree is processed by a depth-first traversal algorithm to extract the path and obtain the path set.
[0039] The path set is encoded using a semantic vector model to obtain a template vector set;
[0040] Store the template vector set into the vector database.
[0041] Secondly, this application provides a data classification apparatus, comprising:
[0042] The retrieval module is used to retrieve the current database metadata.
[0043] The preprocessing module is used to preprocess the current database metadata to obtain the first data;
[0044] The fusion module is used to perform vector fusion processing on the first data according to the multimodal fusion algorithm to obtain the fused vector;
[0045] The filtering module is used to perform similarity filtering based on the template vector set and the fused vector to obtain a classification result matrix; wherein, the template vector set is generated by the template tree;
[0046] The judgment module is used to determine whether the classification result matrix contains a new category based on a category recognition strategy.
[0047] The update module is used to update the template tree, re-acquire the current database metadata, and perform data classification if the classification result matrix contains a new category.
[0048] Thirdly, this application provides a data classification device, including: a memory and a processor;
[0049] The memory stores the instructions that the computer executes;
[0050] The processor executes computer execution instructions stored in memory, causing the processor to perform the data classification method as described in the first aspect of the invention.
[0051] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data classification method as described in the first aspect of the invention.
[0052] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data classification method described in the first aspect of the invention.
[0053] This application provides a data classification method, apparatus, device, medium, and product, comprising: acquiring current database metadata; preprocessing the current database metadata to obtain first data; performing vector fusion processing on the first data according to a multimodal fusion algorithm to obtain a fusion vector; performing similarity filtering processing on the template vector set and the fusion vector to obtain a classification result matrix; determining whether the classification result matrix contains a new category through a category recognition strategy; if the classification result matrix contains a new category, updating the template tree, reacquiring the current database metadata, and performing data classification. Compared to existing technologies that mostly rely on static templates, cannot adapt to data evolution, and lack the application of statistical regularities in field example data, resulting in low data classification efficiency, this application combines the semantic features of field text with the statistical regularity features of example data to form a unified multimodal vector representation, and triggers manual / model annotation and incremental updates based on uncertainty detection of classification results, thereby achieving continuous evolution of classification templates and vector libraries, and thus improving data classification efficiency. Attached Figure Description
[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 is a schematic diagram of the system architecture of a data classification method provided in an embodiment of this application;
[0056] Figure 2 is a schematic flowchart of a data classification method provided in an embodiment of this application;
[0057] Figure 3 is a schematic diagram of a data classification method provided in an embodiment of this application;
[0058] Figure 4 is a schematic flowchart of a data classification method provided in an embodiment of this application;
[0059] Figure 5 is a schematic diagram of the operation flow of the data classification system provided in the embodiment of this application;
[0060] Figure 6 is a schematic diagram of the data classification device provided in an embodiment of this application;
[0061] Figure 7 is a schematic diagram of the structure of a data classification device provided in an embodiment of this application. Detailed Implementation
[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0063] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, nor do they necessarily imply difference. It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.
[0064] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not impose a specific limitation on this. Furthermore, the data classification method provided in the embodiments of this application is merely an example, and data classification methods may include more or less content.
[0065] In today's digital age, data has become a key element in driving enterprise development and enhancing competitiveness. However, the disorderly accumulation of data assets also poses challenges to enterprises, making data classification and hierarchical management crucial.
[0066] Enterprises face the need for efficient classification and grading of fields in massive databases, including: in the financial industry, fields in customer information tables (such as "ID number" and "account balance") need to be graded according to sensitivity to meet compliance requirements; in the medical field, fields such as "diagnosis results" and "medication records" in patient medical records need to be stored in categories to ensure privacy and security.
[0067] Existing technical solutions mainly achieve classification and grading through field text semantic modeling and the utilization of structured contextual information. Typical methods include: semantically encoding field names and descriptions based on large language models and matching them with classification criteria to generate classification labels; combining industry classification frameworks and using hierarchical multi-step classification and prompting engineering techniques to determine field categories layer by layer; and using regular expressions to validate field content and assist in classification results.
[0068] Traditional methods rely on field names, descriptions, or table structure information for semantic matching, but this presents numerous challenges in practical applications: ambiguous field names (such as "code" and "value") lead to classification errors; fields with similar semantics may have significantly different meanings in different business scenarios (such as "date of birth" and "date"); and data content evolves over time (e.g., new field types are added or business rules change). Furthermore, enterprises frequently need to manually correct classification errors, but the corrections are difficult to retain in the system, resulting in high maintenance costs.
[0069] The existing technical solutions have the following limitations:
[0070] Optional, rely on static templates: Once the classification model and template are built, they lack a dynamic update mechanism and cannot adapt to changes in data content or business rules, resulting in a decrease in classification accuracy over time.
[0071] Optionally, ignoring the statistical patterns of the sample data: relying solely on the semantic information of field names or descriptions makes it difficult to handle fields with ambiguous names or similar semantics but different meanings, resulting in a high classification error rate.
[0072] Optionally, it lacks proactive learning capabilities: when low-confidence samples appear in the classification results, the system cannot automatically trigger annotation and model updates; manual correction is inefficient and cannot be retained in the system, leading to the recurrence of similar errors, increasing maintenance costs, and failing to form a closed-loop optimization.
[0073] Optionally, high computational resource consumption: Some solutions rely on large models for multi-round classification or complex algorithms, resulting in low processing efficiency and difficulty in meeting the needs of large-scale data scenarios.
[0074] To address the aforementioned issues, the inventors, in their research on the low efficiency of data classification, discovered that existing technologies largely rely on static templates, which cannot adapt to data evolution and lack the application of statistical patterns in field example data, thus resulting in low data classification efficiency. Therefore, the inventors considered combining the semantic features of field text with the statistical features of example data to form a unified multimodal vector representation. Based on uncertainty detection of classification results, manual / model annotation and incremental updates are triggered, enabling continuous evolution of the classification template and vector library. Based on this, embodiments of this application provide a data classification method, apparatus, device, medium, and product, which can be used in the field of data classification management, aiming to solve the problem of low data classification efficiency in existing technologies.
[0075] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0076] Figure 1 is a schematic diagram of the system architecture of a data classification method provided in an embodiment of this application. The data classification system is a computer device. In Figure 1, the architecture includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.
[0077] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the processing system architecture of the data classification method. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or divide some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. The components shown in Figure 1 can be implemented in hardware, software, or a combination of software and hardware.
[0078] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface. The data acquisition device 101 can connect to the processing device through the input / output interface or the communication interface to obtain the current database metadata and template vector set.
[0079] The processing device 102 can classify data based on the current database metadata and template vector set.
[0080] The display device 103 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.
[0081] It should be understood that the aforementioned processing device can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.
[0082] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0083] The technical solution of this application will be described in detail below with reference to specific embodiments:
[0084] Figure 2 is a schematic flowchart of a data classification method provided in an embodiment of this application. As shown in Figure 2, the method includes:
[0085] S201. Obtain the current database metadata.
[0086] It should be noted that the following steps are included before step S201:
[0087] Specifically, the template tree is processed into a string to obtain the first template data.
[0088] The template tree is a JSON-formatted tree structure of classification and grading templates.
[0089] Specifically, a template tree is a data hierarchy and classification standard model based on a tree-like data structure. It uses nested key-value pair logic to systematically organize discrete data items according to business affiliation, logical relevance, and security sensitivity.
[0090] Furthermore, the hash value of the first template data is calculated according to the hash algorithm.
[0091] In this embodiment, the input classification and grading template (JSON format tree structure) is stringified to generate a hash value H(T).
[0092] The formula for calculating the hash value is as follows:
[0093]
[0094] Where H(T) is the hash value.
[0095] Furthermore, the hash value is matched against a vector database to obtain the matching result.
[0096] Furthermore, if a matching result exists, it is loaded directly.
[0097] Furthermore, if no matching result is found, the template tree is processed by a depth-first traversal algorithm to extract the path set.
[0098] In this embodiment, a vector set named H(T) is searched in the vector database: if it exists, it is loaded directly; if it does not exist, all leaf node paths are extracted using a depth-first traversal algorithm to obtain the set. .
[0099] Furthermore, the path set is encoded using a semantic vector model to obtain a template vector set.
[0100] In this embodiment, for each path Using semantic vector models (e.g., BGE-M3) is encoded as a vector:
[0101]
[0102] Furthermore, the template vector set is stored in a vector database.
[0103] In this embodiment, the path vectors corresponding to each path are combined into a template vector set (the path vector set is...). The template vector set is stored in the vector database and named H(T).
[0104] S202. Preprocess the current database metadata to obtain the first data.
[0105] The current database metadata includes multiple fields.
[0106] The current database metadata also includes: table name, table description, field name, field description, and sample data.
[0107] Specifically, enhanced text is generated for each field.
[0108] The enhanced text for each field consists of the table name, table description, field name, field description, and sample data.
[0109] For example, enhanced text is:
[0110] x Table name: "+t+", table description: "+d_t+"; field name: "+f+", field description: "+d_f+", example: "+s".
[0111] Where t, d_t, f, d_f, and s represent the table name, table description, field name, field description, and sample data, respectively.
[0112] In one possible implementation, during the generation of enhanced text for fields, an industry-specific knowledge base (such as a terminology dictionary for finance, healthcare, or government affairs) is introduced to enhance the enhanced text with domain knowledge. For example, in a healthcare scenario, if the field description contains keywords such as "diagnosis" or "medication," the corresponding category path in the "medical terminology table" (such as "medical data → diagnosis results → sensitive data") is automatically associated.
[0113] Specifically, the enhanced text is semantically expanded using a domain knowledge base. Knowledge graphs or rule engines are used to match field descriptions with industry terminology, and the matching results are injected into the enhanced text. For example, the field description "patient medication records" is associated with the "medication records → sensitive data" path in the "medical terminology table," forming the enhanced text: "Table name: medical records, table description: stores patient treatment data; field name: medication records, field description: patient medication records, example: aspirin 100mg, domain knowledge: medical terminology table → sensitive data."
[0114] In particular, domain knowledge enhancement significantly improves classification accuracy in specific industry scenarios. For example, in medical data classification, the field "medication record" is associated with the "sensitive data" path, thus avoiding misclassification as a "normal record." Furthermore, the injection of domain knowledge can enhance the model's semantic understanding of ambiguous fields (such as "code" and "value"), reducing classification errors caused by ambiguous naming.
[0115] In addition, preprocessing also includes dividing batches at the table level and building an asynchronous task queue.
[0116] In one possible implementation, a distributed computing framework (such as Apache Spark) is introduced to break down the asynchronous task queue into multiple subtasks for parallel processing. For example, the task of classifying fields from multiple tables in a database can be distributed to different computing nodes, with each node independently generating enhanced text and submitting it to a vector database.
[0117] Technical approach: Leveraging the parallel processing capabilities of a distributed computing framework, table-level tasks are divided into field-level tasks, and message queues (such as Kafka) are used to coordinate task distribution and result aggregation. For example, the table "Customer Information" is split into fields "Name," "ID Number," and "Account Balance." Each field task is executed independently, and the results are aggregated and submitted to a vector database.
[0118] In particular, distributed parallel processing significantly improves classification efficiency in large-scale data scenarios. For example, when the financial industry needs to process tens of millions of fields, traditional single-node processing may take several hours, while the distributed framework can compress the task time to minutes. In addition, the optimization of asynchronous task queues can avoid single points of failure and improve system stability.
[0119] S203. Perform vector fusion processing on the first data according to the multimodal fusion algorithm to obtain the fused vector.
[0120] S204. Perform similarity filtering based on the template vector set and the fusion vector to obtain the classification result matrix.
[0121] The template vector set is generated from the template tree.
[0122] The template vector set contains multiple path vectors.
[0123] The template tree contains multiple paths, each corresponding to a path vector.
[0124] Specifically, based on the cosine similarity algorithm, the cosine similarity between each fusion vector and each path vector is calculated.
[0125] Each fusion vector corresponds to multiple cosine similarities.
[0126] For example, the fused vector With template vector set Perform cosine similarity calculation:
[0127]
[0128] Furthermore, the maximum cosine similarity corresponding to each fusion vector is selected.
[0129] Furthermore, each maximum cosine similarity is used as a classification result, and all classification results form a classification result matrix.
[0130] For example, select The path with the highest similarity is used as the classification result, and all classification results form a result matrix R.
[0131] S205. Determine whether the classification result matrix contains newly added categories through a category recognition strategy.
[0132] S206. If the classification result matrix contains newly added categories, update the template tree, re-acquire the current database metadata, and perform data classification.
[0133] Optional, if category label If it belongs to an existing category, then... Add to the set of vectors for this category:
[0134]
[0135] Optional, if category label If a new category is added, the template tree is updated and the new path vector is recalculated.
[0136] Furthermore, continuous iteration forms a closed loop of "classification, uncertainty detection, annotation, incremental update, and classification," enabling the system performance to continuously optimize over time. When encountering low-confidence samples, the system can automatically trigger manual or large-model-assisted correction and update the correction results to the classification template and vector library in real time, giving the classifier continuous self-learning and evolution capabilities, rather than gradually aging after a one-time construction.
[0137] In one possible implementation, during the active learning loop, when the system detects a new category, it not only expands the classification template but also synchronously updates the retrieval index structure in the vector database. For example, when a new category "virtual currency account" is added, the system automatically generates a new path "financial data, virtual currency, account" and optimizes the index structure of the vector database (e.g., adding the vector of the new path to the inverted index).
[0138] Specifically, during the template expansion phase, an incremental index update strategy is adopted to avoid a full index rebuild. For example, the inverted index of a vector database only needs to add vectors for new paths, rather than recalculating all path vectors. Simultaneously, retrieval efficiency is improved through index structure optimization (such as hierarchical indexes or locality-sensitive hashing).
[0139] In particular, the synergy between incremental learning and index optimization significantly reduces the computational cost of template expansion. For example, when the financial industry adds a "virtual currency account" field, the system does not need to rebuild the entire classification template vector library; it only needs to update the index locally, saving resources. Furthermore, index structure optimization can improve the retrieval speed of new categories and avoid performance degradation caused by template expansion.
[0140] In this embodiment, dynamic weight adjustment enhances the adaptability of multimodal fusion. For example, the "ID number" field, due to its significant structural features (such as fixed length and regular expression matching), can more accurately match the "sensitive data → ID number" path after dynamic weighting, while the "remarks" field, due to its complex textual semantics, can more accurately capture its descriptive semantics after weighting. Furthermore, this strategy reduces redundant computation and avoids over-reliance on semantic models for fields with strong rules.
[0141] In one possible implementation, an anomaly field detection module is introduced into the active learning loop to automatically identify fields in the sample data that contain outliers (such as the "age" field containing "150 years old" or "negative values"), and trigger manual annotation or model correction. For example, if the system detects that the "age" field sample data contains "150 years old" or "negative values", it will trigger manual annotation or model correction. When an error occurs, it is automatically marked as an abnormal field and pushed to the manual annotation interface.
[0142] Specifically, through statistical rules (such as...) Anomaly detection (such as IQR range detection) or machine learning models (like Isolation Forest) can be used to identify outliers. For example, sample data for the field "Age" has a mean of 35 and a standard deviation of 15. When "150 years old" is detected, A value of 9 triggers an exception flag.
[0143] Among these improvements, the detection of outlier fields significantly enhances the reliability of classification results. For example, if the field "age" is misclassified as "system error log" due to an outlier, the detection module can trigger correction in advance to avoid misclassification. Furthermore, this mechanism reduces the workload of manual annotation, allowing focus only on high-priority outlier fields and improving maintenance efficiency.
[0144] This embodiment provides a data classification method, including: acquiring current database metadata; preprocessing the current database metadata to obtain first data; performing vector fusion processing with the first data according to a multimodal fusion algorithm to obtain a fusion vector; performing similarity filtering processing based on a template vector set and the fusion vector to obtain a classification result matrix; determining whether the classification result matrix contains a new category through a category recognition strategy; if the classification result matrix contains a new category, updating the template tree, reacquiring the current database metadata, and performing data classification. Compared to existing technologies that mostly rely on static templates, cannot adapt to data evolution, and lack the application of statistical patterns in field example data, resulting in low data classification efficiency, this application combines field text semantic features with statistical pattern features of example data to form a unified multimodal vector representation, and triggers manual / model annotation and incremental updates based on uncertainty detection of classification results, realizing the continuous evolution of classification templates and vector libraries, thereby improving data classification efficiency.
[0145] Figure 3 is a schematic flowchart of a data classification method provided in an embodiment of this application. As shown in the figure, step S203 specifically includes:
[0146] S301. Encode each enhanced text using a semantic vector model to obtain multiple text modality vectors.
[0147] For example, text can be augmented by encoding using a semantic vector model to obtain multiple text modality vectors:
[0148]
[0149] S302. Perform structured processing on each sample data segment to obtain multiple structured mode vectors.
[0150] The structured processing includes: extraction processing, statistical processing, rule-based feature processing, and numerical vectorization processing.
[0151] Each field corresponds to a text modality vector and a structure modality vector.
[0152] For example, from the sample dataset Extraction, statistical analysis, and rule-based feature generation are performed to obtain a numerical vector.
[0153]
[0154] in, The average length; Maximum length; The ratio of numbers / letters / special characters; To determine whether to use regular expressions to match dates, ID numbers, etc.; Shannonentropy is the enumeration value.
[0155] This approach leverages table structure and field text semantics, and models the statistical features of the example data (such as length distribution, character composition ratio, regular expression matching results, enumeration entropy, etc.) as structural modal vectors, which are then fused with semantic modal vectors. This significantly improves the classification accuracy for fields with similar text but different meanings or ambiguous field names.
[0156] S303. All structural modal vectors are mapped to the same-dimensional space of text modal vectors through linear mapping to obtain structural modal vectors of the same dimension.
[0157] For example, a linear mapping can be used to map structural modality vectors to a space with the same dimensions as the text modality:
[0158]
[0159] S304. The text modality vector corresponding to each field is fused with the structural modality vector of the same dimension through a multimodal fusion algorithm to generate multiple fused vectors.
[0160] The formula for calculating the fusion vector is as follows:
[0161]
[0162] In one possible implementation, during multimodal fusion, the weights of text modality and structural modality features are dynamically adjusted based on the field type. For example, for fields with strong regularity, such as "date" or "ID number," structural modality features (such as regular expression matching results and enumeration entropy) are given higher weights; while for "descriptive fields" (such as "remarks" or "explanations"), text modality features are given higher weights.
[0163] Specifically, the field type classifier predicts the field type and dynamically assigns weights based on preset rules. For example, after the field type classifier identifies "ID number", it automatically increases the weight of the structural modality feature from the default 0.5 to 0.8, and decreases the weight of the text modality to 0.2.
[0164] In particular, domain knowledge enhancement significantly improves classification accuracy in specific industry scenarios. For example, in medical data classification, the field "medication record" is associated with the "sensitive data" path, thus avoiding misclassification as a "normal record." Furthermore, the injection of domain knowledge can enhance the model's semantic understanding of ambiguous fields (such as "code" and "value"), reducing classification errors caused by ambiguous naming.
[0165] In this embodiment, dynamic weight adjustment enhances the adaptability of multimodal fusion. For example, the "ID number" field, due to its significant structural characteristics (such as fixed length and regular expression matching), can more accurately match "sensitive data" after dynamic weighting. The "ID number" path is used, while the "remarks" field, due to its complex textual semantics, can more accurately capture its descriptive semantics after weighting. In addition, this strategy can reduce redundant calculations and avoid over-reliance on semantic models for fields with strong rules.
[0166] Figure 4 is a schematic flowchart of a data classification method provided in an embodiment of this application. As shown in the figure, the specific implementation steps of S205 above include:
[0167] S401. Determine whether each classification result meets the uncertainty detection strategy based on the category recognition strategy.
[0168] For example, uncertainty detection includes:
[0169] Specifically, if a field classification result meets any of the following conditions, it is marked as a low-confidence sample:
[0170]
[0171]
[0172] S402. If the classification result meets the uncertainty detection strategy, then label the classification result with low confidence sample labels.
[0173] S403. According to the preset model, label the classification results with low confidence sample labels with category labels.
[0174] Specifically, based on a pre-set model or manually, category labels are assigned to classification results with low-confidence sample labels. .
[0175] S404. Determine whether the category label is a newly added category based on the template vector set.
[0176] In this embodiment, through the coordinated judgment of category recognition and uncertainty detection strategies, low-confidence sample labeling, pre-defined model secondary classification labeling, and the streamlined process of adding category detection to the template vector set, accurate identification and dynamic optimization of uncertain samples in classification tasks are achieved. This approach strengthens the model's learning ability for ambiguous boundary samples through low-confidence labeling and enhances the model's adaptive expansion capability for new categories through the added category detection mechanism. Ultimately, it generates more robust and interpretable classification results, effectively solving the performance bottleneck problem of traditional classification models in scenarios with ambiguous boundary samples and new categories. This improves data classification efficiency.
[0177] This application also provides a possible embodiment. Figure 5 is a schematic diagram of the operation flow of the data classification system provided in this application embodiment. As shown in Figure 5, the system takes the initialization of classification and grading templates as the core starting point. It checks whether the vector library has a corresponding set by checking the hash value. If it exists, it is loaded directly; otherwise, it extracts the leaf node path through DFS and encodes and stores it in the vector database. Then, it enters the data preprocessing stage, constructs an asynchronous queue to generate field-enhanced text, and performs classification and grading retrieval after multimodal fusion vectorization (linear mapping fusion of structural modality statistical features and text modality encoding). In the active learning closed loop, the system judges the confidence threshold of the Top-K candidate paths. Low-confidence samples trigger manual / large model annotation. Finally, the system iterates through template tree structure update and path vector increment calculation, forming a complete "preprocessing-fusion-retrieval-annotation-update" closed loop process.
[0178] Specifically, data uniqueness verification and rapid retrieval are achieved through the interaction of hash values and vector libraries, while asynchronous queues and enhanced text provide the foundation for multimodal fusion. The multimodal fusion stage achieves effective fusion of cross-modal features through a linear mapping between structural modality statistical features and text modality encoding. The active learning loop dynamically optimizes the classification model through the annotation feedback of low-confidence samples, and ultimately achieves adaptive iteration and continuous performance improvement of the system through template tree structure updates and incremental calculation of vector sets.
[0179] In this embodiment, a multimodal fusion mechanism significantly improves classification accuracy, while an active learning loop enables intelligent labeling and model optimization of low-confidence samples, enhancing system robustness. An incremental update mechanism allows the system to continuously adapt to new data categories, maintaining dynamic optimization of classification performance. The overall architecture, while improving classification accuracy, achieves scalability and adaptability through template tree structure updates and vector set expansion, effectively supporting data classification needs in complex scenarios.
[0180] Figure 6 is a schematic diagram of the data classification device provided in the embodiment of this application. As shown in Figure 7, the device includes: an acquisition module 61, a preprocessing module 62, a fusion module 63, a filtering module 64, a judgment module 65, and an update module 66.
[0181] Module 61 is used to obtain the current database metadata;
[0182] Preprocessing module 62 is used to preprocess the current database metadata to obtain the first data;
[0183] The fusion module 63 is used to perform vector fusion processing on the first data according to the multimodal fusion algorithm to obtain a fused vector;
[0184] The filtering module 64 is used to perform similarity filtering based on the template vector set and the fused vector to obtain a classification result matrix; wherein, the template vector set is generated by the template tree;
[0185] Module 65 is used to determine whether the classification result matrix contains a new category through a category recognition strategy;
[0186] The update module 66 is used to update the template tree, re-acquire the current database metadata, and perform data classification if the classification result matrix contains a new category.
[0187] In one possible design, the current database metadata includes multiple fields;
[0188] The current database metadata also includes: table name, table description, field name, field description, and sample data;
[0189] Preprocess the current database metadata to obtain the first data, including:
[0190] The preprocessing module 62 is also used to generate enhanced text for each field; wherein, the enhanced text for each field consists of the table name, table description, field name, field description and sample data corresponding to each field.
[0191] In one possible design, a vector fusion process is performed between the first data and a multimodal fusion algorithm to obtain a fused vector, including:
[0192] The fusion module 63 is also used to encode each enhanced text using a semantic vector model to obtain multiple text modality vectors;
[0193] Each example data segment undergoes structured processing to obtain multiple structured modality vectors. The structured processing includes extraction, statistical processing, rule-based feature processing, and numerical vectorization. Each field corresponds to a text modality vector and a structured modality vector.
[0194] By mapping all structural modal vectors to the same-dimensional space of text modal vectors through linear mapping, structural modal vectors of the same dimension are obtained;
[0195] The text modality vector corresponding to each field is fused with the structural modality vector of the same dimension using a multimodal fusion algorithm to generate multiple fused vectors.
[0196] In one possible design, the template vector set contains multiple path vectors;
[0197] The template tree contains multiple paths, each corresponding to a path vector;
[0198] Similarity filtering is performed based on the template vector set and the fused vector to obtain a classification result matrix, including:
[0199] The filtering module 64 is also used to calculate the cosine similarity between each fusion vector and each path vector according to the cosine similarity algorithm; wherein each fusion vector corresponds to multiple cosine similarities;
[0200] Filter out the maximum cosine similarity corresponding to each fusion vector;
[0201] Each maximum cosine similarity is used as a classification result, and all classification results form a classification result matrix.
[0202] In one possible design, a category recognition strategy is used to determine whether the classification result matrix contains a new category, including:
[0203] The judgment module 65 is also used to determine whether each classification result meets the uncertainty detection strategy based on the category recognition strategy;
[0204] If the classification result meets the uncertainty detection strategy, then label the classification result with low-confidence sample labels;
[0205] Based on the pre-set model, label the classification results with low-confidence sample labels with category labels;
[0206] Determine whether the category label is a newly added category based on the template vector set.
[0207] In one possible design, before retrieving the current database metadata, the following is also included:
[0208] The template tree is processed into a string to obtain the first template data;
[0209] The hash value of the first template data is calculated based on the hash algorithm.
[0210] The hash value is matched against the vector database to obtain the matching results;
[0211] If a matching result exists, load the matching result directly.
[0212] If no matching result is found, the template tree is processed by a depth-first traversal algorithm to extract the path and obtain the path set.
[0213] The path set is encoded using a semantic vector model to obtain a template vector set;
[0214] Store the template vector set into the vector database.
[0215] The data classification device provided in this embodiment can execute a data classification method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.
[0216] In a specific implementation of the aforementioned data classification method, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned data classification method.
[0217] Figure 7 is a schematic diagram of a data classification device according to an embodiment of this application. As shown in Figure 7, the data classification device 70 includes at least one processor 71 and a memory 72. The data classification device 70 also includes a communication component 73. The processor 71, memory 72, and communication component 73 are connected via a bus 74.
[0218] In the specific implementation process, at least one processor 71 executes computer execution instructions stored in memory 72, causing at least one processor 71 to execute a data classification management method as executed on the data classification device side as described above.
[0219] The specific implementation process of processor 71 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0220] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0221] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.
[0222] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0223] The above description of the functions implemented by the data classification device and the main control device illustrates the solutions provided by the embodiments of the present invention. It is understood that, in order to achieve the above functions, the data classification device or the main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.
[0224] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the above-described method in the field of data classification and management.
[0225] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0226] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in a data classification device or a master control device.
[0227] This application also provides a computer program product, comprising: a computer program stored in a readable storage medium, at least one processor of the data classification device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the data classification device to perform the scheme provided in any of the above embodiments.
[0228] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.
[0229] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A data classification method, characterized in that, include: Retrieve current database metadata; Preprocess the current database metadata to obtain the first data; The first data is fused using a multimodal fusion algorithm to obtain a fused vector; a similarity filter is performed between the fused vector and a template vector set to obtain a classification result matrix; wherein the template vector set is generated by a template tree; and a category recognition strategy is used to determine whether the classification result matrix contains a new category. If the classification result matrix contains the newly added category, then update the template tree, re-acquire the current database metadata, and perform data classification.
2. The method according to claim 1, characterized in that, The current database metadata includes multiple fields; The current database metadata also includes: table name, table description, field name, field description, and sample data; the preprocessing of the current database metadata to obtain the first data includes: generating enhanced text for each field; wherein, the enhanced text corresponding to each field consists of the table name, table description, field name, field description, and sample data corresponding to each field.
3. The method according to claim 2, characterized in that, The step of performing vector fusion processing with the first data according to the multimodal fusion algorithm to obtain fused vectors includes: encoding each of the enhanced texts using a semantic vector model to obtain multiple text modal vectors; performing structuring processing on each segment of the example data to obtain multiple structural modal vectors; wherein the structuring processing includes: extraction processing, statistical processing, rule feature processing, and numerical vectorization processing; wherein each field corresponds to a text modal vector and a structural modal vector; mapping all the structural modal vectors to the same-dimensional space of the text modal vectors through linear mapping to obtain same-dimensional structural modal vectors; and fusing the text modal vectors corresponding to each field with the same-dimensional structural modal vectors using the multimodal fusion algorithm to generate multiple fused vectors.
4. The method according to claim 3, characterized in that, The template vector set contains multiple path vectors; the template tree contains multiple paths, each path corresponding to one path vector; the similarity filtering process based on the template vector set and the fusion vector to obtain a classification result matrix includes: calculating the cosine similarity between each fusion vector and each path vector using a cosine similarity algorithm; wherein each fusion vector corresponds to multiple cosine similarities; filtering out the maximum cosine similarity corresponding to each fusion vector; using each maximum cosine similarity as a classification result, and all classification results forming the classification result matrix.
5. The method according to claim 4, characterized in that, The step of determining whether the classification result matrix contains a new category using a category identification strategy includes: determining whether each classification result satisfies an uncertainty detection strategy according to the category identification strategy; if the classification result satisfies the uncertainty detection strategy, labeling the classification result with low-confidence sample labels; labeling the classification result with the low-confidence sample labels with category labels according to a preset model; and determining whether the category labels are the new category according to the template vector set.
6. The method according to claim 5, characterized in that, Before obtaining the current database metadata, the method further includes: stringifying the template tree to obtain first template data; calculating the hash value of the first template data according to a hash algorithm; matching the hash value in a vector database to obtain a matching result; if the matching result exists, directly loading the matching result; if the matching result does not exist, performing path extraction processing on the template tree using a depth-first traversal algorithm to obtain a path set; encoding the path set using the semantic vector model to obtain the template vector set; and storing the template vector set in the vector database.
7. A data classification device, characterized in that, include: The retrieval module is used to retrieve the current database metadata. A preprocessing module is used to preprocess the current database metadata to obtain the first data; The fusion module is used to perform vector fusion processing with the first data according to the multimodal fusion algorithm to obtain a fused vector; the filtering module is used to perform similarity filtering processing with the fused vector according to the template vector set to obtain a classification result matrix; wherein, the template vector set is generated by the template tree; the judgment module is used to determine whether the classification result matrix contains a new category through a category recognition strategy. The update module is used to update the template tree, re-acquire the current database metadata, and perform data classification if the classification result matrix contains the newly added category.
8. A data classification device, characterized in that, include: Memory, processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.