Data grading method, system, device, medium and program product

Abstract

The invention discloses a data grading method, system and device, a medium and a program product, and relates to the technical field of data governance, and the method comprises the steps: obtaining a data table in each business system in response to a data obtaining operation, and obtaining a to-be-graded data table; displaying fields in the to-be-graded data table; and in response to the configuration operation, configuring a corresponding grading mode for the fields of the to-be-graded data table, and grading each field based on the grading mode to obtain a target grading result. The problem that a single mode of an existing grading mode cannot deal with complex service data can be effectively solved.

Description

Technical Field

[0001] This application relates to the technical field of data governance, and in particular to data classification methods, data classification systems, electronic devices, storage media, and computer program products. Background Technology

[0002] Data security is receiving increasing attention. To meet these requirements, enterprises need to inventory and organize their data, establish data tiered governance, clearly define data protection objects, and create and maintain timely data catalogs. Current data tiering technologies suffer from several limitations: their methods for tiering fields are too simplistic and cannot handle complex business scenarios. Specifically, traditional methods rely on pre-built rule bases, which are essentially static collections. When new fields are added to a business system, the rule base needs to be manually expanded, leading to tiering delays. Although existing tiering methods use machine learning models capable of parsing semantics, these models are trained on a general training set. When the same field appears in different business systems, the model cannot perceive the differences in business scenarios, outputting fixed tier labels that cannot automatically adjust with business rules. Summary of the Invention

[0003] The main purpose of this application is to provide a data classification method, system, device, medium and program product, which aims to solve the technical problem that existing data classification methods have a single classification mode and cannot cope with the complex business scenarios in reality.

[0004] To achieve the above objectives, this application proposes a data classification method, which includes:

[0005] In response to the data acquisition operation, data tables from various business systems are retrieved to obtain the data table to be classified.

[0006] Display the fields in the data table to be classified;

[0007] In response to the configuration operation, a corresponding classification method is configured for the fields of the data table to be classified, and each field is classified based on the classification method to obtain the target classification result.

[0008] In one embodiment, the step of obtaining data tables from each business system to obtain the data table to be hierarchical includes:

[0009] Extract the table name, field attributes, and storage location information of each physical sub-table of the data table;

[0010] Determine the semantic similarity of the table names of each physical sub-table, determine the primary and foreign key relationships of the field attributes of each physical sub-table, or determine the lineage of each physical sub-table based on the storage location information;

[0011] Based on the semantic similarity, the primary-foreign key relationship, and / or the lineage relationship, each physical sub-table is mapped to the same logical table, and the logical table is used as the data table to be classified.

[0012] In one embodiment, the grading method includes text regular expression grading, lightweight model grading, and large model grading. After the step of obtaining data tables from each business system to obtain the data table to be graded, the method further includes:

[0013] Perform text regular expression classification on the fields in the data table to be classified to obtain the first classification result;

[0014] For fields in the data table to be classified that failed to be classified using text regular expression, lightweight model classification is performed to obtain a second classification result, and the confidence score of the second classification result is determined.

[0015] From the second classification results, target fields with confidence scores lower than a preset score threshold are selected, and large model classification is performed to obtain the third classification results;

[0016] The first classification result, the second classification result, and / or the third classification result are taken as the target classification result.

[0017] In one embodiment, before performing the large model grading step, the step of filtering target fields with confidence scores lower than a preset score threshold from the second grading results includes:

[0018] The target field is used to identify the business entity type, and the preset hierarchical rule clauses that match the business entity type are recalled.

[0019] Based on the business system identifier of the data table where the target field is located, determine the historical amendment examples that match the business system identifier from the preset knowledge base;

[0020] The preset grading rules and the historical amendment examples are combined to form a context, which is then input into the large model to perform the grading.

[0021] In one embodiment, the method further includes:

[0022] When configuring the corresponding classification method for the fields of the data table to be classified, one or more classification methods are recommended for the fields of the data table to be classified based on preset recommendation rules and / or historical configuration data.

[0023] The configuration interface displays one or more of the aforementioned hierarchical methods for users to select.

[0024] In one embodiment, the method further includes:

[0025] Compare the classification results of the same field under different classification methods;

[0026] When a conflict is detected in the classification results, corresponding processing is performed according to the type of conflict: if the conflict type is the preset first type, the final classification result of the field is determined according to the pre-configured weight parameters;

[0027] If the conflict type is the preset second type, the manual review rule is triggered, the corresponding manual review result is obtained, and the manual review result is used as the final classification result of the field.

[0028] In addition, to achieve the above objectives, this application also proposes a data classification system that performs the steps of the data classification method described above.

[0029] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data grading method as described above.

[0030] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data classification method described above.

[0031] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data classification method described above.

[0032] One or more technical solutions proposed in this application have at least the following technical effects:

[0033] Existing data classification methods suffer from limitations due to their simplistic classification approaches and inability to adapt to complex business scenarios. This application's technical solution addresses these issues by responding to data acquisition operations, obtaining data tables from various business systems, and then displaying the fields within the data table to be classified. A configuration operation is then performed to assign corresponding classification methods to different fields. Finally, based on the configured classification methods, the fields of the data table to be classified are classified to obtain the target classification result.

[0034] This application's technical solution for dynamically configuring hierarchical methods enables the combination of multiple hierarchical methods, allowing fields in the data table to be hierarchically configured to suit the current business scenario. Specifically, when a new field is added to a business system, there is no need to wait for manual expansion of the rule base; the appropriate hierarchical method can be configured instantly based on the actual business scenario, avoiding hierarchical lag. Furthermore, addressing the issue of rigid hierarchical configuration of the same field across systems due to a lack of business awareness in general machine learning models, the configuration operation can assign different hierarchical methods to the same field based on specific business scenarios. This allows the hierarchical results to dynamically respond to the compliance requirements and data sensitivity differences of different business systems, thereby effectively solving the problem at its root that the single mode of existing hierarchical methods cannot cope with complex business data. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating an embodiment of the data classification method of this application.

[0038] Figure 2 A schematic diagram illustrating the data classification method of this application is provided in yet another embodiment.

[0039] Figure 3 A schematic diagram illustrating the effect of another embodiment of the data classification method of this application;

[0040] Figure 4 This is a schematic diagram of the data classification system of this application;

[0041] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the data classification method of this application.

[0042] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0044] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0045] It should be noted that the executing entity in this embodiment can be a data classification system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or processor capable of performing the above functions. The following description uses a data classification system as an example to illustrate this embodiment and the subsequent embodiments.

[0046] Based on this, embodiments of this application provide a data classification method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data classification method of this application.

[0047] In this embodiment, the data classification method includes steps S10 to S30:

[0048] Step S10: In response to the data acquisition operation, acquire the data tables in each business system to obtain the data table to be classified.

[0049] It should be noted that when the data classification system responds to a user's data acquisition operation or an automated data acquisition operation, it retrieves data tables from various business systems. For example, the retrieval method can be to pull data tables from various business systems or for the user to import data tables. Subsequently, the data tables from different sources are uniformly integrated into a data table to be classified.

[0050] In one embodiment, step S10 includes steps A10 to A30:

[0051] Step A10: Extract the table name, field attributes, and storage location information of each physical sub-table from the data table;

[0052] It should be noted that when there are multiple physical sub-tables in a data table, the data hierarchy system extracts the table name, field attributes (describes the characteristics of each field in the physical sub-table, including field name, data type, constraint relationship, etc.) and storage location information (the specific storage node path where the physical sub-table is located).

[0053] Step A20: Determine the semantic similarity of the names of each physical sub-table, determine the primary and foreign key relationships of the field attributes of each physical sub-table, or determine the lineage of each physical sub-table based on storage location information;

[0054] It should be noted that the data hierarchy system determines the semantic similarity of table names, i.e., analyzes the textual meaning association between the names of different physical sub-tables, such as the naming similarity between order_table_2023 and order_table_2024 being greater than 90%; the data hierarchy system determines the primary and foreign key relationships of field attributes (identifying key field constraints across physical sub-tables, such as the user_id field being a primary key in table A and a foreign key in table B); the data hierarchy system determines the lineage relationship based on storage location information (e.g., the paths of tables 1, 2, and 3 belonging to the same business system). In this step, the similarity, association, and lineage relationships of table names for each physical sub-table are determined, achieving centralized data management across business systems, overcoming physical storage isolation, and triggering unified sensitivity classification of associated fields.

[0055] Step A30: Based on semantic similarity, primary-foreign key relationships and / or lineage, map each physical sub-table to the same logical table, and use the logical table as the data table to be classified.

[0056] It should be noted that the data classification system aggregates the scattered physical sub-tables into a unified logical table based on the determined semantic similarity, primary and foreign key relationships and / or lineage, which serves as the data table to be classified.

[0057] This can be achieved by aggregating scattered physical sub-tables into a unified logical table based on semantic similarity, primary / foreign key relationships, or lineage. Alternatively, it can be achieved by aggregating scattered physical sub-tables into a unified logical table based on semantic similarity and primary / foreign key relationships or lineage, or by aggregating scattered physical sub-tables into a unified logical table based on primary / foreign key relationships or lineage. Alternatively, it can be achieved by aggregating scattered physical sub-tables into a unified logical table based on these three methods.

[0058] It should also be noted that this implementation method is a database sharding and table partitioning process, which automatically identifies data tables that are physically distributed in different databases or sub-tables but logically belong to the same entity, and establishes cross-database table association mapping relationships (for example, mapping MySQL sharded tables A1 and A2 together to logical table A).

[0059] In this implementation, by determining the relationship between physical sub-tables from multiple dimensions and aggregating the scattered physical sub-tables into a unified logical table, the fragmentation of business entities caused by physical storage fragmentation can be eliminated. Finally, the logical table is used as the data table to be classified. This ensures that subsequent field classification operations are directed towards complete business objects rather than discrete storage units, enabling efficient data classification. In other words, classifying the fields in the data table to be classified can classify the fields in multiple physical sub-tables.

[0060] Step S20: Display the fields in the data table to be classified;

[0061] It should be noted that after obtaining the data table to be classified, the data classification system displays the fields to be classified. For example, the data classification system creates a task sheet for each classification task. This task sheet includes the fields of the data table to be classified and the classification methods configured for those fields. The abstraction of the task sheet allows a single data table to use multiple classification methods simultaneously. For example, this method of configuring combined classification strategies in the task sheet can be quickly extended to support new classification methods.

[0062] Step S30: In response to the configuration operation, configure the corresponding classification method for the fields of the data table to be classified, classify each field according to the classification method, and obtain the target classification result.

[0063] It should be noted that the data classification system responds to user configuration operations via the interactive interface, setting the classification method for each field in the data table to be classified (such as columns in the data table, such as ID card number, address, etc.). Specifically, users can configure corresponding classification methods for different fields (referring to selecting sensitivity judgment rules for the field, such as manual annotation, rule matching, or model analysis). Then, the data classification system performs classification on each field according to the configured classification method, ultimately generating the target classification result.

[0064] It is understandable that step S30 transforms the traditional single hierarchical method into a flexible matching of the best hierarchical method for fields under different business scenarios. For example, one application scenario of this embodiment can configure hierarchical methods for fields such as the case number of the dependent document, home address update time, company name update time, English name update time, initiator system code, historical latest connection weight from the perspective of retail customers, professional mobile phone update time, customer country and region update time, initiator information, and home phone update time in the financial field to obtain hierarchical results.

[0065] In this embodiment, in response to a data acquisition operation, data tables from various business systems are retrieved to obtain a data table to be classified. The fields in the data table are then displayed, and a configuration operation is performed to configure corresponding classification methods for different fields. Finally, the fields in the data table to be classified are classified based on the configured classification methods to obtain the target classification result. The technical solution of dynamically configuring classification methods in this application allows for the mixing and matching of multiple classification methods, ensuring that each field in the data table to be classified has a classification method suitable for the current business scenario. Specifically, when a new field is added to a business system, there is no need to wait for manual expansion of the rule base; the appropriate classification method can be configured immediately according to the actual business scenario, avoiding classification lag. Simultaneously, addressing the problem of rigid classification of the same field across systems due to a lack of business awareness in general machine learning models, the configuration operation can assign different classification methods to the same field based on specific business scenarios. This allows the classification result to dynamically respond to the compliance requirements and data sensitivity differences of different business systems, thereby effectively solving the problem that the single mode of existing classification methods cannot cope with complex business data at its root.

[0066] Faced with tens of thousands of data points, directly using large models for classification would be inefficient and wasteful of valuable model resources. Therefore, a multi-level hierarchical architecture is constructed within the data classification system. Under this architecture, a combined classification method is used to perform multi-level collaborative classification of fields in the data tables to be classified. Specifically, in one implementation, after step S10, the following steps are also included:

[0067] Perform text regular expression classification on the fields in the classification data table to obtain the first classification result;

[0068] For fields in the tiered data table that failed to be tiered using text regular expressions, perform lightweight model tiering to obtain a second tiering result, and determine the confidence score of the second tiering result.

[0069] From the second classification results, target fields with confidence scores lower than the preset score threshold are selected, and large model classification is performed to obtain the third classification results;

[0070] The first, second, and / or third classification results are used as the target classification results.

[0071] It should be noted that the data classification system performs text regular expression classification on the fields in the data table to be classified (matching field content based on predefined regular expression rules, such as detecting sensitive data with fixed patterns like ID card numbers and bank card numbers). This text regular expression classification can quickly scan the text of field values, automatically mark fields that meet the rules with sensitivity levels, generate the first classification result, filter out commonly used data fields, significantly reduce the subsequent computational load, and efficiently process fields with obvious text features.

[0072] For fields that failed the text regular expression grading process, the data grading system uses lightweight model grading (i.e., employing a low-computational-cost machine learning model such as the BERT model) to analyze field features such as field name and data type, outputting a second grading result and generating a confidence score. This confidence score quantifies the reliability of the model's judgment, ranging from 0 to 1, with higher scores indicating greater reliability. Lightweight model grading enables data grading with lower resources.

[0073] The data classification system selects target fields (ambiguous fields that are difficult for lightweight models to determine, such as code, value, and other generic names) from the second classification results whose confidence scores are lower than a preset score threshold (e.g., ≤0.5) and uses a large model (e.g., GLM4 model) for classification. It analyzes the actual content of the field, the contextual semantics, and the relevance to the business scenario. The large model classification can perform deep semantic mining on ambiguous fields to ensure that no classification is missed.

[0074] Finally, the data classification system uses the first classification result generated by the regularization classification, the second classification result output by the lightweight model, and the third classification result supplemented by the large model as the target classification result.

[0075] It should also be noted that after obtaining the target classification results, a data asset catalog is constructed and stored in the data classification system, providing an interface for external or internal queries.

[0076] For example, one embodiment involves classifying data in the financial field. In the first layer, text regular expressions are used to classify common fields. Fields that fail to be classified using text regular expressions are then classified in the second layer using a BERT small model. Finally, unreliable small model classification results are filtered out and classified in the third layer using a large model. This multi-layered classification architecture improves data classification efficiency compared to directly classifying data using the entire large model, while ensuring classification accuracy.

[0077] For example, such as Figure 2 This diagram illustrates a multi-level hierarchical approach. Text regularization is applied to the fields of the data table to be hierarchically graded, yielding the first hierarchical result. Fields without text regularization are then graded using a lightweight model, resulting in the second hierarchical result. Fields with lower confidence scores are then graded using a large model, yielding the third hierarchical result.

[0078] In this implementation, a multi-level collaborative grading approach is constructed, employing three grading methods in a progressive manner. First, text regularization is used to quickly identify fields with fixed pattern characteristics, yielding the first grading result. Then, for the remaining fields that cannot be matched by rules, a lightweight model grading is activated, expanding the covered field range while generating confidence scores. Next, ambiguous fields that the model struggles to determine are identified using confidence thresholds, triggering a large-scale model grading to achieve grading of complex fields. Finally, the grading results of the three methods are used as the target grading result. This approach avoids the computational waste of performing large-scale model grading on all fields while ensuring deep analysis capabilities for unconventional and sensitive fields. It significantly reduces system overhead while ensuring comprehensive and high-precision grading processing, effectively overcoming the limitations of traditional single-level grading methods, such as incomplete field coverage, low accuracy, and low grading efficiency.

[0079] In another embodiment, the following steps may be taken: performing text regular expression grading on the first field of the grading data table to obtain a first grading result; performing lightweight model grading on the second field of the grading data table to obtain a second grading result; performing large model grading on the third field of the grading data table to obtain a third grading result; and using the first grading result, the second grading result, and / or the third grading result as the target grading result.

[0080] This involves configuring a corresponding grading method for each field in the data table to be graded, and then grading the fields based on the configured grading methods to obtain the target grading result. For example, when there is only one grading method, the target grading result is the result of that one grading method; when there are two grading methods, the target grading result is the result of the combined two grading methods; and when there are three grading methods, the target grading result is the result of the combined three grading methods. This process can achieve both efficiency and accuracy by leveraging the synergistic effect of the three grading methods or by allowing each method to function independently. Regular expression matching quickly covers typical sensitive fields, lightweight models handle regular fields, and large models handle target fields, thus covering the grading of each field.

[0081] In another implementation, prior to performing the large model grading step, the following is included:

[0082] The target field is used to identify the business entity type, and the preset hierarchical rule clauses that match the business entity type are recalled.

[0083] Based on the business system identifier of the data table where the target field is located, determine the historical amendment examples that match the business system identifier from the preset knowledge base;

[0084] The preset grading rules and the historical amendment examples are combined to form a context, which is then input into the large model to perform the grading.

[0085] It's important to note that the data classification system identifies the business entity type of the target field, i.e., it parses the business system to which the field belongs (e.g., the customer or employee to which the target field belongs). Then, it recalls pre-defined classification rules that match this business entity type (e.g., when the business entity type is customer, only the clauses concerning sensitive personal information in the privacy protection regulations are recalled, excluding irrelevant financial data regulations). This process filters out irrelevant rule noise by identifying the business entity, transforming general rules into rules that fit the business scenario, ensuring that the rules input into the large model are highly adapted to the business scenario to which the target field belongs.

[0086] Based on the business system identifier of the data table containing the target field, historical amendment examples matching the business system identifier are determined from a pre-set knowledge base (a private knowledge base storing historical classification cases). These can be records of manual corrections to classification results for similar fields, such as records where contract_no was previously adjusted from level 3 to level 2. This step inherits historical classification experience, avoids repeating errors in similar business scenarios, and improves accuracy.

[0087] The data classification system combines pre-defined classification rules and historical amendment examples that align with the business scenario into a context. Since the main model classifies target fields, the context implicitly includes the target fields. The main model then applies these classification rules and historical amendment examples to the target fields, outputting their sensitivity level. For example, for a transaction serial number field in a banking system, the main model combines financial data classification rules (which require encryption) and historical amendment examples (where similar fields are classified as level 2) to output that the transaction serial number field is level 2 sensitive.

[0088] For example, the steps for document retrieval before large-scale model grading can be as follows: Before document retrieval, the large-scale model learns the manual grading process and makes identifiable subject judgments based on different data volumes. It leverages the transforming and inferring capabilities of the large-scale model to perform English-to-Chinese translation and synonym expansion, improving the coverage and accuracy of document retrieval. During document retrieval, a self-updating closed loop of a private domain knowledge base is constructed, automatically supplementing cases whose large-scale model recognition results have been manually corrected to the proprietary private domain knowledge base under the system number to which the case belongs. As data grading deepens, a grading knowledge base built on general grading rules and specific to each business system is constructed, which helps improve the fit between grading rules and their respective systems, thus improving the accuracy of document retrieval. After document retrieval, an inverse ranking fusion algorithm (RRF) is introduced, fusing the statistical model-based TF-IDF algorithm and the representation learning-based word vector similarity algorithm. The RRF algorithm "normalizes" the differences between different text matching algorithms through ranking, increasing the probability that the most relevant document appears at the top of the final retrieved documents.

[0089] In this implementation, identifying the business entity type to determine strongly relevant hierarchical rule clauses enables filtering of hierarchical rule clauses based on the business entity type, improving document recall accuracy. Combining with historical amendment examples to generate context input to the large model allows the inheritance of historical hierarchical experience, avoids repetitive errors in similar business scenarios, and enables the large model to accurately hierarchize target fields.

[0090] Based on the above embodiments of this application, in another embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter.

[0091] Traditional hierarchical systems rely on users manually consulting rule documents to configure field hierarchies. When dealing with massive fields across business systems, this requires repeatedly switching rule bases and comparing historical cases, resulting in low configuration efficiency and susceptibility to hierarchical biases due to insufficient business understanding. The data hierarchical method in this application also includes:

[0092] When configuring the corresponding grading method for the fields of the data table to be graded, one or more grading methods are recommended and displayed for the fields of the data table to be graded based on preset recommendation rules and / or historical configuration data.

[0093] It should be noted that when configuring grading methods for fields in a data table to be graded, the data grading system automatically recommends one or more grading methods for the current field (e.g., simultaneously recommending Level 3 for sensitive identity data and Level 2 for personal information) based on preset recommendation rules (e.g., prioritizing personal privacy grading for ID card fields) and / or historical configuration data (records of manual grading selections for similar fields in the past). This step overcomes the limitations of traditional manual rule traversal. For example, for a newly added "credit score" field in a financial system, it combines financial data clauses in the rule base with historical grading records for risk assessment value fields to generate recommended options that fit the business scenario, effectively reducing configuration complexity.

[0094] The data classification system dynamically displays one or more classification methods in the configuration interface (such as drop-down menus for options like Level 3 Sensitive, Level 2 Confidential, etc.), clearly indicating the recommendation criteria for each method (e.g., matching financial rule 5 or selecting this level based on 92% similarity fields), allowing users to choose (users can confirm or correct with a single click). This step combines intelligent recommendation with human business judgment through human-machine collaborative decision-making, avoiding the scenario mismatch risk of purely automated classification while significantly reducing the time cost of manually retrieving rules.

[0095] In this embodiment, by integrating preset rules and historical configuration data to replace manual retrieval, the efficiency of hierarchical classification is improved; based on differentiated recommendations according to business context, the accuracy of hierarchical classification across business scenarios is improved; and the hierarchical criteria are transformed into visual options, which reduces configuration time and improves hierarchical configuration efficiency.

[0096] In one implementation, when traditional hierarchical systems apply multiple hierarchical methods to the same field and obtain conflicting hierarchical results, they only use simple judgment rules, which leads to a large number of fields being over-upgraded to the high sensitivity level, resulting in unnecessary data encryption costs.

[0097] The data classification method of this application includes: comparing the classification results of the same field under different classification methods;

[0098] When a conflict is detected in the classification results, the corresponding processing is performed according to the type of conflict: if the conflict type is the preset first type, the final classification result of the field is determined according to the pre-configured weight parameters;

[0099] If the conflict type is the preset second type, the manual review rule is triggered, the corresponding manual review result is obtained, and the manual review result is used as the final classification result of the field.

[0100] It should be noted that the data classification system automatically compares the classification results of the same field (such as "phone number") under different classification methods (such as regular expression matching, lightweight model, and large model) (such as "sensitive level 1" output by regular expression, "sensitive level 2" output by lightweight model, and "sensitive level 2" output by large model), and identifies conflicts by analyzing the differences in classification results.

[0101] If the conflict type is the preset first type (e.g., the difference in classification results is within a preset range), the data classification system dynamically calculates the final classification result for the field based on pre-configured weight parameters (e.g., setting the weight of the large model classification result to 40%, the weight of the lightweight model classification result to 40%, and the weight of the regular expression matching classification result to 30%). For example, for the "Account Balance" field, the large model outputs "Sensitive Level 3", while the regular expression matching classification outputs "Sensitive Level 2". The difference between Level 2 and Level 3 classifications is within the preset Level 1 difference range.

[0102] If the conflict type is the preset second type (such as a conflict that does not meet the compliance requirements of the business system or a difference in classification results exceeding the preset range), the data classification system triggers a manual review rule. The manual review result (the classification result assessed by classification experts based on the business scenario) is obtained through an interactive interface and used as the final classification result for that field. For example, for the "transaction location" field in cross-border financial transactions, business system A outputs "Sensitive Level 4," while business system B outputs "Sensitive Level 2." In this case, manual judgment is required to adopt the more stringent "Level 3." This solution, which incorporates manual review, ensures priority compliance with the business system's compliance requirements and fills the blind spots of automated decision-making.

[0103] In this implementation, when a conflict occurs, dynamic calculation of weight parameters or manual review is used, which provides fault tolerance and balances security and cost. Specifically, the conflict handling mechanism can prevent non-sensitive fields from being mistakenly identified as highly sensitive, saving a lot of encryption costs; and the use of manual review as a fallback, but not entirely dependent on manual labor, reduces labor costs and improves the accuracy of classification.

[0104] like Figure 3 The figure shown is a comparison of the data grading method of this application (expert thesaurus plus model grading, where model grading includes lightweight model grading and large model grading) with other grading methods. Specifically, in Figure 3 The paper presents a performance comparison of four grading methods: expert thesaurus grading, thesaurus plus human verification, expert thesaurus plus model grading, and grading by expert and human verification. Figure 3 The data includes: a bar chart showing the distribution of data at each level, and a line chart showing the similarity rate between all levels and the manual classification by classification experts, as well as the similarity rate between the "sensitive" and above levels and the manual classification by classification experts. The classification method, which combines expert lexicon with manual verification and adjustment, matches well on other general first-level data, reaching 691. The dark blue line, representing all levels, shows that the classification method of this application has a similarity rate of 97.74% with the manual classification by classification experts on all levels. The orange line, representing the "sensitive" and above levels, has a similarity rate of 85.23% with the manual classification by classification experts. On the sensitive second-level data, the data classification method of this application can identify 43 data points without relying on manual verification (compared to 35 for manual classification), demonstrating its strong semantic understanding ability, avoiding the risk of omissions in lexicon schemes, and eliminating the high cost of manual verification.

[0105] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data classification method of this application. Any simple variations based on this technical concept, such as the interaction and combination of various embodiments, are all within the protection scope of this application.

[0106] This application also provides a data hierarchical system; please refer to [reference needed]. Figure 4 The data grading system includes:

[0107] The system comprises a data management module and a data classification module. The data management module includes a data acquisition unit for obtaining data tables from various business systems, a data processing unit for sharding and partitioning physical sub-tables, a data query unit for saving the target classification results for internal or external querying, and a permission management unit for setting permissions during querying or data acquisition. The data classification module includes a text regular expression classification unit, a lightweight model classification unit, and a large model classification unit.

[0108] The data classification system provided in this application, employing the data classification method described in the above embodiments, can solve the technical problem that existing data classification methods suffer from a single classification approach and are unable to cope with complex real-world business scenarios. Compared with the prior art, the beneficial effects of the data classification system provided in this application are the same as those of the data classification method provided in the above embodiments, and other technical features of the data classification system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0109] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data grading method described in the first embodiment above.

[0110] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0111] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0112] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0113] The electronic device provided in this application, employing the data classification method described in the above embodiments, can solve the technical problem that existing data classification methods suffer from a single classification approach and are unable to cope with complex real-world business scenarios. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the data classification method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0114] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0115] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0116] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data classification method described in the above embodiments.

[0117] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0118] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0119] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: in response to a data acquisition operation, acquire data tables from various business systems to obtain a data table to be classified; display the fields in the data table to be classified; and in response to a configuration operation, configure corresponding classification methods for the fields of the data table to be classified, classify each field based on the classification method, and obtain the target classification result.

[0120] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the module itself.

[0123] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data classification method. This solves the technical problem that existing data classification methods suffer from a single classification approach and cannot cope with complex real-world business scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data classification method provided in the above embodiments, and will not be repeated here.

[0124] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data classification method described above.

[0125] The computer program product provided in this application can solve the technical problem that existing data classification methods have a single classification mode and cannot cope with complex real-world business scenarios. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data classification methods provided in the above embodiments, and will not be repeated here.

[0126] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data classification method, characterized in that, The data classification method includes: In response to the data acquisition operation, the data tables of each business system are retrieved to obtain the data table to be classified. Display the fields in the data table to be classified; In response to the configuration operation, a corresponding classification method is configured for the fields of the data table to be classified, and each field is classified based on the classification method to obtain the target classification result.

2. The data grading method as described in claim 1, characterized in that, The steps of obtaining data tables from various business systems to obtain the data tables to be hierarchically classified include: Extract the table name, field attributes, and storage location information of each physical sub-table of the data table; Determine the semantic similarity of the table names of each physical sub-table, determine the primary and foreign key relationships of the field attributes of each physical sub-table, or determine the lineage of each physical sub-table based on the storage location information; Based on the semantic similarity, the primary-foreign key relationship, and / or the lineage relationship, each physical sub-table is mapped to the same logical table, and the logical table is used as the data table to be classified.

3. The data grading method as described in claim 1, characterized in that, The classification methods include text regular expression classification, lightweight model classification, and large model classification. After the step of obtaining data tables from each business system to obtain the data table to be classified, the method further includes: Perform text regular expression classification on the fields in the data table to be classified to obtain the first classification result; For fields in the data table to be classified that failed to be classified using text regular expression, lightweight model classification is performed to obtain a second classification result, and the confidence score of the second classification result is determined. From the second classification results, target fields with confidence scores lower than a preset score threshold are selected, and large model classification is performed to obtain the third classification results; The first classification result, the second classification result, and / or the third classification result are taken as the target classification result.

4. The data grading method as described in claim 3, characterized in that, The step of performing the large model hierarchical classification includes: The target field is used to identify the business entity type, and the preset hierarchical rule clauses that match the business entity type are recalled. Based on the business system identifier of the data table where the target field is located, determine the historical amendment examples that match the business system identifier from the preset knowledge base; The preset grading rules and the historical amendment examples are combined to form a context, which is then input into the large model to perform the grading.

5. The data grading method as described in claim 1, characterized in that, The method further includes: When configuring the corresponding grading method for the fields of the data table to be graded, one or more grading methods are recommended and displayed for the fields of the data table to be graded based on preset recommendation rules and / or historical configuration data.

6. The data grading method as described in claim 1, characterized in that, The method further includes: Compare the classification results of the same field under different classification methods; When a conflict is detected in the classification results, corresponding processing is performed according to the type of conflict: if the conflict type is the preset first type, the final classification result of the field is determined according to the pre-configured weight parameters; If the conflict type is the preset second type, the manual review rule is triggered, the corresponding manual review result is obtained, and the manual review result is used as the final classification result of the field.

7. A data hierarchical system, characterized in that, The data classification system performs the steps of the data classification method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data grading method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data classification method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data grading method as described in any one of claims 1 to 6.

Images