Sensitive data identification and grading method for enterprise data governance

Abstract

The invention provides a sensitive data identification and grading method oriented to enterprise data governance, and relates to the technical field of data processing. Enterprise multi-type source data including structured, semi-structured and non-structured data is acquired and preprocessed, and based on the preprocessed data, the sensitive data is identified and graded; a plurality of algorithm models selected for different data types are utilized to carry out distributed training to obtain a plurality of candidate recognition models, a multi-dimensional evaluation system covering classification performance, efficiency and resource consumption, model robustness and service applicability is adopted to carry out comprehensive evaluation on the candidate models, a target recognition model is preferably selected according to an evaluation result, and the target recognition model is identified. And the method is applied to sensitive information identification and classification of enterprise data, the target identification model can balance multi-aspect constraints through evaluation and screening, and the method can be specifically and really suitable for a solution of a complex enterprise data management scene, so that the accuracy and decision scientificity of enterprise data security management and control are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for identifying and classifying sensitive data for enterprise data governance. Background Technology

[0002] In the field of enterprise data governance, accurate identification and classification of sensitive data is a core prerequisite for ensuring data security and meeting compliance requirements. Existing technologies typically employ rule-based matching methods or train dedicated recognition models for specific business scenarios (such as processing only database tables or documents in specific formats). As enterprise data becomes increasingly complex, encompassing structured data in databases, semi-structured data in logs and interfaces, as well as unstructured data such as documents and images, traditional methods face several challenges. First, rule-based methods struggle to adapt to varying data content and formats, resulting in high maintenance costs and limited coverage. Second, models for single data types or scenarios have poor generalization capabilities, failing to handle heterogeneous data assets within an enterprise under a unified framework. In recent years, although advanced artificial intelligence technologies, such as large language models, have demonstrated powerful representation capabilities, their direct application to the identification of sensitive information in massive, multimodal enterprise data generally suffers from problems such as enormous computational consumption, high processing latency, and a lack of systematic guidance for model optimization and selection. Meanwhile, existing solutions often focus on basic metrics such as the accuracy of the algorithm itself, lacking a comprehensive evaluation system for model processing efficiency, resource consumption, robustness, and applicability in real business scenarios. This leads to uncertainty in the effectiveness of model implementation and makes it difficult to closely integrate with enterprises' actual data governance processes and security control needs. Existing technologies have the following technical problems when used: Problem 1: Existing sensitive data identification methods mostly rely on rules or isolated models designed for specific data types or business scenarios. They lack a unified and efficient comprehensive governance framework that can handle multiple types of data, such as structured, semi-structured, and unstructured data within an enterprise. This results in poor adaptability, complex maintenance, and difficulty in expansion. Problem 2: Existing methods based on adaptive learning or large models suffer from high computational resource consumption and low processing timeliness when dealing with massive, multimodal enterprise data. At the same time, the evaluation of model performance is often limited to basic classification indicators, lacking a systematic, multi-dimensional comprehensive evaluation and optimization mechanism that covers algorithm performance, resource efficiency, robustness, and business applicability. As a result, the reliability, economy, and interpretability of the model in actual business deployment cannot be guaranteed. Summary of the Invention

[0003] To achieve the above objectives, the present invention provides the following technical solution: a method for identifying and classifying sensitive data for enterprise data governance, the method comprising: Step S100: Obtain enterprise multi-type source data, preprocess the enterprise multi-type source data, and generate standardized training data. The enterprise multi-type source data includes structured data, semi-structured data, and unstructured data. Step S200: Based on the standardized training data, different algorithm models are selected for distributed training for different types of data. N algorithm models are selected from the preset algorithm model library for training to generate N candidate recognition models. Step S300: Perform multi-dimensional evaluation on N candidate recognition models to generate evaluation results. The multi-dimensional evaluation includes classification performance evaluation dimension, efficiency and resource consumption evaluation dimension, and model robustness evaluation dimension. Step S400: Based on the evaluation results, determine the target identification model from N candidate identification models, and based on the target identification model, identify and classify sensitive data of the enterprise data to be processed, and generate identification and classification results.

[0004] Furthermore, the acquisition of multi-type source data from enterprises, and the preprocessing of this multi-type source data to generate standardized training data, includes: Based on industry data classification and grading standard documents, a labeling system for machine learning models to recognize is defined and constructed, which includes classification labels and grading labels; The system acquires multi-type source data from enterprises and labels the multi-type source data based on a tagging system. The classification tags include sensitive data tags and non-sensitive data tags, and the hierarchical tags include public tags, internal tags, and confidential tags. The labeled enterprise source data of various types is preprocessed to generate standardized training data.

[0005] Furthermore, the standardized training data includes text data, image data, and audio data. For text data, select and load a transformer-based bidirectional encoder representation or a long short-term memory network model as the base model; For image data, a convolutional neural network model is selected and loaded as the base model; For audio data, select and load a recurrent neural network model or a spectrogram-based convolutional neural network model as the base model.

[0006] Furthermore, the standardized training data is trained in parallel using a distributed training framework, which enables mixed precision training and gradient configuration during the training process, and configures training hyperparameters for each training process.

[0007] Furthermore, the metrics used for the classification performance evaluation dimension include accuracy, precision, recall, F1 score, and area under the AUC-ROC curve; the metrics used for the efficiency and resource consumption evaluation dimension include training time, prediction time, memory usage, and GPU utilization; and the model robustness evaluation dimension is obtained through K-fold cross-validation or bias-variance analysis.

[0008] Furthermore, the multi-dimensional assessment also includes a business usability assessment dimension, which is obtained through model interpretability analysis, business indicator comparison, and A / B testing.

[0009] Furthermore, the text data is divided into structured text data and unstructured text data. For training on the unstructured text data, adjustments are made based on pre-trained models of LLaMA-2 or DeepseeKR1.

[0010] Furthermore, based on the evaluation results, a target identification model is determined from N candidate identification models. Based on the target identification model, sensitive data is identified and classified in the enterprise data to be processed, generating identification and classification results, including: Obtain the evaluation results for each evaluation dimension, calculate the multi-dimensional score for each candidate recognition model based on a predefined weighted scoring strategy, and select the candidate recognition model with the highest multi-dimensional score as the target recognition model; Based on the target recognition model, the acquired enterprise data is scanned and its features are extracted to generate recognition and classification results.

[0011] Furthermore, the identification and classification results include data identifier, sensitivity type, sensitivity level, and prediction confidence. The identification and classification results are stored in a preset database and indexed.

[0012] This invention provides a method for identifying and classifying sensitive data for enterprise data governance. It has the following beneficial effects: 1. This invention solves the problem of unified governance of multi-type data by constructing a unified processing framework covering structured, semi-structured, and unstructured data, and adapting the optimal basic model for different data types. After preprocessing and standardizing data from different sources and formats, different algorithm models are selected. For different modalities such as text, images, and audio, validated and most targeted basic models such as BERT / LSTM, CNN, and RNN are selected and loaded for training, ensuring that each data type can obtain high-precision feature extraction and recognition capabilities. The model training is integrated into a unified distributed training framework. By using multi-GPU parallelism, mixed precision, gradient accumulation, and DeepSpeed ​​optimization techniques, efficient and collaborative training of massive multi-modal data is achieved. This eliminates the need for enterprises to maintain multiple independent recognition systems for different types of data. Sensitive information scanning and classification of all data can be completed within a single technical system, greatly improving the coverage, processing efficiency, and system maintainability of data governance work.

[0013] 2. This invention establishes a multi-dimensional performance evaluation system and optimizes models based on the evaluation results. The training models are evaluated and comprehensively decided from the dimensions of classification performance evaluation, efficiency and resource consumption evaluation, model robustness evaluation, and business practicality evaluation. The target recognition model with the best performance in terms of performance, efficiency, stability, and business value is selected for application. This ensures that the final deployed model has targeted and effective computing capabilities at both the theoretical and practical levels. It can balance multiple constraints and is a solution that is truly suitable for complex enterprise data governance scenarios. This improves the accuracy and scientific nature of enterprise data security management and fundamentally enhances the reliability, economy, and business fit of the target recognition model in the actual enterprise environment. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of a sensitive data identification and classification method for enterprise data governance according to the present invention. Figure 2 This is a data flow diagram of a sensitive data identification and classification method for enterprise data governance according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] like Figures 1 to 2As shown, a method for sensitive data identification and classification for enterprise data governance includes: Step S100: Acquire multi-type source data from the enterprise, preprocess the multi-type source data to generate standardized training data. The multi-type source data includes structured data, semi-structured data, and unstructured data. Specifically, internal business data containing customer information and transaction records is obtained from the enterprise's internal data storage system; external public datasets are obtained from publicly available corpora and image libraries; and synthetic data that meets specific testing requirements is constructed using data generation tools. These together constitute the multi-type source data. Structured data consists of tabular data from databases, such as financial statements, sales records, and customer information tables; semi-structured data is in JSON and XML formats, used for scenarios such as logs and API interfaces; unstructured data includes text files, images, PDF files, etc., covering different types of information such as documents and images. Step S101: Based on at least one industry data classification and grading standard document, define and construct a tag system for machine learning model recognition. The tag system includes classification tags and grading tags. The industry data classification and grading standard document includes a tag system constructed based on the "YDT3813-2020 Basic Telecommunications Enterprise Data Classification and Grading Method" and in conjunction with the "China Mobile Data Classification and Grading and Important Data Management Guidance" and the "China Unicom Data Classification and Grading Security Management Specification". Step S102: Obtain enterprise multi-type source data, and label the enterprise multi-type source data based on the tag system. The classification tags include sensitive data tags and non-sensitive data tags, and the hierarchical tags include public tags, internal tags and confidential tags. First, classification labels are determined based on the content and nature of multi-type source data from enterprises. The determination criteria are as follows: If the data content contains personally identifiable information, trade secrets, financially sensitive information, undisclosed major business decision information, or other specific sensitive categories as defined in industry standards, it will be labeled as sensitive data; otherwise, it will be labeled as non-sensitive data. Secondly, based on the identification of sensitive data labels, the classification label is determined according to the scope and degree of harm that may be caused by its leakage or misuse. The determination process is as follows: If a data breach causes only minor inconvenience to the company's internal operations or has very low information value, it will be marked as public. If a data breach may have some impact on the company's internal management and business collaboration, but will not directly cause significant economic losses or reputational damage, it will be labeled as internal. Data breaches that could lead to significant economic losses, loss of core competitiveness, serious violations of regulations, or significant reputational risks for a company will be labeled as confidential.

[0017] In actual annotation, for structured data, such as database tables, the judgment is made column by column or in combination based on the meaning and value range of the fields and business rules; for semi-structured data, such as JSON, its key-value pair structure is parsed, and the judgment is made based on the key field names and contents; for unstructured data, such as documents, the text content or image features need to be extracted first, and then the judgment is made according to the above content standards. This process is usually completed by combining automated rule scripts with manual review to ensure the accuracy of the annotation and its consistency with the business reality.

[0018] Step S103: Preprocess the labeled enterprise multi-source data. The enterprise multi-source data is divided into text data, image data, and audio data according to the data content. Preprocessing includes data cleaning and format conversion. For text data, noise removal, word segmentation, and word vector conversion are performed. For image data, size standardization, noise reduction, and color channel adjustment are performed. For audio data, noise reduction, format conversion, and feature extraction are performed. The processed text data is converted into a numerical sequence acceptable to the algorithm model using a word segmenter, and a custom dataset class adapted to the training program interface is created. A data loader is built by setting the batch size to generate standardized training data.

[0019] Step S200: Based on standardized training data, different algorithm models are selected for distributed training for different types of data. N algorithm models are selected from the preset algorithm model library for training, generating N candidate recognition models, where N is an integer. The pre-built algorithm model library is a collection of pre-built machine learning model components optimized for different data modalities and tasks. It is set according to the data types and their respective data characteristics and processing paradigms specified in steps S100 and S201. The pre-built algorithm model library mainly includes model components for processing serialized text data, such as transformer-based bidirectional encoder representation (BERT) and long short-term memory network (LSTM), especially pre-trained large language model foundations for unstructured text; model components for processing grid data with spatial local correlation, such as convolutional neural network (CNN), suitable for image data; and model components for processing time series or spectral data, such as recurrent neural network (RNN) and spectrogram-based CNN, suitable for audio data. When the preset algorithm model library is started in step S200, it provides a set of verified basic models that can be directly called, avoiding the need to design models from scratch for each data type. Based on the modal characteristics of the input standardized training data, it automatically or according to the configuration matches and loads one or more corresponding basic model architectures from the preset algorithm model library as the training starting point, thereby ensuring that the technical solution can quickly and effectively adapt to multiple types of data, and providing diverse and targeted model candidate starting points for subsequent parallel training and multi-dimensional evaluation.

[0020] Step S201: The data types of standardized training data include text data, image data, and audio data; For text data, a transformer-based bidirectional encoder representation (BERT) or long short-term memory network (LSTM) model is selected and loaded as the base model; for training on unstructured text data, the pre-trained model based on LLaMA-2 or DeepseeKR1 is tuned. For image data, a convolutional neural network (CNN) model is selected and loaded as the base model; For audio data, select and load a recurrent neural network (RNN) model or a spectrogram-based convolutional neural network (CNN) model as the base model.

[0021] Among them, Convolutional Neural Network (CNN) models are deep learning models used to process data with grid-like structures (such as images and videos). Their core is the convolutional layer, which effectively extracts spatial features. Long Short-Term Memory (LSTM) models, by introducing sophisticated input, forget, and output gates, effectively solve the gradient vanishing or exploding problems that easily occur in original RNNs when processing long sequences. They can learn and remember long-distance dependencies, making it easier to understand complex text data. Recurrent Neural Network (RNN) models are neural networks used to process sequential data (such as text, time series, and audio). The nodes in their network structure have recurrent connections, allowing information to be persisted and enabling them to capture temporal dependencies in sequences. Transformer-based Bidirectional Encoder Representation (BERT) is a pre-trained language model based on the Transformer architecture. It learns lexical representations by simultaneously considering words on both sides of the context, and performs exceptionally well in natural language processing tasks. Step S202: For the base model selected in step S201, the standardized training data is trained in parallel using the PyTorch distributed training framework. The training task is distributed to multiple GPUs via a multi-process startup method. First, a distributed data parallelism strategy is used to wrap the selected base model, and a distributed sampler is used to ensure that each process obtains a different subset of data, achieving data parallelism. Second, efficient training techniques are applied in the training loop. Mixed-precision training is enabled, and half-precision floating-point numbers are used for computation to accelerate the training process. A gradient accumulation step is configured so that, under the limited memory of the hardware, multiple forward propagations accumulate gradients to equivalently simulate the training effect of large-scale data. Transfer learning fine-tuning is performed based on the weights of the training model. Then, to further optimize resource utilization, the DeepSpeed ​​optimizer is integrated, and its zero-redundancy optimizer second-stage configuration is used to reduce GPU memory usage. Gradient checkpointing is also enabled, selectively retaining some intermediate activation values ​​during forward propagation, while recalculating the rest when needed during backpropagation. This significantly improves GPU memory utilization efficiency at the cost of a small increase in computation time, supporting the training of larger models. Step S203 involves monitoring the training process of each basic model, configuring training hyperparameters for each training process, including setting the training batch size and evaluation batch size for each device, setting the initial learning rate and cosine learning rate scheduler, and setting the training warm-up ratio; starting the training process and monitoring the changes in training loss and validation metrics; and generating N candidate recognition models after training is completed. By controlling the behavior and convergence characteristics of the base model during training through hyperparameter training, the final performance, training speed, and stability of the model are directly affected. First, hyperparameter initialization is performed, setting a set of initial values ​​based on common configuration experience and hardware conditions of the selected base model. For example, the training batch size and evaluation batch size for each device are determined based on the GPU memory capacity, such as setting them to 64 and 128 respectively. The initial learning rate is set to a small value, such as 2e-5, based on the complexity of the base model and the task. The training warm-up ratio is set to a percentage of the total training steps (e.g., 0.1), allowing the learning rate to gradually increase from zero to the initial value in the early stages of training, stabilizing the model's initial training. Second, dynamic scheduling is integrated. The strategy involves using a cosine learning rate scheduler in conjunction with the initial learning rate. This scheduler causes the learning rate to decay from its initial value to near zero according to the cosine function curve after the warm-up period, which helps the model converge more precisely in the later stages of training. Finally, during the training execution and monitoring phase, after the training process starts, the changes in training loss and validation metrics (such as validation set accuracy) are continuously monitored. If the training loss does not decrease or the metrics prematurely converge, the current training may be interrupted, and hyperparameters may be adjusted based on monitoring feedback. For example, the learning rate may be increased or decreased appropriately, the batch size may be adjusted, and then training may be restarted or continued until the model performs stably or optimally on the validation set, thereby completing the generation of the candidate recognition model.

[0022] Step S300: Perform multi-dimensional evaluation on N candidate recognition models to generate evaluation results. The multi-dimensional evaluation includes classification performance evaluation dimension, efficiency and resource consumption evaluation dimension, and model robustness evaluation dimension. The performance metrics used in the classification evaluation include accuracy, precision, recall, F1 score, and the area under the AUC-ROC curve. Accuracy represents the proportion of correctly predicted samples out of the total number of samples; precision represents the proportion of samples predicted as positive by the candidate model that are actually positive; recall represents the proportion of samples that are actually positive that were correctly predicted by the candidate model; the F1 score is the harmonic mean of precision and recall; and the AUC-ROC value is the area under the receiver operating characteristic curve, used to evaluate the candidate model. The overall classification ability under different classification thresholds; by running the candidate recognition model, collecting the prediction results and real labels of the candidate recognition model, and then calculating the accuracy (number of correct predictions / total number), precision (true positives / (true positives + false positives)), recall (true positives / (true positives + false negatives)), F1 score (2 * precision * recall / (precision + recall)), and plotting the ROC curve and calculating its area under the curve (AUC-ROC). The calculation results of these five indicators together constitute a quantitative evaluation report of the classification performance evaluation dimensions. The efficiency and resource consumption evaluation dimensions use indicators including training time, prediction time, memory usage, and GPU utilization. Training time is the total time required for each candidate recognition model to complete training; prediction time is the average time required for each candidate recognition model to make a prediction on a single or batch of data; memory usage is the peak memory usage during training and prediction; and GPU utilization is the average utilization of the graphics processor during training and prediction. During the candidate recognition model training and prediction process, monitoring tools record and summarize training time, prediction time, memory usage, and GPU utilization. These four indicators constitute the performance profile of the efficiency and resource consumption evaluation dimensions. The robustness evaluation dimension of the model is obtained through K-fold cross-validation or bias-variance analysis. The standardized training data is randomly divided into K mutually exclusive subsets, where K is an integer. One subset is used as the validation set and the remaining K-1 subsets are used as the training set. The training and evaluation are repeated K times. The key performance indicators (such as F1 score) on the validation set are recorded in each round. Finally, the mean and standard deviation of these K indicator values ​​are calculated. The mean measures the average performance and the standard deviation measures the performance fluctuation. The combination of the two is used as the robustness evaluation conclusion. The multi-dimensional assessment also includes a business usability assessment dimension. This dimension is obtained through model interpretability analysis, business metric comparison, and A / B testing. Model interpretability analysis uses attention mechanism visualization, feature importance ranking, or locally interpretable model methods to understand the model's decision-making basis. Business metric comparison involves correlating and comparing the recognition results of candidate recognition models on historical business data with key performance indicators defined by the business department. A / B testing involves deploying different candidate recognition models in an online production environment and evaluating the impact of different models on the actual business experience by collecting user feedback or business flow log data. After obtaining the specific indicator results for all evaluation dimensions, the final evaluation results are generated, including: First, data standardization is necessary. Since the dimensions and ranges of various indicators are different, such as accuracy being between 0 and 1 and training time being in seconds, all quantitative indicators need to be normalized and converted into comparable values ​​in the range of [0,1]. For example, for cost-related indicators (such as training time), a conversion function that is as small as possible should be used. Secondly, multi-dimensional scoring aggregation is performed, and weights are assigned to each evaluation dimension, or even key indicators within a dimension, based on predefined business preferences or decision-making rules. For example, in scenarios that emphasize accuracy, the classification performance dimension has the highest weight; in resource-constrained scenarios, the efficiency and resource consumption dimensions have increased weights. Then, the standardized index values ​​are weighted and summed to calculate a comprehensive score for each candidate recognition model. Finally, the results are generated and sorted, and an evaluation report is output, which includes the original index, standardized value, score of each dimension and final comprehensive score of each candidate recognition model in all dimensions. All candidate models are sorted in descending order according to the comprehensive score. This sorted list and the detailed data behind it are the evaluation results on which the model selection in step S400 is based. The comprehensive score is the multi-dimensional score of each candidate recognition model. In step S300, each indicator for each evaluation dimension can be calculated using existing publicly available calculation methods. The specific formulas and calculation details are not elaborated here.

[0023] Step S400: Based on the evaluation results, determine the target identification model from N candidate identification models, and based on the target identification model, identify and classify sensitive data of the enterprise data to be processed, and generate identification and classification results; Step S401: Obtain the evaluation results for each evaluation dimension, calculate the multi-dimensional score for each candidate recognition model based on a predefined weighted scoring strategy, and select the candidate recognition model with the highest multi-dimensional score as the target recognition model. Step S402: Based on the target recognition model, scan and extract features from the acquired enterprise data to be processed, and generate recognition and classification results. The recognition and classification results include data identifier, sensitivity type, sensitivity level, and prediction confidence; including: First, data access and preprocessing: The enterprise data to be processed is routed to the preprocessing module standardized in step S103 according to its data type (text, image, audio), and performs the same cleaning, format conversion and feature extraction operations to generate standardized input features consistent with the format of "standardized training data". Secondly, model inference is performed by inputting standardized input features in batches or one by one into the loaded target recognition model. The target recognition model calculates the standardized input features based on the patterns it has learned internally. Then, the results are decoded. The target recognition model outputs the original prediction vector or probability distribution. The output is decoded as follows: for classification labels, the category with the highest probability is taken as the sensitive type label; for grading labels, the grade with the highest probability is taken as the sensitive level label, and the highest probability value (or the calibrated confidence score) is taken as the prediction confidence. Finally, the results are encapsulated, a globally unique data identifier is generated for the currently processed data unit, and the above decoding results are bound to this identifier to form a structured recognition and classification result.

[0024] Among them, the enterprise data to be processed refers to new data assets generated or received in the daily operation of the enterprise that require sensitive information review. Its scope, type and source are consistent with the data source used for training in step S100, including structured data, such as real-time transaction logs and customer attribute tables from business databases (Oracle, MySQL); semi-structured data, such as JSON format application logs from log servers (ELKStack) and XML / JSON interface messages from API gateways; and unstructured data, such as contract documents (PDF / WORD), product design drawings (JPG / PNG), and meeting recording files (MP3 / WAV) from file servers or cloud storage. These data come from various business endpoints of the enterprise IT system. The data identifier is a string that uniquely identifies a single identification task or a single data object. It is usually composed of elements such as "data source identifier_timestamp_serial number" and is used to establish a unique association key between data, identification results and all subsequent operations (such as query, audit, and review). When a data unit is processed in step S402, the identifier generation service is called by the system to generate it in real time. Sensitive types refer to the specific sensitive categories of data that have been identified, such as personal identification information, trade secrets, financial information, etc., or more basic binary classification of sensitive and non-sensitive data. Sensitive types are a refined way to identify the nature of risk. They are derived from the output of the target identification model on the classification task and are obtained by mapping the category label with the highest probability from the probability distribution of the model output. Sensitivity level refers to the classification of data according to its sensitivity, such as public, internal, confidential, etc. It provides a direct basis for subsequent formulation of differentiated data security control strategies, such as access permissions, encryption strength, and de-identification rules. It originates from the output of the target recognition model on the classification task and is obtained by taking the level label with the highest probability in the level probability distribution of the model output.

[0025] Prediction confidence is a value between 0 and 1, representing the model's certainty regarding its judgment of the sensitivity type and sensitivity level. It provides a priority reference for manual review, with low-confidence results receiving priority review, and serves as input for risk threshold judgment in automated decision-making processes. It is typically obtained by directly taking the output probability value corresponding to the model's prediction of the final sensitivity type and sensitivity level, or by post-processing and calibrating that probability value through temperature scaling or other means.

[0026] Step S403: The identification and classification results are stored in a preset database and an index is created. The preset database refers to a relational or non-relational database system (such as MySQL, PostgreSQL, MongoDB) specifically designed for the persistent storage and management of sensitive data identification and classification results. It includes a data table (or collection) structure, which defines the fields for storing the identification results. Typically, it includes at least the data identifier (primary key), original data source / path, sensitivity type, sensitivity level, prediction confidence, model version number, and processing timestamp. In addition, the preset database also includes indexes created to improve query efficiency, such as indexes on frequently queried fields like "data identifier," "sensitivity level," and "processing timestamp." Its main functions are threefold: first, persistent storage to ensure that the identification results are not lost and can be traced and audited historically; second, efficient querying, enabling business systems or security administrators to quickly retrieve results based on various conditions (such as "find all data classified as 'confidential'"); and third, support for analysis and management, serving as the core data source for downstream data security posture analysis, compliance report generation, and data lifecycle management strategy execution (such as automatically triggering encryption for "confidential" data).

[0027] Step S404: Encapsulate the target recognition model into an application programming interface (API) service with a standard input / output format. This API service supports integration with existing enterprise data governance platforms or data security management systems and provides two calling modes: The first type is batch processing mode, which accepts a data file or a data list and returns the identification and classification results of all data. Specifically, it receives a call request from an upstream system, the request body of which contains a data file (such as CSV, Parquet) or a specific data list. It parses the request, splits the large batch of data into small batches suitable for single inference by the model, and performs the operation in step S402 on each small batch of data in sequence: performs standardization preprocessing, then inputs it into the target identification model for prediction, and finally collects the prediction results of all small batches. It integrates the identification and classification results of all data, including data identifier, sensitivity type, sensitivity level, and confidence level, into a structured list (such as a JSON array), and returns it to the caller in one go through the API response. The second type is the real-time processing mode, which accepts a single data request and immediately returns the identification and classification results for that data. Specifically, when the service receives a call request, the request body usually contains only a single piece of data to be identified (such as a piece of text or a Base64 encoded image). The service immediately performs rapid standardization preprocessing on the data, inputs the processed features into the target recognition model already loaded in memory, performs a single forward propagation calculation, and after the target recognition model outputs the identification and classification results, the service immediately decodes and generates the sensitivity type, sensitivity level, and prediction confidence of the data, and generates a temporary data identifier. Then, the complete identification and classification result is returned to the caller in real time via API response, usually within milliseconds to seconds.

[0028] The main purpose of encapsulating the target recognition model as a standardized application programming interface (API) service is to realize the service-oriented and decoupled capabilities of the model. This hides the complex internal computational logic of the model behind a unified calling interface, allowing existing upstream business applications such as enterprise data governance platforms and data security management systems to obtain professional sensitive data recognition capabilities through simple network requests without having to worry about the specific framework, deployment environment or technical details of the model. This greatly improves the system's integrability, maintainability and scalability.

[0029] In this embodiment, for practical application, the following is an example: Applications in the governance of structured and semi-structured customer data in the telecommunications industry; First, a total of 9,062,042 data entries were collected from business databases and API logs, including basic customer information, package subscription records, and service interaction logs. A fine-grained tagging system was built around "YDT3813-2020" as the core, and the data was accurately labeled. Classification tags include sensitive data tags (corresponding to personal identity information, communication content, etc.) and non-sensitive data tags, and hierarchical tags include internal tags and confidential tags. Subsequently, the database table fields were standardized and cleaned, the JSON format logs were parsed and key information was extracted, and all text data was converted into a unified word vector representation to generate standardized training data. For this type of text-based sequence data, BERT, Random Forest and SVM were selected as candidate recognition models from the preset algorithm model library. The PyTorch distributed data parallel framework was used to enable mixed precision training and DeepSpeed ​​optimization on 8 GPUs for efficient training, and hyperparameters such as batch size 64 and learning rate 2e-5 were configured. After training, a multi-dimensional performance evaluation step was performed. On the independent test set, not only were basic classification performance metrics such as accuracy, precision, recall, and F1 score calculated, but the average prediction latency (efficiency dimension) of each candidate recognition model was also evaluated, and 5-fold cross-validation (robustness dimension) was conducted. The evaluation results showed that the BERT model had the best overall performance (F1 score of 90.0% and good robustness). Therefore, based on the weighted scoring strategy, the BERT model was selected as the target recognition model. Finally, the target recognition model was encapsulated as a RESTful API service (S404) and integrated into the data governance platform of the telecommunications enterprise. This enabled real-time scanning and automated classification of sensitive information in new customer data. The recognition and classification results (including data ID, type, level and confidence level) were stored in a structured manner, directly driving the automated execution of data access control policies.

[0030] Applications for unified governance of unstructured document data within enterprises; First, 2,098 real business documents, including contracts, reports, and 70,490 related corpora generated through enhancement, were collected from the file server, forming a variety of unstructured source data. After constructing a tagging system based on unified industry standards, PDF, Word and other format documents were uniformly parsed and text extracted. Image areas containing sensitive information were identified by OCR and converted into text. Subsequently, all text data were denoised, segmented, and vectorized to generate standardized training data. To address the characteristics of unstructured text, LSTM, BERT, and RoBERTa were selected as candidate recognition models from a pre-defined library, and parallel training was performed using distributed training and optimization techniques. After training, a multi-dimensional performance evaluation was conducted. In addition to using basic indicators such as cross-validation to calculate the F1 score, a business applicability evaluation was added. The decision-making basis of the model for key fields such as "commercial terms" and "personal signature" was analyzed through interpretability tools, and the model was correlated and compared with the key performance indicators for compliance review defined by the legal department. The comprehensive evaluation results show that the RoBERTa model achieves the best balance between classification accuracy and business interpretability. Therefore, the RoBERTa model is the preferred target recognition model. In the application phase, the target recognition model is deployed to the backend service of the enterprise document management system. When a user uploads a new document, the scanning and feature extraction process is automatically triggered, and the structured recognition and classification results, including "Business secret level: confidential, confidence level 0.96", are output. The results are stored in a preset database and synchronized to the access control system, thereby automatically imposing a strict access control list on the document according to the "confidential" level, realizing closed-loop governance of sensitive data from identification, classification to security control.

[0031] In this embodiment, a unified processing framework covering structured, semi-structured, and unstructured data is constructed, and the optimal basic model is adapted for different data types. This solves the problem of unified governance of multiple data types. After preprocessing and standardizing data from different sources and formats, different algorithm models are selected. For different modalities such as text, images, and audio, validated and most targeted basic models such as BERT / LSTM, CNN, and RNN are selected and loaded for training. This ensures that each data type can obtain high-precision feature extraction and recognition capabilities. The model training is integrated into a unified distributed training framework. By using technologies such as multi-GPU parallelism, mixed precision, gradient accumulation, and DeepSpeed ​​optimization, efficient and collaborative training of massive multimodal data is achieved. This eliminates the need for enterprises to maintain multiple independent recognition systems for different types of data. Sensitive information scanning and classification of all data can be completed within a single technical system, greatly improving the coverage, processing efficiency, and system maintainability of data governance work.

[0032] This invention establishes a multi-dimensional performance evaluation system and optimizes models based on the evaluation results. The trained models are evaluated and comprehensively decided upon from the dimensions of classification performance, efficiency and resource consumption, model robustness, and business usability. The target recognition model that performs best in performance, efficiency, stability, and business value is selected for application. This ensures that the final deployed model has targeted and effective computational capabilities at both the theoretical and practical levels, balancing multiple constraints and providing a solution truly suitable for complex enterprise data governance scenarios. This improves the accuracy and scientific nature of enterprise data security management and fundamentally enhances the reliability, economy, and business fit of the target recognition model in real-world enterprise environments.

[0033] This application also provides an electronic device. The electronic device may include one or more processors and one or more memories. The memories store computer-readable code that, when executed by the one or more processors, can perform a sensitive data identification and classification method for enterprise data governance as described above.

[0034] The method described in this application can also be implemented using the architecture of the electronic device shown in this application. The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, input / output, a hard disk, etc. The storage device in the electronic device, such as ROM or hard disk, may store a sensitive data identification and classification method for enterprise data governance provided in this application. Furthermore, the electronic device may also include a user interface. Of course, the architecture shown in this application is merely exemplary; when implementing different devices, one or more components in the electronic device shown in this application may be omitted according to actual needs.

[0035] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0036] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for identifying and classifying sensitive data for enterprise data governance, characterized in that, The method includes: Step S100: Obtain enterprise multi-type source data, preprocess the enterprise multi-type source data, and generate standardized training data. The enterprise multi-type source data includes structured data, semi-structured data, and unstructured data. Step S200: Based on the standardized training data, different algorithm models are selected for distributed training for different types of data. N algorithm models are selected from the preset algorithm model library for training to generate N candidate recognition models. Step S300: Perform multi-dimensional evaluation on N candidate recognition models to generate evaluation results. The multi-dimensional evaluation includes classification performance evaluation dimension, efficiency and resource consumption evaluation dimension, and model robustness evaluation dimension. Step S400: Based on the evaluation results, determine the target identification model from N candidate identification models, and based on the target identification model, identify and classify sensitive data of the enterprise data to be processed, and generate identification and classification results.

2. The sensitive data identification and classification method for enterprise data governance according to claim 1, characterized in that, The process of acquiring multi-type source data from enterprises, preprocessing the multi-type source data, and generating standardized training data includes: Based on industry data classification and grading standard documents, a labeling system for machine learning models to recognize is defined and constructed, which includes classification labels and grading labels; The system acquires multi-type source data from enterprises and labels the multi-type source data based on a tagging system. The classification tags include sensitive data tags and non-sensitive data tags, and the hierarchical tags include public tags, internal tags, and confidential tags. The labeled enterprise source data of various types is preprocessed to generate standardized training data.

3. The sensitive data identification and classification method for enterprise data governance according to claim 1, characterized in that, The standardized training data includes text data, image data, and audio data. For text data, select and load a transformer-based bidirectional encoder representation or a long short-term memory network model as the base model; For image data, a convolutional neural network model is selected and loaded as the base model; For audio data, select and load a recurrent neural network model or a spectrogram-based convolutional neural network model as the base model.

4. A sensitive data identification and classification method for enterprise data governance according to claim 3, characterized in that, The standardized training data is trained in parallel using a distributed training framework. Mixed precision training and gradient configuration are enabled during the training process, and training hyperparameters are configured for each training process.

5. A method for identifying and classifying sensitive data for enterprise data governance according to claim 1, characterized in that, The metrics used for classification performance evaluation include accuracy, precision, recall, F1 score, and area under the AUC-ROC curve; the metrics used for efficiency and resource consumption evaluation include training time, prediction time, memory usage, and GPU utilization; and the metrics used for model robustness evaluation are obtained through K-fold cross-validation or bias-variance analysis.

6. A method for sensitive data identification and classification for enterprise data governance according to claim 1, characterized in that, The multi-dimensional assessment also includes a business usability assessment dimension, which is obtained through model interpretability analysis, business indicator comparison, and A / B testing.

7. A sensitive data identification and classification method for enterprise data governance according to claim 3, characterized in that, The text data is divided into structured text data and unstructured text data. For training on the unstructured text data, adjustments are made based on pre-trained models of LLaMA-2 or DeepseeKR1.

8. A sensitive data identification and classification method for enterprise data governance according to claim 7, characterized in that, The process involves determining a target identification model from N candidate identification models based on the evaluation results, identifying and classifying sensitive data in the enterprise data to be processed based on the target identification model, and generating identification and classification results, including: Obtain the evaluation results for each evaluation dimension, calculate the multi-dimensional score for each candidate recognition model based on a predefined weighted scoring strategy, and select the candidate recognition model with the highest multi-dimensional score as the target recognition model; Based on the target recognition model, the acquired enterprise data is scanned and its features are extracted to generate recognition and classification results.

9. A sensitive data identification and classification method for enterprise data governance according to claim 8, characterized in that, The identification and classification results include data identifier, sensitivity type, sensitivity level, and prediction confidence. The identification and classification results are stored in a preset database and indexed.

Images