Method and system for automatic discovery of sensitive data

Abstract

This disclosure presents a method and system for automatically discovering sensitive data, comprising: monitoring a task queue to obtain sensitive data discovery tasks to be processed; parsing the sensitive data discovery tasks, dynamically adapting connection strategies using a database fingerprinting mechanism, and connecting to a target database; extracting configuration information, determining the target database table name and target field name, generating and executing an intelligent sampling strategy and data query SQL statement, scanning and obtaining data samples, performing sensitive information pattern matching and identification, and generating discovery results and a data evolution tracking system; writing the discovery results into a result table and generating a data lineage graph; updating the execution progress of the sensitive data discovery tasks, closing the database connection after execution, and updating the status of the sensitive data discovery tasks to complete. This method can improve the efficiency, accuracy, and traceability of sensitive data discovery.

Description

Technical Field

[0001] This invention relates to the fields of data security and database auditing technology, and in particular to a method and system for automatically discovering sensitive data, used to automatically identify and locate sensitive data in a large-scale database environment. Background Technology

[0002] As enterprises deepen their digital transformation, the amount of sensitive information stored in databases is increasing, including personal privacy data, trade secrets, and financial information. Traditional methods for discovering sensitive data mainly rely on manual configuration and periodic scanning, which suffers from inefficiency and incomplete coverage.

[0003] Existing sensitive data discovery technologies typically employ scheduled tasks or manual triggering for scanning. A typical implementation involves performing a full scan of database tables using predefined rules and regular expressions to identify fields and records that may contain sensitive information. Another common approach is static scanning based on data classification labels, which performs targeted scanning using pre-labeled sensitive fields.

[0004] The most relevant existing technology is a rule-based sensitive data identification system. Its working principle involves pre-defining the characteristic patterns and identification rules of sensitive data, and then performing batch scanning of the target database. The system obtains data samples through SQL queries, applies predefined rules for pattern matching, and ultimately generates a sensitive data distribution report.

[0005] However, existing technologies suffer from the following key problems: first, they lack a flexible task scheduling mechanism, making it difficult to support large-scale, multi-data source concurrent scanning needs; second, the scanning process lacks real-time progress monitoring, affecting system reliability and maintainability. Therefore, there is an urgent need for a sensitive data discovery method that can automate and concurrently process sensitive data discovery tasks, flexibly adapt to different database structures and scanning requirements, and track progress in real time. Summary of the Invention

[0006] In view of this, the present disclosure provides a method and system for automatic discovery of sensitive data, aiming to solve the technical problems of insufficient flexibility in sensitive data discovery and weak concurrent identification performance in the prior art.

[0007] In a first aspect, embodiments of this disclosure provide an automatic sensitive data discovery method, employing the following technical solution: monitoring a task queue to obtain sensitive data discovery tasks to be processed; parsing the sensitive data discovery tasks, dynamically adapting connection strategies using a database fingerprinting mechanism, and connecting to a target database; extracting the configuration information of the sensitive data discovery tasks, determining the target database table name and target field name, and generating an intelligent sampling strategy and data query SQL statements; executing the intelligent sampling strategy and the data query SQL statements, scanning and obtaining data samples, performing sensitive information pattern matching and identification on the data samples, and generating discovery results and a data evolution tracking system; writing the discovery results into a result table, and generating a data lineage graph based on the discovery results and the data evolution tracking system; updating the execution progress of the sensitive data discovery tasks, closing the database connection after execution, and updating the status of the sensitive data discovery tasks to complete.

[0008] Preferably, the monitoring task queue for acquiring sensitive data discovery tasks to be processed includes: establishing a continuous monitoring service to scan the objects to be processed in the task queue in real time; and classifying and filtering by object type identifiers to acquire the sensitive data discovery tasks to be processed.

[0009] Preferably, after acquiring the sensitive data discovery tasks to be processed, the monitoring task queue further includes: dynamically calculating the task priority score according to a preset data sensitivity level mapping table and a business importance weight matrix; monitoring the resource utilization rate of each execution node and determining the optimal node allocation scheme through adaptive load balancing; and determining the execution node and the corresponding sensitive data discovery task according to the optimal node allocation scheme and the task priority score.

[0010] Preferably, the step of parsing the sensitive data discovery task and dynamically adapting the connection strategy using a database fingerprinting mechanism to connect to the target database includes: parsing the sensitive data discovery task to determine a unique task ID and scanning parameter configuration information; wherein, the scanning parameter configuration information includes the target data source address, authentication information, and scanning range; obtaining the current number of concurrent tasks and system load, and dynamically adjusting the connection pool size; obtaining the version information, characteristic strings, and error message patterns of the database response, identifying the type and version of the target database, and dynamically generating an adaptive connection strategy based on database fingerprinting technology; and establishing a connection with the target database based on the unique task ID.

[0011] Preferably, the step of extracting the configuration information of the sensitive data discovery task, determining the target database table name and target field name, and generating an intelligent sampling strategy and data query SQL statement includes: extracting the configuration information of the sensitive data discovery task, determining the target database table name and target field name; dynamically generating the data query SQL statement by concatenating the target database table name and target field name using an SQL template engine; obtaining the total number of records in the target database table through pre-querying, and generating the intelligent sampling strategy according to a preset sampling rate threshold and table size classification standard; analyzing the data distribution characteristics and aggregation patterns of the target database table, dividing the target database table into N data subsets with similar characteristics, and generating a parallel scanning queue.

[0012] Preferably, the step of executing the intelligent sampling strategy and the data query SQL statement to scan and obtain data samples, performing sensitive information pattern matching and identification on the data samples, and generating discovery results and a data evolution tracking system includes: executing the data query SQL statement according to the intelligent sampling strategy to extract the data samples from the target database table; performing sensitive information pattern matching and identification on the data samples to determine the sensitive information pattern corresponding to the data samples; inputting the obtained data samples into a multi-level recognition engine for sensitive information detection; wherein the multi-level recognition engine selects a detection mode based on the sensitive information pattern; and generating the discovery results and the data evolution tracking system based on the detection.

[0013] Preferably, the multi-level recognition engine includes a data version management mechanism; wherein, the data version management mechanism tracks the change trajectory and propagation path of sensitive data by comparing data snapshots at different time points, and generates the data evolution tracking system across time dimensions.

[0014] Preferably, the multi-level recognition engine further includes at least one of the following: a regular expression matching engine, a machine learning model and contextual semantic analysis mechanism, a dynamic pattern learning mechanism, a data correlation reasoning mechanism, a business scenario awareness recognition engine, and a data spoofing detection mechanism; wherein, the regular expression matching engine performs basic pattern recognition; the machine learning model performs reasoning and judgment on complex data patterns; the contextual semantic analysis mechanism evaluates the sensitive features of the data; the dynamic pattern learning mechanism automatically mines new sensitive data feature patterns based on the analysis of historical recognition results, and integrates effective patterns into the recognition rule base through pattern verification and confidence evaluation; the data correlation reasoning mechanism analyzes the logical correlation and business semantic correlation of data by constructing a dependency graph between fields, and identifies the sensitivity of data combinations; the business scenario awareness recognition engine dynamically adjusts the sensitivity threshold and matching rule weight of the recognition strategy according to a pre-configured industry feature library and business domain knowledge graph; and the data spoofing detection mechanism restores data that has undergone Base64 encoding and / or hash processing and / or format transformation through pattern restoration algorithms and encoding recognition technology.

[0015] Preferably, the step of writing the discovery results into a result table and generating a data lineage graph based on the discovery results and the data evolution tracking system includes: storing the discovery results in a result table according to a predefined result data model; constructing a result confidence assessment mechanism and calculating a comprehensive confidence score based on confidence indicators during the identification process; automatically marking discovery results that may be misjudged using an anomaly detection algorithm; removing the discovery results that may be misjudged to obtain corrected discovery results; and, based on the corrected discovery results and the data evolution tracking system, identifying duplicate sensitive data instances using intelligent deduplication and association analysis, and establishing a data lineage graph through data fingerprint calculation and similarity matching algorithms; wherein, the data lineage graph includes the association relationships and flow paths of sensitive data in different storage locations.

[0016] Preferably, updating the execution progress of the sensitive data discovery task, closing the database connection after execution, and updating the status of the sensitive data discovery task to complete includes: updating the execution progress of the sensitive data discovery task after writing the discovery results into the result table; determining whether the sensitive data discovery task is complete; if not complete, then repeatedly executing the scanning to obtain data samples according to the target database table name and the target field name, and performing sensitive information pattern matching and identification on the data samples; if complete, then closing the connection with the target database; and updating the status of the sensitive data discovery task to complete.

[0017] Secondly, embodiments of this disclosure also provide an automatic sensitive data discovery system, including,

[0018] The task monitoring module is used to acquire sensitive data to be processed and identify tasks.

[0019] The task parsing module is used to parse the sensitive data discovery task, dynamically adapt the connection strategy, and connect to the target database.

[0020] The query statement generation module is used to extract the configuration information of the sensitive data discovery task, determine the target database table name and target field name, and generate intelligent sampling strategies and data query SQL statements.

[0021] The query statement execution module is used to execute the intelligent sampling strategy and the data query SQL statement, scan and obtain data samples, perform sensitive information pattern matching and identification on the data samples, and generate discovery results and a data evolution tracking system.

[0022] The result writing module is used to write the discovery results into a result table and generate a data lineage map based on the discovery results and the data evolution tracking system.

[0023] The task status update module is used to update the execution progress of the sensitive data discovery task. After the execution is completed, the database connection is closed and the status of the sensitive data discovery task is updated to complete.

[0024] Thirdly, this disclosure also provides a computer device, which adopts the following technical solution:

[0025] The computer device includes:

[0026] At least one processor; and,

[0027] A memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform any of the above-described methods for automatically discovering sensitive data.

[0029] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute any of the above-described methods for automatically discovering sensitive data.

[0030] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0031] The beneficial effects of the automatic sensitive data discovery method and system provided in this disclosure are:

[0032] 1. By using a queue-driven task scheduling mechanism, the automation and concurrent processing of sensitive data discovery tasks are realized, which significantly improves the efficiency of task acquisition and subsequent execution.

[0033] 2. Dynamic SQL concatenation technology enables the system to flexibly adapt to different database structures and scanning requirements, enhancing the system's versatility.

[0034] 3. The real-time progress tracking and result recording mechanism ensures the transparency of task execution and the integrity of results, facilitating auditing and compliance management.

[0035] 4. Automated connection resource management effectively avoids database connection leakage problems and improves system stability and reliability.

[0036] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart of an automatic sensitive data discovery method provided in this disclosure embodiment;

[0039] Figure 2 A schematic diagram of the structure of an automatic sensitive data discovery system;

[0040] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation

[0041] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0042] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0043] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0044] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0045] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0046] like Figure 1 As shown, this embodiment provides a method for automatically discovering sensitive data, including the following steps:

[0047] Step S1: Monitor the task queue and obtain the sensitive data to be processed to identify tasks;

[0048] Step S2: Analyze the sensitive data discovery task, dynamically adapt the connection strategy using a database fingerprinting mechanism, and connect to the target database;

[0049] Step S3: Extract the configuration information of the sensitive data discovery task, determine the target database table name and target field name, and generate intelligent sampling strategy and data query SQL statement;

[0050] Step S4: Execute the intelligent sampling strategy and the data query SQL statement to scan and obtain data samples, perform sensitive information pattern matching and identification on the data samples, and generate discovery results and a data evolution tracking system;

[0051] Step S5: Write the findings into a results table, and generate a data lineage map based on the findings and the data evolution tracing system;

[0052] Step S6: Update the execution progress of the sensitive data discovery task. After the execution is completed, close the database connection and update the status of the sensitive data discovery task to complete.

[0053] The queue-driven task scheduling mechanism automates and enables concurrent processing of sensitive data discovery tasks, significantly improving the efficiency of task acquisition and subsequent execution. Dynamic SQL concatenation technology allows the system to flexibly adapt to different database structures and scanning requirements, enhancing its versatility. Real-time progress tracking and result recording mechanisms ensure the transparency of task execution and the integrity of results, facilitating auditing and compliance management. Automated connection resource management effectively avoids database connection leakage issues, improving the system's stability and reliability.

[0054] In this embodiment, step S1 includes establishing a continuous monitoring service to scan the pending objects in the task queue in real time; and classifying and filtering by object type identifiers to obtain the pending sensitive data discovery tasks.

[0055] Specifically, a continuous monitoring service is established. The task queue is accessed at fixed time intervals through a periodic polling mechanism to obtain all task objects in the queue that are pending processing. The type identifier field of each task object is read and compared with the predefined sensitive data discovery task types. The target tasks that meet the conditions are filtered out, namely the sensitive data discovery tasks to be processed. The filtering results are sorted according to the task creation time and priority identifier to form an ordered list of tasks to be executed. At the same time, monitoring logs are recorded and the service running status is updated.

[0056] One feasible implementation is to establish a continuous monitoring service. This service uses a timed polling mechanism to access the task queue every 5 seconds, retrieving all pending task objects. For example, in a bank's data security management scenario, the system reads the type identifier field of each task object (e.g., `task_type="sensitive_data_scan"`), compares it with predefined sensitive data discovery task types, and filters out sensitive data scanning tasks from the mixed task queue, eliminating backup tasks, synchronization tasks, and other types. The filtered target tasks (e.g., customer information table scanning tasks, transaction record table scanning tasks, employee file table scanning tasks) are then sorted according to their creation time and priority identifier. Customer ID number scanning tasks marked as "high sensitivity" have a priority of 1, and transaction amount scanning tasks marked as "medium sensitivity" have a priority of 2, forming an ordered list of tasks to be executed. Simultaneously, monitoring logs are recorded (including task quantity, filtering results, sorting criteria, etc.), and the service running status is updated to "normal monitoring."

[0057] By using a queue-driven task scheduling mechanism, the system establishes a continuous monitoring service, scans the objects to be processed in the task queue in real time, classifies and filters them by object type identifiers, accurately extracts sensitive data discovery tasks, realizes the automation and concurrent processing of sensitive data discovery tasks, and significantly improves scanning efficiency.

[0058] In this embodiment, after step S1, the method further includes:

[0059] Step S11: Dynamically calculate the task priority score based on the preset data sensitivity level mapping table and business importance weight matrix;

[0060] Specifically, the process begins by extracting predefined sensitive data from the task configuration to identify task types, i.e., sensitivity identifiers. These sensitivity identifiers are then converted into corresponding numerical levels using a mapping table, with higher sensitivity levels resulting in larger numerical values. Simultaneously, the importance identifiers of the business systems associated with the tasks are obtained, and the corresponding weight coefficients are found using a weight matrix. Next, the sensitivity level values ​​are weighted and calculated with the business importance weight coefficients to obtain the task priority score. Finally, the task queue is sorted according to the priority scores, with tasks having higher scores being executed first, ensuring that data scanning tasks with high sensitivity and high business importance are processed preferentially.

[0061] For example, in a banking scenario, the sensitivity of a customer ID number scanning task is "highly sensitive" (corresponding to a value of 10), the sensitivity of a customer name scanning task is "medium sensitive" (corresponding to a value of 6), and the sensitivity of a customer contact information scanning task is "low sensitive" (corresponding to a value of 3). Simultaneously, the importance of the associated business systems is obtained, and the corresponding weight coefficients are found using a weight matrix. For example, the weight coefficient for the core transaction system is 1.5, the weight coefficient for the customer management system is 1.2, and the weight coefficient for the log system is 0.8. Then, the sensitivity level values ​​are compared with the business importance weight coefficients... The task is weighted and calculated to obtain a comprehensive priority score. For example, the priority score of the ID card number scanning task in the core transaction system is 10 × 1.5 = 15, the priority score of the name scanning task in the customer management system is 6 × 1.2 = 7.2, and the priority score of the contact information scanning task in the log system is 3 × 0.8 = 2.4. Finally, the ID card number scanning task (15 points) takes priority over the name scanning task (7.2 points), and the name scanning task takes priority over the contact information scanning task (2.4 points), ensuring that data scanning tasks with high sensitivity and high business importance can be processed first.

[0062] Step S12: Monitor the resource utilization of each execution node and determine the optimal node allocation scheme through adaptive load balancing;

[0063] Specifically, firstly, a node resource monitoring module is established to periodically collect CPU utilization, memory usage, and network bandwidth usage of each execution node. Resource utilization is divided into three levels—low load, medium load, and high load—according to preset thresholds. Simultaneously, a historical execution efficiency statistics table is maintained, recording the average time and success rate of each node in processing different types of tasks. Then, a load assessment algorithm is used to calculate the overall load score of each node, weighting the current resource utilization level with historical execution efficiency. Nodes with lower resource utilization and higher historical efficiency receive better scores, thus determining the optimal node allocation scheme. Finally, the optimal node is selected based on the optimal node allocation scheme, prioritizing the allocation of new tasks to the node with the highest score. When the load of the optimal node exceeds a preset limit, the next best node is automatically selected, ensuring balanced task allocation and execution efficiency.

[0064] For example, in a sensitive data scanning scenario in a bank's data center, the system monitors node A's CPU utilization at 15%, memory utilization at 30%, and network bandwidth utilization at 20%. Based on preset thresholds (CPU < 30%, memory < 50%, network < 40% is low load), node A is classified as low load. Node B's CPU utilization is 65%, memory utilization is 75%, and network bandwidth utilization is 60%, exceeding the high load threshold (CPU > 60%, memory > 70%, network > 50%), and it is classified as high load. Node C's CPU utilization is 45%, memory utilization is 65%, and network bandwidth utilization is 50%, exceeding the high load threshold (CPU: 30%-60%, memory: 50%-70%, network: 40%-50%), and it is classified as medium load. Simultaneously, a historical execution efficiency statistics table is maintained, recording the average time and success rate for each node to process different types of tasks. For example, node A's average time to process a customer information table scan task is 120 seconds with a success rate of 98%, and its average time to process a transaction record table scan task is 1 second. With a success rate of 96% and a low load of 80 seconds, node C's average processing time for the same task was 150 seconds and 220 seconds, with success rates of 95% and 94%, respectively. Node A, with a low load level, has a base score of 90 points. Combined with its high execution efficiency (98% success rate), it receives an efficiency weighting coefficient of 1.2, resulting in a final comprehensive score of 90 × 1.2 = 108 points. Node B, with a high load level, has a base score of 40 points, and even with historically high efficiency, it can only achieve a comprehensive score of 40 × 1.2 = 48 points. Node C, with a medium load level, has a base score of 70 points. Combining its high execution efficiency (success rates of 95% and 94%), an efficiency weighting coefficient of 1.1 is obtained, resulting in a final comprehensive score of 70 × 1.1 = 77 points. Finally, based on the comprehensive load score, the optimal node is selected, and new sensitive data scanning tasks are preferentially assigned to node A (108 points), which has the highest score. When the number of concurrent tasks on node A reaches the preset limit (e.g., 5 concurrent tasks), the second-best node C (77 points) is automatically selected for task allocation, ensuring that the sensitive data scanning tasks of the bank's core business database can be executed efficiently under optimal resource conditions.

[0065] Step S13: Determine the execution node and the corresponding sensitive data discovery task based on the optimal node allocation scheme and the task priority score.

[0066] Specifically, firstly, sensitive data discovery tasks are retrieved sequentially from the task queue sorted by priority score, while simultaneously obtaining the currently available optimal execution node from the optimal node allocation scheme. Next, it is checked whether the current task load of the optimal node exceeds the preset concurrent task limit. If it does not exceed the limit, the sensitive data discovery task is directly assigned to that node; otherwise, the second-best available node with the lowest load score is selected. Then, a binding relationship is established between the task and the execution node, recording information such as the task ID, the assigned node identifier, and the allocation time in the task scheduling table. Simultaneously, a task allocation instruction, containing complete task configuration information and execution parameters, is sent to the target execution node. Finally, the current task load count of the node and the task allocation status identifier are updated to ensure that subsequent task allocation accurately reflects the actual load of each node, achieving reasonable distribution and efficient execution of tasks across multiple execution nodes.

[0067] For example, in a data security management scenario of a large bank, the system retrieves the "Customer ID Number Scanning Task" (priority score 15), which is ranked first in the priority queue. Simultaneously, based on the load assessment results, node A is identified as the optimal execution node (overall load score 108, current concurrent tasks 2), which does not exceed the preset concurrent task limit of 5. Therefore, the ID number scanning task is directly assigned to node A. When node A subsequently reaches the 5 concurrent task limit, the system automatically selects the second-best node C (overall load score 77, current concurrent tasks 1) to assign a new "Customer Mobile Number Scanning Task." Next, a binding relationship is established between the task and the execution node, recording information such as the task ID, the assigned node identifier, and the assignment time in the task scheduling table. The system records in the task scheduling table: Task ID "TASK_20240315_001", assigned node "NODE_A", and assignment time "2024-03-15". At 14:30:25, the target database is "customer_db.personal_info", and the estimated execution time is "180 seconds". Simultaneously, a complete task instruction containing database connection information (IP address: XXX.XXX.X.XXX, port: 3306, database name: customer_db), scan configuration (target table: personal_info, target field: id_card_number, sampling strategy: intelligent sampling 20%), and recognition rules (ID card number regular expression, machine learning model parameters) is sent to node A. Finally, the current task load count of node A is updated from 2 to 3, the status of the "Customer ID Card Number Scanning Task" is updated from "Pending Assignment" to "Assigned", and the load status of each node is displayed in real time on the node monitoring panel: Node A (3 / 5 tasks, load 60%), Node B (5 / 5 tasks, load 100%), Node C (1 / 5 tasks, load 20%), achieving reasonable distribution and efficient execution of tasks across multiple execution nodes.

[0068] Based on a pre-defined data sensitivity level mapping table and business importance weight matrix, task priority scores are dynamically calculated to adaptively adjust the task execution order. Simultaneously, the adaptive load balancing mechanism monitors the CPU, memory, and network resource utilization of each execution node, combines historical task execution efficiency statistics, and uses a load assessment algorithm to calculate the optimal node allocation scheme, ensuring that tasks are executed on nodes with the most abundant system resources.

[0069] In this embodiment, step S2 includes: parsing the sensitive data discovery task to determine a unique task ID and scanning parameter configuration information; wherein, the scanning parameter configuration information includes the target data source address, authentication information, and scanning range; obtaining the current number of concurrent tasks and system load, and dynamically adjusting the connection pool size; obtaining the version information, characteristic string, and error message pattern of the database response, identifying the type and version of the target database, and dynamically generating an adaptive connection strategy based on database fingerprinting technology; and establishing a connection with the target database based on the unique task ID.

[0070] Specifically, the system first retrieves task objects from the task queue and extracts their unique identifiers as task IDs, which are used for subsequent configuration queries and progress tracking. Based on the task ID, it retrieves the corresponding scan parameter configurations from the configuration management module, including the target data source address (database server IP, port number, database name), authentication information (username, password, or key file path), and scan scope (specified list of database table names, set of field names, and data filtering conditions). The system then verifies the validity of the retrieved configuration information, checking if the data source address format is correct, if the authentication information is complete, and if the scan scope conforms to the predefined security policy. Finally, the verified configuration information is loaded into memory to prepare for subsequent database connection establishment and scan execution, while configuration parsing logs are recorded for troubleshooting and auditing.

[0071] Then, the system monitoring module continuously monitors the number of currently executing sensitive data scanning tasks, as well as CPU utilization and memory usage. When the number of concurrent tasks exceeds a preset threshold and the system load is high, the number of available connections in the connection pool is automatically reduced to avoid resource contention. When the number of concurrent tasks is low and the system load is low, the connection pool size is appropriately increased to improve processing efficiency. The system obtains the version information, characteristic strings, and error message patterns from the database response to identify the type and version of the target database. Standard query statements are sent to obtain database version information, and key strings in the returned results are analyzed to determine the database type. For example, MySQL returns a version information containing the identifier "MySQL", Oracle returns an identifier "Oracle Database", and PostgreSQL returns an identifier "PostgreSQL". The system also records the major and minor version numbers of the database. Finally, an adaptive connection strategy is dynamically generated based on database fingerprinting technology. Based on the identified database type and version, the corresponding connection parameters are selected from the pre-configured strategy library, including connection timeout, query timeout settings, character encoding, etc. For example, a 30-second connection timeout and UTF-8 encoding are used for MySQL 5.7, and a 45-second connection timeout and UTF-8 encoding are used for Oracle 12c, to ensure connection stability and compatibility.

[0072] Finally, based on the task ID, the connection parameters of the target database are extracted from the configuration information, including the database server address, port number, database name, username, and password authentication information. Then, a probe query is sent using database fingerprinting technology to analyze the returned version information and characteristic strings, automatically identifying the database type (such as MySQL, Oracle, PostgreSQL, etc.) and version number. Based on the identification results, the corresponding connection parameter configuration is selected from the policy library, including connection timeout, character encoding method, and driver type. At the same time, it checks whether there are any reusable idle connections in the connection pool. If they exist, they are directly acquired and reused; otherwise, a new database connection is created. The validity of the connection is verified by executing a simple test query to confirm that the connection status is normal. The successfully established connection object is bound to the current task ID, and the connection establishment time and connection identifier are recorded to prepare for subsequent data scanning operations.

[0073] Sensitive data parsing and task discovery identifies unique task IDs. Based on these IDs, the system retrieves corresponding scan parameter configuration information from the configuration management module, including key parameters such as the target data source address, authentication information, and scan range. A connection instance pool avoids frequent connection establishment and destruction operations, while an adaptive connection number adjustment strategy dynamically adjusts the connection pool size based on the current number of concurrent tasks and system load, maximizing resource utilization efficiency. A database fingerprinting mechanism automatically identifies the target database type and version by analyzing database response version information, characteristic strings, and error message patterns, providing a basis for subsequent connection strategy selection and query optimization scheme formulation.

[0074] In this embodiment, step S3 includes: extracting the configuration information of the sensitive data discovery task, determining the target database table name and the target field name; dynamically generating the data query SQL statement by concatenating the target database table name and the target field name using an SQL template engine; obtaining the total number of records in the target database table through pre-query, and generating the intelligent sampling strategy according to the preset sampling rate threshold and table size classification standard; analyzing the data distribution characteristics and aggregation patterns of the target database table, dividing the target database table into N data subsets with similar characteristics, and generating a parallel scanning queue.

[0075] Specifically, the target database table names are first extracted from the task configuration file. For example, in a banking scenario, the table names customer_info, account_details, and transaction_records are obtained. At the same time, the target field names are extracted (which may be one or more, i.e., a set of field names), such as the id_number, phone, and email fields of the customer_info table. Then, the SQL template engine selects the corresponding query statement template according to different database types, dynamically replaces the table names and field names in the placeholder positions in the template, and generates a standard SELECT query statement, such as "SELECT id_number, phone, email FROM customer_info WHERE id_number IS NOTNULL".

[0076] Secondly, a COUNT query is executed to obtain the total number of records in the target table, for example, the customer_info table contains 500,000 records. Then, based on the preset sampling rate threshold and table size classification standard, the intelligent sampling strategy is generated. For example, according to the preset table size classification standard, small tables (less than 10,000 records) adopt a full scan strategy, medium tables (10,000-100,000 records) adopt a fixed ratio sampling strategy, and large tables (more than 100,000 records) adopt a stratified sampling strategy. Next, the sampling rate is determined according to the table size level, with a sampling rate of 50% for medium tables and 20% for large tables, and the corresponding sampling SQL statement is generated, such as "SELECT id_number, phone, email FROM customer_info ORDER BY RAND() LIMIT100000" to achieve 20% sampling.

[0077] Analyze the data distribution characteristics and clustering patterns of the target database table, divide the target database table into N data subsets with similar characteristics, and generate a parallel scan queue.

[0078] Finally, based on the data distribution characteristics and aggregation patterns of key fields in the target database tables, such as dividing the data by time period according to the create_time field in the customer_info table, or partitioning by the numerical range of the customer_id field, large data tables are then divided into N data subsets according to the analysis results. For example, N is 5 for large tables, N is 3 for medium tables, and small tables are not divided into subsets. Each subset contains a similar number of records. For example, a table with 500,000 records is divided into 5 subsets, each containing 100,000 records. Finally, an independent scan task is generated for each data subset, forming a parallel scan queue. For example, subtasks such as "scan customer_info table for 2020 data" and "scan customer_info table for 2021 data" are generated. These subtasks can be processed in parallel on different execution nodes, improving the overall scan efficiency.

[0079] In another feasible implementation, the data distribution characteristics and aggregation patterns of key fields in the target database table are analyzed. This includes statistically analyzing the numerical distribution of key fields in the target table. For example, in the `customer_info` table, the minimum, maximum, and total number of records for the `customer_id` field are calculated, assuming the `customer_id` range is 1 to 500,000 and the total number of records is 500,000. Then, an appropriate partitioning strategy is selected based on the field type: range partitioning for numeric fields, time-based partitioning for time-based fields, and hash partitioning for character fields. For example, `customer_id` can be evenly divided into five intervals based on its numerical range: 1-100,000, 100,001-200,000, 200,001-300,000, 300,001-400,000, and 400,001-500,000. Next, corresponding query conditions are generated for each partition, rewriting the original full table scan SQL statement into a partitioned query statement with a WHERE condition, such as "SELECT id_number, phone, email FROM customer_info WHERE". The query is set to "customer_id BETWEEN 1 AND 100000" to ensure that each partition contains approximately the same amount of data. Finally, each partition query task is added to the parallel scan queue. Each partition task can be executed independently without interfering with each other. By having multiple execution nodes process the data of different partitions simultaneously, parallel scanning is achieved to improve the overall processing efficiency.

[0080] The system parses the list of table names and corresponding field names of the target database from the task configuration. Based on this metadata, it dynamically generates data query statements using an SQL template engine. The intelligent sampling strategy obtains the total number of records in the data table through pre-queries and automatically selects different sampling methods such as full scan, fixed-ratio sampling, or stratified sampling based on preset sampling rate thresholds and table size classification standards. Building upon the sampling strategy, the data distribution awareness technology analyzes the distribution characteristics and clustering patterns of the data to divide large data tables into multiple subsets with similar characteristics, generating parallel scan queues to support subsequent distributed scan execution.

[0081] In this embodiment, step S4 includes:

[0082] Step S41: According to the intelligent sampling strategy, execute the data query SQL statement to extract the data sample from the target database table;

[0083] Specifically, based on the intelligent sampling strategy type generated in the previous step, the corresponding SQL execution method is selected. For small tables (less than 10,000 records), a full query statement is executed directly to obtain all data. For medium-sized tables (10,000 to 100,000 records), a fixed-ratio sampling query with a LIMIT clause is executed. For large tables (more than 100,000 records), a stratified sampling query statement is executed. Then, the corresponding SQL statement is executed through the established database connection. The system automatically handles abnormal situations during the query process, including connection timeouts and query interruptions, to ensure the stability of the data extraction process. Next, the result set returned by the query is read row by row, and the target field data in each row is extracted. For example, the specific values ​​of fields such as id_number, phone, and email are extracted from the customer_info table. At the same time, the extracted data undergoes basic format verification to filter out records with null values ​​or abnormal formats, ensuring the accuracy of subsequent sensitive information identification. Finally, the valid data samples are loaded into the memory buffer to form a structured data set, i.e., data samples, to prepare data for the next step of sensitive information pattern matching and identification processing, and the actual number of data records extracted and the sampling completion status are recorded.

[0084] Step S42: Perform sensitive information pattern matching and identification on the data sample to determine the sensitive information pattern corresponding to the data sample;

[0085] Specifically, the extracted data samples are input into a regular expression matching engine. Basic pattern recognition is performed using a predefined sensitive data pattern library to determine the sensitive information patterns corresponding to the data samples. For example, when all data samples are numerical, matching ID card numbers and mobile phone numbers indicates a numerical sensitive information pattern, and subsequent detection focuses on data transmission and time tracking. When character or text data accounts for more than 80% of all data samples, the sensitive information pattern is a text sensitive pattern, and subsequent detection focuses on contextual semantic analysis. When encoded data exists in all data samples, the sensitive information pattern is a text-encoded sensitive pattern, and subsequent detection, while focusing on contextual semantics, adds encoding recognition and data spoofing detection. This is merely an example and not a limitation of the implementation of this solution. The predefined sensitive data pattern library can be set based on expert experience or automatically set and optimized after cluster analysis based on historical detection results; no restrictions are imposed here.

[0086] Step S43: Input the acquired data sample into the multi-level recognition engine for sensitive information detection; wherein the multi-level recognition engine selects a detection mode based on the sensitive information pattern;

[0087] Specifically, the acquired data samples are input into a multi-level recognition engine. The engine automatically identifies or acquires sensitive information patterns and selects the corresponding detection mode based on the sensitive information patterns to perform sensitive information detection.

[0088] In this embodiment, the multi-level recognition engine includes a data version management mechanism; wherein, the data version management mechanism tracks the change trajectory and propagation path of sensitive data by comparing data snapshots at different time points, and generates the data evolution tracking system across time dimensions.

[0089] In a feasible implementation, firstly, a timestamp is generated for each sensitive data record scanned during each detection, and a data snapshot storage table is established to save the status information of sensitive data at different points in time. Then, a periodic comparison mechanism is used to detect the addition, deletion, and modification of sensitive data, identifying changes in data content and location migration. Next, a mapping of relationships between sensitive data is established, recording the copying and referencing relationships between data in different tables and databases. At the same time, a data change history is maintained to form a change trajectory of sensitive data, including key information such as change time, change type, and change source. Finally, a data propagation path is constructed to show the flow and evolution trajectory of sensitive data from the source to various storage locations, supporting full traceability and monitoring of the sensitive data lifecycle, thereby generating the aforementioned data evolution tracking system across time dimensions.

[0090] For example, in a banking scenario, the system generates a timestamp "2024-03-15 14:30:25" for a customer ID number scanning task, and records 100 ID number records and their location information found in the customer_info table in the data snapshot table. The system performs a comparative analysis every day at 2 AM, finding that 15 new ID number records have been added to the customer_info table, 3 bank card number records have been deleted from the account_details table, and customer mobile phone numbers have changed from "138" to "2024-03-15 14:30:25" in the transaction_records table. "1234" changed to "139" 5678"; then it was discovered that the customer ID number "320102XXXXXXXX1234" appeared simultaneously in the customer_info table of the customer information database, the risk_assessment table of the risk control database, and the customer_archive table of the archive database, establishing the data lineage relationship between the three tables; at the same time, change events such as "2024-03-15 09:15:30, added, imported from the customer account opening system", "2024-03-16 16:45:12, modified, customer information update interface", and "2024-03-17 11:20:05, deleted, data cleanup task" were recorded; finally, a data propagation path graph was constructed to show the flow and evolution trajectory of sensitive data from the source to various storage locations. For example, a customer ID number is generated from the customer account opening module of the core business system, propagated to the customer subject database of the data warehouse through the ETL data synchronization process, and then copied to the marketing analysis database through the reporting system, forming a complete propagation link graph of "account opening system → data warehouse → marketing system".

[0091] In this embodiment, the multi-level recognition engine further includes at least one of the following: a regular expression matching engine, a machine learning model and contextual semantic analysis mechanism, a dynamic pattern learning mechanism, a data correlation reasoning mechanism, a business scenario awareness recognition engine, and a data spoofing detection mechanism. Specifically, the regular expression matching engine performs basic pattern recognition; the machine learning model performs reasoning and judgment on complex data patterns; the contextual semantic analysis mechanism evaluates the sensitivity features of the data; the dynamic pattern learning mechanism automatically mines new sensitive data feature patterns based on the analysis of historical recognition results, and integrates effective patterns into the recognition rule base through pattern verification and confidence evaluation; the data correlation reasoning mechanism analyzes the logical and business semantic correlations of data by constructing a dependency graph between fields, and identifies the sensitivity of data combinations; the business scenario awareness recognition engine dynamically adjusts the sensitivity threshold and matching rule weights of the recognition strategy based on a pre-configured industry feature library and business domain knowledge graph; and the data spoofing detection mechanism restores data that has undergone Base64 encoding and / or hash processing and / or format transformation through pattern restoration algorithms and encoding recognition technology.

[0092] One feasible implementation is to use the ID card number pattern "^[1-9]\d{5}(18|19|(

[23] \d))\d{2}((0[1-9])|(10|11|12))(([0-2][1-9])|10|20|30|31)\d{3}[0-9Xx]$" to match an 18-digit ID card number, and use the mobile phone number pattern "^1[3-9]\d{ 9}$" matches 11-digit mobile phone numbers, using email address pattern"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$" matches email addresses, using bank card number pattern"^[1-9]\d{15,18}$" matches 16-19 digit bank card numbers, using passport number pattern"^[EGP]\d{8}$" matches Chinese passport numbers."

[0093] This approach uses complex data patterns for reasoning and judgment. A deep learning-based text classification model is employed to identify implicit sensitive information in customer remarks fields, such as non-standard format sensitive data descriptions like "Mr. Zhang, ID card number ending in 1234, mobile phone number ending in 5678". The BERT model is used to understand the semantics of the text and identify the identity information it contains. Simultaneously, an anomaly detection model is used to analyze the distribution characteristics of numerical fields and identify sensitive data that may have undergone transformation, such as converting "320102XXXXXXXX1234" to "320102XXXXXXXX". The anonymized data was determined to be of the original data type as an ID card number through model reasoning.

[0094] Contextual semantic analysis mechanisms assess the sensitivity of data. First, the semantic meaning of field names can be analyzed. For example, in a banking scenario, if a field named "card_no" has a data format consistent with bank card numbers, the system uses semantic analysis to confirm it as a bank card number field, setting its sensitivity weight to 0.9. Next, the contextual position of the field within the table structure can be analyzed. For instance, in the `customer_info` table, the "mobile_phone" field appears adjacent to the "customer_name" and "id_card" fields; contextual association analysis can increase the confidence level of the sensitivity assessment for this field. Then, the business semantic features of the data content can be analyzed. For example, if descriptive text such as "cardholder Zhang San, card number starting with 6222" is found in the remarks field, natural language processing technology can be used to identify the implicit bank card information, even if the field was originally marked as a plain text field. Simultaneously, sensitivity assessment can be performed in conjunction with the business process context. For example, in the order table of a transaction system, the "pay_account" field, although containing only partially anonymized account information "622202", can still be sensitive. Although the data was initially classified as "1234", it was determined to still be highly sensitive based on the transaction business scenario. Finally, a comprehensive sensitivity score was generated through weighted calculation of multi-dimensional semantic features, such as a field semantic weight of 0.3, a contextual association weight of 0.2, a content feature weight of 0.3, and a business scenario weight of 0.2. This determined the sensitivity level of the data and the processing strategy.

[0095] The dynamic pattern learning mechanism first collects recognition result data from historical scanning processes, statistically analyzes misjudged and missed data samples. For example, in a banking scenario, the system finds that traditional ID number rules cannot recognize ID numbers ending in the letter X, such as "440101XXXXXXXX123X". By analyzing 1 million historical recognition records, a new ID number pattern "^\d{17}[0-9Xx]$" is extracted. Then, machine learning algorithms are used to extract features from the newly discovered data pattern and establish a pattern verification sample set, such as collecting 1000 ID number samples containing the letter X and 1000 non-ID number samples. For samples with similar ID numbers but formats, a binary classification model is trained to verify the accuracy of the new pattern. Next, confidence evaluation metrics for the new pattern are calculated, including precision, recall, and F1 score. When the new ID number pattern achieves 98.5% precision and 96.2% recall on the validation set, the system automatically marks it as a high-confidence pattern. Finally, the validated new pattern is automatically integrated into the recognition rule base, and the pattern weight configuration is updated, for example, setting the weight of the new ID number pattern to 0.95. This pattern is then enabled in subsequent scans. Simultaneously, a pattern performance monitoring mechanism is established to continuously track the actual recognition performance of the new pattern, ensuring continuous optimization of the rule base and improvement in accuracy.

[0096] Data association reasoning mechanisms can first establish data relationships through field semantic analysis. For example, in a banking scenario, the system analyzes the "birth_year" field (value "XXXX"), "birth_month" field (value "XX"), "birth_day" field (value "XX"), and "id_suffix" field (value "1234") in the customer_info table. Individually, each field is not sensitive, but through association analysis, it is found that these four fields combined can form the ID number information "XXXXXXXX1234". Then, a dependency graph between fields is established to identify field combinations with business logic relationships. For example, in an e-commerce system, the "user_name" field (value "Zhang San"), "phone_last4" field (value "5678"), and "address_district" field (value "Chaoyang District") have low sensitivity when existing individually, but when combined... After comprehensive analysis, it was found that specific individuals could be precisely located, forming highly sensitive identity information. Then, through business semantic association analysis, sensitive information combinations across tables were identified. For example, in the medical system, the "patient_code" field in the patient_basic table and the "diagnosis_code" field in the medical_record table are associated through patient codes. Although a single diagnosis code does not directly expose the patient's identity, it can form complete medical privacy data when combined with the patient's basic information. Finally, a sensitivity scoring model was established to quantitatively evaluate the sensitivity of field combinations. For example, the sensitivity score of the combination "name + last four digits of mobile phone number + address area" was set at 8.5 points (out of 10), and the sensitivity score of the combination "year of birth + month of birth + day of birth + last four digits of ID card" was set at 9.2 points. When the combination score exceeds the preset threshold of 8.0 points, it is automatically marked as a highly sensitive data combination, and the complete field association path and combination sensitivity analysis report are recorded in the sensitive data discovery results.

[0097] The business scenario-aware recognition engine can load corresponding recognition strategy configurations from an industry feature library based on the business system identifier of the scanning task. For example, in a banking scenario, the system automatically loads the financial industry feature library, setting the weight for bank card number recognition to 0.95 and ID card number recognition to 0.9. In a healthcare scenario, the system loads the medical feature library, setting the weight for patient ID recognition to 0.85 and medical record number recognition to 0.8. Then, it adjusts the sensitivity threshold based on the business domain knowledge graph. For instance, in an e-commerce system, the sensitivity threshold for the order amount field is set to 6.0 (out of 10), while in a payment system, the sensitivity threshold for the same type of field is increased to 8.5, reflecting different business scenarios. The system addresses the differentiated requirements for data sensitivity across various scenarios. It then dynamically adjusts matching rules through business context analysis. For example, in an HR system, the "employee_code" field, while in a standard number format, has its sensitivity weight increased from the default 0.3 to 0.7 based on the HR business context. In an inventory management system, the "product_code" field, with the same format, maintains a lower weight of 0.2. Finally, it establishes a scenario-based identification priority matrix. For instance, in a securities trading system, priority is given to identifying customer account numbers and trading passwords; in a social media platform, priority is given to identifying user privacy information and contact data. This ensures that sensitive data across different industries and business areas is accurately identified and appropriately protected.

[0098] The data spoofing detection mechanism can first detect the encoding type of the data through encoding format identification algorithms. For example, in a banking scenario, the system might find data in the remark field of the customer_info table with a Base64 encoding format like "MzIwMTAyMTk5MDAxMDExMjM0". After decoding, this data would be restored to an ID card number like "320102XXXXXXXX1234". Then, it can use hash reverse analysis technology to identify sensitive data processed by MD5 or SHA. For instance, if a 32-bit MD5 value like "5d41402abc4b2a76b9719d911017c592" is found in the user table, it can be further analyzed using color hashing. The system uses a rainbow table query or pattern matching to confirm potentially sensitive information. Next, it detects and identifies spoofed data through format transformation, character substitution, position swapping, or separator insertion. For example, it transforms the ID number "320102XXXXXXXX1234" into "32-01-02-XX-XX-XX-XX-12-34" or "320102@XXXX@XXXX@1234". The system then removes separators and character substitutions to restore the original ID number format. Finally, it establishes a spoofing pattern learning mechanism. By analyzing historically discovered spoofed data samples, it automatically extracts new spoofing feature patterns, such as converting the mobile phone number "138XXXX5678" into "138". The 5678" composite spoofing pattern is then Base64 encoded, and the identified spoofing pattern is added to the detection rule base to improve the ability to identify complex data spoofing.

[0099] Step S44: Based on the detection output, output the discovery results and the data evolution tracking system.

[0100] Specifically, the detection results from the multi-level recognition engine are first structured and organized to form discovery results. For example, in a banking scenario, the system detected 15 ID card number records, 8 mobile phone number records, and 12 email address records in the customer_info table. Each record contains information such as field location, sensitivity type, and confidence score. Then, a sensitive data discovery result report is generated, recording the specific sensitive data location information, such as "Table name: customer_info, field name: id_number, record location: row 156, sensitivity type: ID card number, confidence score: 0.95, discovery time: 2024-03-15 14:35:20". Finally, the discovery results and data evolution tracking system are output.

[0101] The system executes the constructed SQL query statement, extracts data samples from the target database, inputs the acquired data into a multi-level recognition engine, selects the detection mode according to the sensitive information pattern, and performs sensitive information detection. A data version management mechanism is established based on a cross-time dimension data evolution tracking system. By comparing data snapshots at different time points, the system tracks the change trajectory and propagation path of sensitive data.

[0102] This engine includes basic pattern recognition via regular expression matching, reasoning and judging complex data patterns using pre-trained machine learning models, and comprehensive evaluation of data sensitivity features using contextual semantic analysis. A dynamic pattern learning mechanism automatically mines new sensitive data feature patterns based on statistical analysis of historical recognition results, and integrates effective patterns into the recognition rule base through pattern verification and confidence assessment. Data association reasoning technology analyzes the logical and business semantic relationships of data by constructing a dependency graph between fields, discovering data patterns that are not identifiable by individual fields but are sensitive when combined. A business scenario-aware recognition engine dynamically adjusts the sensitivity threshold and matching rule weights of the recognition strategy based on a pre-configured industry feature library and business domain knowledge graph. A data spoofing detection mechanism uses pattern reconstruction algorithms and encoding recognition technology to reconstruct and analyze data that has undergone Base64 encoding, hashing, or format transformation.

[0103] In this embodiment, step S5 includes: writing the discovery results into a result table for storage according to a predefined result data model; constructing a result confidence assessment mechanism, calculating a comprehensive confidence score based on the confidence index in the identification process; automatically marking discovery results that may be misjudged using an anomaly detection algorithm; removing the discovery results that may be misjudged to obtain corrected discovery results; based on the corrected discovery results and the data evolution tracking system, using intelligent deduplication and association analysis, identifying duplicate sensitive data instances through data fingerprint calculation and similarity matching algorithms, and establishing a data lineage graph; wherein, the data lineage graph includes the association and flow path of sensitive data in different storage locations.

[0104] Specifically, the sensitive data discovery results output by the recognition engine are first organized according to a standardized format, including key information such as sensitive data type, field position, confidence score, and discovery time. For example, in a banking scenario, the system formats the detected ID card number result as "Table name: customer_info, Field name: id_number, Sensitive type: ID card number, Confidence score: 0.95, Discovery time: 2024-03-15 14:35:20"; then the formatted result data is written in batches to a predefined result storage table.

[0105] Secondly, various confidence indicators are collected during the identification process, including regular expression matching score, machine learning model inference confidence score, and contextual semantic analysis confidence score. For example, the regular expression matching score of a certain ID number record is 0.98, the model inference confidence score is 0.92, and the semantic analysis confidence score is 0.89. Then, the comprehensive confidence score is calculated using a weighted average method. The weights of each indicator are determined based on historical accuracy statistics, such as a weight of 0.4 for regular expression matching, 0.4 for model inference, and 0.2 for semantic analysis. The final comprehensive confidence score is 0.98×0.4+0.92×0.4+0.89×0.2=0.938.

[0106] Next, a confidence threshold is set for anomaly detection. When the overall confidence level is lower than the preset threshold, it is automatically marked as a suspected misjudgment record. For example, if the threshold is set to 0.8, the recognition results below this value will be marked as "awaiting manual review". This removes the discovery results that may have been misjudged and obtains the corrected discovery results. At the same time, an anomaly result statistics mechanism is established to regularly analyze the misjudgment patterns and feed them back to the recognition engine for rule optimization.

[0107] Finally, based on the corrected findings and the data evolution tracking system, a unique data fingerprint is generated for each sensitive data record. A hash algorithm is used to convert the sensitive data content into a fixed-length identifier; for example, the ID number "320102XXXXXXXX1234" is converted into the fingerprint "FP_320102XXXX". Then, fingerprint matching and similarity matching algorithms are used to search for identical data fingerprints in different tables and databases, identifying duplicate sensitive data instances. For example, the ID number is found to exist simultaneously in the `customer_info` table, the `account_details` table, and the `risk_assessment` table. Next, a data lineage is established. The system records the occurrence and time sequence of sensitive data in different locations. For example, the ID number first appears in the customer_info table, then is synchronized to the account_details table through the ETL process, and finally propagates to the risk_assessment table through the risk control analysis process. Finally, a visualized data lineage graph is constructed to show the relationship and complete flow path of sensitive data from the source to each storage location, forming a lineage link graph of "customer_info→account_details→risk_assessment", which supports the full lifecycle tracking and impact analysis of sensitive data.

[0108] The system identifies sensitive data from the engine and writes it into a result storage table according to a predefined result data model, while simultaneously updating the task execution progress status and completion percentage. The result confidence assessment mechanism calculates a comprehensive confidence score based on multiple confidence indicators during the identification process and automatically marks potentially misjudged result records using anomaly detection algorithms. Intelligent deduplication and association analysis technology uses data fingerprinting and similarity matching algorithms to identify duplicate sensitive data instances across tables and databases, establishing a data lineage graph to show the relationships and flow paths of sensitive data across different storage locations.

[0109] In this embodiment, step S6 includes: after writing the discovery result into the result table, updating the execution progress of the sensitive data discovery task; determining whether the sensitive data discovery task is completed; if not completed, then according to the target database table name and the target field name, repeatedly executing the scanning to obtain data samples, and performing sensitive information pattern matching and identification on the data samples; if completed, then closing the connection with the target database; and updating the status of the sensitive data discovery task to complete.

[0110] Specifically, after writing the findings into the results table, the detection progress is updated, recording the number of tables currently completed and the total number of tables, and calculating the completion percentage, such as "3 / 10 tables completed, progress 30%"; then the status identifier is updated in real time, updating the status of the sensitive data discovery task to the corresponding progress status, ensuring that the task monitoring system can accurately reflect the current execution status.

[0111] Starting with the list of tables to be processed, each table is processed sequentially according to a preset priority order. For each table, a list of corresponding field names is extracted, and then a complete process of data query, sensitive information identification, and result recording is performed on each field in turn. When all fields of a single table have been scanned, the execution progress of the sensitive data discovery task is updated. For example, the system updates the scanning status of the table to "completed". The system then checks whether the sensitive data discovery task has been completed, for example, by checking whether all the data tables in the list of tasks to be executed have been queried, data samples extracted, and detected. If not, the system continues to process the next table until all tables and fields in the configuration list have been processed. After the task is completed, the connection with the target database is closed, and the system releases database connection resources. For example, in a banking scenario, after completing the sensitive data scan of the three tables customer_info, account_details, and transaction_records, the system automatically closes the connection with the core business database (XXX.XXX.X.XXX:XXXX), removes the connection object from the connection pool, releases approximately 50MB of occupied memory resources, and records "Connection release time: 2024-03-15" in the connection log. At 16:45:30, the connection duration was 2 hours and 15 minutes, the total number of records processed was 1.5 million, and the connection status was "normally closed". The status of the sensitive data discovery task was updated to "completed".

[0112] The system iteratively executes the aforementioned data scanning and identification process according to the list of tables and fields in the scanning configuration until the scanning task for all configured objects is completed. After the task is completed, the system releases database connection resources and updates the status of the sensitive data discovery task to "completed".

[0113] In another embodiment, a method for automatically discovering sensitive data also includes a breakpoint resume scanning mechanism. By maintaining a persistent record of the task execution progress, if the task is interrupted due to an abnormality and then restarted, the execution can continue from the point of abnormal interruption, thus avoiding the repeated processing of completed data objects.

[0114] Specifically, the current scanning progress is recorded in real time during task execution, including the names of completed tables, field names, and scan timestamps. This progress information is persistently stored in a progress record table. For example, in a scanning scenario of a bank's core business system, the system is scanning a customer information database. It has completed scanning the id_card and phone fields of the customer_basic table and is processing the email field when a network interruption occurs. At this time, the progress record table records: "Task ID: TASK_20240315_001, Database: customer_db, Completed Table: customer_basic, Completed Fields: id_card, phone, Currently Processing Field: email, Interruption Time: 2024-03-15" 15:30:45, Number of records scanned: 15000; When the task is interrupted due to system failure or network anomaly, after the system restarts, it retrieves the scan position before the last interruption by querying the progress record table. If the above progress record is found after the system restarts, it determines that the scan needs to continue from the 15001st record in the email field of the customer_basic table; then, the scan resumes from the interruption point, skipping completed tables and fields, and directly processing the unfinished scan objects to ensure the continuity and integrity of the scan task. For example, it skips the id_card and phone fields of the customer_basic table and directly starts from the 15001st record in the email field. The scan begins, continuing to process the remaining 5000 records. After completing the email field, the scan continues with the address field, and then the next table, customer_account, is processed. Simultaneously, a progress verification mechanism is established to compare the data status before and after the interruption, confirming the accuracy of the resumed scan. For example, if the system finds that 100 new records were added to the customer_basic table during the interruption, the scan range is automatically adjusted to include the new records in the scan queue, ensuring no sensitive data is missed. The recovery log records: "Resumption scan successful. Skipped 2 completed fields, continued scanning 5100 records. 100 new records detected, automatically included in the scan range."

[0115] In another embodiment, a method for automatically discovering sensitive data also includes a predictive scan scheduling mechanism. Based on historical statistics of data change frequency and analysis results of business access patterns, the mechanism predicts the low-load time window of the database and schedules scan tasks to be executed during the time period with the least impact on business, thereby achieving a balance between business continuity and scan efficiency.

[0116] Specifically, the system first collects historical access logs of the target database through a monitoring module, and statistically analyzes the query frequency and data change counts for different time periods. For example, analyzing access patterns over the past 30 days reveals that 2 AM to 6 AM is the low-peak period for business. Then, a load prediction model is established to predict the database load trend over the next 24 hours based on historical data, identifying time windows with lower loads. Next, sensitive data scanning tasks are prioritized for execution during the predicted low-load periods. For example, full scans of large tables are scheduled for the early morning hours, while incremental scans of small tables are scheduled during business downtime. Finally, a dynamic adjustment mechanism is established to automatically pause or adjust the execution intensity of scanning tasks when the actual load does not match the prediction, ensuring that the normal operation of the business system is not affected.

[0117] like Figure 2 As shown, this embodiment also provides a sensitive data automatic discovery system, including:

[0118] The task monitoring module is used to acquire sensitive data to be processed and identify tasks.

[0119] The task parsing module is used to parse the sensitive data discovery task, dynamically adapt the connection strategy, and connect to the target database.

[0120] The query statement generation module is used to extract the configuration information of the sensitive data discovery task, determine the target database table name and target field name, and generate intelligent sampling strategies and data query SQL statements.

[0121] The query statement execution module is used to execute the intelligent sampling strategy and the data query SQL statement, scan and obtain data samples, perform sensitive information pattern matching and identification on the data samples, and generate discovery results and a data evolution tracking system.

[0122] The result writing module is used to write the discovery results into a result table and generate a data lineage map based on the discovery results and the data evolution tracking system.

[0123] The task status update module is used to update the execution progress of the sensitive data discovery task. After the execution is completed, the database connection is closed and the status of the sensitive data discovery task is updated to complete.

[0124] In another embodiment, the task monitoring module is also used for predictive scan scheduling. Based on historical statistics of data change frequency and analysis results of business access patterns, it predicts the low-load time window of the database and schedules the scan task to be executed during the time period with the least impact on business, thereby achieving a balance between business continuity and scan efficiency.

[0125] In another embodiment, the task parsing module is also used for resume scanning from the point of interruption. By maintaining a persistent record of the task execution progress, the module can resume execution from the point of interruption after the task is interrupted and restarted.

[0126] The functions and implementation methods of the above modules are the same as the corresponding steps in the aforementioned method embodiments, and will not be repeated here.

[0127] The sensitive data automatic discovery system provided by this invention automates and concurrently processes sensitive data discovery tasks through a queue-driven task scheduling mechanism, significantly improving the efficiency of task discovery and procedure execution. Dynamic SQL concatenation technology enables the system to flexibly adapt to different database structures and scanning requirements, enhancing its versatility. Real-time progress tracking and result recording mechanisms ensure the transparency of task execution and the integrity of results, facilitating auditing and compliance management. Furthermore, automated connection resource management effectively avoids database connection leaks, improving the system's stability and reliability.

[0128] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0129] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0130] like Figure 3 This is a schematic diagram of a computer device provided for an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 3 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0131] A computer device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0132] The processor may be a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the aforementioned methods for automatically discovering sensitive data in the embodiments of this disclosure.

[0133] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0134] like Figure 3 As shown, a computer device may include a processor (such as a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from storage devices into random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0135] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 3 A computer apparatus with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0136] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the sensitive data automatic discovery method of embodiments of this disclosure are performed.

[0137] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0138] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the aforementioned methods for automatically discovering sensitive data according to embodiments of the present disclosure are performed.

[0139] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0140] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0141] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0142] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0143] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0144] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0145] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0146] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0147] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for automatically discovering sensitive data, characterized in that, include: Monitor the task queue to identify tasks that require processing sensitive data. The sensitive data is analyzed to discover the task and determine the unique task ID and scanning parameter configuration information. Get the current number of concurrent tasks and system load, and dynamically adjust the connection pool size; get the version information, characteristic strings and error message patterns of the database response, automatically identify the type and version of the target database, and dynamically generate an adaptive connection strategy based on database fingerprinting technology; A connection is established with the target database based on the unique task ID; Extract the configuration information of the sensitive data discovery task, determine the target database table name and target field name, and generate intelligent sampling strategy and data query SQL statement; wherein, the total number of records in the target database table is obtained through pre-query, and the intelligent sampling strategy is generated according to the preset sampling rate threshold and table size classification standard; According to the intelligent sampling strategy, the data query SQL statement is executed to extract data samples from the target database table; sensitive information pattern matching and identification are performed on the data samples to determine the sensitive information pattern corresponding to the data samples; the acquired data samples are input into a multi-level recognition engine for sensitive information detection; wherein the multi-level recognition engine selects a detection mode based on the sensitive information pattern; and a discovery result and data evolution tracking system are generated based on the detection. The findings are written into a results table, and a data lineage map is generated based on the findings and the data evolution tracking system. Update the execution progress of the sensitive data discovery task, close the database connection after completion, and update the status of the sensitive data discovery task to complete; The multi-level recognition engine includes a data version management mechanism; wherein, the data version management mechanism tracks the change trajectory and propagation path of sensitive data by comparing data snapshots at different time points, and generates the data evolution tracking system across time dimensions; The multi-level recognition engine also includes a regular expression matching engine, a machine learning model and contextual semantic analysis mechanism, a dynamic pattern learning mechanism, a data correlation reasoning mechanism, a business scenario perception recognition engine, and a data spoofing detection mechanism.

2. The method according to claim 1, characterized in that, The monitoring task queue retrieves sensitive data discovery tasks to be processed, including: Establish a continuous monitoring service to scan the pending objects in the task queue in real time; The sensitive data discovery task to be processed is obtained by classifying and filtering based on object type identifiers.

3. The method according to claim 1 or 2, characterized in that, After the monitoring task queue acquires the sensitive data to be processed and discovers the task, it also includes: Based on the preset data sensitivity level mapping table and business importance weight matrix, the task priority score is dynamically calculated; Monitor the resource utilization of each execution node and determine the optimal node allocation scheme through adaptive load balancing; Based on the optimal node allocation scheme and the task priority score, the execution node and the corresponding sensitive data discovery task are determined.

4. The method according to claim 1, characterized in that, The scanning parameter configuration information includes the target data source address, authentication information, and scanning range.

5. The method according to claim 1, characterized in that, The step of extracting the configuration information of the sensitive data discovery task, determining the target database table name and target field name, and generating intelligent sampling strategies and data query SQL statements also includes: Extract the configuration information of the sensitive data discovery task to determine the target database table name and the target field name; Based on the target database table name and the target field name, the data query SQL statement is dynamically generated by concatenating the SQL template engine. Analyze the data distribution characteristics and clustering patterns of the target database table, divide the target database table into N data subsets with similar characteristics, and generate a parallel scan queue.

6. The method according to claim 1, characterized in that, The regular expression matching engine performs basic pattern recognition; the machine learning model infers and judges complex data patterns; the contextual semantic analysis mechanism evaluates the sensitive features of the data; the dynamic pattern learning mechanism, based on the analysis of historical recognition results, automatically mines new sensitive data feature patterns and integrates effective patterns into the recognition rule base through pattern verification and confidence evaluation; the data association reasoning mechanism, by constructing a dependency graph between fields, analyzes the logical and business semantic relationships of the data and identifies the sensitivity of data combinations; the business scenario awareness recognition engine, based on a pre-configured industry feature library and business domain knowledge graph, dynamically adjusts the sensitivity threshold and matching rule weight of the recognition strategy; the data spoofing detection mechanism, through pattern restoration algorithms and encoding recognition technology, restores data that has undergone Base64 encoding and / or hashing and / or format transformation.

7. The method according to claim 1, characterized in that, The step of writing the findings into a results table and generating a data lineage map based on the findings and the data evolution tracing system includes: The findings are written into a result table and stored according to a predefined result data model; Construct a result confidence assessment mechanism and calculate a comprehensive confidence score based on confidence indicators in the identification process; Anomaly detection algorithms automatically flag findings that may be misjudged. The findings that may have been misjudged are discarded to obtain corrected findings. Based on the corrected findings and the data evolution tracking system, intelligent deduplication and association analysis are used to identify duplicate sensitive data instances and establish a data lineage graph through data fingerprint calculation and similarity matching algorithms. The data lineage map includes the associations and flow paths of sensitive data in different storage locations.

8. The method according to claim 1, characterized in that, The process of updating the execution progress of the sensitive data discovery task, closing the database connection after completion, and updating the status of the sensitive data discovery task to complete includes: After writing the findings into the results table, update the execution progress of the sensitive data discovery task; Determine whether the sensitive data discovery task has been completed; If not completed, the scanning and extraction of data samples will be performed in a loop according to the target database table name and the target field name, and the sensitive information pattern matching and identification will be performed on the data samples. If completed, the connection to the target database is closed; Update the status of the sensitive data discovery task to complete.

9. A sensitive data automatic discovery system, characterized in that, include: The task monitoring module is used to acquire sensitive data to be processed and identify tasks. The task parsing module is used to parse the sensitive data discovery task, determine the unique task ID and scanning parameter configuration information; obtain the current number of concurrent tasks and system load, and dynamically adjust the connection pool size; obtain the version information, characteristic strings and error message patterns of the database response, automatically identify the type and version of the target database, and dynamically generate an adaptive connection strategy based on database fingerprinting technology. A connection is established with the target database based on the unique task ID; The query statement generation module is used to extract the configuration information of the sensitive data discovery task, determine the target database table name and target field name, and generate intelligent sampling strategies and data query SQL statements; wherein, the total number of records in the target database table is obtained through pre-query, and the intelligent sampling strategy is generated according to the preset sampling rate threshold and table size classification standard; The query statement execution module is used to execute the data query SQL statement according to the intelligent sampling strategy, extract data samples from the target database table; perform sensitive information pattern matching and identification on the data samples to determine the sensitive information pattern corresponding to the data samples; input the acquired data samples into a multi-level recognition engine for sensitive information detection; wherein the multi-level recognition engine selects a detection mode based on the sensitive information pattern; and generates discovery results and a data evolution tracking system based on the detection. The result writing module is used to write the discovery results into a result table and generate a data lineage map based on the discovery results and the data evolution tracking system. The task status update module is used to update the execution progress of the sensitive data discovery task. After the execution is completed, the database connection is closed and the status of the sensitive data discovery task is updated to complete. The multi-level recognition engine includes a data version management mechanism; wherein, the data version management mechanism tracks the change trajectory and propagation path of sensitive data by comparing data snapshots at different time points, and generates the data evolution tracking system across time dimensions; The multi-level recognition engine also includes a regular expression matching engine, a machine learning model and contextual semantic analysis mechanism, a dynamic pattern learning mechanism, a data correlation reasoning mechanism, a business scenario perception recognition engine, and a data spoofing detection mechanism.

Images