Data scanning method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data scanning method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: scanning a data source to obtain a sampling data set; aiming at each piece of target data in the sampling data set, matching the target data by adopting a sensitive rule regular expression, if a matching result shows that sensitive data exists in the target data, recording the hit times of the sensitive data, and obtaining the total hit times after the total number of target data in the sampling data set is matched; determining a hit rate according to the total hit times and the total number; and generating risk alarm information under the condition that the hit rate is greater than a hit rate threshold value. And when the hit rate is greater than the hit rate threshold value, the risk alarm information is generated to inform the maintenance personnel to carry out risk processing in time, so that unnecessary risk alarms are reduced, and data assets in a risk set are processed preferentially.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, and in particular to a data scanning method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of Internet technology, a large amount of data is generated. Generally, sensitive data needs to be filtered out from the large amount of data to avoid sensitive data leakage and improve data security. The traditional data scanning tool scans and identifies sensitive data from a database or a log file, and then generates an alarm, resulting in an alarm information flood. SUMMARY

[0003] Therefore, it is necessary to provide a data scanning method, device, computer equipment, computer readable storage medium and computer program product capable of avoiding alarm information flooding in view of the above technical problems.

[0004] In a first aspect, the present application provides a data scanning method, comprising:

[0005] scanning a data source to obtain a sample data set;

[0006] For each target data in the sample data set, a sensitive rule regular expression is used to match the target data. If the matching result indicates that the target data contains sensitive data, the number of hits of sensitive data is recorded. After matching the total number of target data in the sample data set, the total number of hits is obtained;

[0007] determining a hit rate according to the total number of hits and the total number;

[0008] generating a risk alarm information if the hit rate is greater than a hit rate threshold.

[0009] In one embodiment, the sensitive rule regular expression is used to match the target data. If the matching result indicates that the target data contains sensitive data, the number of hits of sensitive data is recorded, comprising:

[0010] using a sensitive rule regular expression to match the target data to obtain an initial matching result;

[0011] using a secondary verification rule to screen the initial matching result to obtain a target matching result; the secondary verification rule is a secondary verification function for excluding values of preset sensitive data;

[0012] If the target matching result indicates that the target data contains sensitive data, the number of hits of sensitive data is recorded.

[0013] In one of the embodiments, the secondary verification rule constitutes a plug-in verification module; the screening of the initial matching result by using the secondary verification rule to obtain the target matching result comprises:

[0014] The initial matching result is screened by using the plug-in verification module to obtain the target matching result; wherein the plug-in verification module comprises at least one of a neural network model and a hard-coded verification module.

[0015] In one of the embodiments, before the matching of the target data by using the sensitive rule regular expression, the method further comprises:

[0016] According to each target data in the sample data set, a callback function is used to construct a whitelist information object according to the target data;

[0017] The whitelist information object is matched with the corresponding whitelist rule;

[0018] If the matching fails, the step of matching the target data by using the sensitive rule regular expression is performed;

[0019] If the matching succeeds, the sensitive rule regular expression matching of the target data is skipped.

[0020] In one of the embodiments, the determination of the hit rate according to the total number of hits and the total number comprises:

[0021] The total number of hits corresponding to each field is obtained, and the ratio of the total number of hits corresponding to each field to the total number is taken as the hit rate corresponding to each field.

[0022] In one of the embodiments, the determination of the hit rate according to the total number of hits and the total number comprises:

[0023] The weight of each field and the total number of hits corresponding to each field are obtained;

[0024] The hit rate is determined according to the weight of each field, the total number of hits corresponding to each field and the total number, or the hit rate is determined according to the weight of each field, the total number of hits corresponding to each field, the total number and the data source weight.

[0025] In one of the embodiments, the scanning of the data source to obtain the sample data set comprises:

[0026] If the data source is a database, a database scanning function is used to scan the database to obtain the sample data set;

[0027] If the data source is a log file, a log scanning function is used to scan the log to obtain the sample data set.

[0028] In a second aspect, the present application provides a data scanning device, comprising:

[0029] a scanning module, configured to scan a data source to obtain a sample data set;

[0030] an analysis module, configured to, for each target data in the sample data set, match the target data with a sensitive rule regular expression, and if the matching result indicates that the target data contains sensitive data, record the hit number of sensitive data, and after matching the total number of target data in the sample data set, obtain a total hit number;

[0031] an evaluation module, configured to determine a hit rate according to the total hit number and the total number;

[0032] an alarm generation module, configured to generate risk alarm information if the hit rate is greater than a hit rate threshold.

[0033] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0034] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the method of the first aspect when executed by a processor.

[0035] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and the computer program implements the method of the first aspect when executed by a processor.

[0036] The data scanning method, device, computer device, computer readable storage medium and computer program product have the following advantages. The sample data set is obtained by scanning the data source. For each target data in the abstract data set, the sensitive rule regular expression is matched. If the matching result indicates that the target data contains sensitive data, the hit number of sensitive data is recorded. After matching the total number of target data in the sample data set, the total hit number is obtained. The hit rate is determined according to the total hit number and the total number. If the hit rate is greater than the hit rate threshold, the risk alarm information is generated to inform the maintenance personnel so as to timely process the risk, reduce unnecessary risk alarms, preferentially process the data assets in the risk set, and improve the efficiency and effect of security governance. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained from these drawings without creative effort.

[0038] Figure 1 An application environment diagram of the data scanning method in one embodiment;

[0039] Figure 2 A flowchart of the data scanning method in one embodiment;

[0040] Figure 3 A flowchart of the data scanning method in another embodiment;

[0041] Figure 4 A structural block diagram of the data scanning device in one embodiment;

[0042] Figure 5 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purposes, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0044] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" used in the present application and any variations thereof are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application means two and more than two. The term "and / or" used in the present application means one of the solutions, or any combination of a plurality of solutions.

[0045] The data scanning method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment is shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, unmanned aerial vehicles, low-altitude aircraft, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart vehicle device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0046] In an exemplary embodiment, as shown in Figure 2 , a data scanning method is provided, which is applied to the server or terminal in Figure 1 for example, including the following steps 202 to 206. Among them:

[0047] Step 202, scanning the data source to obtain a sample data set.

[0048] The data source can be a database or a log file, etc. Scanning the database or log text can obtain a plurality of target data to obtain a sample data set. The database can be MySQL (Structured Query Language), Hive or MongoDB, etc. MySQL is a relational database. Hive is a data warehouse tool based on Hadoop. MongoDB is a document-oriented NoSQL database based on distributed file storage. The log file can be a log file on the network or local device.

[0049] Exemplarily, the processor can call a task processing function corresponding to the data source to scan the data source to obtain the sample data set.

[0050] Step 204, for each target data in the sample data set, a sensitive rule regular expression is used to match the target data, and if the matching result indicates that the target data contains sensitive data, the hit number of sensitive data is recorded. After matching the total number of target data in the sample data set, the total number of hits is obtained.

[0051] A sensitive rule regular expression is a regular expression used to identify sensitive data, such as regular expressions for matching ID card numbers, mobile phone numbers, driver's license numbers, license plate numbers, bank card numbers, email addresses, and residential addresses. Sensitive rule regular expressions can be pre-compiled or represented as a list of sensitive rule regular expressions (such as a SensitiveRuleMap). Sensitive data can include, but is not limited to, ID card numbers, phone numbers, bank account numbers, email addresses, user addresses, license plate numbers, and vehicle VIN codes. The total quantity refers to the total number of target data items in the sampled dataset.

[0052] For example, the processor can pre-load the corresponding sensitive rule regular expressions from the configuration file where the data source is located, then read each target data in the abstract data set, transmit the read target data to the core analysis engine for detection, and use the core analysis engine to match each target data using the sensitive rule regular expressions. If a match is successful, it indicates that the target data contains sensitive data, and the number of times the sensitive data is matched is recorded. Since the data in the data source is recorded in rows, each target data in the sampled data set can be a row of data. For example, if the data source is a database, a target data can be a row of data in a data table. If the data source is a log file, a target data can be a row of log record data, etc. Accordingly, the total number can be the total number of sampled rows.

[0053] For example, each type of sensitive data can have its own hit count recorded. If a type of sensitive data is successfully matched within each target data set, then that type of sensitive data is recorded as having hit once. After matching a total number of target data sets, the total hit count for each type of sensitive data can be calculated. For instance, if the total number of target data sets is 10,000, each target data set is retrieved sequentially. If the first target data set contains an ID card number, then the ID card number type of sensitive data is recorded as having hit once; if the first target data set contains a mobile phone number, then the mobile phone number type of sensitive data is recorded as having hit once; if the second target data set contains an ID card number, then the ID card number type of sensitive data is recorded as having hit once, and the total hit count for the ID card number type of sensitive data is updated to 2, and so on. After matching all 10,000 target data sets, the total hit count for each type of sensitive data can be obtained.

[0054] For example, sensitive data can be unclassified. If sensitive data exists in each target data, it is recorded as a hit. After matching the total number of times, the total number of times sensitive data exists can be counted and used as the total number of hits.

[0055] Step 206: Determine the hit rate based on the total number of hits and the total number of hits.

[0056] For example, the hit rate can be determined using either a linear function or a non-linear function based on the total number of hits and the total number of targets. For instance, the hit rate can be the ratio of the total number of hits to the total number of targets, or the ratio of the total number of hits to the number of targets multiplied by a coefficient. Alternatively, weights can be configured according to the security level of different sensitive data, and the overall hit rate can be obtained by weighting the sensitive data with its corresponding total hit rate. The hit rate can then be determined based on the overall hit rate and the total number of targets.

[0057] Step 208: If the hit rate is greater than the hit rate threshold, generate a risk alarm message.

[0058] The hit rate threshold can be set as needed or determined based on historical statistical results. Risk alert information can be represented in different forms, such as using different color markings, like yellow, red, or green. Preset color markings can be applied to field values, the boxes containing field values, or rows containing sensitive data with hit rates exceeding the hit rate threshold. For example, risk alert information can be character markers, such as "high risk" or "medium risk," with text markers added to the corresponding positions of field values.

[0059] For example, the processor generates a risk alarm message when the hit rate is greater than a hit rate threshold, and can ignore it when the hit rate is less than or equal to the hit rate threshold. The risk alarm message can be reported as a risk alarm.

[0060] In this embodiment, a sampled data set is obtained by scanning the data source. For each target data in the abstract data set, a sensitive rule regular expression is used for matching. If the matching result indicates that sensitive data exists in the target data, the number of times the sensitive data is matched is recorded. After matching a total number of target data in the sampled data set, the total number of matches is obtained. The hit rate can be determined based on the total number of matches and the total number of matches. If the hit rate is greater than the hit rate threshold, a risk alarm message is generated to notify maintenance personnel so that risk can be handled in a timely manner. This reduces unnecessary risk alarms, prioritizes the handling of data assets with concentrated risks, and improves the efficiency and effectiveness of security governance.

[0061] In one exemplary embodiment, scanning a data source to obtain a sampled data set includes: if the data source is a database, then using a database scanning function to scan the database to obtain the sampled data set.

[0062] The database can include one or more of MySQL, Hive, or MongoDB. For example, the corresponding database scan function is obtained based on the database type, and the database is scanned using the corresponding database scan function to obtain a sample data set.

[0063] If the database is MySQL, the DoMySQLScan function is used to scan a single MySQL instance to obtain a sampled data set. This can include: obtaining all database names under the target instance; filtering out system database names from all database names using the built-in no-scan database mapping table to obtain the business database names under the target instance; obtaining all table names in the business database corresponding to each business database name; and retrieving the target data from the data tables corresponding to each table name to obtain the sampled data set.

[0064] The database mapping table that does not need to be scanned can be `notNeedScanDatabaseMap`, which has a built-in list / dictionary (Map is a key-value pair structure) to store the names of system databases that do not need to be scanned. System database names can be the MySQL library included with MySQL, information_schema, etc.

[0065] For example, the MySQL scanning module first calls the `DoMySQLScan` function to call `GetDataBase` (the database retrieval function) to obtain all database names under the target instance, and filters out system database names based on the built-in `notNeedScanDatabaseMap`. Next, for each business database, it calls the `GetTables` function to retrieve all table names. Finally, for each table, it calls the `LoadDbData` function, which executes `SELECT * FROM ... LIMIT 10000` (an SQL query to retrieve the first 10,000 rows of data in the table) to obtain sampled data. Then, it reads each row of data line by line using `rows.Scan` (row data scanning). After each row of data is encapsulated, the value of each field in each row is passed to the core analysis engine for detection. Sensitive rule regular expressions are used to match the value of each field in each row of data. Business databases can be user-created databases used to store business data, such as e-commerce order databases or social media user databases.

[0066] If the database is Hive, the DoHiveScan function is used to scan a single Hive table. This process can include: obtaining a connection from the connection pool; performing Kerberos authentication; constructing and executing an HQL query statement using the gohive library; and retrieving the data retrieved by the HQL query statement to obtain a sampled data set.

[0067] For example, the Hive scanning module obtains a connection from a pre-initialized Hive connection pool (ConPoolChan); then, it performs Kerberos authentication via kinit, constructs an HQL query statement of `select * from ... limit 10000`, and executes the HQL query statement using the gohive library; it then retrieves the data obtained from the HQL query statement row by row using `cursor.RowMap(ctx)`, and passes the value of each field in each row of data to the core analysis engine for detection. Sensitive rule regular expressions are used to match the value of each field in each row of data.

[0068] If the database is MongoDB, the DoMongoScan function is used to scan a single MongoDB instance to obtain a data set. This may include: connecting to the target MongoDB instance, traversing all databases under the target MongoDB instance, and then traversing the collections under each database; for each collection, obtaining the total number of documents in the collection to obtain a sampled data set.

[0069] A MongoDB database instance refers to a running MongoDB service process; multiple instances can be deployed on a single server.

[0070] For example, the MongoDB scanning module calls the DoMongoScan function to connect to the target MongoDB instance and traverses all databases and collections under the target MongoDB instance. For each collection, it uses find().SetSort(BSON.D{{"_id", -1}}).SetLimit(10000) to retrieve the latest 10,000 BSON (Binary JavaScriptObject Notation) documents; for each document, it calls the getStructureTree function, which recursively extracts data from embedded documents (sub-documents nested within documents) and RravelPrimitiveA (array parsing function, specifically handling arrays (list-type data) in BSON, recursively extracting data from each element in the array) until the leaf nodes of the BSON tree are reached; then, the values ​​of the leaf nodes (such as strings, numbers, etc.) are extracted and passed to the core analysis engine for detection. Use sensitive rule regular expressions to match the values ​​of leaf nodes.

[0071] In MongoDB, a database is a container used to isolate different business data. Containers are similar to the concept of a "database" in MySQL, such as an "e-commerce database" or a "social database." A collection is a container used to store documents (similar to a "table" in MySQL, but without a fixed structure). A database can contain multiple collections. `find().SetSort` is a function that sets the sorting rules to sort the query results; the parameter is the sorting rule. `BSON.D` (a key-value pair structure of BSON documents) `{{"_id", -1}}` means "sorted in descending order by _id".

[0072] In one exemplary embodiment, scanning a data source to obtain a sampled data set includes: if the data source is a log file, then using a log scanning function to scan the logs to obtain the sampled data set.

[0073] The log scanning function (DoLogScan function) is responsible for scanning the logs of a single host to obtain a sampled data set. This may include: obtaining the server IP address where the application is located from the configuration management database; calling the function to obtain scan data to remotely connect to the target server corresponding to the server IP address; obtaining the log file on the target server; and sampling a preset number of log contents from the log file to obtain the sampled data set.

[0074] For example, the log scanning module first retrieves the IP address of the server where the application is deployed from the CMDB (Configuration Management Database) by calling the log scanning function through QueryAppHost. Then, it calls the GetScanData function, which internally uses SSH (Secure Shell) to connect to the target server corresponding to the IP address via executeRemoteCommand. On the target server, it executes find (a Linux system search command) / xx / logs / ... -name "application.log (log file name format).*" to locate the log file, and then executes tail -n10000 (view the last 10000 lines of the file) on each found file to obtain a preset number of the latest log entries. Each line of log text is then passed to the core analysis engine for analysis. Sensitive rule regular expressions are used to match each line of log text.

[0075] By using corresponding task processing functions to process different data sources, the system was able to parse and scan data in complex nested structures such as JSON and BSON, thus reducing the false negative rate.

[0076] In one exemplary embodiment, scanning a data source to obtain a sampled data set includes: obtaining a task list for scanning the data source; iterating through each task in the task list; upon reading a task, obtaining a token from a buffered task pool channel; if a token is obtained from the task pool channel, executing the task; otherwise, not executing the task. After the task is completed, the token corresponding to the task is released back into the task pool channel.

[0077] Initialization: When the system starts, it enters the corresponding task processing function, such as DbScanTask, according to the scan type specified by the user. Concurrency control: Inside the task processing function, the setTaskNum(threadnum) function is first called to create a buffered task pool channel (chan int) as a concurrent task pool, the capacity of which is determined by the configured number of threads threadnum. This task pool channel acts as a semaphore. Task dispatch: The system obtains a list of tasks to be scanned (such as a list of target assets) from the CMDB or configuration file, iterates through the task list, and in each loop, first executes WaitTask(tastchan). This operation will take a token (element) from the task pool channel. If the channel is empty, the program will block here until another task completes and releases the semaphore. Execution unit: After obtaining the semaphore, the system uses wg.Add(1) to increase the wait group count and starts an independent goroutine to perform the specific scanning work (such as DoMySQLScan). Task completion: When a goroutine completes its scan task, it calls `ReleaseTask(tastchan)` to add a new element to the task pool channel, releasing the semaphore and allowing the next waiting task to begin execution. Simultaneously, it calls `wg.Done()` to notify the waiting group that a task has completed. Finally, the main thread uses `wg.Wait()` to wait for all concurrent tasks to finish before exiting.

[0078] In an exemplary embodiment, a sensitive rule regular expression is used to match the target data. If the matching result indicates that the target data contains sensitive data, the number of times the sensitive data is hit is recorded. This includes: matching the target data using a sensitive rule regular expression to obtain an initial matching result; screening the initial matching result using a secondary verification rule to obtain a target matching result; and recording the number of times the sensitive data is hit if the target matching result indicates that the row data contains sensitive data.

[0079] The sensitive rule regular expression can be a list of pre-loaded and compiled regular expressions (SensitiveRuleMap) that the Detect function iterates through from a configuration file. The secondary validation rule can be a function used to exclude values ​​of preset sensitive data. Preset sensitive data values ​​can include one or more of the following: phone number, email address, VIN, mailing address, etc. If a value of preset sensitive data is matched from the initial matching result using the secondary validation rule, that value is removed from the initial matching result to avoid false alarms. When a regular expression initially matches successfully, the Detect function enters a switch statement, calling a dedicated secondary validation function based on the matched rule name (e.g., "VIN", "email"). For example, the VINCheck function verifies the validity of the VIN (Vehicle Identification Number) code using a standard checksum algorithm; the EmailCheck function checks and excludes internal domain email addresses; and PhoneCheck and AddressCheck query a hard-coded Map (mapping table / key-value pair set (data structure)) to exclude known specific numbers / addresses. Only data that passes the secondary validation is ultimately confirmed as a valid match.

[0080] Through secondary logical verification, the detection results are ensured to be both accurate (low false alarm rate) and comprehensive (low false negative rate).

[0081] In an exemplary embodiment, the secondary verification rules constitute a plug-in verification module; the initial matching result is screened using the secondary verification rules to obtain the target matching result, including: the initial matching result is screened using the plug-in verification module to obtain the target matching result; wherein, the plug-in verification module includes at least one of a neural network model and a hard-coded verification module.

[0082] The secondary verification rules can be designed as plug-ins. The hard-coded verification module contains preset sensitive data values ​​to be excluded. These preset sensitive data values ​​can include one or more of the following: contact number, email address, VIN, and residential address. The hard-coded verification module can include one or more plug-in verification modules for contact numbers, email addresses, VIN addresses, and residential addresses. The neural network model can be an NLP (Natural Language Processing) model, a CNN (Convolutional Neural Network) model, an RNN (Recurrent Neural Network) model, or a large-scale model. The initial neural network model is trained by collecting sample data of the sensitive data values ​​to be excluded, resulting in a trained neural network model. This trained model is used to screen sensitive data values ​​in the initial matching results, excluding values ​​that do not trigger risk alerts, such as publicly available company phone numbers, email addresses, and mailing addresses. The modular and plug-in nature of the secondary verification rules allows the system to easily call appropriate plug-ins or modules according to its needs and capabilities, ensuring high system performance and easy scalability.

[0083] In an exemplary embodiment, before matching the target data using sensitive rule regular expressions, the method further includes: constructing a whitelist information object based on each target data in the sampled data set using a callback function; matching the whitelist information object with the corresponding whitelist rule; if the matching fails, performing the step of matching the target data using sensitive rule regular expressions; if the matching succeeds, skipping the sensitive rule regular expression matching of the target data.

[0084] When the scanning module calls the `sensitive.Detect` function, it passes in a context-dependent callback function. The `Detect` function calls this callback before performing any matching. The callback function constructs a `WhitelistInfo` object (a whitelist information object that stores the "current scanning context" (equivalent to a "query condition package"), containing key information such as database name, table name, and field name) based on the current scan context (e.g., database name, table name, field name). Then, functions like `MatchMySQLWhiteList` use this object to perform multi-dimensional matching against the whitelist rules loaded from the database, including: 1) exact match (match all databases, tables, and fields); 2) database prefix match; 3) table prefix match; 4) both database and table prefix match. If a whitelist match is found, the `Detect` function skips all subsequent checks on that target data.

[0085] Among them, there are several matching methods: Exact match: The database name, table name, and field name are exactly the same as the whitelist rule. For example, "e-commerce database → order table → order number field" is a perfect match. Database prefix match: The database name starts with the prefix of the whitelist rule. For example, if the whitelist rule is "test_", then "test_e-commerce database" and "test_test database" will both match. Table prefix match: The table name starts with the prefix of the whitelist rule. For example, if the whitelist rule is "tmp_", then "tmp_20251203 data" and "tmp_temporary table" will both match. Both database and table prefix match: Both the database name and the table name start with the prefix of the whitelist rule. For example, if the database prefix is ​​"dev_" and the table prefix is ​​"log_", then "dev_operations database → log_access record" will match.

[0086] The context refers to the specific environment of the current scan, such as which database, table, and field are being scanned, and is the key basis for building the whitelist. The callback function is a predefined function that is "called in reverse" when `sensitive.Detect` is executed. It is used to collect the context and build whitelist query objects, such as `dbWhiteListhandle`, the database whitelist handling function.

[0087] In an exemplary embodiment, determining the hit rate based on the total number of hits and the total number of hits includes: obtaining the total number of hits corresponding to each field, and using the ratio of the total number of hits corresponding to each field to the total number of hits as the hit rate corresponding to each field.

[0088] The `OutputHandle` function, acting as the sole consumer, receives a `SenDataMapping` object containing match details from the global channel `SenDataChan` (sensitive data channel). This object contains the total number of matches (`MatchCount`) and the total number of rows scanned (`rowCount`). The `OutputHandle` function filters the total number of matches for each field, removing data for fields with a total number of matches less than a threshold. For example, it uses `IsNotEffectData` to filter out noisy data with an excessively low total number of matches (e.g., less than 5).

[0089] To calculate the hit rate of a given segment, we can determine the total number of hits for each field in a data table corresponding to a task, and then calculate the ratio of the total number of hits for that field to the total number of samples. The formula is: Rate = float64(data.MatchCount) / float64(data.rowCount). Here, Rate is the hit rate, MatchCount is the total number of hits, and rowCount is the total number of samples (i.e., the total number of rows sampled). For example, if the total number of samples is 10,000, and the total number of hits for the ID card number field is 900, then the hit rate for ID card numbers is 9%. If the total number of hits for the mobile phone number field is 500, then the hit rate for mobile phone numbers is 5%. The hit rate directly reflects the "degree of contamination" of sensitive data in that field. The hit rate, total number of samples, database name, table name, and other information can be written together into a temporary summary file.

[0090] In an exemplary embodiment, determining the hit rate based on the total number of hits and the total number of hits includes: obtaining the weight of each field and the total number of hits corresponding to each field; and determining the hit rate based on the weight of each field, the total number of hits corresponding to each field, and the total number of hits.

[0091] Different fields can be assigned the same or different weights. For example, the weight of an ID card number is greater than that of an email address. The weights of each field and the total number of hits for each field are weighted and summed. The ratio of this sum to the total number of hits is used as the hit rate. For example, Rate = Σ(field weight * total number of hits) / total number of hits.

[0092] In an exemplary embodiment, determining the hit rate based on the total number of hits and the total number of hits includes: obtaining the weight of each field and the total number of hits corresponding to each field, and obtaining the weight of each data source; determining the hit rate based on the weight of each field, the total number of hits corresponding to each field, the total number of hits, and the weight of each data source.

[0093] Different fields can be assigned the same or different weights. Different data sources can be assigned different weights. The sensitivity level of the data source can be used as a factor, with the production database having a higher discovery weight than the test database. For each data source, the weights of each field and the total number of hits corresponding to each field are weighted and summed. The ratio of the weighted sum to the total number of hits is calculated, and the hit rate is obtained by weighting the data source weight and the ratio corresponding to the data source. For example, the hit rate Rate = Σ(field weight * total number of hits) / total number of hits * data source weight. By setting different weights for different fields and data sources and dynamically calculating the hit rate, the risk detection of sensitive data can be made more accurate, unnecessary alarms can be reduced, and the efficiency of alarm handling can be improved. Maintenance resources can be concentrated on prioritizing urgent and high-risk alarms to improve the utilization of maintenance resources.

[0094] In an exemplary embodiment, the method further includes: for each target data in the sampled dataset, if the target data contains three or more sensitive data simultaneously, then the target data is recorded as high-risk sensitive data. High-risk sensitive data can refer to sensitive data with a hit rate greater than a hit rate threshold. Sensitive data can be names, mobile phone numbers, email addresses, residential addresses, ID card numbers, vehicle registration numbers, etc. By directly marking target data containing three or more sensitive data as high-risk sensitive data, the accuracy of detection is improved.

[0095] In one exemplary embodiment, such as Figure 3 As shown, a data scanning method includes:

[0096] Step 302: Scan the data source to obtain the sampled data set.

[0097] Step 304: For each target data in the sampled data set, use a callback function to construct a whitelist information object based on the target data.

[0098] Step 306: Perform multi-dimensional matching between the whitelist information object and the corresponding whitelist rule. If the match is successful, proceed to step 308 and return to step 304. If the match fails, proceed to step 310.

[0099] Step 308: Skip the sensitive rule regular expression matching of the target data.

[0100] Step 310: Use sensitive rule regular expressions to match the target data to obtain the initial matching results.

[0101] Step 312: The initial matching results are screened using a secondary verification rule to obtain the target matching results.

[0102] Step 314: If the target matching result indicates that there is sensitive data in the target data, then record the number of times the sensitive data is hit. After matching the total number of target data in the sampled data set, the total number of hits is obtained.

[0103] Step 316: Obtain the total number of hits for each field, and use the ratio of the total number of hits for each field to the total number of hits as the hit rate for each field.

[0104] Step 318: If the hit rate is greater than the hit rate threshold, generate a risk alarm message.

[0105] Step 320: If the hit rate is less than or equal to the hit rate threshold, it is considered low risk or medium risk, and no risk alarm information is generated.

[0106] By formatting and outputting the target data from the sampled data set with a hit rate greater than the hit rate threshold, and sending it to the security team in CSV and email reports, effective convergence of alerts and prioritization of risks are achieved.

[0107] In this embodiment, thousands of raw alerts are transformed into a clear and quantifiable risk list. Security teams no longer need to deal with massive amounts of homogeneous alerts; instead, they can prioritize the most concentrated and severe data assets based on hit rate, greatly improving the efficiency and effectiveness of security interventions. The combination of secondary logical verification, complex data structure traversal, and multi-dimensional whitelists ensures that the detection results are both accurate and comprehensive. Through a dual filtering mechanism based on the lowest hit count and lowest hit rate, a large number of sporadic, low-risk events that do not require human intervention are effectively filtered out, resulting in a very high signal-to-noise ratio in the final report, allowing security personnel to focus their efforts on the real risks.

[0108] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0109] Based on the same inventive concept, this application also provides a data scanning apparatus for implementing the data scanning method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data scanning apparatus embodiments provided below can be found in the limitations of the data scanning method described above, and will not be repeated here.

[0110] In one exemplary embodiment, such as Figure 4 As shown, a data scanning device 400 is provided, the device including: a scanning module 410, an analysis module 420, an evaluation module 430 and an alarm generation module 440.

[0111] The scanning module 410 is used to scan the data source to obtain a sampled data set.

[0112] The analysis module 420 is used to match each target data in the sampled data set using a sensitive rule regular expression. If the matching result indicates that the target data contains sensitive data, the number of times the sensitive data is hit is recorded. After matching a total number of target data in the sampled data set, the total number of hits is obtained.

[0113] Evaluation module 430 is used to determine the hit rate based on the total number of hits and the total quantity.

[0114] The alarm generation module 440 is used to generate risk alarm information when the hit rate is greater than the hit rate threshold.

[0115] In this embodiment, a sampled data set is obtained by scanning the data source. For each target data in the abstract data set, a sensitive rule regular expression is used for matching. If the matching result indicates that sensitive data exists in the target data, the number of times the sensitive data is matched is recorded. After matching a total number of target data in the sampled data set, the total number of matches is obtained. The hit rate can be determined based on the total number of matches and the total number of matches. If the hit rate is greater than the hit rate threshold, a risk alarm message is generated to notify maintenance personnel so that risk can be handled in a timely manner. This reduces unnecessary risk alarms, prioritizes the handling of data assets with concentrated risks, and improves the efficiency and effectiveness of security governance.

[0116] In an exemplary embodiment, the analysis module 420 is further configured to match the target data using a sensitive rule regular expression to obtain an initial matching result; to screen the initial matching result using a secondary verification rule to obtain a target matching result; the secondary verification rule is used to exclude the secondary verification function of preset sensitive data values; if the target matching result indicates that the target data contains sensitive data, then the number of times the sensitive data is hit is recorded.

[0117] In an exemplary embodiment, the secondary verification rules constitute a plug-in verification module; the analysis module 420 is further configured to use the plug-in verification module to screen the initial matching results to obtain the target matching results; wherein, the plug-in verification module includes at least one of a neural network model and a hard-coded verification module.

[0118] In an exemplary embodiment, the analysis module 420 is further configured to construct a whitelist information object based on each target data in the sampled data set using a callback function; match the whitelist information object with the corresponding whitelist rule; if the match fails, perform the matching of the target data using a sensitive rule regular expression; if the match succeeds, skip the matching of the target data using a sensitive rule regular expression.

[0119] In an exemplary embodiment, the evaluation module 430 is further configured to obtain the total number of hits corresponding to each field, and use the ratio of the total number of hits corresponding to each field to the total number as the hit rate corresponding to each field.

[0120] In an exemplary embodiment, the evaluation module 430 is further configured to obtain the weight of each field and the total number of hits corresponding to each field; determine the hit rate based on the weight of each field, the total number of hits corresponding to each field and the total number of hits, or determine the hit rate based on the weight of each field, the total number of hits corresponding to each field, the total number of hits and the data source weight.

[0121] In an exemplary embodiment, the evaluation module 430 is further configured to obtain the weight of each field and the total number of hits corresponding to each field, as well as the weight of each data source; and determine the hit rate based on the weight of each field, the total number of hits corresponding to each field, the total number of hits, and the weight of each data source.

[0122] In an exemplary embodiment, the scanning module 410 is further configured to, if the data source is a database, use a database scanning function to scan the database to obtain a sampled data set; and if the data source is a log file, use a log scanning function to scan the log to obtain a sampled data set.

[0123] In an exemplary embodiment, the scanning module 410 is further configured to obtain a task list of the scanning data source, traverse each task in the task list, and after reading a task, obtain a token from the buffered task pool channel. If a token is obtained from the task pool channel, the task is executed; if no token is obtained from the task pool channel, the task is not executed.

[0124] In an exemplary embodiment, the analysis module 420 is further configured to, for each target data in the sampled data set, record the target data as high-risk sensitive data if three or more sensitive data exist simultaneously in the target data.

[0125] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0126] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data scanning method.

[0127] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0128] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0129] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0130] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data scanning method, characterized in that, The method includes: Scan the data source to obtain a sample data set; For each target data in the sampled data set, a sensitive rule regular expression is used to match the target data. If the matching result indicates that the target data contains sensitive data, the number of times the sensitive data is hit is recorded. After matching a total number of target data in the sampled data set, the total number of hits is obtained. The hit rate is determined based on the total number of hits and the total quantity. If the hit rate is greater than the hit rate threshold, a risk alarm message is generated.

2. The method according to claim 1, characterized in that, The process involves matching the target data using sensitive rule regular expressions. If the matching result indicates that the target data contains sensitive data, the number of times the sensitive data is matched is recorded, including: The target data is matched using sensitive rule regular expressions to obtain the initial matching results; The initial matching results are screened using a secondary verification rule to obtain the target matching results; the secondary verification rule is a secondary verification function used to exclude values ​​of preset sensitive data. If the target matching result indicates that the target data contains sensitive data, then the number of times the sensitive data is matched is recorded.

3. The method according to claim 2, characterized in that, The secondary verification rules constitute a plug-in verification module; the process of using the secondary verification rules to screen the initial matching results to obtain the target matching results includes: The initial matching results are screened using a plug-in verification module to obtain the target matching results; wherein, the plug-in verification module includes at least one of a neural network model and a hard-coded verification module.

4. The method according to claim 2, characterized in that, Before matching the target data using sensitive rule regular expressions, the method further includes: For each target data in the sampling dataset, a whitelist information object is constructed using a callback function based on the target data. Match the whitelist information object with the corresponding whitelist rules; If the match fails, then the step of matching the target data using sensitive rule regular expressions is executed; If a match is found, the sensitive rule regular expression matching of the target data is skipped.

5. The method according to claim 1, characterized in that, The step of determining the hit rate based on the total number of hits and the total quantity includes: Obtain the total number of hits for each field, and use the ratio of the total number of hits for each field to the total number of hits as the hit rate for each field.

6. The method according to claim 1, characterized in that, The step of determining the hit rate based on the total number of hits and the total quantity includes: Obtain the weight of each field and the total number of hits for each field; The hit rate can be determined based on the weight of each field, the total number of hits for each field, and the total number of hits, or based on the weight of each field, the total number of hits for each field, the total number of hits, and the data source weight.

7. The method according to claim 1, characterized in that, The scanned data source obtains a sampled data set, including: If the data source is a database, then the database scan function is used to scan the database to obtain the sampled data set; If the data source is a log file, then the log scanning function is used to scan the logs to obtain a sample data set.

8. A data scanning device, characterized in that, The device includes: The scanning module is used to scan data sources to obtain a sample data set; The analysis module is used to match each target data in the sampled data set using a sensitive rule regular expression. If the matching result indicates that the target data contains sensitive data, the number of times the sensitive data is hit is recorded. After matching a total number of target data in the sampled data set, the total number of hits is obtained. An evaluation module is used to determine the hit rate based on the total number of hits and the total quantity. The alarm generation module is used to generate risk alarm information when the hit rate is greater than the hit rate threshold.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images