A method and device for detecting sensitive data
By sampling data tables and detecting sensitive data detection algorithms in big data scenarios, the problem of inefficient manual detection is solved, efficient sensitive data detection and timely monitoring in big data scenarios are realized, and data permission control and auditing are enhanced. ability.
Patent Information
- Application Number
- CN202010767486.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-03
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-08-03
AI Technical Summary
In big data scenarios, manual detection efficiency is low and it is difficult to detect data changes in time, resulting in low efficiency in control of permissions and auditing of sensitive data.
Using sensitive data detection methods and devices, the columns to be detected in the data table are sampled by pre-configuring detection execution conditions and strategies, and using sensitive data detection algorithms to detect, reduce the amount of data to be detected, improve detection efficiency, and timely perceive Data changes.
On the premise of ensuring the accuracy of detection, data detection efficiency is improved, data changes are monitored in a timely manner, and the authority control and audit capabilities of sensitive data are enhanced.
Smart Images

Figure CN111914130B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of computer application technology, and in particular, to a sensitive data detection method and device. Background Art
[0002] At present, in the big data scenario, for considerations such as privacy protection and data security, it is necessary to detect sensitive data in massive data and determine non-sensitive data and sensitive data in order to facilitate permission control and auditing of the detected sensitive data.
[0003] When detecting sensitive data, it is usually done manually. However, in big data scenarios, the amount of data is large, and manual detection alone makes the detection efficiency very low. In addition, data changes at any time, making it difficult to detect the changed data in a timely manner. Summary of the invention
[0004] In order to improve detection efficiency and detect changed data in a timely manner, this specification provides a sensitive data detection method and device. The technical solution is as follows.
[0005] A sensitive data detection method is used to perform sensitive detection on any column in a data table of source data, wherein the source data includes at least one data table; pre-configuring detection execution conditions and corresponding to-be-detected column determination strategies; the method comprises:
[0006] Determining that the source data meets any detection execution condition, wherein the detection execution condition includes: a new data table is added, a new column is added to any data table, or a name of any column in any data table is changed;
[0007] Determine a strategy for the columns to be detected corresponding to the satisfied detection execution conditions, determine the columns to be detected in the data table of the source data, and perform sampling on any column to be detected;
[0008] For the sampling result of the column to be detected, detection is performed based on any sensitive data detection algorithm, and the detection result is determined as the detection result of the column to be detected.
[0009] A sensitive data detection device is used to perform sensitive detection on any column in a data table of source data, wherein the source data includes at least one data table; a detection execution condition and a corresponding to-be-detected column determination strategy are pre-configured; the device includes:
[0010] Determining unit: determining that the source data meets any detection execution condition, wherein the detection execution condition includes: a new data table is added, a new column is added to any data table, or a name of any column in any data table is changed;
[0011] Sampling unit: determining a strategy for the columns to be detected corresponding to the satisfied detection execution conditions, determining the columns to be detected in the data table of the source data, and sampling any column to be detected;
[0012] Detection unit: performs detection on the sampling result of the column to be detected based on any sensitive data detection algorithm, and determines the detection result as the detection result of the column to be detected.
[0013] Through the above technical solution, the sampled data can be detected with the help of a machine using a sensitive data detection algorithm. In scenarios with a large amount of data, data sampling can be used to reduce the amount of data that needs to be detected while ensuring the accuracy of data detection, thereby improving the efficiency of data detection; changes in the source data can also be perceived in a timely manner, so that new data can be detected in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0015] Figure 1 It is a flowchart of a sensitive data detection method provided in an embodiment of this specification;
[0016] Figure 2 It is a flowchart of an application example of a sensitive data detection method provided in an embodiment of this specification;
[0017] Figure 3 It is a structural schematic diagram of a sensitive data detection device provided in an embodiment of this specification;
[0018] Figure 4 is a schematic diagram of the structure of a sampling unit provided in an embodiment of this specification;
[0019] Figure 5 It is a structural diagram of a device used to configure the method of the embodiment of this specification. DETAILED DESCRIPTION
[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of this specification, the technical solutions in the embodiments of this specification will be described in detail below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection.
[0021] At present, in the big data scenario, for considerations such as privacy protection and data security, it is necessary to detect sensitive data in massive data and determine non-sensitive data and sensitive data in order to facilitate permission control and auditing of the detected sensitive data.
[0022] Sensitive data may include business-related data, personal privacy data, company confidential data, etc. For example, personal information such as ID number, mobile phone number, home address, or company customer information, business secrets, etc.
[0023] Illegal modification or disclosure of sensitive data may cause losses to stakeholders. For example, when personal information is leaked, it may lead to telephone harassment, theft of bank assets, illegal business operations, etc., causing unpredictable losses.
[0024] In order to protect sensitive data, the permissions for viewing, modifying, and outputting detected sensitive data can be managed. Audits can also be conducted after illegal modifications or leaks occur to identify loopholes in data security management and remedy them.
[0025] Of course, sensitive data can be further divided into different sensitivity levels to facilitate more detailed permission control and auditing.
[0026] For the convenience of description, the data that needs to be tested for sensitive data in this specification is referred to as source data. The source data can be stored in the form of a database table, and can include at least one data table.
[0027] In the data table of the source data, a single row of data may represent a data entity, a data entity may include at least one field, and a single column of data may belong to the same field of all data entities in the data table.
[0028] Due to the large amount of data, for example, the source data may contain dozens of data tables, each of which contains millions of rows of data. Even if it is possible to detect whether each field value in each row of data is sensitive data, it is difficult to perform permission control and audit for each field value that is sensitive data in the future.
[0029] When sensitive data is detected, it is usually done manually, with each field value in each data table and each row of data in the source data being manually detected. This method is obviously inefficient.
[0030] In addition, source data may also have business processing requirements, requiring the addition, deletion, modification and query of data in the source data. In other words, source data may change, and the changes include at least the addition, deletion or modification of data. Therefore, manual confirmation and detection of the changed data is required, but it is difficult for manual detection to timely discover changes in source data, conduct detection efficiently, and obtain detection results in a timely manner.
[0031] In view of the above three technical problems, this specification provides a sensitive data detection method. The following is an explanation of the features of a sensitive data detection method provided in this specification that solve the above three problems respectively.
[0032] 1) In response to the problem that it is difficult to perform permission control and auditing in the future, this manual first takes into account that in scenarios with large data volumes, it is difficult to perform permission control and auditing on fine-grained individual field values, and it is necessary to increase the granularity of permission control and auditing, that is, it is necessary to increase the granularity of detected sensitive data; secondly, a single column of data can belong to the same field of all data entities in a data table, and sensitive data is usually not part of a data entity, but part of a field of a data entity. For example, an individual as a data entity can include fields such as name, gender, and ID number, among which the ID number is sensitive data, and the name and gender are non-sensitive data.
[0033] In a sensitive data detection method provided in this specification, the detection of sensitive data can be performed on a single column of data in any data table of source data, and the detection result may include: the single column of data is sensitive data, or the single column of data is non-sensitive data.
[0034] Performing sensitive data detection on single-column data can increase the granularity of the detected sensitive data, facilitating subsequent permission control and auditing in scenarios where the source data volume is large.
[0035] It is worth noting that although single-column data can belong to the same field, they may also have different content meanings. For example, in the account field, the corresponding single-column data may include mobile phone numbers, email addresses, user names, etc. Among them, mobile phone numbers can be sensitive data, while user names and email addresses are not sensitive data. In other words, the data with various content meanings contained in the single-column data are not necessarily sensitive data, and the final detection result of the single-column data can be determined based on the detection result of each data in the single-column data.
[0036] 2) To address the problem of low efficiency of manual detection, in a sensitive data detection method provided in this specification, a machine can be used to detect sensitive data based on any sensitive data detection algorithm, thereby improving detection efficiency.
[0037] 3) In order to solve the problem of difficulty in obtaining test results in a timely manner, in a sensitive data detection method provided in this specification, a machine can monitor whether the source data has changed, and the machine can detect sensitive data for the changed data. Since the efficiency of machine monitoring and detection is very high, the test results can be obtained in a timely manner.
[0038] Based on the above description, this specification provides a sensitive data detection method for performing sensitive detection on any column in a data table of source data, which may include: when a change is detected in any column in any data table of the source data, sampling the data of any changed column; performing detection based on the sampling result of the column based on any sensitive data detection method, and using the detection result as the detection result of the column.
[0039] Among them, since the source data has a large amount of data, when detecting any column of data, the data in the column is sampled, and the detection result of the sampling result is used as the detection result of the column. This can reduce the amount of data that needs to be detected while ensuring the correctness of the detection result, thereby further improving the detection efficiency.
[0040] In order to explain in more detail a sensitive data detection method provided in this specification, such as Figure 1 The figure is a flow chart of a sensitive data detection method provided in this specification, which is used to perform sensitive detection on any column in a data table of source data, and the source data may include at least one data table.
[0041] Pre-configure the detection execution conditions and the corresponding to-be-detected column determination strategy.
[0042] The detection execution conditions may at least include changes in source data, which can be used to perform sensitive data detection when the detection execution conditions are met.
[0043] The columns to be detected may at least include columns that have changed correspondingly when the source data meets the detection execution condition. The determined columns to be detected may be subjected to sensitive data detection in subsequent steps.
[0044] The correspondence between the detection execution conditions and the determination strategies of the columns to be detected can be shown in the following table 1. Of course, the following table only contains some examples of the correspondence, which are used for illustrative purposes and cannot limit the scope of the disclosure of this specification.
[0045]
[0046] Table 1. Correspondence between detection execution conditions and strategies for determining the columns to be detected
[0047] Figure 1 The method shown may include at least the following steps.
[0048] S101: Determine whether source data satisfies any detection execution condition.
[0049] S102: Determine a strategy according to the to-be-detected columns corresponding to the satisfied detection execution conditions, determine the to-be-detected columns in the data table of the source data, and perform sampling on any to-be-detected column.
[0050] S103: Perform detection on the sampling result of the column to be detected based on any sensitive data detection algorithm, and determine the detection result as the detection result of the column to be detected.
[0051] In S101, by real-time monitoring of source data, the execution of S101 can be triggered in real time when the source data meets any detection execution condition.
[0052] Of course, the source data may satisfy at least two detection execution conditions at the same time, and correspondingly, the to-be-detected columns may be determined according to at least two corresponding to-be-detected columns determination strategies.
[0053] The method of real-time monitoring whether the source data meets any detection execution condition is not limited in this specification. The following three examples correspond to some of the detection execution conditions in Table 1 above to achieve real-time monitoring, which is only for illustrative purposes.
[0054] a) The three detection execution conditions of adding a new data table, adding a new column to any data table, or changing the name of any column in any data table all refer to changes in the structure of the source data, which can be monitored in real time by synchronizing with the structure of the source data.
[0055] Specifically, it can be a combination of real-time synchronization and scheduled synchronization. Real-time synchronization may include: when the structure of the source data changes, real-time notification is provided to facilitate determination that the source data meets any detection execution condition. Scheduled synchronization may include: at a preset periodic time point, checking whether the structure of the source data has changed, and if so, determining that the source data meets any detection execution condition.
[0056] By combining the two synchronization methods, real-time monitoring can be achieved and the failure of real-time monitoring caused by real-time synchronization failure can be avoided.
[0057] b) For the detection execution condition of updating the version of any sensitive data detection algorithm, it is possible to monitor in real time whether the algorithm version is updated. A method combining real-time synchronization and scheduled synchronization can also be used, which will not be described in detail here.
[0058] c) For the two detection execution conditions that the duration between the current time point and the time point of the last detection is greater than the preset duration, or the current time point is the same as the preset time point for periodic detection, a timer can be set to determine whether the source data meets any of the detection execution conditions when sensitive data detection has not been performed for a long time or when the preset periodic time point is reached.
[0059] These two detection execution conditions can be targeted at non-structural changes in the source data. Specific non-structural changes may include the addition and deletion of data, for example, adding data to a single column of data, adding data entities to a data table, etc. Since non-structural changes may also cause changes in the sensitive data detection results of the single column of data, it is necessary to periodically detect whether there are changes in the sensitive data detection results caused by non-structural changes. Specifically, how non-structural changes cause changes in sensitive data detection results is explained in S103.
[0060] In S102 , before sampling, for any column to be detected, it can be determined whether the name of the column to be detected exists in the pre-stored sensitive data name set.
[0061] If the name of the column to be detected exists in the pre-stored sensitive data name set, the detection result of the column to be detected is determined; if the name of the column to be detected does not exist in the pre-stored sensitive data name set, sampling is performed on the column to be detected.
[0062] For example, the pre-stored sensitive data name set may include: ID number, mobile phone number, bank password, password, etc. When the name of any column to be detected is ID number, it is obvious that the content of the data in the column to be detected is the ID number, and subsequent sampling and sensitive data detection are not required. It can be directly determined that the column to be detected is sensitive data.
[0063] The sampled data may be stored using a column dimension to facilitate the subsequent processing in S103. Specifically, the data sampled from a single column to be detected may be stored using a dynamic array, and the obtained array may be processed in the subsequent S103.
[0064] The specific sampling method is described after the individual steps are explained.
[0065] Because in S102, when the source data meets the detection execution conditions, sampling and subsequent sensitive data detection are only performed on the columns to be detected corresponding to the satisfied detection execution conditions, and sampling or subsequent sensitive data detection will not be performed on other columns that do not belong to the columns to be detected; at the same time, the columns to be detected that belong to sensitive data can be determined in advance through a pre-stored set of sensitive data names, and there is no need to perform sampling and subsequent sensitive data detection, thereby further improving the efficiency of sensitive data detection and obtaining detection results in a more timely manner.
[0066] Regarding S103, the following is an explanation from three aspects: sensitive data detection algorithm, specific detection process, and algorithm version update.
[0067] 1. Sensitive data detection algorithm.
[0068] The sensitive data detection algorithm may include a data content identification algorithm, and the content identification algorithm may specifically include a regular expression algorithm, a drools rule, an xgboost algorithm, etc. After identifying the content of the data, it may be determined whether the identification result belongs to sensitive data according to a preset sensitive data content set.
[0069] In the specific process of identifying the data content, the data may be matched with a plurality of pre-stored data content forms one by one to determine at least one data content form that is successfully matched.
[0070] The regular expression algorithm may be to match the data with a plurality of pre-stored regular expressions one by one. If the match is successful, it may be determined that the data may contain the content meaning corresponding to the regular expression that matches successfully.
[0071] For example, for the content meaning of mobile phone number, the regular expression can be ^1(3|4|5|7|8)\d{9}$. In the data containing 11 digits, the first digit can only be 1, the second digit can be 3, 4, 5, 7 or 8, and the following 9 digits can be any digit from 0 to 9. If a certain data matches the regular expression, it means that the content meaning of the data may be a mobile phone number, and then the data is determined to be sensitive data.
[0072] It is worth noting that since different data content forms may match the same data at the same time, the same data can contain different content meanings and can also be detected as sensitive data and non-sensitive data at the same time.
[0073] For example, for a string "Wang X", it can match both the content form of a name (regular expression) and the content form of a username (regular expression). Therefore, "Wang X" can have two content meanings, namely a name and a username. In the case where the name belongs to sensitive data and the username belongs to non-sensitive data, "Wang X" can be detected as both sensitive data and non-sensitive data simultaneously.
[0074] 2. Specific detection process.
[0075] In the specific detection process, the following will explain three aspects respectively: pre-identification before sensitive data detection, how to obtain the detection result of the sampling result after detecting each data in the sampling result using the above-mentioned sensitive data detection algorithm, and how to calculate the proportion of sensitive data.
[0076] 1) Pre-identification.
[0077] Before conducting sensitive data detection on the sampling result, pre-identification can be carried out first to determine the characteristics of each data in the sampling result, so as to reduce the number of detections, speed up the detection speed, and improve the detection efficiency in the subsequent sensitive data detection algorithm.
[0078] For example, when it is pre-identified that a certain data in the sampling result contains 15 digits, then this data must not be a mobile phone number, and in the subsequent sensitive data detection algorithm, there is no need to conduct mobile phone number detection, thus speeding up the detection speed and improving the detection efficiency.
[0079] 2) How to obtain the detection result of the sampling result.
[0080] When conducting detection based on any sensitive data detection algorithm, specifically, it can be to use a data content recognition algorithm to perform data content recognition on each data in the sampling result of the column to be detected, and determine whether the data is sensitive according to the recognition result; then, according to the proportion of sensitive data in the sampling result of the column to be detected, determine the detection result of the sampling result of the column to be detected.
[0081] For example, when the proportion of sensitive data exceeds the preset threshold, it can be determined that the sampling result of the column to be detected belongs to sensitive data, and further determine that the column to be detected belongs to sensitive data.
[0082] Corresponding to the explanation in the above S101 that "non-structural changes may also cause changes in the sensitive data detection results of single-column data", non-structural changes in the source data can also cause changes in the sensitive data detection results of the column to be detected. When the data in a single column to be detected includes sensitive data and non-sensitive data, the proportion of sensitive data may change with the addition and deletion of data in the column, thus causing changes in the sensitive data detection results of the column.
[0083] A specific example may be a column to be detected corresponding to the account field, which may include mobile phone numbers, email addresses, and user names. Among them, mobile phone numbers are sensitive data, and email addresses and user names are non-sensitive data. When the proportion of mobile phone numbers in the data of the account column exceeds 30%, the detection result of the account column is sensitive data, otherwise it is non-sensitive data.
[0084] In response to this change, we can perform regular monitoring and sensitive data detection by detecting that the duration between the current time point in the execution condition and the time point of the last detection is greater than the preset duration, or that the current time point is the same as the preset, periodically executed detection time point, to ensure that the detection results are updated in a timely manner as the unstructured source data changes.
[0085] 3) How to calculate the proportion of sensitive data?
[0086] It should be noted that since the same data may be detected as sensitive data and non-sensitive data at the same time, when determining the proportion of sensitive data in a single column of data, this specification does not limit the calculation method of the proportion, and the following two examples are only for illustrative purposes.
[0087] a) When determining the percentage of sensitive data, data that is detected as both sensitive data and non-sensitive data is not considered.
[0088] b) When determining the proportion of sensitive data, the number of all test results for all data is used as the denominator, and the number of test results for sensitive data is used as the numerator.
[0089] 3.Algorithm version update.
[0090] It is important to note that since the sensitive data detection algorithm may be updated, for example, due to business changes, user names are also sensitive data, and the sensitive data detection algorithm needs to be updated and the rules in the algorithm need to be re-determined. Therefore, when performing detection based on any sensitive data detection algorithm, the latest version of the sensitive data detection algorithm can be used.
[0091] In Table 1 above, the detection execution condition also includes updating the version of any sensitive data detection algorithm, so that when the sensitive data detection algorithm is updated, the single column data using the old version of the sensitive data detection algorithm can be re-detected using the latest version of the sensitive data detection algorithm. This ensures that the sensitive data detection results are updated in real time according to the version update of the sensitive data detection algorithm.
[0092] S102 and S103 describe the execution of sensitive data detection for any column to be detected. It is understandable that sensitive data detection can be executed in the same manner for all columns to be detected determined in S102.
[0093] After explaining each step, the sampling method in S102 is explained below.
[0094] Since the sensitive data detection method provided in this specification performs sensitive data detection on a single column of data, the single column of data may belong to the same field.
[0095] It is understandable that single-column data belonging to the same field can have the same content meaning. For example, the single-column data corresponding to the identity field can all be ID card numbers; the single-column data corresponding to the contact field can all be mobile phone numbers. For single-column data with the same content meaning, the number of samples does not affect the sensitive data detection results, so the number of samples can be reduced as much as possible.
[0096] Single-column data belonging to the same field can also have different content meanings. For example, the single-column data corresponding to the contact field can include mobile phone numbers and email addresses; the single-column data corresponding to the account field can include mobile phone numbers, email addresses, and user names.
[0097] However, due to the large amount of source data, it is difficult to determine in advance whether the data in a single column of data has the same content and meaning. For example, for a single column of data corresponding to a contact field, it may contain millions of data, and it is difficult to determine whether they all have the same content and meaning.
[0098] This specification does not limit the specific implementation of the sampling method, and the following two examples are only provided for illustrative purposes.
[0099] a) For single-column data, random sampling with a fixed sampling quantity or fixed sampling ratio is performed first, and then recognition is performed.
[0100] The specific identification method is not limited in this specification, and it can be manual identification or identification based on a data content identification algorithm.
[0101] If the recognition result is that the sampling results of this column of data have the same content meaning, then the column of data is considered to have the same content meaning, and sensitive data detection is performed on the sampling results; if the recognition result is that the sampling results of this column of data have different content meanings, re-sampling can be performed, and sensitive data detection is performed based on the new sampling results.
[0102] The re-sampling method is not limited in this specification.
[0103] b) For data of any content meaning, the preset requirement is: if the actual proportion of data of the content meaning exceeds the first preset threshold, then the probability of not including data of the content meaning in the sampling result is less than the second preset threshold. Based on this requirement, the number of samples can be determined according to the number of data contained in a single column of data, and random sampling can be performed based on the determined number of samples.
[0104] A specific formula example can be shown below.
[0105]
[0106] For data with any content meaning, p refers to the probability that the sampling result does not contain data with the content meaning, that is, the second preset threshold, m is the number of data in this column of data, n is the number of samples of this column of data, and x is the first preset threshold.
[0107] After the first preset threshold, the second preset threshold, and the number of data in the column of data are determined, the number of samples can be obtained.
[0108] Through the above sampling method, it is possible to sample as little data as possible for subsequent sensitive data detection while ensuring the correctness of the detection results, thereby improving the detection efficiency.
[0109] Through the above method embodiments, the sampled data can be detected with the help of a machine using a sensitive data detection algorithm. In scenarios with a large amount of data, data sampling can be used to reduce the amount of data that needs to be detected while ensuring the accuracy of data detection, thereby improving the efficiency of data detection; changes in the source data can also be monitored in a timely manner, thereby detecting the changed data in a timely manner.
[0110] In order to facilitate further understanding, this specification also provides an application example of a sensitive data detection method, such as Figure 2 As shown, it is a flowchart of an application example of a sensitive data detection method provided in this specification, which may specifically include the following steps.
[0111] S201: Create a new data table in the source data, and the data entities in the data table include name, ID number and contact information.
[0112] S202: Determine whether the source data meets the detection execution condition of the “newly added data table”.
[0113] S203: Based on the correspondence in Table 1, all columns in the newly added data table are determined as columns to be detected. According to the names of the three fields in the data table and based on the preset sensitive data name set, it is determined that the ID card number column to be detected is sensitive data.
[0114] S204: For the name column to be detected, 500 data are randomly sampled, and based on the regular expression algorithm, it is determined that the data in the sampling results have the same content and meaning, and all successfully match the regular expression corresponding to the name, and then S205 is executed.
[0115] S205: Based on the preset sensitive data content set, it is determined that the name does not exist therein, and therefore it is determined that the name column to be detected belongs to non-sensitive data.
[0116] S206: For the contact information column to be tested, randomly sample 500 data, and determine based on the regular expression algorithm that the content and meaning of the data in the sampling results are different. There is data that successfully matches the regular expression corresponding to the mobile phone number, and there is data that successfully matches the regular expression corresponding to the email address, and execute S207.
[0117] S207: Further, based on the 500,000 data in the contact information column to be detected, 10,000 data are randomly sampled, and based on the sensitive data detection algorithm, it is determined that the proportion of mobile phone numbers (mobile phone numbers belong to the sensitive data content set) in the 10,000 sampling results is 60%, that is, the proportion of sensitive data in the contact information column to be detected is greater than the preset threshold of 30%. Therefore, it is determined that the contact information column to be detected is sensitive data.
[0118] S204 and S206 may be executed in parallel.
[0119] It was finally determined that the ID number and contact information columns were sensitive data, and the name column was non-sensitive data.
[0120] In addition to the above method embodiments, this specification also provides a sensitive data detection device.
[0121] like Figure 3 The figure is a schematic diagram of the structure of a sensitive data detection device provided in this specification. The device is used to perform sensitive detection on any column in a data table of source data, and the source data may include at least one data table; the detection execution conditions and the corresponding determination strategy of the columns to be detected are pre-configured, and the details can be seen in Table 1 above.
[0122] Figure 3 The device shown may include the following three units.
[0123] Determining unit 301: Determine whether source data satisfies any detection execution condition.
[0124] Sampling unit 302: Determines a strategy for the columns to be detected corresponding to the satisfied detection execution conditions, determines the columns to be detected in the data table of the source data, and performs sampling on any column to be detected.
[0125] Detection unit 303: performs detection on the sampling result of the column to be detected based on any sensitive data detection algorithm, and determines the detection result as the detection result of the column to be detected.
[0126] The detection unit 303 may include the following two sub-units.
[0127] Identification subunit 303a: uses a data content identification algorithm to perform data content identification on each data in the sampling result of the to-be-detected column, and determines whether the data is sensitive data according to the identification result.
[0128] Determining subunit 303b: Determines the detection result of the sampling result of the column to be detected according to the proportion of sensitive data in the sampling result of the column to be detected.
[0129] The sampling unit 302 may include the following two sub-units.
[0130] Pre-detection subunit 302a: For any column to be detected, if the name of the column to be detected exists in the pre-stored sensitive data name set, then determine the detection result of the column to be detected.
[0131] The first judgment subunit 302b: if the name of the column to be detected does not exist in the pre-stored sensitive data name set, sampling is performed on the column to be detected.
[0132] The sampling unit 302 may further include the following two sub-units.
[0133] Random sampling subunit 302c: for any column to be detected, firstly perform random sampling with a fixed sampling number or a fixed sampling ratio, and then identify the data in the sampling result;
[0134] The second judgment subunit 302d: if the recognition result is that each data in the sampling result has a different content meaning, re-sampling is performed for the column to be detected.
[0135] like Figure 4 As shown, it is a schematic diagram of the structure of a sampling unit provided in this specification, wherein the sampling unit 302 can include four subunits, namely, a pre-detection subunit 302a, a first judgment subunit 302b, a random sampling subunit 302c and a second judgment subunit 302d. It can pre-judge the sensitive data detection result according to the pre-stored sensitive data name set, and can perform different sampling processes for single-column data that need to be detected for sensitive data and contain the same or different content meanings.
[0136] The embodiment of the present specification also provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a computer program is implemented as follows: Figure 1The sensitive data detection method shown.
[0137] Figure 5 A more specific hardware structure diagram of a computer device provided in an embodiment of this specification is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0138] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0139] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0140] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0141] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0142] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).
[0143] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0144] The embodiment of the present specification also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a method such as Figure 1 The sensitive data detection method shown.
[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0146] It can be known from the above description of the implementation mode that the technicians in this field can clearly understand that the embodiments of this specification can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of this specification can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments of this specification.
[0147] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.
[0148] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment scheme of this specification. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.
[0149] The above is only a specific implementation of the embodiments of this specification. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiments of this specification. These improvements and modifications should also be regarded as protection for the embodiments of this specification.
Claims
1. A sensitive data detection method, used to perform sensitivity detection on any column in a data table of source data, wherein the source data includes at least one data table ; Pre-configure the detection execution conditions and the corresponding to-be-detected column determination strategy; The method comprises: Determining that the source data meets any detection execution condition, the detection execution condition at least including: a newly added data table, a newly added column in any data table, or a change in the name of any column in any data table; Determine a strategy for the columns to be detected corresponding to the satisfied detection execution conditions, determine the columns to be detected in the data table of the source data, the columns to be detected include columns that change when the source data satisfies the detection execution conditions; and perform sampling on any column to be detected; For the sampling result of the column to be detected, detection is performed based on any sensitive data detection algorithm, and the detection result is determined as the detection result of the column to be detected; The sensitive data detection algorithm includes a data content identification algorithm; the sampling results for the column to be detected are detected based on any sensitive data detection algorithm, including: using the data content identification algorithm to perform data content identification on each data in the sampling results of the column to be detected, and determining whether the data is sensitive data based on the identification result; determining the detection result of the sampling results of the column to be detected based on the proportion of sensitive data in the sampling results of the column to be detected.
2. The method according to claim 1, wherein the detection execution condition further comprises: include: Update the version of any sensitive data detection algorithm; The strategy for determining the columns to be detected corresponding to the detection execution condition includes: determining a column in any data table that is detected using an old version of a sensitive data detection algorithm as a column to be detected.
3. The method according to claim 1, wherein the detection execution condition is further include: The duration between the current time point and the time point of the last detection is greater than the preset duration, or the current time point is the same as the preset time point for periodically performing detection.
4. The method according to claim 1, wherein sampling is performed for any column to be detected. include: For any column to be detected, if the name of the column to be detected exists in the pre-stored sensitive data name set, the detection result of the column to be detected is determined; If the name of the column to be detected does not exist in the pre-stored sensitive data name set, sampling is performed on the column to be detected.
5. The method according to claim 1, wherein sampling is performed for any column to be detected. include: For any column to be detected, first perform random sampling with a fixed number of samples or a fixed sampling ratio, and then identify the data in the sampling results; If the recognition result is that each data in the sampling result has a different content meaning, the column to be detected is sampled again.
6. A sensitive data detection device, used to perform sensitive detection on any column in a data table of source data, wherein the source data includes at least one data table ; Pre-configure the detection execution conditions and the corresponding to-be-detected column determination strategy; The device comprises: Determining unit: determining that the source data meets any detection execution condition, wherein the detection execution condition at least includes: a newly added data table, a newly added column in any data table, or a change in the name of any column in any data table; Sampling unit: determining a strategy according to the to-be-detected columns corresponding to the satisfied detection execution conditions, determining the to-be-detected columns in the data table of the source data, wherein the to-be-detected columns include columns that change when the source data satisfies the detection execution conditions; and sampling any to-be-detected columns; Detection unit: performs detection on the sampling result of the column to be detected based on any sensitive data detection algorithm, and determines the detection result as the detection result of the column to be detected; The sensitive data detection algorithm includes a data content identification algorithm; The detection unit comprises: Identification subunit: using a data content identification algorithm, performs data content identification on each data in the sampling result of the to-be-detected column, and determines whether the data is sensitive data according to the identification result; Determine subunit: Determine the detection result of the sampling result of the column to be detected according to the proportion of sensitive data in the sampling result of the column to be detected.
7. The device according to claim 6, wherein the detection execution condition is further include: Update the version of any sensitive data detection algorithm; The strategy for determining the columns to be detected corresponding to the detection execution condition includes: determining a column in any data table that is detected using an old version of a sensitive data detection algorithm as a column to be detected.
8. The device according to claim 6, wherein the detection execution condition is further include: The duration between the current time point and the time point of the last detection is greater than the preset duration, or the current time point is the same as the preset time point for periodically performing detection.
9. The device according to claim 6, wherein the sampling unit include: Pre-detection subunit: for any column to be detected, if the name of the column to be detected exists in the pre-stored sensitive data name set, then determine the detection result of the column to be detected; The first judgment subunit: if the name of the column to be detected does not exist in the pre-stored sensitive data name set, sampling is performed on the column to be detected.
10. The device according to claim 6, wherein the sampling unit include: Random sampling subunit: For any column to be detected, first perform random sampling with a fixed sampling number or fixed sampling ratio, and then identify the data in the sampling results; The second judgment subunit: if the recognition result is that each data in the sampling result has a different content meaning, re-sampling is performed on the column to be detected.
11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, in, When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.
