A data query method, device, medium and equipment

Abstract

This invention relates to the field of data query technology, and more particularly to a data query method, apparatus, medium, and device. It acquires incremental data and a first predicted record count from data at preset time points. The first predicted record count and the total number of incremental data records are used to divide and filter out first data blocks requiring anonymization. When the number of first data blocks exceeds a preset threshold, a second predicted record count for each first data block is calculated. Based on the second predicted record count and the total number of records in the first data blocks, second data blocks requiring anonymization are divided and filtered out. Each record in the second data block is queried and anonymized, thereby accurately meeting the anonymization requirements. This process gradually filters out initial data tables that do not require anonymization, historical data in the target data table that does not require anonymization, and multi-level data blocks, gradually reducing the data processing volume. This accurately identifies records requiring anonymization and greatly improves data query efficiency when querying massive amounts of data for records requiring anonymization.

Description

Technical Field

[0001] This invention relates to the field of data query technology, and in particular to a data query method, apparatus, medium and device. Background Technology

[0002] In the field of data security, de-identifying data containing sensitive information is a key step in ensuring data security. Efficiently and accurately identifying the records that need to be de-identified before de-identification is an important prerequisite for achieving this step.

[0003] Existing technologies typically employ methods such as overall scanning or fixed-strategy slicing followed by scanning to query data requiring anonymization. This involves querying all records in the data one by one, or dividing the entire dataset into segments according to a preset size and querying in batches to identify the records to be anonymized. However, existing methods only divide the data according to the preset size, without considering the actual number and distribution of records to be anonymized. When the data volume is large and a small fixed segment size is used, a large number of data blocks are generated, significantly increasing the complexity of data block management. Conversely, when the data volume is small and a large fixed segment size is used, each data block contains too much data, resulting in scanning a large amount of unnecessary data when querying records to be anonymized, severely reducing query efficiency. Furthermore, existing methods comprehensively query and identify the data to be anonymized in the entire dataset. When the incremental data volume is large, the query process becomes excessively time-consuming and inefficient, making it difficult to meet the needs of scenarios with high real-time requirements. For example, in patent CN108255871A, entitled "A Data Query Method and Data Query Node," data slices are obtained from multiple data cache nodes, objects are divided into first and second categories, and target objects are filtered out through feature value calculation and aggregation functions. This improves query efficiency by using distributed cache nodes and reducing network transmission rounds and data volume. Although this patent performs data slicing and object classification, its filtering process focuses on indiscriminately slicing and transmitting the entire dataset, resulting in wasted computing resources. Furthermore, it uses a single-dimensional object partitioning method, which cannot dynamically adjust the query granularity based on data increment characteristics, predicted anonymization needs, and other multi-dimensional factors, leading to low efficiency in accurately locating records requiring anonymization within massive datasets.

[0004] Therefore, improving data query efficiency is an urgent problem to be solved when querying records that need to be de-identified in massive amounts of data. Summary of the Invention

[0005] To address the aforementioned technical problems, the present invention provides a data query method, which includes the following steps: S1, select several target data tables to be de-identified from the target database.

[0006] S2, for any target data table, obtain the incremental data and the number of first predicted records corresponding to the current target data table based on the data corresponding to the current target data table at the preset time point.

[0007] S3, divide the current incremental data according to the first predicted record number and the first number of all records included in the current incremental data, and filter to obtain several first data blocks corresponding to the current incremental data.

[0008] S4. If the number of first data blocks corresponding to the current incremental data is greater than the preset number threshold, then the number of second prediction records corresponding to each first data block is obtained according to the amount of data corresponding to each first data block.

[0009] S5, divide the current incremental data according to the number of second predicted records corresponding to each first data block and the second number of all records included in each first data block, and filter to obtain several second data blocks to be queried.

[0010] S6. Query each record in each second data block to obtain the records to be de-identified, and perform de-identification processing on the records to be de-identified.

[0011] The present invention also provides a data query device, which includes: The data table filtering module is used to filter out several target data tables to be de-identified from the target database.

[0012] The first data analysis module is used to obtain the incremental data and the number of first predicted records for any target data table based on the data corresponding to the current target data table at a preset time point.

[0013] The first data block filtering module is used to divide the current incremental data according to the first predicted record number and the first number of all records included in the current incremental data, and filter to obtain several first data blocks corresponding to the current incremental data.

[0014] The second data analysis module is used to obtain the number of second predicted records corresponding to each first data block based on the amount of data corresponding to each first data block if the number of first data blocks corresponding to the current incremental data is greater than a preset number threshold.

[0015] The second data block filtering module is used to divide the current incremental data according to the number of second predicted records corresponding to each first data block and the second number of all records included in each first data block, and filter to obtain several second data blocks to be queried.

[0016] The data query module is used to query each record in each second data block, obtain the records to be de-identified, and perform de-identification processing on the records to be de-identified.

[0017] The present invention also provides a non-transitory computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described data query method.

[0018] The present invention also provides an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0019] This invention has at least the following beneficial effects: By filtering the target data table to be anonymized from the initial data table of the target database, the core data that needs to be anonymized is focused; by obtaining incremental data and the first predicted record number through preset time point data, only newly added or changed data is processed, eliminating the need to repeatedly process historically anonymized data, thus significantly reducing the processing volume; by dividing and filtering the first data block that needs to be anonymized through the first predicted record number and the total number of incremental data records, when the number of the first data block exceeds a preset threshold, by calculating the second predicted record number of each first data block, the second data block that needs to be anonymized is divided and filtered according to the second predicted record number and the total number of records in the first data block, and the records to be processed in the second data block are queried and anonymized one by one, thereby accurately meeting the anonymization requirements, gradually filtering out the initial data table that does not need to be anonymized, the historical data in the target data table that does not need to be anonymized, and multi-level data blocks, gradually reducing the data processing volume, thereby accurately identifying the records that need to be anonymized, and greatly improving the data query efficiency when querying the records to be anonymized in massive amounts of data. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a data query method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a data query device provided in Embodiment 2 of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the terms used to distinguish similar objects can be interchanged so that the invention can also be implemented in other embodiments besides the illustrated or described embodiments. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0024] Example 1 This embodiment provides a data query method, which includes the following steps: Figure 1 As shown: S1. Select several target data tables to be de-identified from the target database. The target database includes several initial data tables, and the target data tables are the initial data tables that need to be de-identified. Each target data table contains several records.

[0025] The target database is a database containing various types of data that require anonymization. It contains multiple initial data tables, each with a specific structure and purpose, storing different types of data records with varying levels of sensitivity. For example, user information tables and transaction record tables may contain a large amount of sensitive information, while system configuration tables and log tables may not require anonymization. Therefore, it is necessary to filter through all initial data tables to identify the target data tables containing sensitive information that require anonymization.

[0026] A record is a row of data in either the initial or target data table, containing values ​​for multiple fields. For example, in a user information table, a record might contain values ​​for fields such as user ID, name, gender, and date of birth.

[0027] As described above, by selecting the target data tables to be de-identified, the subsequent de-identification process can focus on the data tables that actually need to be processed, avoiding unnecessary operations on irrelevant data tables, thereby reducing the amount of data processed and improving the efficiency of data processing.

[0028] In one specific implementation, each initial data table includes several records, and S1 includes the following steps: S11, for any initial data table, obtain the first reference data volume corresponding to the current initial data table based on the first number of all records included in the current initial data table and the preset data volume corresponding to each record in the current initial data table, wherein the preset data volume corresponding to each record in the current initial data table is determined according to the preset desensitization rule corresponding to the current initial data table.

[0029] S12, if the actual data volume of the current initial data table is greater than the first reference data volume corresponding to the current initial data table, then the current initial data table is determined to be the target data table.

[0030] The preset data masking rules are predefined masking strategies for different types of data tables, such as masking, replacement, encryption, and hashing. The preset masking rules for each initial data table can be determined based on its type. For example, a user information table needs to mask field values ​​such as ID card numbers, so the corresponding preset masking rule is partial replacement. A medical record table needs to mask field values ​​such as medical record numbers, so the corresponding preset masking rule is hashing. A logistics information table needs to mask field values ​​such as recipient names, so the corresponding preset masking rule is to replace them with the surname followed by "Mr / Ms."

[0031] The preset data volume is an estimate of the average data volume per record after de-identification, based on the de-identification rules. A fixed length can be directly set according to the field type and de-identification rules, and then the preset data volume is obtained by combining the lengths of all fields corresponding to each record. For example, de-identifying an ID number into 12 characters... The system uses a combination of a single character and four real characters, corresponding to a preset data size of 16 bytes. The medical record number is anonymized into a 128-bit (16-byte) hash value using the MD5 algorithm, or into a 160-bit (20-byte) hash value using the SHA-1 algorithm, or into a 256-bit (32-byte) hash value using the SHA-256 algorithm. Statistical analysis can also be performed on historical or sample data to obtain the average length of the anonymized fields, thereby determining the average preset data size for each record.

[0032] The first reference data volume is theoretically the total amount of data after anonymizing all records in the current data table. For example, the first reference data volume equals the product of the first quantity and the preset data volume. The actual data volume is the physical storage space (e.g., bytes) occupied by the initial data table in the database. If the actual data volume is greater than the theoretical first reference data volume, it indicates that there are sensitive fields in the initial data table that need to be anonymized. By quantifying the theoretical first reference data volume, a comparison benchmark is provided for subsequently selecting target data tables, avoiding direct processing of all initial data tables and thus improving processing efficiency.

[0033] The above-mentioned method calculates the theoretical data volume by setting preset rules and compares it with the actual data volume, so that subsequent operations are only performed on the initial data table that may contain sensitive data, thereby reducing system overhead and improving query efficiency.

[0034] S2, for any target data table, based on the data corresponding to the current target data table at a preset time point, obtain the incremental data and the first predicted record number corresponding to the current target data table, wherein the first predicted record number is used to characterize the number of records in the current incremental data that need to be de-identified.

[0035] Incremental data refers to newly added or modified records in the current target data table. During the data processing flow, the target data table will continuously generate new data (such as newly added or modified records) over time, while historical data (i.e., data other than incremental data) has already undergone anonymization in previous query tasks, and its sensitive information has been protected and does not need to be processed again.

[0036] Therefore, this embodiment obtains incremental data and only queries and de-identifies the incremental data to avoid repeated scanning of the full data, thereby reducing the amount of data processing, reducing system resource consumption, shortening the processing time of a single query and de-identification, and improving the overall process response speed.

[0037] In one specific implementation, the preset time point includes a first preset time point and a second preset time point, wherein the first preset time point is the time point corresponding to the previous data query task, and the second preset time point is the time point corresponding to the current data query task. S2 includes the following steps: S21, obtain the preset data volume difference of a record corresponding to the current target data table, all first records and first data volume corresponding to the first preset time point, and all second records and second data volume corresponding to the second preset time point.

[0038] S22, based on all the first records and all the second records corresponding to the current target data table, obtain the incremental data of the current target data table between the first preset time point and the second preset time point, as well as the first number of all records included in the current incremental data.

[0039] S23. Based on the preset data volume difference, the first data volume, and the second data volume corresponding to the current target data table, obtain the number of first predicted records corresponding to the current incremental data. The number of first predicted records is positively correlated with the absolute value of the difference between the corresponding first data volume and the second data volume, and negatively correlated with the corresponding preset data volume difference.

[0040] The preset data volume difference is the average change in data volume of a single record before and after de-identification (e.g., each record is reduced by 10 bytes after de-identification).

[0041] Incremental data refers to records that are added or modified between the first preset time point and the second preset time point.

[0042] The absolute value of the difference between the first and second data volumes reflects the change in the total data volume of the current target data table, that is, the change in the data volume corresponding to the incremental data. The larger the difference in data volume, the more records are predicted to require anonymization. Conversely, the larger the change in the data volume of a single record, the fewer records are predicted to require anonymization. Therefore, by inferring the potential spatial changes caused by the lack of anonymization through changes in data volume, the number of records requiring anonymization can be estimated. For example, the first predicted number of records L1 = |V2 - V1| / ΔV, where V2 is the second data volume corresponding to the current target data table, V1 is the first data volume corresponding to the current target data table, and ΔV is the preset data volume difference corresponding to the current target data table.

[0043] The above-mentioned method automatically captures data growth patterns by comparing the amount of data between preset time points and predicts the number of records that need to be de-identified. Based on the prediction results, it optimizes the data block division, reduces invalid scans, accelerates the location of records to be de-identified, and improves the efficiency of data query and de-identification.

[0044] S3. Divide the current incremental data according to the first predicted record number and the first number of all records included in the current incremental data, and filter to obtain several first data blocks corresponding to the current incremental data. Among them, the first data blocks are data blocks that need to be de-identified.

[0045] In one specific embodiment, S3 includes the following steps: S31. Based on the first predicted record number and the first number of all records corresponding to the current incremental data, obtain the first partition number N corresponding to the current incremental data.

[0046] S32, divide the current incremental data according to the first division number N corresponding to the current incremental data, and obtain N intermediate data blocks corresponding to the current incremental data.

[0047] S33, obtain the second number of all records corresponding to each intermediate data block.

[0048] S34. Based on the second number of all records included in each intermediate data block and the preset data amount corresponding to each record in each intermediate data block, obtain the second reference data amount corresponding to each intermediate data block.

[0049] S35, for any intermediate data block, if the actual data volume corresponding to the current intermediate data block is greater than the corresponding second reference data volume, then the current intermediate data block is determined to be the first data block.

[0050] Based on the first predicted record number and the first quantity, namely the number of records to be de-identified and the total number of incremental data records, the incremental data is divided into multiple intermediate data blocks to provide a basis for subsequent parallel processing.

[0051] For example, the number of first partitions N = argmin y {|y-J1 / (L1×α1)|}, where J1 is the first number of all records corresponding to the current incremental data, L1 is the first predicted number of records corresponding to the current incremental data, α1 is the preset partitioning coefficient, y is a divisor of J1, and argmin{} is a function to find the minimum value. Through the above calculation, the value with the smallest difference from the theoretical partitioning number J1 / (L1×α1) among all divisors of J1 can be found as the first partitioning number N.

[0052] The specific value of the preset division coefficient can be set by the implementer according to the actual situation to adjust the division granularity. Correspondingly, the smaller the preset division coefficient, the larger the number of first divisions and the finer the division granularity. For example, the preset division coefficient can be a value such as 0.5 or 1.

[0053] Furthermore, the incremental data is evenly distributed across N intermediate data blocks. Understandably, each intermediate data block includes metadata such as block ID, starting record position, and number of records to facilitate subsequent processing.

[0054] The second reference data volume is theoretically the total amount of data after anonymization of all records in each intermediate data block. For example, the second reference data volume is equal to the product of the second quantity corresponding to each intermediate data block and the preset data volume. If the actual data volume of an intermediate data block is greater than the theoretical second reference data volume, it indicates that there are sensitive fields in the intermediate data block that need to be anonymized. By quantifying the theoretical second reference data volume, a comparison benchmark is provided for the subsequent selection of the first data block, avoiding direct processing of all intermediate data blocks and thus improving processing efficiency.

[0055] The above-mentioned method divides the data block size reasonably by using the first predicted record number and the first quantity, and divides the incremental data into several intermediate data blocks. Based on the actual data volume of the intermediate data blocks and the theoretical second reference data volume, the data blocks that need to be de-identified are selected. While ensuring the data processing load balance, the method accurately meets the de-identification requirements, reduces the data processing volume, and ultimately improves the efficiency of the overall data query and de-identification process.

[0056] S4. If the number of first data blocks corresponding to the current incremental data is greater than the preset number threshold, then the number of second predicted records corresponding to each first data block is obtained according to the amount of data corresponding to each first data block. The number of second predicted records is used to characterize the number of records in the corresponding first data block that need to be de-identified.

[0057] The preset quantity threshold is the critical value for determining whether the first data block needs to undergo secondary refinement processing.

[0058] When the number of first data blocks is less than or equal to a preset threshold, the data of each first data block can be queried directly without having to calculate the number of second prediction records, thus reducing unnecessary resource consumption.

[0059] When the number exceeds the preset threshold, the amount of data in the first data block is still large, and the efficiency of direct data query is low. Therefore, it is necessary to further refine the prediction of the number of records that need to be de-identified in each first data block, that is, the second predicted number of records, so as to ensure that the de-identification requirements of each first data block are accurately captured through secondary partitioning, reduce the processing of de-identified data, and avoid processing deviation and low efficiency caused by excessive data volume.

[0060] The specific value of the preset quantity threshold can be set by the implementer according to the actual situation. For example, the threshold can be dynamically adjusted based on the ratio of the average of the first quantity of all records included in the incremental data to the average of the second quantity of records corresponding to a single first data block. Specifically, the preset quantity threshold Z = β × J1 / P, where J1 is the first quantity of all records corresponding to the current incremental data, P is the average of the second quantity of records corresponding to the first data block corresponding to the current incremental data, and β is the preset adjustment quantity, which can be set by the implementer according to the sensitivity of the data table, for example, β can be 0.1 or 0.05. The threshold can also be determined by analyzing the relationship between the number of data blocks and processing efficiency in historical processing records. Specifically, the processing efficiency such as time consumption and accuracy corresponding to different numbers of first data blocks in historical data is collected, the efficiency inflection point is found, and the inflection point value is used as the preset quantity threshold.

[0061] In one specific embodiment, S4 includes the following steps: S41, obtain the second reference data volume corresponding to each first data block.

[0062] S42, based on the data volume and second reference data volume corresponding to each first data block, and the difference between the preset data volume corresponding to each first data block, the second predicted record number corresponding to each first data block is obtained, wherein the second predicted record number is positively correlated with the absolute value of the difference between the data volume and the second reference data volume corresponding to each first data block, and negatively correlated with the corresponding preset data volume difference.

[0063] The absolute value of the difference between the data volume corresponding to each first data block and the second reference data volume reflects the change in the total data volume of each first data block. Therefore, the larger the difference in data volume corresponding to each first data block, the more records are predicted to require de-identification; conversely, the larger the change in the data volume of a single record, the fewer records are predicted to require de-identification. Thus, by inferring the potential spatial changes due to lack of de-identification from the changes in the data volume corresponding to each first data block, the number of records requiring de-identification in each first data block can be estimated.

[0064] For example, the number L of the second predicted records corresponding to the i-th first data block. 2i =|U 2i -U 1i | / ΔV i , among which, U 2i U represents the amount of data corresponding to the i-th first data block. 1i Let ΔV be the second reference data quantity corresponding to the i-th first data block. i This is the preset data volume difference corresponding to the target data table of the i-th first data block.

[0065] As mentioned above, when the number of first data blocks is too large, the number of second prediction records corresponding to each first data block is calculated block by block, avoiding the prediction error of a one-size-fits-all approach, more accurately reflecting the de-identification requirements of each first data block, eliminating the query task for first data blocks that do not need to be de-identified, and providing a basis for further segmentation of first data blocks that need to be de-identified, thereby reducing the resource consumption of data scanning and querying.

[0066] S5, divide the current incremental data according to the number of second predicted records corresponding to each first data block and the second number of all records included in each first data block, and filter to obtain several second data blocks to be queried.

[0067] In one specific embodiment, S5 includes the following steps: S51, for any first data block, obtain the second partition number M corresponding to the current first data block based on the second predicted record number corresponding to the current first data block and the second number of all records included in the current first data block.

[0068] S52, divide the current first data block according to the second partition number M corresponding to the current first data block, and obtain M reference data blocks corresponding to the current first data block.

[0069] S53, obtain the third number of all records corresponding to each reference data block.

[0070] S54. Based on the third number of all records included in each reference data block and the preset data amount corresponding to each record in each reference data block, obtain the third reference data amount corresponding to each reference data block.

[0071] S55, for any reference data block, if the actual data volume corresponding to the current reference data block is greater than the corresponding third reference data volume, then the current reference data block is determined to be the second data block.

[0072] In this process, based on the second number of predicted records and the second quantity, each first data block is divided into multiple second data blocks, providing a basis for subsequent parallel processing.

[0073] For example, the number of second partitions M corresponding to the i-th first data block. i =argmin r {|rJ 2i / (L 2i ×α1)|},where J 2i L represents the second number of all records corresponding to the i-th first data block. 2i Let α1 be the number of second predicted records corresponding to the i-th first data block, α1 be the preset partitioning coefficient, and r be the number of records in J. 2i The divisors of J are given by argmin{}, which is a function to find the minimum value. Through the above calculations, the minimum value can be found in J. 2i Among all the divisors, find the number of partitions J that corresponds to the theoretical number of partitions. 2i / (L 2i The value with the smallest difference between ×α1) is taken as the second partition number M. i .

[0074] Furthermore, each first data block is evenly distributed among several corresponding reference data blocks. Understandably, each reference data block includes metadata such as block ID, starting record position, and number of records to facilitate subsequent processing.

[0075] The third reference data volume is theoretically the total amount of data after anonymization of all records in each reference data block. For example, the third reference data volume equals the product of the third quantity corresponding to each reference data block and the preset data volume. If the actual data volume of a reference data block is greater than the theoretical third reference data volume, it indicates that there are sensitive fields in the reference data block that need to be anonymized. By quantifying the theoretical third reference data volume, a comparison benchmark is provided for the subsequent selection of the second data block, avoiding the direct processing of all reference data blocks and thus improving processing efficiency.

[0076] As described above, based on the second predicted record number and the second quantity of each first data block, that is, the number of records to be de-identified and the total number of records in each first data block, multiple second data blocks to be queried are divided and filtered from the first data block, which further reduces the query of data that does not need to be de-identified, achieves more refined de-identification processing optimization, and further improves query and de-identification efficiency.

[0077] S6. Query each record in each second data block to obtain the records to be de-identified, and perform de-identification processing on the records to be de-identified.

[0078] In one specific embodiment, S6 includes the following steps: S61, for any second data block, compare the data volume of each record in the current second data block with the preset data volume of each record in the corresponding initial data table in turn, and obtain the records to be de-identified in the current second data block.

[0079] S62, traverse all second data blocks corresponding to the initial data table corresponding to the current second data block, and obtain all records to be de-identified corresponding to the current initial data table.

[0080] S63, based on all records to be de-identified corresponding to the current initial data table, calls several target service interfaces corresponding to the current initial data table.

[0081] S64, de-identify all records to be de-identified in the current initial data table by using the preset de-identification rules corresponding to the target service interface.

[0082] In one specific embodiment, S63 includes the following steps: S631, based on the number of records processed and the preset processing speed of each de-identification service interface corresponding to the current initial data table, the fourth number of all records to be de-identified corresponding to the current initial data table, and the preset de-identification time corresponding to the current initial data table, obtain the judgment category corresponding to each de-identification service interface corresponding to the current initial data table, wherein the judgment category is to call or not call.

[0083] S632, determine the de-identified service interface that is classified as a call as the target service interface.

[0084] S633 calls all target service interfaces corresponding to the current initial data table.

[0085] If the data volume of a certain record exceeds the preset data volume, it is determined to be a record to be de-identified.

[0086] The record processing quantity is the upper limit of the number of records that the de-identification service interface can process per unit time. The preset processing speed is the average time (seconds / record) that the de-identification service interface takes to process a single record. The fourth quantity is the total number of records to be de-identified. The preset de-identification time is the maximum allowable de-identification time (seconds) set by the implementer.

[0087] By comprehensively analyzing the processing capabilities of the de-identification service interfaces and the records to be processed, several target service interfaces that perform query and de-identification tasks for each initial data table are selected. Then, the preset de-identification rules corresponding to the target service interfaces are called to de-identify all records to be de-identified in the current initial data table, thereby improving the efficiency of the de-identification process.

[0088] In one specific implementation, the number of records to be processed and the preset processing speed for each de-identification service interface corresponding to the current initial data table, the fourth number of all records to be de-identified corresponding to the current initial data table, and the preset de-identification time corresponding to the current initial data table are input into the preset classification model to obtain the judgment category corresponding to each de-identification service interface corresponding to the current initial data table.

[0089] Those skilled in the art will recognize that the structure of any classification model in the prior art falls within the protection scope of this invention, and will not be elaborated further here. For example, a preset classification model includes convolutional layers and fully connected layers. The convolutional layers extract features from the input, and the fully connected layers map the features to obtain the probability for each classification category and output the corresponding classification category based on the probability value. It is understood that the preset classification model needs to be trained to improve the classification accuracy in this embodiment. Those skilled in the art will recognize that the training methods of classification models in the prior art fall within the protection scope of this invention, and will not be elaborated further here.

[0090] Specifically, if the number of first data blocks corresponding to the current incremental data is less than or equal to a preset threshold, then each record in each first data block is directly queried to obtain the records to be de-identified, and de-identification processing is performed. The query and de-identification method for each record in the first data block can refer to the query and de-identification method for each record in the second data block described above, and will not be repeated here.

[0091] The above describes a process that, by filtering the initial data tables of the target database to identify the target data to be anonymized, focuses on the core data requiring anonymization. Incremental data and the first predicted record count are obtained using data from preset time points, allowing processing only on newly added or changed data, eliminating the need to repeatedly process historically anonymized data and significantly reducing processing volume. The first data blocks requiring anonymization are divided and filtered based on the first predicted record count and the total number of incremental data records. When the number of first data blocks exceeds a preset threshold, the second predicted record count for each first data block is calculated. Based on this second predicted record count and the total number of records in the first data blocks, second data blocks requiring anonymization are divided and filtered. Each record in the second data block is then queried and anonymized, thus precisely meeting the anonymization requirements. This process gradually eliminates unanonymized initial data tables, unanonymized historical data in the target data tables, and multi-level data blocks, gradually reducing data processing volume and accurately identifying records requiring anonymization. This significantly improves data query efficiency when querying massive amounts of data for records requiring anonymization.

[0092] Example 2 This second embodiment provides a data query device, which includes, for example: Figure 2 As shown: The data table filtering module 21 is used to filter out several target data tables to be de-identified from the target database. The target database includes several initial data tables, and the target data tables are the initial data tables that need to be de-identified. Each target data table includes several records.

[0093] The first data analysis module 22 is used to obtain the incremental data and the first predicted record number corresponding to the current target data table based on the data corresponding to the current target data table at a preset time point for any target data table. The first predicted record number is used to characterize the number of records in the current incremental data that need to be de-identified.

[0094] The first data block filtering module 23 is used to divide the current incremental data according to the first predicted record number and the first number of all records included in the current incremental data, and filter to obtain several first data blocks corresponding to the current incremental data. The first data blocks are data blocks that need to be de-identified.

[0095] The second data analysis module 24 is used to obtain the second predicted record number corresponding to each first data block based on the data volume corresponding to each first data block if the number of first data blocks corresponding to the current incremental data is greater than a preset number threshold. The second predicted record number is used to characterize the number of records in the corresponding first data block that need to be de-identified.

[0096] The second data block filtering module 25 is used to divide the current incremental data according to the number of second predicted records corresponding to each first data block and the second number of all records included in each first data block, and filter to obtain several second data blocks to be queried.

[0097] The data query module 26 is used to query each record in each second data block, obtain the records to be de-identified, and perform de-identification processing on the records to be de-identified.

[0098] In one specific implementation, each initial data table includes several records, and the data table filtering module 21 includes: The first reference data volume acquisition submodule is used to obtain the first reference data volume corresponding to any initial data table based on the first number of all records included in the current initial data table and the preset data volume corresponding to each record in the current initial data table. The preset data volume corresponding to each record in the current initial data table is determined according to the preset desensitization rule corresponding to the current initial data table.

[0099] The target data table acquisition submodule is used to determine the current initial data table as the target data table if the actual data volume of the current initial data table is greater than the first reference data volume corresponding to the current initial data table.

[0100] In one specific implementation, the preset time points include a first preset time point and a second preset time point, wherein the first preset time point is the time point corresponding to the previous data query task, and the second preset time point is the time point corresponding to the current data query task. The first data analysis module 22 includes: The first data acquisition submodule is used to acquire the preset data volume difference of a record corresponding to the current target data table, all first records and first data volume corresponding to the first preset time point, and all second records and second data volume corresponding to the second preset time point.

[0101] The first quantity acquisition submodule is used to obtain the incremental data corresponding to the current target data table between the first preset time point and the second preset time point, as well as the first quantity of all records included in the current incremental data, based on all the first records and all the second records corresponding to the current target data table.

[0102] The first predicted record quantity acquisition submodule is used to obtain the first predicted record quantity corresponding to the current incremental data based on the preset data quantity difference, the first data quantity, and the second data quantity corresponding to the current target data table. The first predicted record quantity is positively correlated with the absolute value of the difference between the corresponding first data quantity and the second data quantity, and negatively correlated with the corresponding preset data quantity difference.

[0103] In one specific embodiment, the first data block filtering module 23 includes: The first partitioning quantity acquisition submodule is used to obtain the first partitioning quantity N corresponding to the current incremental data based on the first predicted record quantity and the first quantity of all records corresponding to the current incremental data.

[0104] The intermediate data block acquisition submodule is used to divide the current incremental data according to the first division number N corresponding to the current incremental data, and obtain N intermediate data blocks corresponding to the current incremental data.

[0105] The second quantity acquisition submodule is used to obtain the second quantity of all records corresponding to each intermediate data block.

[0106] The second reference data volume acquisition submodule is used to obtain the second reference data volume corresponding to each intermediate data block based on the second number of all records included in each intermediate data block and the preset data volume corresponding to each record in each intermediate data block.

[0107] The first data block acquisition submodule is used to determine the current intermediate data block as the first data block if the actual data volume corresponding to the current intermediate data block is greater than the corresponding second reference data volume.

[0108] In one specific embodiment, the second data analysis module 24 includes: The second data acquisition submodule is used to acquire the second reference data volume corresponding to each first data block.

[0109] The second predicted record quantity acquisition submodule is used to obtain the second predicted record quantity corresponding to each first data block based on the data quantity and the second reference data quantity corresponding to each first data block, as well as the difference between the preset data quantity corresponding to each first data block. The second predicted record quantity is positively correlated with the absolute value of the difference between the data quantity and the second reference data quantity corresponding to each first data block, and negatively correlated with the corresponding preset data quantity difference.

[0110] In one specific embodiment, the data query module 26 includes: The first record acquisition submodule is used to compare the data volume of each record in the current second data block with the preset data volume of each record in the corresponding initial data table for any second data block, so as to obtain the record to be de-identified in the current second data block.

[0111] The second record acquisition submodule is used to traverse all second data blocks corresponding to the initial data table corresponding to the current second data block and obtain all records to be de-identified corresponding to the current initial data table.

[0112] The target service interface acquisition submodule is used to call several target service interfaces corresponding to the current initial data table based on all records to be de-identified in the current initial data table.

[0113] The data desensitization submodule is used to desensitize all records to be desensitized in the current initial data table according to the preset desensitization rules corresponding to the target service interface.

[0114] In one specific implementation, the target service interface acquisition submodule includes: The category determination unit is used to determine the category corresponding to each de-identification service interface of the current initial data table based on the number of records processed and the preset processing speed of each de-identification service interface corresponding to the current initial data table, the fourth number of all records to be de-identified in the current initial data table, and the preset de-identification time of the current initial data table. The category is either called or not called.

[0115] The target service interface acquisition unit is used to identify the de-identified service interface that is classified as a call as the target service interface.

[0116] The target service interface call unit is used to call all target service interfaces corresponding to the current initial data table.

[0117] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0118] Example 3 Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, which stores at least one instruction or at least one program segment, wherein the at least one instruction or at least one program segment is loaded and executed by a processor to implement the following steps: S1. Select several target data tables to be de-identified from the target database. The target database includes several initial data tables, and the target data tables are the initial data tables that need to be de-identified. Each target data table contains several records.

[0119] S2, for any target data table, based on the data corresponding to the current target data table at a preset time point, obtain the incremental data and the first predicted record number corresponding to the current target data table, wherein the first predicted record number is used to characterize the number of records in the current incremental data that need to be de-identified.

[0120] S3. Divide the current incremental data according to the first predicted record number and the first number of all records included in the current incremental data, and filter to obtain several first data blocks corresponding to the current incremental data. Among them, the first data blocks are data blocks that need to be de-identified.

[0121] S4. If the number of first data blocks corresponding to the current incremental data is greater than the preset number threshold, then the number of second predicted records corresponding to each first data block is obtained according to the amount of data corresponding to each first data block. The number of second predicted records is used to characterize the number of records in the corresponding first data block that need to be de-identified.

[0122] S5, divide the current incremental data according to the number of second predicted records corresponding to each first data block and the second number of all records included in each first data block, and filter to obtain several second data blocks to be queried.

[0123] S6: Query each record in each second data block to obtain the record to be de-identified and perform de-identification processing.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0126] Example 4 Embodiment 4 of the present invention provides an electronic device, which includes a processor and a non-transitory computer-readable storage medium as described in Embodiment 3 of the present invention.

[0127] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A data query method, characterized in that, The data query method includes the following steps: S1, Select several target data tables to be de-identified from the target database; S2, for any target data table, based on the data corresponding to the current target data table at a preset time point, obtain the incremental data and the number of first predicted records corresponding to the current target data table. The preset time point includes a first preset time point and a second preset time point. The first preset time point is the time point corresponding to the previous data query task, and the second preset time point is the time point corresponding to the current data query task. S2 includes the following steps: S21, obtain the preset data volume difference of a record corresponding to the current target data table, all first records and first data volume corresponding to the first preset time point, and all second records and second data volume corresponding to the second preset time point; S22, based on all the first records and all the second records corresponding to the current target data table, obtain the incremental data of the current target data table between the first preset time point and the second preset time point, and the first number of all records included in the current incremental data; S23. Based on the preset data volume difference, the first data volume, and the second data volume corresponding to the current target data table, obtain the first predicted record number corresponding to the current incremental data. The first predicted record number is used to characterize the number of records in the current incremental data that need to be desensitized. The first predicted record number is positively correlated with the absolute value of the difference between the corresponding first data volume and the second data volume, and negatively correlated with the corresponding preset data volume difference. S3, the current incremental data is divided according to the first predicted number of records and the first number of all records included in the current incremental data, and several first data blocks corresponding to the current incremental data are obtained by filtering. S3 includes the following steps: S31, based on the first predicted record number and the first number of all records corresponding to the current incremental data, obtain the first division number N corresponding to the current incremental data; S32, divide the current incremental data according to the first division number N corresponding to the current incremental data, and obtain N intermediate data blocks corresponding to the current incremental data; S33, obtain the second number of all records corresponding to each intermediate data block; S34, based on the second number of all records included in each intermediate data block and the preset data amount corresponding to each record in each intermediate data block, obtain the second reference data amount corresponding to each intermediate data block; S35, for any intermediate data block, if the actual data volume corresponding to the current intermediate data block is greater than the corresponding second reference data volume, then the current intermediate data block is determined to be the first data block that needs to be desensitized. S4. If the number of first data blocks corresponding to the current incremental data is greater than the preset number threshold, then the number of second predicted records corresponding to each first data block is obtained according to the amount of data corresponding to each first data block. The number of second predicted records is used to characterize the number of records in the corresponding first data block that need to be de-identified. S5, divide the current incremental data according to the number of second predicted records corresponding to each first data block and the second number of all records included in each first data block, and filter to obtain several second data blocks to be queried; S6. Query each record in each second data block to obtain the records to be de-identified, and perform de-identification processing on the records to be de-identified.

2. The data query method according to claim 1, characterized in that, The target database includes several initial data tables, each containing several records. S1 includes the following steps: S11, For any initial data table, based on the first number of all records included in the current initial data table and the preset data amount corresponding to each record in the current initial data table, obtain the first reference data amount corresponding to the current initial data table, wherein the preset data amount corresponding to each record in the current initial data table is determined according to the preset desensitization rule corresponding to the current initial data table; S12, if the actual data volume of the current initial data table is greater than the first reference data volume corresponding to the current initial data table, then the current initial data table is determined to be the target data table, and the target data table is the initial data table that needs to be de-identified.

3. The data query method according to claim 1, characterized in that, S4 includes the following steps: S41, obtain the second reference data volume corresponding to each first data block; S42, based on the data volume and second reference data volume corresponding to each first data block, and the difference between the preset data volume corresponding to each first data block, the second predicted record number corresponding to each first data block is obtained, wherein the second predicted record number is positively correlated with the absolute value of the difference between the data volume and the second reference data volume corresponding to each first data block, and negatively correlated with the corresponding preset data volume difference.

4. The data query method according to claim 1, characterized in that, S6 includes the following steps: S61, for any second data block, compare the data volume of each record in the current second data block with the preset data volume of each record in the corresponding initial data table in turn, and obtain the records to be de-identified in the current second data block; S62, traverse all second data blocks corresponding to the initial data table corresponding to the current second data block, and obtain all records to be de-identified corresponding to the current initial data table; S63, based on all records to be de-identified corresponding to the current initial data table, call several target service interfaces corresponding to the current initial data table; S64, de-identify all records to be de-identified in the current initial data table by using the preset de-identification rules corresponding to the target service interface.

5. The data query method according to claim 4, characterized in that, S63 includes the following steps: S631, based on the number of records processed and the preset processing speed of each de-identification service interface corresponding to the current initial data table, the fourth number of all records to be de-identified corresponding to the current initial data table, and the preset de-identification time corresponding to the current initial data table, obtain the judgment category corresponding to each de-identification service interface corresponding to the current initial data table, wherein the judgment category is to call or not call. S632, determine the de-identified service interface whose category is "call" as the target service interface; S633 calls all target service interfaces corresponding to the current initial data table.

6. A data query device, characterized in that, The data query device includes: The data table filtering module is used to filter out several target data tables to be de-identified from the target database; The first data analysis module is used to, for any target data table, obtain the incremental data and the first predicted record count corresponding to the current target data table based on the data corresponding to the current target data table at a preset time point. The preset time point includes a first preset time point and a second preset time point. The first preset time point is the time point corresponding to the previous data query task, and the second preset time point is the time point corresponding to the current data query task. The first data analysis module includes: The first data acquisition submodule is used to acquire the preset data volume difference of a record corresponding to the current target data table, all first records and first data volume corresponding to the first preset time point, and all second records and second data volume corresponding to the second preset time point. The first quantity acquisition submodule is used to obtain the incremental data of the current target data table between the first preset time point and the second preset time point, and the first quantity of all records included in the current incremental data, based on all the first records and all the second records corresponding to the current target data table. The first predicted record quantity acquisition submodule is used to obtain the first predicted record quantity corresponding to the current incremental data based on the preset data quantity difference, the first data quantity and the second data quantity corresponding to the current target data table. The first predicted record quantity is used to characterize the number of records in the current incremental data that need to be de-identified. The first predicted record quantity is positively correlated with the absolute value of the difference between the corresponding first data quantity and the second data quantity, and negatively correlated with the corresponding preset data quantity difference. The first data block filtering module is used to divide the current incremental data according to the first predicted record number and the first number of all records included in the current incremental data, and filter to obtain several first data blocks corresponding to the current incremental data. The first data block filtering module includes: The first partitioning quantity acquisition submodule is used to obtain the first partitioning quantity N corresponding to the current incremental data based on the first predicted record quantity and the first quantity of all records corresponding to the current incremental data. The intermediate data block acquisition submodule is used to divide the current incremental data according to the first division number N corresponding to the current incremental data, and obtain N intermediate data blocks corresponding to the current incremental data. The second quantity acquisition submodule is used to obtain the second quantity of all records corresponding to each intermediate data block; The second reference data volume acquisition submodule is used to acquire the second reference data volume corresponding to each intermediate data block based on the second number of all records included in each intermediate data block and the preset data volume corresponding to each record in each intermediate data block. The first data block acquisition submodule is used to determine the current intermediate data block as the first data block that needs to be de-identified if the actual data volume of the current intermediate data block is greater than the corresponding second reference data volume. The second data analysis module is used to obtain the second predicted record number corresponding to each first data block based on the data volume corresponding to each first data block if the number of first data blocks corresponding to the current incremental data is greater than a preset number threshold. The second predicted record number is used to characterize the number of records in the corresponding first data block that need to be de-identified. The second data block filtering module is used to divide the current incremental data according to the number of second predicted records corresponding to each first data block and the second number of all records included in each first data block, and filter to obtain several second data blocks to be queried; The data query module is used to query each record in each second data block, obtain the records to be de-identified, and perform de-identification processing on the records to be de-identified.

7. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the data query method as described in any one of claims 1-5.

8. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 7.

Images