Multi-source data security desensitization and efficient analysis integrated method

Abstract

The invention relates to the technical field of big data resource service and data security, and discloses a multi-source data security desensitization and efficient analysis integrated method, which comprises the following steps that: a security policy gateway is arranged, after a global analysis task is intercepted, the security policy gateway sends out a statistical response of a probe to query and obtain a data subset, and generates a disposable confusion token according to the response; the method comprises the following steps: generating a one-time token which is destroyed after use and is adaptive to real-time data statistical characteristics for each query, rewriting an original task by using the token, embedding a desensitization parameter into the token, and completing instant desensitization processing by a data source in an execution period, so that the data is in an unpredictable desensitization pattern in each response, and the desensitization accuracy is improved. Therefore, stable anchor points for information association in a cross-query mode are eliminated, and advanced reasoning attacks are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-source data security desensitization and efficient analysis integrated method, belonging to the technical field of big data resource service and data security. BACKGROUND

[0002] Currently, in order to protect data security, sensitive data is desensitized while providing analysis capability to the outside, which is a basic technical requirement. Currently, a commonly implemented technical method in the industry is to define a static global unified data desensitization strategy by a data administrator before providing data services, such as masking generalization or disturbance of specific fields. This set of fixed rules will be applied to all data query and analysis tasks subsequently, and the purpose is to ensure the consistency and auditability of data protection measures. However, with the increasing demand for multi-source data fusion analysis, especially in scenarios requiring deep, multi-dimensional, exploratory data analysis, the above desensitization method relying on static strategy begins to pose a potential risk to data security. The core of this restriction is that the invariability of the strategy provides a stable logical reference system for specific attack methods. Specifically, analysts can initiate a series of logically different but legally compliant queries that meet access permissions. Although a single query returns results that do not contain individual plaintext information after desensitization, since each query is a stable observation of the same desensitized data set, the attacker can gradually narrow down the target range by performing intersection operations on the result sets returned by multiple queries, and ultimately identify the private information of a specific individual with a high probability without directly accessing any single plaintext record.

[0003] To address this problem, one direct improvement idea is to strengthen the intensity of static desensitization strategies, such as using coarser granularity generalization, but this way will reduce the value of data analysis, and even make it unable to meet the normal business analysis requirements, which constitutes a conflict between data availability and security. Another idea is to monitor the query behavior sequence of analysts to identify attacks, but this also faces the problem of accurately defining what is a malicious query pattern, which is prone to a large number of false positives or false negatives, and is essentially a post-response. As can be seen, the existing technical framework faces an inherent technical selection dilemma: the consistency of the strategy adopted to ensure data governance, whose static characteristics itself becomes a prerequisite for the implementation of a specific attack. Not only that, even if it is detached from the rigid static strategy system, the existing technology still has a technical bottleneck in the dynamic generation control logic level of the desensitization rules, which cannot resist advanced attacks. For example, the Chinese invention patent with the authorized publication number CN112989414B discloses a mobile service data desensitization rule generation method based on width learning, which attempts to automatically match or generate desensitization rules according to data attributes (such as type, sensitive level) through a machine learning model. However, this scheme is committed to finding an optimal or most suitable fixed rule for a specific type of data. Although this design idea realizes the automation of rule generation, it still essentially establishes a stable mapping relationship from data characteristics to desensitization rules. Once this mapping relationship is mastered or inferred by the attacker, different queries will again form a logical association anchor point that can be exploited, making advanced reasoning attacks still possible.

[0004] Specifically, the prior art mainly has the following deficiencies: 1. The defense model of the existing desensitization method is static, and the data security boundary constructed by it is a fixed data conversion relationship. It can effectively handle the risk of direct data leakage in a single query, but it cannot avoid reasoning attacks that use multiple legal queries for logical association; 2. In the existing method, the effectiveness of the data security policy is separated from the real-time statistical characteristics of the data it acts on. When the preset desensitization rule is applied to a data subset with extremely uneven statistical distribution, the desensitization operation may be executed in the program but not in fact. The effective privacy protection function; 3. In a real multi-source heterogeneous data analysis environment, data security issues and data semantic heterogeneity issues usually coexist. The existing technical path usually treats the two as two independent governance links, lacking an integrated processing mechanism that can ensure security while eliminating semantic gaps in real time at the query level. Therefore, how to construct a data security processing method that can not sacrifice data analysis usability, make the data present different and unpredictable desensitization patterns in response to each query to break the logical association between multiple queries, and adaptively ensure protection effectiveness according to the real-time statistical characteristics of the data, while also solving the semantic heterogeneity problem of cross-source data in an integrated manner during the query process, has become a technical problem to be solved by the present application. SUMMARY

[0005] The present application provides a multi-source data security desensitization and efficient analysis integrated method, which mainly aims to solve the problem that the existing static desensitization strategy has logical weak points when dealing with advanced reasoning attacks and cannot adapt to real-time statistical characteristics of data nor can it integrally process cross-source semantic heterogeneity.

[0006] To achieve the above purpose, the present application provides a multi-source data security desensitization and efficient analysis integrated method, comprising the following steps: Step a, setting a security policy gateway for intercepting global analysis tasks pointing to at least one data source; Step b, the security policy gateway generates and sends one or more probe queries for obtaining only aggregate statistical characteristics to at least one data source based on the filtering conditions of the global analysis task; Step c, the security policy gateway receives the statistical response returned by the data source after executing the probe query, and the statistical response describes the statistical characteristics of the data subset associated with the global analysis task; Step d, the security policy gateway dynamically generates a confusion token corresponding to the global analysis task for guiding the transformation of structured data based on the received statistical response, and destroys the confusion token associated with the previous global analysis task after generation; Step e, the security policy gateway rewrites the global analysis task using the obfuscated token to generate a sanitized analysis task with the obfuscated token embedded as an execution parameter; Step f, the sanitized analysis task is dispatched to the corresponding one or more data sources, and the data sources perform the sanitized analysis task in the process of which the involved raw data is instantaneously sanitized according to the embedded obfuscated token and the analysis computation is completed to generate the final result.

[0007] Preferably, the obfuscated token in step d contains transformation parameters used to guide the transformation of structured data, the transformation parameters are selected from at least one of the following: a random salt value used for hash calculation, a random seed used for data perturbation, a temporary binning rule used for data generalization, and a pseudo-random sequence seed used for data permutation; the selection and specific value of the transformation parameters are determined by the statistical features of the data subset contained in the statistical response in step c.

[0008] Preferably, the aggregated statistical features obtained by the probe query in step b include at least one of the maximum value, the minimum value, the number of unique values, and the data distribution histogram of the data corresponding to the fields to be sanitized in the global analysis task; in step d, if the statistical response shows that the data distribution concentration in the data subset is higher than a preset first threshold, the security policy gateway generates an obfuscated token using the data perturbation strategy.

[0009] Preferably, the method further comprises an anti-data-source collusion step between step b and step c: in step b, the probe query is generated and sent, specifically: the security policy gateway randomly divides one or more filter conditions required by the probe query into multiple condition subsets with different contents; the security policy gateway generates a probe sub-query with unique content for each target data source based on one or more condition subsets, and sends the probe sub-query to the corresponding target data source; in step c, the statistical response is received, specifically: the security policy gateway receives the local statistical response based on the condition subset returned by each data source after executing the probe sub-query; the security policy gateway executes a statistical reconstruction algorithm on all received local statistical responses to estimate the global statistical features related to the global analysis task, and takes them as the statistical response.

[0010] Preferably, the statistical reconstruction algorithm in step c2 is used to estimate the global number of unique values; the local statistical response is a sketch of the cardinality estimation algorithm with mergable properties returned by each data source, and the statistical reconstruction algorithm obtains the estimate of the global number of unique values by performing union operation on multiple sketches of the cardinality estimation algorithm , which follows the following rules: , wherein is the estimate of the reconstructed global number of unique values, is the estimate of the global number of unique values from the a cardinality estimation algorithm summary returned by the data source with the mergable property, a total number of data sources participating in the query, a union operation corresponding to the cardinality estimation algorithm summary.

[0011] Preferably, the method further comprises a step of adjusting the randomness strength of the obfuscated token based on the query frequency: the security policy gateway is built-in with an access pattern observer for monitoring the query frequency of the global analysis task; the obfuscated token is dynamically generated in step d, and the randomness strength of the generated obfuscated token is dynamically adjusted according to the query frequency monitored by the access pattern observer: when the query frequency is lower than a preset second threshold, the obfuscated token is generated with a first randomness strength; when the query frequency is higher than or equal to the second threshold, the obfuscated token is generated with a second randomness strength higher than the first randomness strength.

[0012] Preferably, the first randomness strength corresponds to an algorithm with a first computational complexity for generating the transformation parameters in the obfuscated token; the second randomness strength corresponds to an algorithm with a second computational complexity higher than the first computational complexity for generating the transformation parameters in the obfuscated token, and the algorithm with the second computational complexity increases the processing load of the security policy gateway.

[0013] Preferably, the method further comprises a step of integrating the semantic heterogeneity across the data sources: the security policy gateway is associated with a semantic metadata repository storing the mapping relationship between the standard business terms and the local implementations of each data source; the obfuscated token generated in step d is a composite token, and its internal structure includes a security domain and a semantic domain; the security domain carries the security parameters for guiding the instant de-sensitization processing; the semantic domain carries the semantic translation rules generated based on the semantic metadata repository, for converting the standard business terms in the global analysis task into the local semantic implementations of each data source; the task rewriting in step e further includes a semantic conversion of the global analysis task according to the semantic translation rules in the semantic domain.

[0014] Preferably, the semantic translation rules carried in the semantic domain include: mapping rules of the standard business terms to the local field names of each data source, and mapping rules of the enumerated values of the standard business terms to the local field encoding values of each data source; when performing the task rewriting in step e, the security policy gateway respectively generates, for each target data source, a de-sensitization analysis task that conforms to the syntax and encoding of the local specification of the data source, according to the semantic translation rules.

[0015] Preferably, the global analysis task is a structured query language (SQL) query; the task rewriting in step e is a rewriting of the original SQL query statement, which passes the obfuscated token as a parameter to the user-defined function (UDF) or built-in function contained in the SQL query statement and executed by the data source, to drive the instant de-sensitization processing in step f.

[0016] Compared with the prior art, the present application has the following advantages: 1. By setting a security policy gateway to intercept the global analysis task, a one-time and unique obfuscation token corresponding to the task is dynamically generated, the original analysis task is rewritten using the token, and an execution parameter for instant desensitization is embedded in the rewritten task, so that the data source performs instant processing on the original data involved in the memory when executing the rewritten task. The execution life cycle of the data security policy is compressed from static configuration to synchronization with single query behavior. Since the data response form faced by each query is determined by an independent token that is destroyed after use, any attack path that attempts to associate information by comparing multiple legal query results is no longer valid due to the lack of stable data targets for reference.

[0017] 2. Before generating the obfuscation token, the security policy gateway first generates a probe query to obtain aggregated statistical features according to the original analysis task, and determines the content of the generated obfuscation token according to the response returned by the data source, which describes the statistical features of the related data subset. This closed-loop feedback of real-time data feature perception before desensitization strategy generation makes the application of desensitization rules no longer a programmatic execution independent of data distribution, but a fine-tuning process that can adaptively adjust its protection mode according to the local concentration or dispersion state of data under specific queries, avoiding the protection failure that may occur due to mismatch between desensitization strategy and data statistical features.

[0018] 3. The security parameters used to guide instant desensitization processing and a set of semantic translation rules used to convert standard business terms into local implementation of each data source are placed in a composite obfuscation token. In the task rewriting step, the desensitization logic is embedded according to the security parameters in the token, and the semantic translation rules in the token are used for local semantic conversion of the data source. By reusing the generation and transmission path of the same token, data security processing and cross-source data integration, which are originally independent governance links, are integrated into an atomic operation that is completed synchronously at the moment of query, presenting a logically unified data view to analysts. This process uses the existing query channel in the system to complete, replacing the process of building a data warehouse in advance for data extraction, transformation, and loading.

[0019] 4. When sending probe queries to multiple data sources to obtain statistical characteristics, first randomly divide the filter conditions of the probe queries into multiple condition subsets that are different in content, and generate probe sub-queries that are unique in content for each data source. After receiving the local statistical responses returned by each data source, estimate the global statistical characteristics through a statistical reconstruction algorithm at the security policy gateway side; this method cuts off the information channel for collusion comparison between data sources through asymmetric segmentation of probe queries, and when the access pattern observer built into the gateway detects that the query frequency exceeds the preset threshold, it switches to a token generation algorithm with higher randomness, which makes the system respond more slowly when dealing with automated high-frequency queries, constituting a computational cost constraint on attack behavior. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of the dynamic desensitization and analysis integration process of the present application based on one-time tokens; Figure 2 A response time variation diagram for the present application to deal with high-frequency query attacks; Figure 3 A token life cycle management timing diagram based on query frequency for the present application. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described in detail below, but it should be understood that the following embodiments are only exemplary descriptions of the present application and are not intended to limit the scope of protection of the present application.

[0022] The disclosed multi-source data security desensitization and efficient analysis integrated method has a security policy gateway as the core of the system architecture, which is deployed between one or more data analysis clients and multiple data sources, and constitutes a necessary channel for data analysis requests; the complete workflow of the method mainly consists of analysis task interception and analysis, probe query based on data statistical characteristics, dynamic generation of one-time obfuscation token, secure rewriting of analysis task, and instant desensitization calculation executed by the data source, etc. These steps work together to achieve a data processing method that completes data dynamic desensitization and semantic conversion during query execution; in a multi-source data joint analysis application scenario, for example, a financial institution needs to jointly assess the risk of user data scattered in the bank core database, credit card business database and third party credit investigation database, and the data analyst submits a global analysis task through the analysis platform, which is usually a structured query language (SQL) query; to deal with the risk that attackers may identify individual information by initiating a series of logically associated queries during data analysis, the method claimed in the present application is configured to first intercept the global analysis task through the security policy gateway, which uses an SQL parser to identify the target field, filter condition and aggregation operation type pointed to by the query statement, which provides the necessary context input for the dynamic generation of subsequent security policies; considering that the preset static desensitization rule may not effectively protect privacy in fact when applied to a statistically uneven data subset, the method sets a pre-posed data real-time feature perception process before generating a specific desensitization strategy; specifically, the security policy gateway does not immediately generate an obfuscation token for desensitization after parsing the original global analysis task, but generates and sends one or more probe queries for obtaining only aggregated statistical features to the relevant data sources based on the filter conditions of the task. These probe queries are designed as low-cost aggregation queries, and their query results do not contain individual information; after the data source executes the probe query, it returns a statistical response describing the statistical characteristics of the target data subset, such as a data object containing the maximum value, minimum value and number of unique values; after receiving the response, the security policy gateway obtains the real-time distribution characteristics of the data associated with this query, for example, if the statistical response shows that the difference between the maximum value and the minimum value is less than a preset first threshold, indicating that the data is highly concentrated under this query condition, the security policy gateway then decides to generate an obfuscation token using a data perturbation strategy instead of a data generalization strategy to ensure the actual effectiveness of desensitization.

[0023] To eliminate the stable anchor point of information correlation across queries, the core mechanism of the method is to dynamically generate a one-time obfuscated token for each query; The generation of the obfuscated token is performed after the security policy gateway receives the statistical response of the probe query, and the content is a set of transformation parameters used to guide the transformation of structured data, which are selected according to the specific security policy and data statistical characteristics, and can be selected from at least one of the random salt value used for hash calculation, the random seed used for data disturbance, the temporary binning rule used for data generalization, and the pseudo-random sequence seed used for data permutation; For example, when the decision adopts the data disturbance strategy, the security policy gateway calls a pseudo-random number generator to generate a random seed, such as 1f7b3d9c, and uses it as a transformation parameter to construct an obfuscated token; After generating the obfuscated token corresponding to this task, the system immediately destroys the obfuscated token associated with the previous global analysis task, and this one-time feature makes any attack path that attempts to compare multiple query results ineffective due to the lack of a stable reference point; After obtaining the one-time obfuscated token, the security policy gateway does not directly process the background data, but uses the token to rewrite the original global analysis task to generate a desensitization analysis task that embeds the obfuscated token as an execution parameter; In the scenario where the global analysis task is an SQL query, task rewriting specifically refers to rewriting the original SQL query statement, which passes the obfuscated token as a parameter to the user-defined function UDF or built-in function executed by the data source in the SQL query statement; Taking a query that needs to disturb the salary field as an example, if the original query is SELECT AVG(salary) FROM employee WHERE dept = finance, and the obfuscated token generated by the security policy gateway for this query is the random seed 1f7b3d9c, then the rewritten desensitization analysis task will become SELECT AVG(ADD_NOISE(salary, 1f7b3d9c)) FROM employee WHERE dept = finance, where ADD_NOISE() is a user-defined function deployed on the data source side; The rewritten desensitization analysis task is then distributed to the corresponding data source for execution, and the query engine of the data source will call the ADD_NOISE() function in memory during the execution of the task, and perform real-time desensitization processing on the involved salary original data according to the token parameter 1f7b3d9c and complete the final analysis calculation, without changing any original data stored on the disk.

[0024] Considering the risk of probe queries being compared between different data sources to infer analytical intent in multi-data source joint query environments, this method further provides a step to prevent data source collusion. When sending probe queries to multiple data sources, the security policy gateway first randomly partitions one or more filtering conditions of the probe query into multiple condition subsets with distinct content. Subsequently, the gateway generates a unique probe subquery for each target data source based on a condition subset and sends these content-distinct probe subqueries to their respective target data sources. After executing the probe subquery, each data source returns a local statistical response based on the condition subset. For example, in the scenario of estimating the number of globally unique values, each data source can return a cardinality estimation algorithm summary with mergeable characteristics. Finally, after receiving all local statistical responses, the security policy gateway estimates the global statistical characteristics by executing a statistical reconstruction algorithm, such as performing a union operation on multiple cardinality estimation algorithm summaries, which follows rules. ,in, This is an estimate of the number of globally unique values ​​after reconstruction. For from the first A summary of the cardinality estimation algorithms returned by the data sources. The total number of data sources participating in the query. The method employs a union operation corresponding to the cardinality estimation algorithm. Through asymmetric partitioning and centralized reconstruction using probe queries, it blocks information channels for collusion between data sources. Simultaneously, to enhance the system's processing capacity when dealing with automated high-frequency queries, this method also includes a step of adjusting the randomness strength of the obfuscation token based on the query frequency. The security policy gateway incorporates an access pattern observer to monitor the query frequency of specific global analysis tasks. The system presets a second threshold for query frequency, which is determined by analyzing the system's historical query logs under normal load and taking the 99th percentile value of its query frequency distribution. When the access pattern... When the query frequency detected by the observer is lower than the second threshold, the system determines it as a regular access. In this case, the first randomness strength is used when dynamically generating the obfuscation token, that is, a pseudo-random number generation algorithm with low computational overhead is called. When the query frequency is higher than or equal to the second threshold, the system determines that it may encounter automated queries, and then switches to using a second randomness strength higher than the first randomness strength to generate the obfuscation token. For example, a cryptographically secure random number generation algorithm based on hardware noise with higher computational complexity is called. By increasing the token computational complexity under high-frequency queries, the system response time is extended, thus constraining the cost of automated query behavior.

[0025] Finally, in order to handle the semantic heterogeneity problem existing in cross-data source analysis, the method extends the structure of obfuscated tokens and the steps of task rewriting; the security policy gateway is associated with a semantic metadata warehouse that stores the mapping relationship between standard business terms and local implementations of each data source, which records mapping rules such as that the standard gender corresponds to the gender field of data source A and the sex_code field of data source B; under this configuration, the obfuscated token generated by the security policy gateway is a composite token, which is divided into a security domain and a semantic domain in its internal structure; the security domain carries the security parameters used to guide the instant de-identification processing as described above, while the semantic domain carries the semantic translation rules generated based on the semantic metadata warehouse, which are used to convert the standard business terms in the global analysis task into the local implementation of each data source; accordingly, in the task rewriting step, the security policy gateway not only embeds the de-identification logic according to the security parameters in the token security domain, but also performs semantic translation on the global analysis task according to the semantic translation rules in the semantic domain, and for each target data source, a de-identified analysis task is generated that conforms to the local specification of the data source in syntax and coding.

[0026] Example 1: In a cross-institutional medical data collaborative analysis scenario, an authorized data analyst needs to access three independent data sources that have been processed by static de-identification rules, namely hospital A database recording patient treatment information, disease control center B database recording specific pathogen detection information, and scientific research institution C database recording gene sequencing information; the static de-identification rules have hashed all patient names and generalized ages according to a fixed five-year interval; at this time, the data analyst attempts to correlate the information of a specific individual by executing a series of global analysis tasks; the analyst first submits the first global analysis task to the system, requiring to query the hash list of all patients who have visited a specific department and whose age is generalized to 40-44 years old in hospital A database; after intercepting the task, the security policy gateway sends a probe query to hospital A database to obtain the statistical characteristics of the patient age data that meet the query conditions; the statistical response returned by the probe query shows that all patient ages in the data subset are concentrated between 41 and 42 years old, according to this response, the security policy gateway judges that the fixed binning generalization rule cannot provide effective protection, and therefore dynamically generates a one-time obfuscated token using the data perturbation strategy, which contains a random seed, and uses the token to rewrite the original query as a de-identified analysis task that performs instant numerical perturbation on the age field when executed, and then returns a perturbed result set to the analyst.

[0027] Then, the analyst submits a second global analysis task, requiring a list of sample hashes of all positive pathogen detections in CDC B database and from a certain community. The security policy gateway intercepts the task and generates a new one-time obfuscated token independent of the previous task, which contains a random salt value for hash calculation. The original task is rewritten as a desensitized analysis task that performs salted hash processing on the geographic location field, and returns a second result set. The analyst then submits a third global analysis task, requiring a list of sequencing sample hashes containing a certain genetic marker in Research Institution C database. The security policy gateway again performs the same process, generates a third independent one-time obfuscated token, and returns a third result set. When the analyst attempts to perform an intersection operation on the three returned result sets, the correlation operation cannot obtain an effective convergent result. Since the three queries are processed through three independent, use-and-destroy obfuscated tokens, the same individual's age is randomly disturbed in the first result set, and the geographic location is salted in the second result set. The form of the individual in different query results loses stable correlatability, and the analyst cannot establish effective logical correlation between multiple queries, so the analyst's attempt to correlate is invalid.

[0028] Example 2: To verify the effectiveness of the method claimed in the present application in inhibiting association query attacks, a control test environment was set up in this example; the test environment used a data set containing one million simulated user transaction records, in which 1000 target users were pre-set that could be identified by 3 to 5 different dimension queries; the test platform consisted of a server configured with a 32-core central processor and 128 GB of memory, and the server was deployed with a PostgreSQL database system and a proxy service program as a security policy gateway; this test set up two control groups and one test group; control group A used a static desensitization method, which pre-generalized all transaction amounts to the hundred integer bit; control group B used a simplified dynamic desensitization method, which could generate a one-time confusion token for each query, but lacked the pre-probe query step, and its desensitization strategy was fixed to apply a constant range of random noise to the transaction amount; the test group used the complete method claimed in the present application, including all steps of performing probe queries and adaptively generating one-time confusion tokens based on statistical responses; the test was tested by an automated script, which simulated the behavior of an attacker, measured the re-identification success rate of target users by increasing the number N of association queries, and recorded the average response time of standardized aggregate queries; in the test, the automated script launched association query attacks on the three groups, with the value of the number N of association queries ranging from 1 to 10, each N value repeated 1000 times of independent identification attempts, and the re-identification success rate was calculated; for the test group, the statistical response returned by the probe query was used to decide the specific generation method of the confusion token after the security policy gateway received the query, when the statistical response showed that the data subset value distribution concentration degree was higher than the pre-set first threshold, the system generated a confusion token using the data perturbation strategy, otherwise the temporary binning rule was used; Table 1 records the data of this comparative test.

[0029] Table 1: Comparison of re-identification success rate and performance under different processing methods.

[0030]

[0031] Referring to Table 1, control group A achieved a re-identification success rate of 99.1% when the number of related queries N increased to 5. Control group B, due to the use of one-time tokens, had a lower re-identification success rate, but still maintained a 1.5% success rate when N was 10. The experimental group maintained a re-identification success rate of 0.0% throughout the entire test, regardless of the increase in the number of related queries. This is because the method used in the experimental group not only severed the association between queries through one-time tokens, but its pre-emptive probe query and adaptive strategy generation mechanism could select a de-identification strategy based on the real-time statistical characteristics of the data, thus compensating for the shortcomings of the fixed de-identification strategy in control group B when dealing with specific data distributions. Performance data showed that the average query response time of the experimental group was 88.7ms.

[0032] Example 3: This example combines Figures 1 to 3 This section explains the integrated method for secure de-identification and efficient analysis of multi-source data, such as... Figure 1 As shown, the process begins with a global analysis task submitted by a data analyst. This task is first processed by the task interception and parsing module to identify its query targets, filtering conditions, and operations. Simultaneously, the access pattern observer monitors the query frequency and inputs the monitoring results as the basis for adjusting the strategy into the dynamic one-time obfuscation token generation module. This token generation module also receives statistical responses from the probe query generation stage, which aims to obtain the aggregate statistical characteristics of a subset of data. The final one-time obfuscation token, which is destroyed after use, is used together with the translation rules from the semantic metadata repository in the task rewriting step. This transforms the original task into a de-identification analysis task that embeds de-identification parameters and semantic rules. When the rewritten task is executed, it performs de-identification in memory through real-time de-identification calculation and returns the final result to the analyst. This process does not change any original data.

[0033] like Figure 2 As shown, the impact of the access pattern observer on system performance in this process is specifically reflected in: Figure 2 The performance diagram shown illustrates the system response time under three different operating conditions: a normal query frequency of <10 queries / minute, a medium frequency of 10-30 queries / minute, and a high-frequency attack of >30 queries / minute. As can be seen from the diagram, under normal and medium query frequencies, the system response time remains at a relatively low and stable level as the number of queries increases. However, under the high-frequency attack mode, the response time increases sharply and linearly with the increase in the number of queries, thus effectively constraining the computational cost of automated high-frequency query behavior.

[0034] like Figure 3As shown, the figure describes the message passing process between the five logical components of data analyst, security policy gateway, access pattern observer, token generator and token storage. When processing the first query task, the access pattern observer returns the normal frequency state, the security policy gateway requests the token generator to generate token 1 with the first intensity accordingly, the token generator stores the generated token 1 in the token storage before returning it to the security policy gateway. When processing the second query task, the access pattern observer returns the high frequency state, the security policy gateway requests the token generator to generate token 2 with the second intensity, the token generator generates the high-intensity random parameter and stores token 2 before issuing a command to the token storage to destroy token 1, and finally returns token 2 to the security policy gateway for task rewriting, which fully embodies the dynamic adjustment of token intensity with query frequency and the one-time feature of token use-and-destroy.

[0035] Embodiment 4: In a data service scenario requiring profiling analysis of a specific user group, the system receives a global analysis task aiming to count the age distribution of users who have purchased student-specific products, where the age field is a sensitive field. In this working condition, if the data subset filtered by the query has a highly concentrated age distribution in the 19-21 age interval, a fixed generalization strategy that divides the age interval by five years will not be able to effectively protect privacy. To address this technical problem, the method claimed in the present application performs an adaptive decision-making process before generating the obfuscated token. The security policy gateway automatically generates a probe query based on the received global analysis task, and the generation procedure is as follows: first, use the SQL parser to identify the desensitization field age and the filter condition in the original task; second, retain the filter condition of the original task and target the age field to construct a set of query statements for obtaining its aggregate statistical characteristics, including the MIN(age) query for obtaining the minimum value, the MAX(age) query for obtaining the maximum value, and the COUNT(DISTINCT age) query for obtaining the number of unique values; the set of probe queries is then distributed to the data source for execution.

[0036] After the data source executes the probe query, a structured statistical response is returned to the security policy gateway. Based on the statistical characteristics in the response, the security policy gateway decides a desensitization policy suitable for the current data distribution. The judgment basis for this decision-making process is a preset first threshold. The calibration of the threshold follows the following offline procedure: first, select historical query logs as the calibration dataset; second, quantify the data distribution concentration of each data subset associated with each query in the calibration dataset. Here, the data distribution concentration is quantified as the ratio of the number of unique values in the data subset to the total amount of data. When the data subset corresponding to an online query has a distribution concentration higher than the first threshold, the system generates a confusion token using the data perturbation strategy. In this scenario, the received statistical response is {min_age: 19, max_age: 21, distinct_age_count: 8, total_count: 2000}, and the first threshold determined by the aforementioned offline calibration procedure has a unique value to total amount ratio of 0.01. The security policy gateway calculates the ratio of the current data subset as 8 / 2000 = 0.004. Since the calculated ratio 0.004 is less than the threshold 0.01, the system determines that the data subset is highly concentrated, and the conventional generalization strategy is ineffective. Accordingly, the security policy gateway generates a confusion token using the data perturbation strategy, which contains a random seed c7a3b2f9. This token is then used to rewrite the original global analysis task.

[0037] Embodiment 5: Before the method claimed in the present application is applied to a new multi-source heterogeneous data analysis environment, the system needs to perform a standardized semantic metadata warehouse construction procedure. This procedure extracts the schema of each data source to be accessed through a data source scanning tool, obtains its table structure, field name, data type, and coding specification, and stores these raw metadata in a structured manner. Subsequently, the system maps the extracted local metadata of each data source to a predefined standard business terminology dictionary. This process is audited and calibrated by data engineers, and the confirmed mapping relationships are loaded into the semantic metadata warehouse. During system operation, the semantic metadata warehouse is configured with periodic synchronization and verification mechanisms. When a change in the schema of any data source is detected, incremental updates and consistency checks are automatically triggered.

[0038] The probe query segmentation strategy against data source collusion in the method is parameterized configured in system deployment; the system internally has two randomization segmentation strategies of probe query filtering conditions, i.e., a completely disjoint segmentation strategy and a partially overlapping segmentation strategy; in processing a global analysis task of high sensitivity, or when there is a high trust risk between target data sources, the system is configured to adopt the completely disjoint segmentation strategy, which randomly divides the filtering conditions of the original query into multiple disjoint subsets, and sends probe sub-queries based on these subsets to each data source; in a regular analysis task, the system is configured to adopt the partially overlapping segmentation strategy, which allows a certain proportion of overlapping elements between subsets when randomly segmenting the filtering conditions, and when using the HyperLogLog base estimation algorithm to perform set operation on the returned local statistical responses, this segmentation method can improve the accuracy of the estimated value of the global unique value number.

[0039] Before the method claimed in the application is deployed in a production environment, a standardized pre-calibration procedure is performed to determine the second threshold value in the access pattern observer for distinguishing between regular access and high-frequency queries, and to configure the two confusion token generation algorithms of different randomness strengths; the procedure first sets a baseline data collection period of 7 natural days, during which the access pattern observer of the security policy gateway records query frequency samples from independent IP addresses with a 60-second sliding time window; after the collection period ends, the system statistically analyzes all recorded query frequency samples, calculates their cumulative distribution function, and selects the frequency value corresponding to the 99.5 percentile as the second threshold value for the deployment environment; based on the calibrated second threshold value, the system is configured as follows: when the query frequency monitored by the access pattern observer is lower than the second threshold value, the system generates confusion tokens with the first randomness strength, which corresponds to calling a pseudo-random number generator to generate transformation parameters; when the query frequency is equal to or higher than the second threshold value, the system switches to using the second randomness strength, which corresponds to calling a cryptographic secure random number generator with higher computational complexity to generate transformation parameters.

[0040] In a scenario where joint analysis of customer data from commercial bank A and insurance company B is required, the global analysis task aims to query all high net worth customers with male gender, but the gender field in bank A database is named gender and coded as M / F, while the corresponding field in insurance company B database is named sex_code and coded as 1 / 0; when the security policy gateway receives the global analysis task using the standard business term sex=male, it first queries the pre-constructed semantic metadata warehouse to obtain the semantic translation rules for the task; then, the security policy gateway generates a composite token, the security domain of which carries a security parameter for random perturbation of the customer asset field, and the semantic domain of which carries the aforementioned obtained semantic translation rules for converting the standard business term into the semantics implemented by the two data sources; finally, the security policy gateway rewrites the original task using the composite token to generate two independent desensitization analysis tasks that conform to the local specifications in syntax and coding, wherein the task distributed to bank A has its filter condition converted to WHERE gender=M, and the task distributed to insurance company B has its filter condition converted to WHERE sex_code=1, and both tasks contain user-defined function UDF calls for real-time desensitization of the asset field.

[0041] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting, and although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method integrating secure de-identification and efficient analysis of multi-source data, characterized in that, Includes the following steps: Step a, configure a security policy gateway to intercept global analysis tasks that point to at least one data source; Step b: Based on the filtering conditions of the global analysis task, the security policy gateway generates and sends one or more probe queries to at least one data source, which are only used to obtain aggregate statistical features. Step c: The security policy gateway receives a statistical response returned by the data source after performing a probe query. The statistical response describes the statistical characteristics of the subset of data associated with the global analysis task. Step d: Based on the received statistical response, the security policy gateway dynamically generates an obfuscation token that is uniquely associated with the current global analysis task to guide the transformation of structured data, and destroys the obfuscation token associated with the previous global analysis task after generation. Step e: The security policy gateway uses the obfuscation token to rewrite the global analysis task to generate a de-identified analysis task that embeds the obfuscation token as an execution parameter. Step f involves distributing the de-identification analysis task to one or more corresponding data sources. During the execution of the de-identification analysis task, the data sources perform real-time de-identification processing on the original data involved based on the embedded obfuscation token and complete the analysis calculation to generate the final result.

2. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 1, characterized in that, In step d, the obfuscation token contains transformation parameters to guide the transformation of structured data. The transformation parameters are selected from at least one of the following: a random salt value for hash calculation, a random seed for data perturbation, a temporary binning rule for data generalization, and a pseudo-random sequence seed for data permutation. The selection and specific value of the transformation parameters are determined by the statistical characteristics of the subset of data contained in the statistical response in step c.

3. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 1, characterized in that, In step b, the aggregated statistical features obtained by the probe query include at least one of the maximum value, minimum value, number of unique values, and data distribution histogram of the data corresponding to the fields to be de-identified in the global analysis task; in step d, if the statistical response shows that the data distribution concentration within the data subset is higher than a preset first threshold, the security policy gateway generates an obfuscation token using a data perturbation strategy.

4. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 1, characterized in that, Between steps b and c, the method further includes a step to prevent data source collusion: In step b, a probe query is generated and sent, specifically: the security policy gateway randomly divides one or more filtering conditions required by the probe query into multiple condition subsets with different content. For each target data source, the security policy gateway generates a unique probe subquery based on one or more condition subsets and sends the probe subqueries to their respective target data sources. Step c involves receiving statistical responses, specifically: the security policy gateway receives local statistical responses based on condition subsets returned by each data source after executing the probe subqueries; the security policy gateway performs a statistical reconstruction algorithm on all received local statistical responses to estimate global statistical features related to the global analysis task and uses them as statistical responses.

5. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 4, characterized in that, In step c2, the statistical reconstruction algorithm is used to estimate the number of globally unique values; the local statistical response consists of cardinality estimation algorithm summaries with merging properties returned by each data source. The statistical reconstruction algorithm obtains the estimate of the number of globally unique values ​​by performing a union operation on multiple cardinality estimation algorithm summaries. Its operation follows these rules: ,in, This is an estimate of the number of globally unique values ​​after reconstruction. For from the first A summary of cardinality estimation algorithms with mergeable characteristics returned by each data source. The total number of data sources participating in the query. This refers to the union operation corresponding to the cardinality estimation algorithm summary.

6. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 1, characterized in that, The method also includes the step of adjusting the randomness strength of the obfuscation token based on the query frequency: the security policy gateway has a built-in access pattern observer to monitor the query frequency of the global analysis task; the obfuscation token is dynamically generated in step d, and the randomness strength of the generated obfuscation token is dynamically adjusted according to the query frequency monitored by the access pattern observer: when the query frequency is lower than a preset second threshold, the obfuscation token is generated with the first randomness strength. When the query frequency is higher than or equal to the second threshold, a confusion token is generated using a second randomness strength that is higher than the first randomness strength.

7. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 6, characterized in that, The first randomness strength corresponds to the transformation parameters within the confusion token generated by an algorithm with the first computational complexity; The second randomness intensity corresponds to the generation of transformation parameters within the obfuscated token using an algorithm with a second computational complexity higher than the first computational complexity. This second computational complexity algorithm increases the processing load on the security policy gateway.

8. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 1, characterized in that, The method also includes a step of integrated processing of semantic heterogeneity across data sources: the security policy gateway is associated with a semantic metadata repository that stores the mapping relationship between standard business terms and the local implementation of each data source; the obfuscation token generated in step d is a composite token, the internal structure of which includes a security domain and a semantic domain; The security domain carries security parameters used to guide immediate desensitization processing; The semantic domain carries semantic translation rules generated based on the semantic metadata repository, which are used to convert standard business terms in global analysis tasks into semantic translation rules implemented locally by each data source; the task rewriting in step e also includes semantic transformation of the global analysis task based on the semantic translation rules in the semantic domain.

9. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 8, characterized in that, The semantic translation rules carried in the semantic domain include: mapping rules from standard business terms to local field names of each data source, and mapping rules from enumerated values ​​of standard business terms to encoded values ​​of local fields of each data source; when the security policy gateway performs task rewriting in step e, it generates a desensitization analysis task that conforms to the local specification of the data source in terms of both syntax and encoding for each target data source according to the semantic translation rules.

10. The integrated method for secure de-identification and efficient analysis of multi-source data according to claim 1, characterized in that, The global analysis task is a Structured Query Language (SQL) query; in step e, the task rewriting is a rewrite of the original SQL query statement, which takes the obfuscation token as a parameter and passes it to the user-defined function (UDF) or built-in function contained in the SQL query statement that is executed by the data source, in order to drive the on-the-fly de-identification process in step f.

Images