Data row and column permission control and desensitization method based on Spark engine

Abstract

The present invention provides a data row and column permission control and desensitization method based on the Spark engine, and relates to the field of data processing technology, including receiving a user data processing request, obtaining a user's data access permission rule from a permission management server, parsing the permission rule into a Spark SQL syntax tree and injecting row-level filtering conditions and column-level field filtering, and constructing a Spark distributed processing task including data scanning, filtering, projection and desensitization nodes and submitting it for execution. The present invention can realize refined data access control in the distributed data processing process, ensure data security, improve data processing efficiency, and is suitable for large-scale data processing scenarios.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a data row and column authority control and desensitization method based on a Spark engine. Background Art

[0002] With the rapid development of big data technology, enterprises and organizations have accumulated massive amounts of data assets. These data often contain sensitive information, which requires strict access control and data protection measures. As a mainstream big data computing engine, Apache Spark is widely used in data processing and analysis scenarios. Implementing fine-grained data access control and desensitization protection on the Spark platform is of great significance to ensuring data security. At present, common data access control solutions are mainly implemented through database-level view mechanisms or application-level permission verification, while data desensitization is mostly processed using independent ETL processes.

[0003] The main problems of the existing technology:

[0004] Existing data access control solutions often couple permission control with business logic at the application layer, resulting in decentralized permission rules that are difficult to manage in a unified manner, and the potential security risk of permission control being bypassed. At the same time, this approach also increases the complexity of application development and reduces development efficiency.

[0005] Traditional data desensitization solutions usually require pre-desensitization of data and storage of desensitized data copies, which not only increases storage overhead, but also easily leads to confusion in data version management. Moreover, different users may have different desensitization requirements for the same data, and pre-processing methods are difficult to flexibly meet diverse desensitization requirements.

[0006] In large-scale distributed data processing scenarios, existing permission control and desensitization solutions often require filtering and processing data only after it is loaded into memory. This approach will generate a large amount of invalid data reading and transmission overhead, affecting system performance and resource utilization efficiency. Summary of the invention

[0007] The embodiment of the present invention provides a data row and column permission control and desensitization method based on the Spark engine, which can solve the problems in the prior art.

[0008] According to a first aspect of the embodiments of the present invention,

[0009] Provides data row and column permission control and desensitization methods based on the Spark engine, including:

[0010] Receive a data processing request submitted by a user, the data processing request including a target data table identifier, a data processing operation type, and user identity information; obtain the user's data access permission rules from a rights management server according to the user identity information, the data access permission rules including a row-level filtering condition, an accessible field list, and a field desensitization rule;

[0011] Parse the data access permission rule into a Spark SQL syntax tree, inject the row-level filtering condition into the Spark SQL syntax tree to generate a first intermediate syntax tree, perform column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree; construct a Spark distributed data processing task based on the second intermediate syntax tree, wherein the Spark distributed data processing task includes a data scanning node, a data filtering node, a field projection node, and a data desensitization node;

[0012] The data scanning node reads the original data of the target data table, the data filtering node executes the row-level filtering condition, the field projection node only selects the fields in the accessible field list, and the data desensitization node desensitizes the sensitive fields according to the field desensitization rules; the Spark distributed data processing task is submitted to the Spark cluster for execution, and a result data set that complies with permission control and completes the desensitization process is obtained.

[0013] According to the user identity information, the user's data access permission rules are obtained from the permission management server. The data access permission rules include row-level filtering conditions, accessible field lists, and field desensitization rules, including:

[0014] Receive user identity information, the user identity information includes user role identification, organizational structure information and timestamp information; construct a user authority feature vector based on the user identity information, the user authority feature vector includes a role inheritance relationship matrix, an organizational structure authority inheritance tree and a time dimension authority change curve; obtain an authority rule set from a distributed authority rule storage system according to the user authority feature vector, the distributed authority rule storage system adopts a hierarchical storage structure to store user basic authority layer rules, role inheritance authority layer rules and temporary authorization authority layer rules;

[0015] Perform consistency verification on the permission rule set, generate rule verification results, and incrementally update the permission rule set based on the rule verification results; parse the updated permission rule set into a conditional expression, and perform semantic normalization on the conditional expression to obtain a basic filtering condition set; construct a condition dependency graph based on the basic filtering condition set, and optimize the condition dependency graph through a graph traversal algorithm to obtain a minimum necessary condition set;

[0016] A field access dependency graph is constructed according to the minimum necessary condition set, and a minimum necessary field set is calculated based on the field access dependency graph; a desensitization metric is evaluated on the fields in the minimum necessary field set to generate a field desensitization scoring matrix; corresponding desensitization strategies are assigned to different types of fields according to the field desensitization scoring matrix to generate a field desensitization rule set; the minimum necessary condition set, the minimum necessary field set, and the field desensitization rule set are combined to generate user data access permission rules.

[0017] Parsing the data access permission rule into a Spark SQL syntax tree, injecting the row-level filtering condition into the Spark SQL syntax tree to generate a first intermediate syntax tree, and performing column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree includes:

[0018] Receive data access permission rules, the data access permission rules including a filter condition set, a column access permission set, and a desensitization rule set; parse the data access permission rules into a formalized permission rule expression, the formalized permission rule expression including an operator node set, a projection node set, a filter node set, and a data source node set; construct a permission rule syntax tree node dependency matrix based on the formalized permission rule expression;

[0019] The original query statement is parsed to generate an original syntax tree, wherein the original syntax tree includes a node set, an edge set, and a root node; based on the authority rule syntax tree node dependency matrix, the filter conditions in the filter condition set are normalized to obtain normalized filter conditions; the injection cost and impact factor of each node in the original syntax tree are calculated, and the node with the smallest weighted sum of the injection cost and the impact factor is selected as the filter condition injection point;

[0020] Inject the normalized filter condition into the filter condition injection point to generate a first intermediate syntax tree; perform predicate push-down optimization on the first intermediate syntax tree, filter the edge set that meets the push-down condition based on a preset optimization threshold, and push down the filter condition in the edge set that meets the push-down condition; construct a field projection node according to the column access permission set, and reconstruct the first intermediate syntax tree based on the field projection node to obtain a second intermediate syntax tree.

[0021] Calculating the injection cost and impact factor of each node in the original syntax tree, and selecting the node with the smallest weighted sum of the injection cost and the impact factor as the filter condition injection point includes:

[0022] ;

[0023] in, Indicates the optimal injection point that is finally selected, that is, the node position in the syntax tree that is most suitable for injecting the filter condition. It means to find the node p that makes the following expression get the minimum value in all the node sets V. represents the cost of injecting the filter condition at node p, represents the set of nodes affected by the injection point, that is, all relevant nodes downstream of the injection point p. Represents the impact factor of the injection operation on node v.

[0024] The data scanning node reads the original data of the target data table, the data filtering node executes the row-level filtering condition, the field projection node selects only the fields in the accessible field list, and the data desensitization node performs desensitization processing on the sensitive fields according to the field desensitization rule, including:

[0025] A data partitioning strategy optimization model is constructed based on data distribution characteristics, data locality, and data partition size of a target data table; data partitions are performed on the target data table according to the data partitioning strategy optimization model; a data reading window size is dynamically adjusted based on a ratio of a target processing time to an actual processing time, and original data is read from the data partition through the data reading window;

[0026] Calculate the complexity of data filtering conditions, determine the parallelism of filtering operations based on the complexity of filtering conditions, and use the parallelism of filtering operations as the number of filtering nodes; construct a multi-filtering node processing unit according to the number of filtering nodes, and the multi-filtering node processing unit performs parallel filtering operations on the original data; calculate the filtering efficiency based on the ratio of the number of records before and after filtering and the benchmark processing time;

[0027] Constructing a field dependency graph, the field dependency graph including a field set, dependency relationships, and dependency costs; calculating processing costs between adjacent fields based on the field dependency graph, and generating an optimal field projection sequence based on the processing costs; performing a field projection operation on the filtered data based on the optimal field projection sequence;

[0028] Calculate the degree of protection, data quality preservation, and computing overhead for the fields that need to be desensitized, and select the optimal desensitization algorithm; build a multi-level cache optimization strategy based on data access frequency and last access time; and use the multi-level cache optimization strategy to perform data desensitization processing.

[0029] Calculate the degree of protection, data quality retention, and computational overhead of the fields that need to be desensitized, and select the optimal desensitization algorithm; construct a multi-level cache optimization strategy based on data access frequency and recent access time; and perform data desensitization processing using the multi-level cache optimization strategy, including:

[0030] Obtain a set of fields to be desensitized, and calculate the sensitivity of each field to be desensitized based on the risk factor score and weight; calculate the data exposure risk value of the field to be desensitized according to the pre-acquired attack probability, information leakage degree, and impact degree; use the sensitivity and the data exposure risk value as data security assessment indicators;

[0031] Collect the original data distribution information of the field to be desensitized, and calculate the information entropy of the original data distribution; execute multiple desensitization algorithms on the field to be desensitized to obtain multiple desensitization result data sets; calculate the information entropy of each desensitization result data set, and calculate the information entropy loss value based on the information entropy of the original data distribution and the information entropy of the desensitized result data set; use the information entropy loss value, data validity, data accuracy and data integrity as data quality assessment indicators;

[0032] Construct a resource consumption evaluation model including CPU consumption, memory consumption and IO consumption; calculate the time complexity and resource consumption of each desensitization algorithm; construct a comprehensive scoring function based on the data security evaluation index, the data quality evaluation index and the resource consumption evaluation model; calculate the comprehensive score of each desensitization algorithm according to the comprehensive scoring function, and select the desensitization algorithm with the highest score as the target desensitization algorithm;

[0033] Construct a multi-level cache structure, where each level of cache includes cache size, replacement strategy, and promotion / relegation threshold; calculate cache item scores based on access frequency, last access time, and data size; hierarchically store cache data based on the cache item scores; construct a cache replacement probability model based on cache cost, access frequency, and data age; and perform cache data replacement based on the cache replacement probability model;

[0034] Count the cache hit rate, and adjust the cache classification parameters based on the cache hit rate; calculate the resource benefit ratio, and optimize the resource allocation strategy based on the resource benefit ratio; use the target desensitization algorithm to desensitize the field to be desensitized, and store the desensitization result into the multi-level cache structure; establish a data version control mechanism to ensure the consistency of cached data and source data.

[0035] A second aspect of an embodiment of the present invention provides a data row and column permission control and desensitization system based on a Spark engine, including:

[0036] The first unit is used to receive a data processing request submitted by a user, wherein the data processing request includes a target data table identifier, a data processing operation type, and user identity information; and obtain the user's data access permission rules from a rights management server according to the user identity information, wherein the data access permission rules include a row-level filtering condition, an accessible field list, and a field desensitization rule;

[0037] The second unit is used to parse the data access permission rule into a Spark SQL syntax tree, inject the row-level filtering condition into the SparkSQL syntax tree to generate a first intermediate syntax tree, and perform column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree; based on the second intermediate syntax tree, construct a Spark distributed data processing task, wherein the Spark distributed data processing task includes a data scanning node, a data filtering node, a field projection node, and a data desensitization node;

[0038] The third unit is used for the data scanning node to read the original data of the target data table, the data filtering node to execute the row-level filtering condition, the field projection node to only select the fields in the accessible field list, and the data desensitization node to desensitize the sensitive fields according to the field desensitization rules; submit the Spark distributed data processing task to the Spark cluster for execution, and obtain a result data set that complies with the authority control and completes the desensitization processing.

[0039] A third aspect of the embodiments of the present invention

[0040] An electronic device is provided, comprising:

[0041] processor;

[0042] a memory for storing processor-executable instructions;

[0043] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0044] A fourth aspect of the embodiments of the present invention is:

[0045] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0046] The beneficial effects of this application are as follows:

[0047] The Spark engine-based data row and column permission control and desensitization method provided by the present invention can effectively implement fine-grained data access control. By parsing the user's data access permission rules into a Spark SQL syntax tree and injecting row-level filtering conditions and column-level field filtering into it, the user's access scope to data can be accurately controlled, effectively preventing unauthorized data access and improving data security.

[0048] This method realizes an efficient data processing process by constructing Spark distributed data processing tasks including nodes such as data scanning, filtering, projection and desensitization. This method makes full use of Spark's distributed computing capabilities and can quickly process large-scale data sets while ensuring the security and compliance of data access, and improving the efficiency and reliability of data processing.

[0049] In addition, the method of the present invention also integrates a data desensitization function, which can process sensitive fields according to predefined desensitization rules. This not only protects user privacy and sensitive information, but also enables data to be used safely in a wider range, enhances the availability and sharing value of data, and reduces the risk of data leakage, providing enterprises and organizations with a more secure and compliant data use environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The present invention is a flowchart of a method for controlling data row and column permissions and desensitizing data based on a Spark engine according to an embodiment of the present invention;

[0051] Figure 2 The present invention is a structural diagram of a data row and column permission control and desensitization system based on the Spark engine according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0054] Figure 1 The following is a flow chart of a method for data row and column permission control and desensitization based on the Spark engine according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0055] S101. Receive a data processing request submitted by a user, the data processing request including a target data table identifier, a data processing operation type, and user identity information; obtain the user's data access permission rules from a rights management server according to the user identity information, the data access permission rules including row-level filtering conditions, an accessible field list, and field desensitization rules;

[0056] S102. Parse the data access permission rule into a Spark SQL syntax tree, inject the row-level filtering condition into the Spark SQL syntax tree to generate a first intermediate syntax tree, perform column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree; construct a Spark distributed data processing task based on the second intermediate syntax tree, wherein the Spark distributed data processing task includes a data scanning node, a data filtering node, a field projection node, and a data desensitization node;

[0057] S103. The data scanning node reads the original data of the target data table, the data filtering node executes the row-level filtering condition, the field projection node only selects the fields in the accessible field list, and the data desensitization node desensitizes the sensitive fields according to the field desensitization rules; the Spark distributed data processing task is submitted to the Spark cluster for execution, and a result data set that complies with the permission control and completes the desensitization process is obtained.

[0058] In an optional implementation, the data access permission rules of the user are obtained from the permission management server according to the user identity information, and the data access permission rules include row-level filtering conditions, accessible field lists, and field desensitization rules including:

[0059] Receive user identity information, the user identity information includes user role identification, organizational structure information and timestamp information; construct a user authority feature vector based on the user identity information, the user authority feature vector includes a role inheritance relationship matrix, an organizational structure authority inheritance tree and a time dimension authority change curve; obtain an authority rule set from a distributed authority rule storage system according to the user authority feature vector, the distributed authority rule storage system adopts a hierarchical storage structure to store user basic authority layer rules, role inheritance authority layer rules and temporary authorization authority layer rules;

[0060] Perform consistency verification on the permission rule set, generate rule verification results, and incrementally update the permission rule set based on the rule verification results; parse the updated permission rule set into a conditional expression, and perform semantic normalization on the conditional expression to obtain a basic filtering condition set; construct a condition dependency graph based on the basic filtering condition set, and optimize the condition dependency graph through a graph traversal algorithm to obtain a minimum necessary condition set;

[0061] A field access dependency graph is constructed according to the minimum necessary condition set, and a minimum necessary field set is calculated based on the field access dependency graph; a desensitization metric is evaluated on the fields in the minimum necessary field set to generate a field desensitization scoring matrix; corresponding desensitization strategies are assigned to different types of fields according to the field desensitization scoring matrix to generate a field desensitization rule set; the minimum necessary condition set, the minimum necessary field set, and the field desensitization rule set are combined to generate user data access permission rules.

[0062] The specific implementation process of obtaining data access permission rules based on user identity information is as follows:

[0063] First, the user identity information data packet is received, which contains the user role identification (such as ordinary employee, department manager, senior administrator, etc.), organizational structure information (such as department, superior-subordinate relationship, etc.) and timestamp information (such as permission effective time, expiration time, etc.). The permission management server parses the data packet and extracts key identity features.

[0064] When constructing the user permission feature vector, we first generate a role inheritance relationship matrix based on the role identifier. For example, a department manager inherits all the permissions of ordinary employees, and a senior administrator inherits all the permissions of the department manager. At the same time, we build a permission inheritance tree based on the organizational structure information to record the permission transfer relationship between organizational units. In addition, we analyze the change of permissions over time in combination with the timestamp information and generate a permission change curve.

[0065] The permission rule storage adopts a hierarchical architecture, including the basic permission layer (storing the user's inherent permissions), the inherited permission layer (storing the permissions obtained through role inheritance), and the temporary permission layer (storing the temporary special permissions). The system obtains permission rules from each layer and merges them into a complete set of rules.

[0066] Verify the consistency of the obtained rule set to check whether there are any rule conflicts. For example, check whether a user has mutually exclusive permissions at the same time, or whether the timeliness of permissions is reasonable. Incrementally update the rules based on the verification results to ensure the consistency of the rule set.

[0067] Parse the rule set into a standard conditional expression, such as "the department to which the data belongs is equal to the department to which the user belongs". Perform semantic normalization on these expressions and unify the expression. Then construct a conditional dependency graph and analyze the dependency between each filtering condition. Use the graph traversal algorithm to optimize the dependency, remove redundant conditions, and obtain the minimum necessary condition set.

[0068] Based on the filtering conditions, a field access dependency graph is constructed to analyze the associations between the fields. The minimum set of fields required to meet the filtering conditions is calculated through graph algorithms. Desensitization metrics are evaluated for these fields, and factors such as field sensitivity and usage scenarios are considered to generate a desensitization score matrix.

[0069] Select appropriate desensitization strategies for different types of fields based on the desensitization scores. For example, use masking for ID card numbers, partially hide mobile phone numbers, replace keywords for sensitive texts, etc. Finally, integrate the filtering conditions, necessary fields, and desensitization rules to generate complete data access permission rules.

[0070] This application can achieve:

[0071] Accurate permission management is achieved through hierarchical storage and multi-dimensional feature analysis, which avoids permission rule conflicts and improves the accuracy and reliability of permission allocation. Optimization processing based on graph algorithms significantly reduces system resource overhead and improves permission verification efficiency.

[0072] The use of dynamic permission feature vectors and incremental update mechanisms enables the system to flexibly respond to permission changes, improving the real-time and adaptability of permission management. The minimum necessary access principle is ensured through dependency analysis, enhancing data security.

[0073] The intelligent desensitization strategy based on the scoring matrix can dynamically adjust the data protection level according to different scenarios, ensuring the security of sensitive information while ensuring data availability. The overall solution realizes the refinement and intelligence of permission management, significantly improving the security and efficiency of data access control.

[0074] In an optional implementation, parsing the data access permission rule into a Spark SQL syntax tree, injecting the row-level filtering condition into the Spark SQL syntax tree to generate a first intermediate syntax tree, and performing column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree includes:

[0075] Receive data access permission rules, the data access permission rules including a filter condition set, a column access permission set, and a desensitization rule set; parse the data access permission rules into a formalized permission rule expression, the formalized permission rule expression including an operator node set, a projection node set, a filter node set, and a data source node set; construct a permission rule syntax tree node dependency matrix based on the formalized permission rule expression;

[0076] The original query statement is parsed to generate an original syntax tree, wherein the original syntax tree includes a node set, an edge set, and a root node; based on the authority rule syntax tree node dependency matrix, the filter conditions in the filter condition set are normalized to obtain normalized filter conditions; the injection cost and impact factor of each node in the original syntax tree are calculated, and the node with the smallest weighted sum of the injection cost and the impact factor is selected as the filter condition injection point;

[0077] Inject the normalized filter condition into the filter condition injection point to generate a first intermediate syntax tree; perform predicate push-down optimization on the first intermediate syntax tree, filter the edge set that meets the push-down condition based on a preset optimization threshold, and push down the filter condition in the edge set that meets the push-down condition; construct a field projection node according to the column access permission set, and reconstruct the first intermediate syntax tree based on the field projection node to obtain a second intermediate syntax tree.

[0078] First, receive the data access permission rules, which include filter condition set, column access permission set and desensitization rule set. The filter condition set can include conditional expressions such as "department = 'IT'" or "salary>5000". The column access permission set defines the list of accessible fields, such as "name, age, department", etc. The desensitization rule set defines the fields that need to be desensitized and the desensitization method.

[0079] Parse these rules into formal permission rule expressions. During the parsing process, convert conditional expressions into operator nodes, such as comparison operators "=", ">", etc.; convert field references into projection nodes; convert filter conditions into filter nodes; and convert table references into data source nodes.

[0080] When constructing the node dependency matrix of the permission rule syntax tree, the dependency graph between nodes is established by analyzing the association between each node. For example, in the filter condition "department = 'IT'", the comparison operator node depends on the field reference node and the constant node.

[0081] Parse the original query statement and generate a syntax tree containing complete node information. Take the query statement "SELECT name, age FROM employees WHERE salary>5000" as an example. The generated syntax tree contains SELECT, FROM, WHERE nodes and the relationships between them.

[0082] Normalize the filter conditions and convert complex filter conditions into standard forms. For example, convert "NOT(age<30)" into "age>= 30". By analyzing the characteristics of each node in the syntax tree, calculate the injection cost and impact factor. The injection cost considers the type, location and complexity of the node, and the impact factor considers the scope of the node's influence on the query result.

[0083] The normalized filter conditions are injected into the selected injection point to generate an intermediate syntax tree containing the access control logic. The syntax tree is optimized for predicate pushdown, and the filter conditions that meet the conditions are pushed down as close to the data source node as possible to improve query efficiency.

[0084] Finally, a field projection node is built according to the column access permission set, and the syntax tree is reconstructed to ensure that only fields that are allowed to be accessed are returned.

[0085] This application can achieve data access control through dynamic parsing and injection, avoiding the maintenance difficulties caused by hard-coding methods and improving the flexibility and scalability of the system. The cost-based injection point selection strategy and predicate push-down optimization are adopted to effectively improve query performance, reduce data processing volume, and optimize system resource utilization. Fine-grained row-level and column-level access control is achieved, and data desensitization processing is supported, which enhances data security and meets data protection needs in different scenarios.

[0086] In an optional implementation, calculating the injection cost and impact factor of each node in the original syntax tree, and selecting the node with the smallest weighted sum of the injection cost and the impact factor as the filter condition injection point includes:

[0087] ;

[0088] in, Indicates the optimal injection point that is finally selected, that is, the node position in the syntax tree that is most suitable for injecting the filter condition. It means to find the node p that makes the following expression get the minimum value in all the node sets V. represents the cost of injecting the filter condition at node p, represents the set of nodes affected by the injection point, that is, all relevant nodes downstream of the injection point p. Represents the impact factor of the injection operation on node v.

[0089] In the process of syntax tree node injection optimization, we first need to build a complete syntax tree structure, which contains all the operation nodes in the query statement. For each possible injection point, we calculate the injection cost and impact factor by depth-first traversal.

[0090] The calculation of injection cost needs to consider the following factors: node type, node complexity, node data size, and node execution cost. For example, for simple filtering operations, the injection cost is low, while for complex aggregation operations, the injection cost is high. Specifically, a basic cost value can be assigned according to the operation type of the node, and then weighted calculation can be performed based on the input data volume of the node.

[0091] The impact factor evaluation requires analyzing the impact of the injection point on the downstream nodes. By traversing all downstream nodes of the injection point, the impact of the filter condition on each node is evaluated. The impact factor considers the following aspects: data flow, data dependencies, and the complexity of the operation. For example, if the downstream node contains aggregation operations, the impact factor is large; if it only contains simple projection operations, the impact factor is small.

[0092] In practical applications, the following processing flow can be adopted:

[0093] First, build the syntax tree of the query statement to determine all possible injection points. For example, for a query that includes multi-table joins and aggregation operations, you can set potential injection points before and after the join operation, and before and after the aggregation operation.

[0094] Each potential injection point is then evaluated. Assuming there is a query that includes a table join, the cost of injecting the filter condition before the join operation is 10 units, affecting two downstream nodes with impact factors of 5 and 3 units respectively; while the cost of injecting after the join operation is 15 units, affecting one downstream node with an impact factor of 2 units.

[0095] Then calculate the comprehensive score of each injection point. The injection cost is weighted with the impact factor of each affected node. In the above example, the comprehensive score of the first injection point is 10+5+3=18 units, and the comprehensive score of the second injection point is 15+2=17 units.

[0096] Finally, the location with the lowest comprehensive score is selected as the final injection point. In this example, the second injection point has a lower comprehensive score, so the filter condition is selected after the connection operation.

[0097] This application can achieve:

[0098] By accurately calculating the injection cost and influencing factors, the optimal injection position of the filter condition can be found, significantly improving query performance and system efficiency. The comprehensive evaluation mechanism based on multi-dimensional factors ensures the scientificity and accuracy of injection point selection and avoids performance loss caused by improper position injection. The dynamic evaluation and adaptive selection strategies enable the system to flexibly adjust the injection strategy according to different query characteristics and data features, improving the adaptability and scalability of the system.

[0099] In an optional implementation, the data scanning node reads the original data of the target data table, the data filtering node executes the row-level filtering condition, the field projection node selects only the fields in the accessible field list, and the data desensitization node desensitizes the sensitive fields according to the field desensitization rule, including:

[0100] A data partitioning strategy optimization model is constructed based on data distribution characteristics, data locality, and data partition size of a target data table; data partitions are performed on the target data table according to the data partitioning strategy optimization model; a data reading window size is dynamically adjusted based on a ratio of a target processing time to an actual processing time, and original data is read from the data partition through the data reading window;

[0101] Calculate the complexity of data filtering conditions, determine the parallelism of filtering operations based on the complexity of filtering conditions, and use the parallelism of filtering operations as the number of filtering nodes; construct a multi-filtering node processing unit according to the number of filtering nodes, and the multi-filtering node processing unit performs parallel filtering operations on the original data; calculate the filtering efficiency based on the ratio of the number of records before and after filtering and the benchmark processing time;

[0102] Constructing a field dependency graph, the field dependency graph including a field set, dependency relationships, and dependency costs; calculating processing costs between adjacent fields based on the field dependency graph, and generating an optimal field projection sequence based on the processing costs; performing a field projection operation on the filtered data based on the optimal field projection sequence;

[0103] Calculate the degree of protection, data quality preservation, and computing overhead for the fields that need to be desensitized, and select the optimal desensitization algorithm; build a multi-level cache optimization strategy based on data access frequency and last access time; and use the multi-level cache optimization strategy to perform data desensitization processing.

[0104] The data scanning node first needs to analyze and optimize the target data table. By collecting information such as the number of rows, data size, and data distribution characteristics of the data table, combined with the system's processing capabilities and storage characteristics, a data partitioning strategy is constructed. For example, for a data table containing user information, it can be partitioned based on the user ID range, and the size of each partition is controlled at around 256MB. The system dynamically monitors the data processing speed, and automatically adjusts the data reading window size when the actual processing time exceeds the target processing time. For example, if the initial window size is set to 1,000 records, if the processing time is 20% slower than expected, the window size is adjusted to 800 records.

[0105] The data filtering node determines the parallel processing scheme by analyzing the complexity of the filtering conditions. For simple equal-value filtering conditions, such as "age>18", a lower degree of parallelism can be set; for complex multi-condition combination filtering, such as "(age>18AND income>5000) OR (vip_level>2)", a higher degree of parallelism is required. The system automatically calculates the number of filtering nodes required based on the complexity of the conditions. For example, for the above complex conditions, 4 parallel filtering nodes may be set. The system continuously monitors the filtering efficiency, records the data volume ratio and processing time before and after filtering, and uses this to optimize subsequent filtering strategies.

[0106] The field projection node first builds a dependency graph between fields. For example, when processing user information, "user level" depends on "consumption amount" and "registration time", and the system will record these dependencies and their processing costs. Based on the dependency graph, the system generates the optimal field processing sequence to ensure that the necessary fields are correctly retained and processed. For example, basic fields such as "user ID" are processed first, and then derived fields such as "user level" are processed.

[0107] The data desensitization node selects the appropriate desensitization algorithm for different types of sensitive fields. For example, for mobile phone numbers, you can choose to keep the first three digits and the last four digits, and replace the middle with asterisks; for ID card numbers, you can choose to only display the date of birth information. The system will establish a multi-level cache based on the frequency of use of the field, caching the commonly used desensitization results in memory and storing the less used results in the disk cache. For example, for frequently queried VIP user information, its desensitization results will be cached in memory first.

[0108] This application can achieve:

[0109] The data processing efficiency is significantly improved through intelligent data partitioning strategy and dynamically adjusted data reading window; the multi-level cache mechanism is adopted to reduce repeated calculations and speed up data access; parallel filtering operations improve the concurrent processing capability of the system. Refined field-level access control is achieved; through intelligent selection of desensitization algorithms, data availability is maintained while protecting data security; multi-level cache strategy ensures the safe storage and access of sensitive data. The processing mechanism based on field dependency graph enables the system to flexibly respond to field changes; the dynamically adjusted number of filtering nodes enables the system to adapt to filtering conditions of different complexities; the optimization model of data partitioning strategy enables the system to process data sets of different sizes.

[0110] In an optional implementation, the degree of protection, data quality retention, and computational overhead of the fields to be desensitized are calculated, and the optimal desensitization algorithm is selected; a multi-level cache optimization strategy is constructed based on data access frequency and last access time; and data desensitization processing is performed using the multi-level cache optimization strategy, including:

[0111] Obtain a set of fields to be desensitized, and calculate the sensitivity of each field to be desensitized based on the risk factor score and weight; calculate the data exposure risk value of the field to be desensitized according to the pre-acquired attack probability, information leakage degree, and impact degree; use the sensitivity and the data exposure risk value as data security assessment indicators;

[0112] Collect the original data distribution information of the field to be desensitized, and calculate the information entropy of the original data distribution; execute multiple desensitization algorithms on the field to be desensitized to obtain multiple desensitization result data sets; calculate the information entropy of each desensitization result data set, and calculate the information entropy loss value based on the information entropy of the original data distribution and the information entropy of the desensitized result data set; use the information entropy loss value, data validity, data accuracy and data integrity as data quality assessment indicators;

[0113] Construct a resource consumption evaluation model including CPU consumption, memory consumption and IO consumption; calculate the time complexity and resource consumption of each desensitization algorithm; construct a comprehensive scoring function based on the data security evaluation index, the data quality evaluation index and the resource consumption evaluation model; calculate the comprehensive score of each desensitization algorithm according to the comprehensive scoring function, and select the desensitization algorithm with the highest score as the target desensitization algorithm;

[0114] Construct a multi-level cache structure, where each level of cache includes cache size, replacement strategy, and promotion / relegation threshold; calculate cache item scores based on access frequency, last access time, and data size; hierarchically store cache data based on the cache item scores; construct a cache replacement probability model based on cache cost, access frequency, and data age; and perform cache data replacement based on the cache replacement probability model;

[0115] Count the cache hit rate, and adjust the cache classification parameters based on the cache hit rate; calculate the resource benefit ratio, and optimize the resource allocation strategy based on the resource benefit ratio; use the target desensitization algorithm to desensitize the field to be desensitized, and store the desensitization result into the multi-level cache structure; establish a data version control mechanism to ensure the consistency of cached data and source data.

[0116] The data desensitization system first obtains the set of fields to be desensitized and performs a risk assessment on each field. Risk factors include dimensions such as data sensitivity, usage scenarios, and data types. For example, for the ID card number field, the sensitivity is high risk, the usage scenario is user authentication, and the data type is a fixed format string. The system calculates a comprehensive risk score based on these factors, with a value range of 0-100.

[0117] In the data exposure risk assessment phase, the system collects historical attack data and establishes a risk prediction model. Taking credit card numbers as an example, by analyzing historical data leakage events, the attack probability is 0.3, the information leakage degree is 0.8, and the impact degree is 0.9. These indicators are calculated comprehensively to obtain the data exposure risk value.

[0118] For the data to be desensitized, the system collects the distribution characteristics of the original data. Taking the mobile phone number field as an example, analyze its number segment distribution, operator proportion and other characteristics. Apply masking, replacement, encryption and other desensitization algorithms to the field to generate multiple desensitization result sets. By comparing the distribution characteristics of the original data and the desensitization results, calculate the degree of information loss. At the same time, evaluate the effectiveness, accuracy and completeness of the desensitization results.

[0119] The system monitors the resource consumption of each desensitization algorithm in real time. Taking the AES encryption algorithm as an example, when processing 1 million records, the CPU utilization rate is 60%, the memory occupies 2GB, and the IO throughput is 500MB / s. Based on the three dimensions of security, data quality, and resource consumption, a scoring model is built to select the optimal algorithm.

[0120] In terms of cache optimization, the system adopts a three-level cache structure. The first level is the memory cache, with a capacity of 10GB and an LRU replacement strategy; the second level is the SSD cache, with a capacity of 100GB and an ARC replacement strategy; the third level is the disk storage. The system records the data access pattern. When a data item is accessed more than 1,000 times within 24 hours, it is promoted to the previous level cache; when the number of accesses is less than 100 times for 7 consecutive days, the storage is downgraded.

[0121] The system continuously optimizes the cache strategy. When the first-level cache hit rate is less than 80%, the cache capacity is automatically increased; when the resource utilization rate is less than 50%, resource allocation is reduced. Through the version number and timestamp mechanism, the cache data is ensured to be synchronized with the source data, supporting incremental updates and real-time synchronization.

[0122] This application can achieve:

[0123] The optimal desensitization algorithm is selected through multi-dimensional evaluation to maximize data availability while protecting data security. The dynamic evaluation system can adapt to the security requirements in different scenarios and provide accurate data protection solutions. Based on the multi-level cache optimization strategy, data processing performance is significantly improved. The intelligent cache classification and replacement mechanism ensures fast access to hot data, reduces system response time, and improves resource utilization efficiency. The system has adaptive optimization capabilities, and continuously optimizes resource configuration and performance through real-time monitoring and dynamic adjustment. The version control mechanism ensures data consistency and supports reliable operation in large-scale data processing scenarios.

[0124] The present application also provides an embodiment as follows:

[0125] The technical implementation of the present invention covers three modules: Spark engine, metadata service, and rule service.

[0126] Spark Engine:

[0127] After the task is submitted, Spark will parse the SQL in the initial parsing stage. The present invention obtains each segment of the parsed SQL after the Spark engine parses the SQL and then performs row permission verification, column permission verification and dynamic desensitization.

[0128] Metadata Services:

[0129] Metadata service mainly refers to the corresponding data source information and current operator information of Spark query. After parsing SQL in the Spark engine, it is necessary to query row and column permission control or desensitization rules and other related data according to the metadata information of the data source to the rule service, which can generally be stored in a relational database;

[0130] Rule Services:

[0131] The rule service includes row permission control rules, column permission control rules and dynamic masking rules. The rule service also includes metadata information of the data source, which can generally be stored in a relational database.

[0132] Implementation:

[0133] When a Spark task is submitted, the metadata information of the corresponding data source and the metadata information of the current operator are stored in the configuration information and submitted to the Spark engine. During the parsing stage, the Spark engine parses the SQL into SQL statements that can be executed separately according to the grammar rule file, and implements the business rule judgment operations of row and column permissions and dynamic desensitization in the extended code in the Spark Extension in the Spark engine. In the extended code, the data source metadata information and personnel information in the submitted configuration are combined with the parsed SQL type to query the rule service whether the corresponding permission or desensitization rule data is stored. If the corresponding rule is found, the permission column limit control or dynamic desensitization operation is performed according to the corresponding rule.

[0134] Row permission control:

[0135] After the sql is submitted, it is confirmed to be a query type sql according to the sql type parsed by the Spark engine. At the same time, the row permission control data can be queried in the rule service through the data source and submitter information. The row permission control rule data is generally the conditional judgment of the field, such as the row permission filtering conditions of the fields id, job and other fields in the configuration table employee: id>10, id<>1, job<>'boss', etc. According to the sql query conditions parsed by the engine, the row permission control rule conditions are reasonably spliced ​​onto the original sql conditions. For example, when querying sql: select * from employ where 1 = 1, the above row filtering rules will be spliced ​​at the end according to the row permission: select * from employ where 1 = 1 and id>10 and id<>1, andjob<>'boss'. At the same time, it is verified that it can run normally, and the spliced ​​sql is run to achieve the effect of filtering sensitive data rows.

[0136] Column permission control:

[0137] After the sql is submitted, it is confirmed as a query type sql according to the sql type parsed by the Spark engine. At the same time, the column permission control data can be queried in the rule service through the data source and submitter information. The column permission control rule data is the column field of the query table. For example, the salary field in the configuration table employee will be set to null when querying the employee table. At the same time, it is verified that it can run normally, so as to achieve the effect of filtering sensitive data columns.

[0138] Dynamic desensitization:

[0139] Data desensitization is like putting a layer of "invisibility cloak" on sensitive data. Through established desensitization rules, it cleverly transforms personal sensitive information such as ID number, mobile phone number, bank card number, as well as corporate trade secrets and financial data, so that the data can be kept available while privacy and security are effectively protected.

[0140] For example, the original ID card number "370102198506081234" may become "370119851234" after desensitization, which not only retains key statistical information such as region and year of birth, but also conceals the specific date of birth and sequence code, effectively preventing the risk of identity theft; looking at the bank card number, the original "6222021234567890123" becomes "622202****0123" after desensitization, the key issuing bank identification code and tail number are retained to meet business verification requirements, but the intermediate card number information is properly hidden.

[0141] The present invention can configure a variety of desensitization methods such as replacement and deletion in the desensitization rules according to user needs. After the sql is submitted, it is confirmed as a query type sql according to the sql type parsed by the Spark engine. At the same time, the corresponding desensitization rule data can be queried in the rule service through the data source and the submitter information. After obtaining the desensitization rule of the corresponding column, dynamic desensitization will be performed according to the actual data queried by the corresponding column. For example, the name field in the configuration table employee will dynamically replace the middle of a three-character name and the last character of a two-character name. When executing the query on the employee table, the result will be displayed as Zhang*, Zhang*Feng, etc., thereby achieving the effect of dynamic desensitization.

[0142] At the same time, dynamic masking supports the transmission of masking along with the lineage relationship. The masked fields are passed down according to the lineage relationship in the Spark task. The upstream dynamically masked fields can support dynamic masking to all the lineage fields downstream, which greatly simplifies the configuration of masking rules.

[0143] Figure 2 The structure diagram of the data row and column permission control and desensitization system based on the Spark engine according to the embodiment of the present invention is as follows: Figure 2 As shown, the system comprises:

[0144] The first unit is used to receive a data processing request submitted by a user, wherein the data processing request includes a target data table identifier, a data processing operation type, and user identity information; and obtain the user's data access permission rules from a rights management server according to the user identity information, wherein the data access permission rules include a row-level filtering condition, an accessible field list, and a field desensitization rule;

[0145] The second unit is used to parse the data access permission rule into a Spark SQL syntax tree, inject the row-level filtering condition into the SparkSQL syntax tree to generate a first intermediate syntax tree, and perform column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree; based on the second intermediate syntax tree, construct a Spark distributed data processing task, wherein the Spark distributed data processing task includes a data scanning node, a data filtering node, a field projection node, and a data desensitization node;

[0146] The third unit is used for the data scanning node to read the original data of the target data table, the data filtering node to execute the row-level filtering condition, the field projection node to only select the fields in the accessible field list, and the data desensitization node to desensitize the sensitive fields according to the field desensitization rules; submit the Spark distributed data processing task to the Spark cluster for execution, and obtain a result data set that complies with the authority control and completes the desensitization processing.

[0147] According to a third aspect of the embodiments of the present invention,

[0148] An electronic device is provided, comprising:

[0149] processor;

[0150] a memory for storing processor-executable instructions;

[0151] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0152] A fourth aspect of the embodiments of the present invention is:

[0153] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0154] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data row and column permission control and desensitization method based on the Spark engine, characterized in that: include: Receive a data processing request submitted by a user, wherein the data processing request includes a target data table identifier, a data processing operation type, and user identity information; Acquire the user's data access permission rules from the permission management server according to the user identity information, wherein the data access permission rules include row-level filtering conditions, accessible field lists, and field desensitization rules; Parse the data access permission rule into a Spark SQL syntax tree, inject the row-level filtering condition into the Spark SQL syntax tree to generate a first intermediate syntax tree, perform column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree; construct a Spark distributed data processing task based on the second intermediate syntax tree, wherein the Spark distributed data processing task includes a data scanning node, a data filtering node, a field projection node, and a data desensitization node; The data scanning node reads the original data of the target data table, the data filtering node executes the row-level filtering condition, the field projection node only selects the fields in the accessible field list, and the data desensitization node desensitizes the sensitive fields according to the field desensitization rule; Submit the Spark distributed data processing task to the Spark cluster for execution, and obtain a result data set that complies with authority control and completes desensitization processing; Parsing the data access permission rule into a Spark SQL syntax tree, injecting the row-level filtering condition into the Spark SQL syntax tree to generate a first intermediate syntax tree, and performing column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree includes: Receive data access permission rules, the data access permission rules including a filter condition set, a column access permission set, and a desensitization rule set; parse the data access permission rules into a formalized permission rule expression, the formalized permission rule expression including an operator node set, a projection node set, a filter node set, and a data source node set; construct a permission rule syntax tree node dependency matrix based on the formalized permission rule expression; The original query statement is parsed to generate an original syntax tree, wherein the original syntax tree includes a node set, an edge set, and a root node; based on the authority rule syntax tree node dependency matrix, the filter conditions in the filter condition set are normalized to obtain normalized filter conditions; the injection cost and impact factor of each node in the original syntax tree are calculated, and the node with the smallest weighted sum of the injection cost and the impact factor is selected as the filter condition injection point; Inject the normalized filter condition into the filter condition injection point to generate a first intermediate syntax tree; perform predicate push-down optimization on the first intermediate syntax tree, filter the edge set that meets the push-down condition based on a preset optimization threshold, and push down the filter condition in the edge set that meets the push-down condition; construct a field projection node according to the column access permission set, and reconstruct the first intermediate syntax tree based on the field projection node to obtain a second intermediate syntax tree.

2. The method according to claim 1, characterized in that According to the user identity information, the user's data access permission rules are obtained from the permission management server. The data access permission rules include row-level filtering conditions, accessible field lists, and field desensitization rules, including: Receive user identity information, the user identity information includes user role identification, organizational structure information and timestamp information; construct a user authority feature vector based on the user identity information, the user authority feature vector includes a role inheritance relationship matrix, an organizational structure authority inheritance tree and a time dimension authority change curve; obtain an authority rule set from a distributed authority rule storage system according to the user authority feature vector, the distributed authority rule storage system adopts a hierarchical storage structure to store user basic authority layer rules, role inheritance authority layer rules and temporary authorization authority layer rules; Perform consistency verification on the permission rule set, generate rule verification results, and incrementally update the permission rule set based on the rule verification results; parse the updated permission rule set into a conditional expression, and perform semantic normalization on the conditional expression to obtain a basic filtering condition set; construct a condition dependency graph based on the basic filtering condition set, and optimize the condition dependency graph through a graph traversal algorithm to obtain a minimum necessary condition set; A field access dependency graph is constructed according to the minimum necessary condition set, and a minimum necessary field set is calculated based on the field access dependency graph; a desensitization metric is evaluated on the fields in the minimum necessary field set to generate a field desensitization scoring matrix; corresponding desensitization strategies are assigned to different types of fields according to the field desensitization scoring matrix to generate a field desensitization rule set; the minimum necessary condition set, the minimum necessary field set, and the field desensitization rule set are combined to generate user data access permission rules.

3. The method according to claim 1, characterized in that Calculating the injection cost and impact factor of each node in the original syntax tree, and selecting the node with the smallest weighted sum of the injection cost and the impact factor as the filter condition injection point includes: Among them, Pi nject Indicates the optimal injection point finally selected, that is, the node position in the syntax tree that is most suitable for injecting filter conditions, argmin p∈V It means to find the node p that makes the following expression get the minimum value in all the node sets V. Cost(p) means the cost of injecting the filter condition at node p. V affected It represents the set of nodes affected by the injection point, that is, all related nodes downstream of the injection point p. Impact(v) represents the impact factor of the injection operation on node v.

4. The method according to claim 1, characterized in that The data scanning node reads the original data of the target data table, the data filtering node executes the row-level filtering condition, the field projection node selects only the fields in the accessible field list, and the data desensitization node performs desensitization processing on the sensitive fields according to the field desensitization rule, including: A data partitioning strategy optimization model is constructed based on data distribution characteristics, data locality, and data partition size of a target data table; data partitions are performed on the target data table according to the data partitioning strategy optimization model; a data reading window size is dynamically adjusted based on a ratio of a target processing time to an actual processing time, and original data is read from the data partition through the data reading window; Calculate the complexity of data filtering conditions, determine the parallelism of filtering operations based on the complexity of filtering conditions, and use the parallelism of filtering operations as the number of filtering nodes; construct a multi-filtering node processing unit according to the number of filtering nodes, and the multi-filtering node processing unit performs parallel filtering operations on the original data; calculate the filtering efficiency based on the ratio of the number of records before and after filtering and the benchmark processing time; Constructing a field dependency graph, the field dependency graph including a field set, dependency relationships, and dependency costs; calculating processing costs between adjacent fields based on the field dependency graph, and generating an optimal field projection sequence based on the processing costs; performing a field projection operation on the filtered data based on the optimal field projection sequence; Calculate the degree of protection, data quality preservation, and computing overhead for the fields that need to be desensitized, and select the optimal desensitization algorithm; build a multi-level cache optimization strategy based on data access frequency and last access time; and use the multi-level cache optimization strategy to perform data desensitization processing.

5. The method according to claim 4, characterized in that Calculate the degree of protection, data quality and computing overhead of the fields that need to be desensitized, and select the optimal desensitization algorithm; build a multi-level cache optimization strategy based on data access frequency and recent access time; Utilizing the multi-level cache optimization strategy to perform data desensitization processing includes: Obtain a set of fields to be desensitized, and calculate the sensitivity of each field to be desensitized based on the risk factor score and weight; calculate the data exposure risk value of the field to be desensitized according to the pre-acquired attack probability, information leakage degree, and impact degree; use the sensitivity and the data exposure risk value as data security assessment indicators; Collect the original data distribution information of the field to be desensitized, and calculate the information entropy of the original data distribution; execute multiple desensitization algorithms on the field to be desensitized to obtain multiple desensitization result data sets; calculate the information entropy of each desensitization result data set, and calculate the information entropy loss value based on the information entropy of the original data distribution and the information entropy of the desensitized result data set; use the information entropy loss value, data validity, data accuracy and data integrity as data quality assessment indicators; Construct a resource consumption evaluation model including CPU consumption, memory consumption and IO consumption; calculate the time complexity and resource consumption of each desensitization algorithm; construct a comprehensive scoring function based on the data security evaluation index, the data quality evaluation index and the resource consumption evaluation model; calculate the comprehensive score of each desensitization algorithm according to the comprehensive scoring function, and select the desensitization algorithm with the highest score as the target desensitization algorithm; Construct a multi-level cache structure, where each level of cache includes cache size, replacement strategy, and promotion / relegation threshold; calculate cache item scores based on access frequency, last access time, and data size; hierarchically store cache data based on the cache item scores; construct a cache replacement probability model based on cache cost, access frequency, and data age; and perform cache data replacement based on the cache replacement probability model; Count the cache hit rate, and adjust the cache classification parameters based on the cache hit rate; calculate the resource benefit ratio, and optimize the resource allocation strategy based on the resource benefit ratio; use the target desensitization algorithm to desensitize the field to be desensitized, and store the desensitization result into the multi-level cache structure; establish a data version control mechanism to ensure the consistency of cached data and source data.

6. A data row and column permission control and desensitization system based on the Spark engine, used to implement the method as described in any one of claims 1 to 5, characterized in that: include: The first unit is used to receive a data processing request submitted by a user, wherein the data processing request includes a target data table identifier, a data processing operation type, and user identity information; Acquire the user's data access permission rules from the permission management server according to the user identity information, wherein the data access permission rules include row-level filtering conditions, accessible field lists, and field desensitization rules; The second unit is used to parse the data access permission rule into a Spark SQL syntax tree, inject the row-level filtering condition into the Spark SQL syntax tree to generate a first intermediate syntax tree, and perform column-level field filtering on the first intermediate syntax tree based on the accessible field list to generate a second intermediate syntax tree; based on the second intermediate syntax tree, construct a Spark distributed data processing task, wherein the Spark distributed data processing task includes a data scanning node, a data filtering node, a field projection node, and a data desensitization node; The third unit is used for the data scanning node to read the original data of the target data table, the data filtering node to execute the row-level filtering condition, the field projection node to select only the fields in the accessible field list, and the data desensitization node to perform desensitization processing on sensitive fields according to the field desensitization rule; Submit the Spark distributed data processing task to the Spark cluster for execution, and obtain a result data set that complies with authority control and completes desensitization processing.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images