Access control method and system for Apache Spark accumulator

By determining the location, type, and data type of the accumulator in Apache Spark, and combining this with the data processing optimizer and access control policies, the problem of unauthorized access to the accumulator is solved, enabling access control based on data tracing and enhancing data protection.

CN118965392AActive Publication Date: 2024-11-15XIDIAN UNIV HANGZHOU RES INST
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411054846.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-15
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

In existing technologies, accumulators in Apache Spark are vulnerable to unauthorized access, resulting in insufficient data security.

Method used

By determining the location, type, and data type of the accumulator, and combining this with the data processing optimizer and access control policies, access control of the accumulator is implemented to ensure that the data processing purpose matches the access control policy and to filter out illegal data access.

Benefits of technology

It enhances data protection in Apache Spark, reduces unauthorized data access, implements access control based on data origination, and protects sensitive data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118965392A_ABST
    Figure CN118965392A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of big data processing, and particularly relates to an access control method and system for an Apache Spark accumulator. The method comprises the following steps: 1) determining the position, the type and the data type of an accumulator AccX used in a data processing program according to the data processing program; 2) confirming the data processing purpose of each accumulator AccX according to the position, the type and the data type of the accumulator AccX; and 3) based on the data processing purpose of the accumulator AccX, determining accessible data in the accumulator AccX. According to the invention, data protection in Spark is enhanced, unauthorized access of data is reduced, and access control of data in an accumulator is realized on the premise of not influencing the logic of a user data processing program.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular to an access control method and system for an Apache Spark accumulator. Background Art

[0002] Apache Spark is an open source big data computing framework for processing and analyzing large-scale data sets. Through in-memory computing and DAG (directed acyclic graph) scheduling, Spark significantly improves data processing speed compared to traditional MapReduce, especially in iterative computing and interactive queries. Spark supports multiple programming languages, including Java, Scala, Python, and R, allowing developers to choose the programming language that best suits their needs. The core components of Spark include Spark Core, Spark Streaming, MLlib, GraphX, Catalyst, DataSet / DataFrame, Spark SQL, Structured Streaming, and ML Pipelines, which are used for batch processing, structured data query, real-time data stream processing, machine learning, and graph computing. Spark can flexibly build complex data processing programs in cluster computing environments. In addition, Figure 1 As shown in the figure, Spark introduces accumulators to perform accumulation operations in a cluster computing environment. The main functions of accumulators include: 1) data aggregation, for example, accumulators can be used to count the number of certain operations, accumulate values, etc., 2) counting operations, for example, using accumulators to count the number of data items that meet a certain condition without having to count them separately on each node and finally summarize them, 3) performance monitoring and debugging, for example, using accumulators to record the amount of data processed, the number of errors that occur, etc., can help identify performance bottlenecks and debug problems. Spark uses a variety of core components and accumulators to form a powerful and flexible big data computing framework.

[0003] Access control technology is a basic way to protect the security of big data. "Big Data Access Control Method, Device, Electronic Device and Storage Medium" (ZL202011025582.0) invented a big data access control method. The method determines the data processing purpose in the data processing logic based on the data processing optimizer (such as Catalyst), and makes access decisions corresponding to the data access request based on the data processing purpose. Although this method can be used for data protection in Spark, for example, Figure 1 As shown in Figure 1, the data processing program is restricted from outputting protected data. However, this method does not consider the application scenarios of accumulators, and the data protection in Spark is insufficient, which may lead to unauthorized access to data. Summary of the invention

[0004] The present invention provides an access control method and system for Apache Spark accumulators. The present invention can prevent sensitive data on the Spark platform from being illegally used by the accumulator according to data sources and data operations, and can enhance system security.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] An access control method for Apache Spark accumulators, comprising the following steps:

[0007] 1) According to the data processing program, determine the location, type and data type of the accumulator AccX used in the data processing program;

[0008] 2) confirming the data processing purpose of each accumulator AccX according to the location, type and data type of the accumulator AccX;

[0009] 3) Based on the data processing purpose of the accumulator AccX, determine the accessible data in the accumulator AccX.

[0010] The step 2) comprises the following steps:

[0011] 2.1) According to the position of each accumulator AccX, based on the data processing optimizer, determine the data operation purpose of the data operation operator before the data operation operator position where AccX is located;

[0012] 2.2) Determine the data operation purpose of each accumulator AccX according to the type and data type of the accumulator AccX;

[0013] 2.3) The data operation purpose of the data operation operator is combined with the data operation purpose of the accumulator AccX as the data processing purpose of each accumulator AccX.

[0014] The step 3) comprises the following steps:

[0015] 3.1) Get the data set Φ in the accumulator AccX, and find the origin data set M of each record data Item in Φ;

[0016] 3.2) Match the corresponding access control policy in the policy library for each data source involved in M;

[0017] 3.3) Traverse all access control policies, fine-grained match the allowed data processing purpose of each item, and determine whether the accumulator data processing purpose meets the allowed data processing purpose of the current item. If not, filter out the item, otherwise do not filter.

[0018] An access control system for Apache Spark accumulators, comprising:

[0019] An accumulator determination module is used to determine the position, type and data type of the accumulator AccX used in the data processing program according to the data processing program;

[0020] A data processing purpose determination module, used to determine the data processing purpose of each accumulator AccX according to the position, type and data type of the accumulator AccX;

[0021] The access control module is used to determine the accessible data in the accumulator AccX based on the data processing purpose of the accumulator AccX.

[0022] The data processing purpose determination module includes:

[0023] A data operation operator data operation purpose determination module is used to determine the data operation purpose of the data operation operator before the data operation operator position where AccX is located based on the data processing optimizer according to the position of each accumulator AccX;

[0024] The accumulator AccX data operation purpose determination module is used to determine the data operation purpose of each accumulator AccX according to the type and data type of the accumulator AccX;

[0025] The data processing purpose determination module is used to merge the data operation purpose of the data operation operator with the data operation purpose of the accumulator AccX as the data processing purpose of each accumulator AccX.

[0026] The access control module comprises:

[0027] The access control policy matching module is used to obtain the data set Φ in the accumulator AccX, find the origin data set M of each record data Item in Φ, and match the corresponding access control policy in the policy library for each data source involved in M;

[0028] The record data filtering module is used to traverse all access control policies, determine the allowed data processing purpose of each item, and judge whether the accumulator data processing purpose meets the allowed data processing purpose of the current item. If not, the item is filtered out, otherwise it is not filtered.

[0029] An access control implementation architecture for an Apache Spark accumulator includes a driver side and a worker task side, wherein:

[0030] The Driver side is used to analyze the data operation purpose of each operator in the data processing program according to the data processing optimizer, and to find the access control policy according to the data source in the data processing program. Finally, the data operation purpose of all operators and the access control policy corresponding to all data sources are broadcast to the Worker side.

[0031] The Worker task end is used to pass the data source header according to the record level when executing tasks, and obtain the location, type and data type of the accumulator.

[0032] An access control device for an Apache Spark accumulator comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement an access control method for an Apache Spark accumulator when executing the computer program.

[0033] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, an access control method for an Apache Spark accumulator is implemented.

[0034] The present invention has the following beneficial effects and advantages:

[0035] 1. The present invention can provide a purpose-aware access control based on data traceability for the Spark accumulator, thereby enhancing data protection in Spark and reducing unauthorized access to data.

[0036] 2. In order to identify the final data processing purpose of the accumulator in the present invention, the data operation purpose in the data processing optimizer and the data operation purpose in the accumulator are systematically considered.

[0037] 3. The access control implementation architecture in the present invention is divided into the Driver side and the Worker task side, which implements access control to the data in the accumulator without affecting the logic of the user data processing program. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the Apache Spark accumulator and data processing program in the background technology.

[0039] Figure 2 The main flow chart of an access control method for Apache Spark accumulators.

[0040] Figure 3 An architectural diagram of an access control implementation for Apache Spark accumulators.

[0041] Figure 4Main flow chart of an access control implementation architecture for Apache Spark accumulators.

[0042] Figure 5 An example diagram using accumulators for a data processing program.

[0043] Figure 6 Capture the data flow graph for the location, source transfer, and accumulator of the accumulator in the worker-side task. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0045] Embodiment 1

[0046] The main flow chart of the access control method for Apache Spark accumulator provided by the present invention is as follows: Figure 2 As shown, including:

[0047] Step S1: After the user submits the data processing program, the location of the accumulator used in the data processing program, the type of the accumulator, and the data type of the accumulator are determined.

[0048] Step S2: According to the position of each accumulator AccX in step S1, based on the data processing optimizer, determine the data operation purpose of the data operation operator before the data operation operator position where AccX is located, that is, when analyzing the data operation purpose, consider the data operation operator from the data source through various data operation operators to (but not including) the data operation operator where AccX is located. This step is implemented by the method in patent ZL202011025582.0.

[0049] Step S3: Determine the data operation purpose of each accumulator AccX according to the accumulator type and the data type of the accumulator in step S1.

[0050] Step S4: Combine the data operation purposes confirmed in step S2 and step S3 to finally confirm the data processing purpose of each accumulator AccX.

[0051] Step S5: Get the data set Φ in the accumulator AccX, and search for the origin data set M of each record data Item in Φ, that is, which record-level data in which data sources each record data Item originates from.

[0052] Step S6: According to the data source found in step S5, find the access control policy corresponding to the data source.

[0053] Step S7: traverse the access control policy in step S6, determine the allowed data processing purpose of the Item in step S5, and judge whether the accumulator data processing purpose confirmed in step S4 meets the allowed data processing purpose. If not, filter out the Item in step S5, otherwise do not filter.

[0054] Embodiment 2

[0055] In order to implement the method in the first embodiment, the present invention provides an access control implementation architecture for an Apache Spark accumulator, which can implement access control on data in the accumulator without affecting the logic of the user data processing program.

[0056] Considering the Driver / Worker architecture of Apache Spark, the present invention divides the access control implementation architecture into the Driver side and the Worker task side. Figure 3 As shown in the figure, the purpose analysis module on the Driver side analyzes the data operation purpose of each operator in the data processing program according to the data processing optimizer, and searches for access control policies according to the data source in the data processing program. Finally, the information broadcast module broadcasts the data operation purpose of all operators and the access control policies corresponding to all data sources to the Worker side. The "Driver broadcasts information" has completed the main process Figure 4 The S1 step in . Figure 4 S2-S4 in the above are completed by the Worker task side, including:

[0057] Step S2: When the Worker task end executes the task, the Worker task end can obtain the location of the accumulator, the type of the accumulator, and the data type of the accumulator. The source transfer module in each task is responsible for transferring the data source according to the record level.

[0058] Step S3: Each task in S2 reads the data broadcast to the Worker end in S1, and the purpose combination module obtains the data operation purposes of all operators, determines the data operation purposes of the data operation operators before the data operation operator position where the accumulator is located; determines the data operation purpose of the accumulator according to the accumulator type in S2 and the data type of the accumulator; combines the data operation purposes of the data operation operators and the data operation purposes of the accumulator, and finally confirms the data processing purpose of the accumulator.

[0059] Step S4: Similar to S3, in the data filtering module, each task reads the data broadcast to the Worker in S1, obtains the access control policies of all data sources, and then Figure 2 S5-S7 in filter the data in the accumulator.

[0060] Embodiment 3

[0061] In order to better illustrate the access control method and implementation architecture, the present invention provides the following Figure 5 The data processing program example shown in Figure 5 As shown in the example code, line L1 instantiates a CollectionAccumulator, line L2 reads the data in the S data source, line L3 maps the data to integer data, line L4 in code ① uses CollectionAccumulator to track data less than or equal to -1, line L5 triggers the sum operator, and L6 returns the tracking result. Among them, Figure 5 The italicized records '-1' in the table are sensitive data.

[0062] Two data users, Bob and Alice, want to analyze a data set, and at the same time, they have different authorizations for the data. Bob cannot output sensitive data and data derived from sensitive data through the accumulator, while Alice is not subject to any restrictions. The access control in the present invention can generate different results for the two data users, realizing purpose-aware access control that supports data traceability and protects data security.

[0063] like Figure 6 As shown, given Figure 5 The data processing program in the DAG representation of the data processing optimizer is shown in Figure 1. The location of the accumulator is RDD2, the type of the accumulator is CollectionAccumulator, and the data type of the accumulator is integer. Each task accumulates data in a distributed manner according to the condition ① in line L4. The accumulator receives the data source information passed by the source and is used to find strategies and filter illegal data. Figure 5 , 6 As shown, "-1" is not extracted by Bob.

Claims

1. An access control method for Apache Spark accumulators, characterized in that: The following steps are involved: 1) According to the data processing program, determine the location, type and data type of the accumulator AccX used in the data processing program; 2) confirming the data processing purpose of each accumulator AccX according to the location, type and data type of the accumulator AccX; 3) Based on the data processing purpose of the accumulator AccX, determine the accessible data in the accumulator AccX.

2. The access control method for Apache Spark accumulator according to claim 1, characterized in that: The step 2) comprises the following steps: 2.1) According to the position of each accumulator AccX, based on the data processing optimizer, determine the data operation purpose of the data operation operator before the data operation operator position where AccX is located; 2.2) Determine the data operation purpose of each accumulator AccX according to the type and data type of the accumulator AccX; 2.3) The data operation purpose of the data operation operator is combined with the data operation purpose of the accumulator AccX as the data processing purpose of each accumulator AccX.

3. The access control method for Apache Spark accumulator according to claim 1, characterized in that: The step 3) comprises the following steps: 3.1) Get the data set Φ in the accumulator AccX, and find the origin data set M of each record data Item in Φ; 3.2) Match the corresponding access control policy in the policy library for each data source involved in M; 3.3) Traverse all access control policies, fine-grainedly match the allowed data processing purpose of each item, and determine whether the accumulator data processing purpose meets the allowed data processing purpose of the current item. If not, filter out the item, otherwise do not filter.

4. An access control system for Apache Spark accumulators, characterized in that: include: An accumulator determination module is used to determine the position, type and data type of the accumulator AccX used in the data processing program according to the data processing program; A data processing purpose determination module, used to determine the data processing purpose of each accumulator AccX according to the position, type and data type of the accumulator AccX; The access control module is used to determine the accessible data in the accumulator AccX based on the data processing purpose of the accumulator AccX.

5. An access control system for Apache Spark accumulator according to claim 4, characterized in that: The data processing purpose determination module includes: A data operation operator data operation purpose determination module is used to determine the data operation purpose of the data operation operator before the data operation operator position where AccX is located based on the data processing optimizer according to the position of each accumulator AccX; The accumulator AccX data operation purpose determination module is used to determine the data operation purpose of each accumulator AccX according to the type and data type of the accumulator AccX; The data processing purpose determination module is used to merge the data operation purpose of the data operation operator with the data operation purpose of the accumulator AccX as the data processing purpose of each accumulator AccX.

6. An access control system for Apache Spark accumulator according to claim 4, characterized in that: The access control module comprises: The access control policy matching module is used to obtain the data set Φ in the accumulator AccX, find the origin data set M of each record data Item in Φ, and match the corresponding access control policy in the policy library for each data source involved in M; The record data filtering module is used to traverse all access control policies, determine the allowed data processing purpose of each item, and judge whether the accumulator data processing purpose meets the allowed data processing purpose of the current item. If not, the item is filtered out, otherwise it is not filtered.

7. An access control implementation architecture for Apache Spark accumulators, used to implement the access control method for Apache Spark accumulators described in claim 1, characterized in that: It includes the Driver side and the Worker task side, where: The Driver side is used to analyze the data operation purpose of each operator in the data processing program according to the data processing optimizer, and to find the access control policy according to the data source in the data processing program. Finally, the data operation purpose of all operators and the access control policy corresponding to all data sources are broadcast to the Worker side. The Worker task end is used to pass the data source header according to the record level when executing tasks, and obtain the location, type, and data type of the accumulator.

8. An access control device for an Apache Spark accumulator, characterized in that: The invention comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement an access control method for an Apache Spark accumulator as described in any one of claims 1 to 3 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, an access control method for an Apache Spark accumulator as described in any one of claims 1 to 3 is implemented.

Citation Information

Patent Citations

  • Extensible access control list framework

    CN102612694A

  • A Spark-oriented access control method and system

    CN109409120A

  • Big data access control method and device, electronic equipment and storage medium

    CN112182023A

  • Data storage system and method for controlling access to data stored in a data storage

    WO2022231514A1