Implementation method for aggregating bitwise OR operation in Spark data processing

Abstract

The invention discloses an implementation method for aggregating bitwise OR operation in Spark data processing. The implementation method comprises the following steps: S1, defining a UDAF (Unified Data Adaptive Function) through programming; s2, registering the UDAF defined in the S1, registering a UDAF aggregation bitwise or operation function, generating an execution plan, and applying the registered UDAF to a Spark SQL (Structured Query Language); and S3, sending the execution plan generated in the S2 to a task execution process, and initializing the task and executing the task plan by the task execution process. According to the method, a Spark custom aggregation bitwise or operation technology can be realized, various complex business requirements can be flexibly processed in Spark through custom bitwise or aggregation functions, the efficiency and accuracy of data analysis are improved, and people can easily cope with various challenges and changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Spark data processing, and particularly relates to an implementation method of aggregating bitwise OR operation in Spark data processing. BACKGROUND

[0002] In the broad field of Spark data analysis, aggregation operation is a common and key operation that allows us to group data according to certain conditions and perform certain forms of calculation or conversion on the data within each group. This function plays an important role in data processing, data cleaning, and feature engineering, etc. However, standard aggregation functions such as sum and average often cannot meet all complex business needs. In addition, bitwise OR operation is a special aggregation operation that is particularly useful when dealing with binary data or performing certain specific logical operations. The basic idea of bitwise OR operation is to perform bitwise OR operation on the binary representation of all data within the same group, and the result can reflect certain global characteristics or states of the data within the group. However, the built-in aggregation functions of Spark cannot directly implement bitwise OR operation. To solve the above problems, we propose an implementation method of aggregating bitwise OR operation in Spark data processing. SUMMARY

[0003] Based on the technical problems existing in the background technology, the present application proposes an implementation method of aggregating bitwise OR operation in Spark data processing.

[0004] The implementation method of aggregating bitwise OR operation in Spark data processing proposed by the present application includes the following steps:

[0005] S1: defining UDAF through programming;

[0006] S2: registering the UDAF defined in S1, registering the UDAF aggregation bitwise OR operation function, generating an execution plan, and applying the registered UDAF to Spark SQL;

[0007] S3: sending the execution plan generated in S2 to the task execution process, initializing the task and executing the task plan;

[0008] S4: the task process executes the aggregation bitwise OR operation, and in the process of execution, the initialization buffer zero method and the aggregation calculation reduce method are called in turn to complete the aggregation operation, and the result is buffered to the buffer;

[0009] S5: the task process aggregates the partial results in S4 and sends them to the driver process;

[0010] S6: The driving process collects the partial aggregation results sent by the task execution process, and collects the aggregation results collected by the Executor node and sends them to the task process to complete the final aggregation;

[0011] S7: The task process calls the merge method to merge the results and returns the final result;

[0012] S8: Complete the aggregation task through the driving node and collect the final result.

[0013] Preferably, the specific steps of S1 are as follows:

[0014] S101: Define the aggregation buffer class AggOrBuff for buffering the operation results of the aggregation or operation;

[0015] S102: Define the UDAF class BitwiseOrUDAF for inheriting the grouping aggregation parent class provided by Spark and rewriting the parent class method.

[0016] Preferably, in S1, the program code used when defining the UDAF is as follows:

[0017]

[0018]

[0019]

[0020] Preferably, the specific steps of S2 are as follows:

[0021] S201: Create a spark program interaction object sparkSession;

[0022] S202: Register the UDAF aggregation bitwise or operation function through the spark program interaction object and generate an execution plan;

[0023] S203: Prepare 12 pieces of test data of some states of the device, the data contains field encoding sn, type type and status status, wherein the status status uses 4-bit binary identification 0001, 0010, 0100, 1000, corresponding to 10-bit 1, 2, 4, 8, the simple requirement realized in the code is to group according to the type type, filter out the device type with status 1, 2, 4, get the aggregation bitwise or operation function realized in spark, and apply the function to the spark program context;

[0024] S204: In the screening process of S203, the desired packet data is quickly screened out using bitwise AND operation, the packet data is defined as an aggregate bitwise OR operation, and the registered custom aggregate bitwise OR operation is called in the Spark SQL grouping processing.

[0025] Preferably, the program code used by the S2 during operation is as follows:

[0026]

[0027]

[0028]

[0029]

[0030] Preferably, in the S102, the rewriting method includes one or more of the zero method, the reduce method, the merge method, the finish method, the bufferEncoder method and the outputEncoder method, wherein the zero method is used to initialize the buffer; the reduce method includes a buffer b and a line data input a, and performs bitwise OR operation on the aggregation result of the buffer b and the input parameter a, and buffers the result to the buffer; the merge method includes two parameters, buffer b1 and buffer b2, buffer b1 and buffer b2 are respectively the calculation results of the spark computing nodes, the bitwise OR operation is performed on the results of the two buffers in the merge method, and the result is buffered to the buffer; the finish method includes a parameter, buffer reduction, and directly returns the bitwise OR operation result; the bufferEncoder method returns the buffer type definition encoding; the outputEncoder method returns the result output type definition encoding.

[0031] Preferably, in the S1, the full name of UDAF is User-Defined Aggregate Functions, which is Chinese for custom aggregate function.

[0032] Compared with the prior art, the beneficial effects of the present application are:

[0033] The present application can realize the technical of Spark custom aggregate bitwise OR operation, through custom bitwise OR aggregate function, various complex business requirements can be flexibly processed in Spark, the efficiency and accuracy of data analysis are improved, and we can easily cope with various challenges and changes. BRIEF DESCRIPTION OF DRAWINGS

[0034] Fig. 1The flow chart of S1 in the implementation method of aggregating bitwise or operation in Spark data processing according to the present application is as follows:

[0035] Fig. 2 The flow chart of S2-S8 in the implementation method of aggregating bitwise or operation in Spark data processing according to the present application is as follows. DETAILED DESCRIPTION

[0036] The present application will be further described below in combination with specific embodiments.

[0037] EMBODIMENT

[0038] REFERENCE Figs. 1-2 The embodiment provides an implementation method of aggregating bitwise or operation in Spark data processing, which comprises the following steps:

[0039] S1: defining a UDFA by programming, wherein UDFA is short for User-Defined Aggregate Functions, and Chinese name is self-defined aggregate function;

[0040] The specific steps are as follows:

[0041] S101: defining an aggregation buffer class AggOrBuff, which is used for buffering the operation result of aggregating or operation;

[0042] S102: defining a UDFA class BitwiseOrUDAF, which is used for inheriting a grouping aggregation parent class provided by Spark and rewriting the parent class method, wherein the rewritten method comprises one or more of zero method, reduce method, merge method, finish method, bufferEncoder method and outputEncoder method, wherein the zero method is used for initializing a buffer area; the reduce method comprises a buffer area b and row data input a, and performs bitwise or operation on the aggregation result of the buffer area b and the input parameter a, and buffers the result to the buffer area; the merge method comprises two parameters, buffer area b1 and buffer area b2, wherein the buffer area b1 and the buffer area b2 are respectively the calculation result of a spark calculation node, the merge method performs bitwise or operation on the results of the two buffer areas and buffers the result to the buffer area; the finish method comprises a parameter, buffer area reduction, and directly returns the bitwise or operation result; the bufferEncoder method returns buffer area type definition coding; and the outputEncoder method returns result output type definition coding;

[0043] In addition, the program code used when defining the UDFA is as follows:

[0044]

[0045]

[0046] S2: register the UDAF defined in S1, and register the UDAF aggregation bitwise or operation function, generate an execution plan, and apply the registered UDAF to Spark SQL, the specific steps are as follows:

[0047] S201: create a spark program interaction object sparkSession;

[0048] S202: register the UDAF aggregation bitwise or operation function through the spark program interaction object, and generate an execution plan;

[0049] S203: prepare 12 pieces of test data of some states of devices, the data contains fields encoding sn, type type and status status, wherein the status status uses 4-bit binary identification 0001, 0010, 0100, 1000, corresponding to 10 binary bits 1, 2, 4, 8, the simple requirement realized in the code is to group according to the type type, and filter out the device type with status 1, 2, 4, so as to realize the aggregation bitwise or operation function in spark, and apply the function to the spark program context;

[0050] S204: use bitwise and operation to quickly filter out the desired grouped data in the filtering process of S203, customize the grouped data as aggregation bitwise or operation, and call the registered custom aggregation bitwise or operation in Spark SQL grouping processing;

[0051] In addition, the program code used in the running process is as follows:

[0052]

[0053]

[0054]

[0055] S3: send the execution plan generated in S2 to the task execution process, initialize the task and execute the task plan;

[0056] S4: the task process executes the aggregation bitwise or operation, and in the process of execution, the initialization buffer zero method and the aggregation calculation reduce method are called in turn to complete the aggregation operation, and the result is buffered to the buffer;

[0057] S5: the task process aggregates part of the results in S4, and sends them to the driver process;

[0058] S6: The driving process collects the partial aggregation results sent by the task execution process, and collects the aggregation results collected by the Executor node to the task process to complete the final aggregation;

[0059] S7: The task process calls the merge method to merge the results and returns the final result;

[0060] S8: Complete the aggregation task through the driving node and collect the final result;

[0061] The embodiment can realize the technical solution of Spark's self-defined aggregation bitwise OR operation. By defining the bitwise OR aggregation function, various complex business requirements can be flexibly handled in Spark, improving the efficiency and accuracy of data analysis, and enabling us to easily cope with various challenges and changes.

[0062] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent replacements or changes within the technical scope disclosed by the present application according to the technical solution and the inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for implementing aggregation of bitwise OR operation in Spark data processing, characterized in that, The method comprises the following steps: S1: defining UDAF by programming; S2: registering the UDAF defined in S1, registering the UDAF aggregation bitwise OR operation function, generating an execution plan, and applying the registered UDAF to Spark SQL; S3: sending the execution plan generated in S2 to a task execution process, the task execution process initializing a task and executing the task plan; S4: the task process executes the aggregation bitwise OR operation, and in the process of execution, the initialization buffer zero method and the aggregation calculation reduce method are called in sequence to complete the aggregation operation, and the result is buffered to the buffer; S5: the task process aggregates the partial results in S4 and sends them to the driver process; S6: the driver process collects the partial aggregation results sent by the task execution process, collects the aggregation results to the Executor node, and sends them to the task process to complete the final aggregation; S7: the task process calls the merge method to merge the results and returns the final result; S8: completing the aggregation task through the driver node and collecting the final result.

2. The implementation method of aggregating bitwise OR operation in Spark data processing according to claim 1, characterized in that, The specific steps of S1 are as follows: S101: defining an aggregation buffer class AggOrBuff for buffering the operation results of the aggregation OR operation; S102: defining a UDAF class BitwiseOrUDAF for inheriting the grouping aggregation parent class provided by Spark and rewriting the parent class method.

3. The method of claim 1, wherein the method further comprises: The program code used when defining UDAF in S1 is as follows:

4. The implementation method of aggregating bitwise OR operation in Spark data processing according to claim 1, characterized in that, The specific steps of S2 are as follows: S201: creating a spark program interactive object sparkSession; S202: registering the UDAF aggregation bitwise OR operation function through the spark program interactive object and generating an execution plan; S203: preparing 12 pieces of test data of some states of devices, the data containing field encoding sn, type type and status status, wherein the status status is identified by 4-bit binary 0001, 0010, 0100 and 1000, corresponding to 10-bit 1, 2, 4 and 8, the simple requirement realized in the code is to group according to the type type, filter out the device types with status 1, 2 and 4, and obtain the aggregation bitwise OR operation function realized in Spark, and apply the function to the spark program context; S204: using bitwise AND operation to quickly filter out the desired grouped data in the filtering process of S203, defining the grouped data as an aggregation bitwise OR operation, and calling the registered custom aggregation bitwise OR operation in Spark SQL grouping processing.

5. The method of claim 1, wherein the method is implemented in Spark data processing. The program code used in the running process of S2 is as follows: import org.apache.spark.sql.types.{IntegerType,StringTyp e,StructField,StructType} import org.apache.spark.sql.{Row,SparkSession,function s} import pandora.spark.udaf.OrOperationUDAF object BitwiseOrTest{ def main(args:Array[String]):Unit={ / / Create a SparkSession object val spark=SparkSession.builder().appName("Bitw iseOrTest").getOrCreate() / / Register aggregate or operation aggregate functions spark.udf.register("bitwise_or",functions.udaf(ne w BitwiseOrUDAF)) / / Test Data Preparation val rdd=spark.sparkContext.parallelize(Seq( Row("120","ios",2), 6. The method of claim 2, wherein the method further comprises: In S102, the overwrite method includes one or more of the following: zero method, reduce method, merge method, finish method, bufferEncoder method, and outputEncoder method. The zero method is used to initialize the buffer. The reduce method includes a buffer b and row data input a. It performs a bitwise OR operation on the aggregation result of buffer b and the input parameter a, and buffers the result into the buffer. The merge method includes two parameters, buffer b1 and buffer b2, which are the calculation results of Spark computing nodes, respectively. The merge method performs a bitwise OR operation on the results of the two buffers and buffers the result into the buffer. The finish method includes one parameter, buffer reduction, and directly returns the bitwise OR operation result. The bufferEncoder method returns the buffer type definition encoding. The outputEncoder method returns the result output type definition encoding.

7. The method of claim 1, wherein the method is implemented in Spark data processing. In S1, UDAF stands for User-Defined Aggregate Functions.

Images