Method and device for improving big data calculation efficiency based on Spark, equipment and medium
By defining key calculation methods and encapsulating them into an aggregator object, combining partition-level processing functions and interruptible iterators, the problem of flexibility and inefficiency in big data processing by traditional Spark calculation methods is solved, and efficient and flexible data processing and resource optimization are achieved.
Patent Information
- Application Number
- CN202510385102.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-08-19
AI Technical Summary
Traditional Spark computing methods lack flexibility in processing large-scale data, resulting in data transmission and redundant computing, which is difficult to meet diverse data processing needs, and lack of effective interruption mechanisms, resulting in waste of resources and prolonged computing time.
Define the key calculation method, encapsulate it into an aggregator object, and inject it into the Spark framework's cross-partition reorganization process, and iterative operations are performed through partition-level processing functions and interruptible iterators to achieve custom aggregation operations and dynamic adjustments.
It improves the accuracy and efficiency of data processing, reduces data transmission and redundant calculations, supports flexible data processing requirements, and avoids waste of resources and prolongs computing time.
Smart Images
Figure CN120508379A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data development technology, and specifically to a method, device, equipment and medium for improving big data computing efficiency based on Spark. Background Art
[0002] In the era of big data, data volumes are exploding, and efficient data processing and analysis are becoming a pressing need in many fields. Spark, a fast and versatile cluster computing system, is widely used in big data processing scenarios due to its advantages in in-memory computing and distributed processing. However, in practice, traditional Spark computing methods still face challenges when processing large amounts of data.
[0003] Traditional Spark computing methods often lack flexibility when handling complex data aggregation and computational tasks. General-purpose aggregation operators struggle to meet diverse data processing requirements, requiring users to write extensive custom code to implement specific computational logic. This not only increases development costs but also reduces code maintainability. Furthermore, when processing cross-partitioned data, traditional methods can generate significant amounts of data transfer and redundant computation, resulting in low computational efficiency. The reorganization and merging of data across partitions can cause performance bottlenecks, especially when processing large datasets. Furthermore, traditional iterative processing methods lack effective interruption mechanisms when processing large amounts of data. Once a task begins, it is difficult to adjust or stop it midway based on actual conditions, which can waste resources and extend computational time.
[0004] Therefore, how to improve the efficiency of Spark in big data computing has become an urgent problem to be solved. Summary of the Invention
[0005] In response to the above-mentioned problems existing in traditional Spark computing methods when processing complex data aggregation and computing tasks, the present invention provides a method, device, equipment and medium for improving the efficiency of big data computing based on Spark.
[0006] In a first aspect, the technical solution of the present invention provides a method for improving big data computing efficiency based on Spark, comprising the following steps: A key calculation method is defined, wherein the key calculation method includes a first function method for processing the first data, a second function method for processing the second and subsequent data, and a third function method for processing data merging logic after reorganizing data across partitions; Encapsulate the first function method, the second function method, and the third function method into an aggregator object, where the aggregator object is an aggregation operator provided by Spark; the aggregator is used to implement a custom aggregation operation; Inject the encapsulated aggregator object into the Spark framework's cross-partition data reorganization process; Use the partition-level processing function to iterate each partition of the elastic distributed dataset, pass in the encapsulated aggregator object and execute it, so that the calculation of each data is processed according to the logic defined by the function method, and generate an elastic distributed dataset operator.
[0007] As a further limitation of the technical solution of the present invention, the first function method is used to initialize the first data according to data processing requirements and determine its initial calculation method and storage form.
[0008] The second function method is used to merge or perform calculation operations on the second and subsequent data with the processed data according to the data processing logic, so as to realize the step-by-step processing and integration of the data.
[0009] After the cross-partition data reorganization operation is completed, the third function method merges and calculates the data of different partitions to ensure that the merged data meets the final calculation requirements.
[0010] The first function method initializes the first piece of data, determining its initial calculation method and storage format. The second function method merges or performs calculations on the second and subsequent pieces of data with the processed data. This gradual data processing and integration method allows data to be updated gradually according to the predetermined logic during processing, ensuring accuracy and consistency of data processing while also improving data processing efficiency.
[0011] As a further limitation of the technical solution of the present invention, the method further includes: When using partition-level processing functions to iterate over each partition of an elastic distributed dataset, the data in each partition is traversed through an interruptible iterator based on the task context, and methods in the aggregator object are called to process the data.
[0012] As a further limitation of the technical solution of the present invention, when performing an iterative operation on each partition of a resilient distributed dataset using a partition-level processing function, the steps of traversing the data in each partition using an interruptible iterator based on the task context and calling a method in an aggregator object to process the data include: Automatically initialize the task context when the Spark task starts; the task context is a context provided during task execution, including the task ID and stage ID; When the partition-level processing function starts, create an interruptible iterator for the data of the current partition; The interruptible iterator begins to access the data elements in the partition one by one and calls the methods in the aggregator object to process the data; At each iteration, the interruptible iterator will check whether there is an interrupt signal. If there is an interrupt signal, the traversal will be stopped; When the interruptible iterator traverses all data elements in the partition, the partition data processing process ends. At this time, the aggregator object stores the intermediate aggregation results of the partition; Returns the partitioned results processed by the aggregator.
[0013] Using partition-level processing functions to iterate over each partition of a resilient distributed dataset, and using interruptible iterators to traverse the data within a partition, this provides fine-grained control over the data processing process. During each iteration, the interruptible iterator checks for an interrupt signal and halts the traversal if one occurs. This allows for dynamic task adjustments based on actual conditions, avoiding wasted resources and extended computation time.
[0014] As a further limitation of the technical solution of the present invention, the step of calling a method in the aggregator object to process data includes: For the first data element in the partition, the first function method in the aggregator object needs to be called to perform the initialization operation; For other data elements except the first data element in the partition, the interruptible iterator sequentially passes these elements to the second function method of the aggregator object for merging data; In the second function approach, the aggregator processes the new data elements according to the business logic.
[0015] In a second aspect, the technical solution of the present invention also provides a device for improving big data computing efficiency based on Spark, comprising a method definition module, an encapsulation module, an object injection module, and an iterative operation calculation module; A method definition module is used to define a key calculation method, wherein the key calculation method includes a first function method for processing the first data, a second function method for processing the second and subsequent data, and a third function method for processing data merging logic after reorganizing data across partitions; An encapsulation module, configured to encapsulate the first function method, the second function method, and the third function method into an aggregator object, where the aggregator object is an aggregation operator provided by Spark; the aggregator is configured to implement a custom aggregation operation; The object injection module is used to inject the encapsulated aggregator object into the cross-partition data reorganization process of the Spark framework; The iterative operation calculation module is used to use the partition-level processing function to perform iterative operations on each partition of the elastic distributed dataset, pass in the encapsulated aggregator object and execute it, so that the calculation of each data is processed according to the logic defined by the function method, and generate an elastic distributed dataset operator.
[0016] The device technical solution of the present invention achieves a comprehensive improvement in big data processing efficiency, stability and development convenience while ensuring calculation accuracy through modular design, customized aggregation logic, shuffle optimization and partition-level iterative processing. It is suitable for high-load scenarios such as financial risk control, real-time recommendation, log analysis, etc., and has significant industrial application value.
[0017] As a further limitation of the technical solution of the present invention, the first function method is used to initialize the first data according to data processing requirements and determine its initial calculation method and storage form.
[0018] The second function method is used to merge or perform calculation operations on the second and subsequent data with the processed data according to the data processing logic, so as to realize the step-by-step processing and integration of the data.
[0019] After the cross-partition data reorganization operation is completed, the third function method merges and calculates the data of different partitions to ensure that the merged data meets the final calculation requirements.
[0020] As a further limitation of the technical solution of the present invention, when the iterative operation calculation module uses the partition-level processing function to perform iterative operations on each partition of the elastic distributed dataset, it traverses the data in each partition through an interruptible iterator based on the task context, and calls the method in the aggregator object to process the data; wherein, the task context is automatically initialized when the Spark task starts; the task context is a context environment provided during the task execution, which includes the task ID and stage ID.
[0021] As a further limitation of the technical solution of the present invention, the iterative operation calculation module includes a creation unit and a return unit; A creation unit is used to create an interruptible iterator for the data of the current partition when the partition-level processing function is started; The interruptible iterator begins to access the data elements in the partition one by one and calls the methods in the aggregator object to process the data. During each iteration, the interruptible iterator checks whether there is an interrupt signal. If an interrupt signal is received, the traversal is stopped. When the interruptible iterator has traversed all the data elements in the partition, the partition data processing process ends. At this point, the aggregator object stores the intermediate aggregation results of the partition. The return unit is used to return the partition results obtained by the aggregator.
[0022] As a further limitation of the technical solution of the present invention, the interruptible iterator is specifically used to call the first function method in the aggregator object to perform initialization operations for the first data element in the partition; for other data elements in the partition except the first data element, the interruptible iterator passes these elements in turn to the second function method of the aggregator object for merging data; in the second function method, the aggregator processes the new data elements according to the business logic.
[0023] In a third aspect, the technical solution of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for improving big data computing efficiency based on Spark as described in the first aspect.
[0024] In a fourth aspect, the technical solution of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the method for improving big data computing efficiency based on Spark as described in the first aspect.
[0025] It can be seen from the above technical solutions that the present application has the following advantages: by defining a key calculation method and encapsulating the first function method, the second function method and the third function method into an aggregator object, Spark is provided with the ability to customize aggregation operations. Users can flexibly define the logic of each function method according to specific data processing requirements, thereby realizing personalized data processing and calculation. This approach avoids writing a large amount of custom code and improves the maintainability and reusability of the code. Injecting the encapsulated aggregator object into the cross-partition data reorganization process of the Spark framework, and merging and calculating the data of different partitions through the third function method can effectively reduce data transmission and redundant calculations. When processing cross-partition data, data merging is performed according to pre-defined logic to ensure that the merged data meets the final calculation requirements, avoid unnecessary storage and transmission of intermediate results, and thus significantly improve the efficiency of cross-partition data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0027] Figure 1A flowchart of a method provided in an embodiment of the present invention.
[0028] Figure 2 A block diagram of a device provided in an embodiment of the present invention.
[0029] Figure 3 Flowchart for big data analysis of routine procedures.
[0030] Figure 4 Flowchart of the big data analysis process designed for this application method. DETAILED DESCRIPTION
[0031] In order to make the application objectives, features, and advantages of this application more obvious and easy to understand, the technical solutions protected by this application will be clearly and completely described below using specific embodiments and drawings. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this patent, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this patent.
[0032] Spark, a general-purpose big data computing framework based on the Scala language, is a computing model based on RDDs (Resilient Distributed Datasets). This allows for distributed processing of large sets of data, splitting them up, performing computations on them separately, and then merging the results. Spark is an open-source cluster computing environment similar to Hadoop, but with some useful differences that make it superior for certain workloads. Specifically, Spark enables in-memory distributed datasets, allowing it to optimize iterative workloads in addition to interactive queries. Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala are tightly integrated, allowing Scala to operate on distributed datasets as easily as on local collections.
[0033] Apache Spark, a big data processing framework, provides a simple and efficient platform for processing large amounts of data. Users should minimize the number of shuffle I / O operations during data processing, as shuffle I / O is the most time-consuming operation in the entire big data computation process. However, to obtain data that better meets user requirements, multiple I / O operations are unavoidable using Spark's built-in integrated data processing methods. Therefore, this paper provides a method to circumvent these multiple I / O operations and improve computational efficiency and stability during big data computations.
[0034] Shuffle in Apache Spark is a process involving data redistribution that occurs when performing operations that require reorganizing data across partitions, such as reduceByKey, groupByKey, join, repartition, and sortByKey. The shuffle process ensures that records with the same key are grouped together, which is crucial for performing aggregation and join operations.
[0035] The Shuffle process typically consists of the following steps: (1) Map phase: In this phase, tasks write data to temporary files on the local disk. Each task creates a file for each task in the downstream phase. (2) Reduce phase: In this phase, tasks read data from the nodes where the tasks in the Map phase are located and may perform sorting and aggregation operations.
[0036] The reasons why the shuffle operation may become a performance bottleneck include: Disk I / O: The shuffle process needs to write data to disk and then read it back, which involves a large number of disk I / O operations and may become a performance bottleneck.
[0037] This invention is based on the Spark framework and designs a clever calculateByKey() method for the overall Spark solution. It converts data processing into three general methods, then encapsulates them into an Aggregator object and injects them into the Spark framework shuffle process. It also returns the RDD operator to assist in subsequent calculations. Specifically, Figure 1 As shown, an embodiment of the present invention provides a method for improving big data computing efficiency based on Spark, comprising the following steps: S1: defining a key calculation method, wherein the key calculation method includes a first function method for processing the first data, a second function method for processing the second and subsequent data, and a third function method for processing data merging logic after reorganizing data across partitions; Since Spark generates a shuffle process whenever it uses general methods such as reduceByKey() and groupByKey(), the shuffle process is the most time-consuming. Moreover, these general methods can only achieve a simple one-step data conversion effect. When faced with complex logic and requirements, they will be used multiple times, resulting in multiple shuffle IOs. In this embodiment of the present invention, the first function method is used to initialize the first data according to the data processing requirements and determine its initial calculation method and storage format.
[0038] The second function method is used to merge or perform calculation operations on the second and subsequent data with the processed data according to the data processing logic, so as to realize the step-by-step processing and integration of the data.
[0039] After the cross-partition data reorganization operation is completed, the third function method merges and calculates the data of different partitions to ensure that the merged data meets the final calculation requirements.
[0040] Taking e-commerce order data processing as an example, when processing each user's order data, when encountering the first order data of that user, we need to initialize it. For example, we can use the order amount as the initial total amount and store it in a suitable data structure, such as Tuple or a custom class object. The code example is as follows: def first_function(order): # Assume that order is a tuple containing order information, and the second element is the order amount initial_total=order[1] return initial_total In this example, first_function is the first function method. According to the data processing requirements, it initializes the first piece of data and determines the initial calculation method (directly taking the order amount) and storage format (stored as a single value).
[0041] For each user's second and subsequent order data, you need to merge or calculate it with the processed data. In this example, the amount of subsequent orders is added to the total amount of previous orders. Code example: def second_function(current_total, new_order): # Assume new_order is a tuple containing order information, and the second element is the order amount new_amount=new_order[1] updated_total=current_total+new_amount return updated_total In this example, second_function is the second function method. According to the data processing logic, the new order data is combined with the processed total amount to achieve gradual data processing and integration.
[0042] After reorganizing data across partitions, it is necessary to merge and calculate the data from different partitions. Continuing with the example of e-commerce order data processing, assume that some users' order data are stored in different partitions. After reorganizing this data, we need to merge the total order amounts of the same user in different partitions. Code example: def third_function(total1, total2): combined_total=total1+total2 return combined_total In this example, third_function is the third function method. After the cross-partition data reorganization operation is completed, the data from different partitions is merged and calculated to ensure that the merged data meets the final calculation requirements, that is, to obtain the accurate total order amount for each user.
[0043] S2: Encapsulate the first function method, the second function method, and the third function method into an aggregator object; the aggregator is used to implement a custom aggregation operation; In Spark, we can use the Aggregator class to encapsulate these three function methods into an aggregator object, which serves as the aggregation operator provided by Spark to implement custom aggregation operations.
[0044] S3: Injects the encapsulated aggregator object into the Spark framework's cross-partition data reorganization process; Inject the encapsulated aggregator object into the Spark framework's cross-partition reorganization of data, such as when using operations such as groupByKey or reduceByKey, so that Spark can process the data according to the aggregation logic we defined.
[0045] S4: Use the partition-level processing function to iterate each partition of the elastic distributed dataset, pass in the encapsulated aggregator object and execute it, so that the calculation of each data is processed according to the logic defined by the function method, and generate an elastic distributed dataset operator.
[0046] We iterate over each partition of the Resilient Distributed Dataset (RDD) using a partition-level processing function, passing in the encapsulated aggregator object and executing it. This way, each piece of data is calculated according to the logic of the function we defined, ultimately generating a new RDD operator containing the total order amount for each user.
[0047] The first function method initializes the first piece of data, determining its initial calculation method and storage format. The second function method merges or performs calculations on the second and subsequent pieces of data with the processed data. This gradual data processing and integration method allows data to be updated gradually according to the predetermined logic during processing, ensuring accuracy and consistency of data processing while also improving data processing efficiency.
[0048] In some embodiments, the method further includes: when performing iterative operations on each partition of the elastic distributed dataset using a partition-level processing function, traversing the data in each partition through an interruptible iterator based on the task context, and calling methods in the aggregator object to process the data.
[0049] The specific steps are as follows: When a Spark task starts, a task context is automatically initialized. This context contains information such as the task ID and stage ID. When a partition-level processing function starts, an interruptible iterator is created for the data in the current partition. The interruptible iterator begins accessing the data elements within the partition one by one, calling methods in the aggregator object to process the data. During each iteration, the interruptible iterator checks for an interrupt signal and stops if so. When the interruptible iterator has traversed all data elements within the partition, processing of the partitioned data ends, and the aggregator object stores the intermediate aggregation results for that partition. The partitioned results obtained by the aggregator are then returned.
[0050] It should be noted that when a Spark task starts, Spark automatically initializes the task context, which contains information such as the task ID and stage ID. In Python, you can obtain this information through the Spark API.
[0051] When a partition-level processing function is started, an interruptible iterator is created for the data of the current partition. In Spark, the itertools module can be used to simulate the functionality of interruptible iterators.
[0052] Using partition-level processing functions to iterate over each partition of a resilient distributed dataset, and using interruptible iterators to traverse the data within a partition, this provides fine-grained control over the data processing process. During each iteration, the interruptible iterator checks for an interrupt signal and halts the traversal if one occurs. This allows for dynamic task adjustments based on actual conditions, avoiding wasted resources and extended computation time.
[0053] The steps of calling the method in the aggregator object to process data include: for the first data element in the partition, it is necessary to call the first function method in the aggregator object to perform initialization operations; for other data elements in the partition except the first data element, the interruptible iterator can pass these elements in turn to the second function method of the aggregator object for merging data; in the second function method, the aggregator processes the new data elements according to the business logic.
[0054] It should be noted that the key calculation method in this embodiment refers to the calculateByKey() method. This method has three parameters, which are three function injections: the first function, createCalculate, handles the first data entry; the second function, mergeValue, handles the second and subsequent data entries; and the third parameter, mergeCalculate, handles merging data sets after shuffle. These three functions are encapsulated into the Aggregator function, an aggregation operator provided by Spark that developers can use to implement custom aggregation operations.
[0055] Finally, define a mapPartition for the partition iteration of the ADD operation and pass the newly encapsulated Aggregator into it for execution. This ensures that each data entry will be evaluated against the three functions passed to the calculateByKey method. By passing in these three parameters, you can group complex operations into this method, specifying the desired data structure and the desired big data computation effect. This allows programmers to significantly reduce big data overhead and improve computational efficiency through coding.
[0056] Example 1: Design a simple program to calculate the two days with the highest temperature each month, where the data may be recorded multiple times on the same day.
[0057] (1) Conventional programming First, sort by temperature to get sorted data, then reduce the sorted data to select only the highest temperature on the same day, then remap the data, select the first two days of the month to form tuple data, and finally group and sort by month to directly get the data of the two days with the highest temperature in each month. Figure 3 shown.
[0058] (2) Program design of this application method Using the latest calculateByKey method, data can be deduplicated and sorted one by one according to a custom function structure. The entire calculation process is a shuffle process, which enables fast calculation in large batches. Figure 4 shown.
[0059] In the big data flow chart analysis, the jumps between different stages indicate the occurrence of shuffles. The conventional design uses three shuffles, while the design using this method uses only one shuffle, demonstrating the excellent characteristics of this design.
[0060] By encapsulating the calculateByKey method and optimizing the data processing logic, this paper can consolidate computational tasks that would otherwise require multiple shuffles (such as multiple reduceByKey or groupByKey operations) into a single shuffle, significantly reducing disk I / O and network transmission overhead, thereby improving the overall execution efficiency of Spark jobs. By employing a fixed method (calculateByKey) and parameterized design (createCalculate, mergeValue, mergeCalculate), developers can flexibly define data processing logic through custom functions, eliminating the need to manually write complex multi-stage shuffle code, reducing development complexity and maintenance costs.
[0061] By reducing the number of shuffles, this invention can effectively reduce memory usage and disk read / write pressure, avoiding job failures caused by data skew or memory overflow, thereby improving the stability and reliability of big data computing tasks. By combining three custom function parameters, this invention can support a variety of complex computing requirements (such as aggregation, sorting, and deduplication), allowing tasks that originally required multiple shuffles to be completed in a single shuffle, while maintaining the clarity and scalability of the computing logic.
[0062] This invention optimizes Spark's native APIs (such as Aggregator and MapPartition), achieving efficient computation without modifying Spark's underlying framework. It offers excellent compatibility and portability, making it easy to deploy and apply within existing Spark clusters. In scenarios with large data volumes and complex computational logic (such as log analysis, recommendation systems, and financial risk control), this invention can significantly reduce computation time, helping enterprises complete data analysis tasks more quickly and improve business decision-making efficiency.
[0063] like Figure 2 As shown, an embodiment of the present invention further provides a device for improving big data computing efficiency based on Spark, comprising a method definition module, an encapsulation module, an object injection module, and an iterative operation computing module; A method definition module is used to define a key calculation method, wherein the key calculation method includes a first function method for processing the first data, a second function method for processing the second and subsequent data, and a third function method for processing data merging logic after reorganizing data across partitions; An encapsulation module, configured to encapsulate the first function method, the second function method, and the third function method into an aggregator object, where the aggregator object is an aggregation operator provided by Spark; the aggregator is configured to implement a custom aggregation operation; The object injection module is used to inject the encapsulated aggregator object into the cross-partition data reorganization process of the Spark framework; The iterative operation calculation module is used to use the partition-level processing function to perform iterative operations on each partition of the elastic distributed dataset, pass in the encapsulated aggregator object and execute it, so that the calculation of each data is processed according to the logic defined by the function method, and generate an elastic distributed dataset operator.
[0064] In an embodiment of the present invention, the first function method is used to initialize the first piece of data according to data processing requirements and determine its initial calculation method and storage form.
[0065] The second function method is used to merge or perform calculation operations on the second and subsequent data with the processed data according to the data processing logic, so as to realize the step-by-step processing and integration of the data.
[0066] After the cross-partition data reorganization operation is completed, the third function method merges and calculates the data of different partitions to ensure that the merged data meets the final calculation requirements.
[0067] In some embodiments, when the iterative operation calculation module uses the partition-level processing function to perform iterative operations on each partition of the elastic distributed dataset, it traverses the data in each partition through an interruptible iterator based on the task context, and calls methods in the aggregator object to process the data; wherein, the task context is automatically initialized when the Spark task starts; the task context is a context environment provided during the task execution, which includes the task ID and the stage ID.
[0068] The iterative operation calculation module includes creating units and returning units; A creation unit is used to create an interruptible iterator for the data of the current partition when the partition-level processing function is started; The interruptible iterator begins to access the data elements in the partition one by one and calls the methods in the aggregator object to process the data. During each iteration, the interruptible iterator checks whether there is an interrupt signal. If an interrupt signal is received, the traversal is stopped. When the interruptible iterator has traversed all the data elements in the partition, the partition data processing process ends. At this point, the aggregator object stores the intermediate aggregation results of the partition. The return unit is used to return the partition results obtained by the aggregator.
[0069] The interruptible iterator is specifically used for the first data element in the partition. It needs to call the first function method in the aggregator object to perform initialization operations; for other data elements in the partition except the first data element, the interruptible iterator passes these elements in turn to the second function method of the aggregator object for merging data; in the second function method, the aggregator processes the new data elements according to business logic.
[0070] Through the modular design of a method definition module, an encapsulation module, an object injection module, and an iterative operation calculation module, the present invention breaks down complex Spark optimization logic into clear functional units, facilitating code reuse and maintenance. Developers can flexibly adjust the implementation of each module without modifying the overall architecture, significantly improving development efficiency and system scalability. Through the three function methods (first function, second function, and third function) provided by the method definition module, users can customize data processing rules according to specific business needs, supporting various computing scenarios such as initialization, incremental merging, and cross-partition aggregation, greatly enhancing Spark's flexibility and applicability. The encapsulation module encapsulates user-defined functions as Spark native aggregator objects and directly embeds them into Spark's shuffle process through the object injection module. This allows tasks that originally required multiple shuffles (such as multi-stage aggregation or join operations) to be completed in a single shuffle, significantly reducing disk I / O and network transmission overhead and improving computing efficiency. The iterative operation calculation module uses partition-level processing functions and interruptible iterators to achieve parallel processing of RDD partitions. By dynamically checking for interrupt signals, the module can safely terminate computations when tasks time out or resources are insufficient, avoiding resource waste while ensuring the integrity and consistency of data processing. By using an interruptible iterator to process data piece by piece, the present invention effectively controls memory usage and avoids memory overflows caused by data skew or overly large partitions, making it particularly suitable for stable computations on terabyte / petabyte-scale data.
[0071] The device is built entirely on Spark's native APIs (such as Aggregator and MapPartition), allowing deployment without modifying the underlying Spark framework. It is compatible with existing Spark clusters and ecosystems, facilitating rapid enterprise adoption. In complex scenarios such as multi-stage aggregation, sorting, and deduplication, the device optimizes the shuffle and partition iteration logic to reduce computational time and improve the throughput and responsiveness of enterprise big data platforms.
[0072] An embodiment of the present invention also provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The communication bus can be used to transmit information between the electronic device and a sensor. The processor can call logic instructions in the memory to execute the following method: S1: defining a key calculation method, wherein the key calculation method includes a first function method for processing a first piece of data, a second function method for processing second and subsequent pieces of data, and a third function method for processing data merging logic after cross-partition reorganization of data; S2: encapsulating the first, second, and third function methods into an aggregator object, which is an aggregation operator provided by Spark; the aggregator is used to implement custom aggregation operations; S3: injecting the encapsulated aggregator object into the cross-partition reorganization of data process of the Spark framework; S4: using a partition-level processing function to iterate each partition of a resilient distributed dataset, passing in and executing the encapsulated aggregator object, so that the calculation of each piece of data is processed according to the logic defined by the function method, and generating a resilient distributed dataset operator.
[0073] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.
[0074] An embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute the method provided by the above method embodiment, for example, including: S1: defining a key calculation method, the key calculation method includes a first function method for processing the first data, a second function method for processing the second and subsequent data, and a third function method for processing data merging processing logic after cross-partition reorganization of data; S2: encapsulating the first function method, the second function method and the third function method together into an aggregator object, which is an aggregation operator provided by Spark; the aggregator is used to implement custom aggregation operations; S3: injecting the encapsulated aggregator object into the cross-partition reorganization data process of the Spark framework; S4: using the partition-level processing function to iterate each partition of the elastic distributed dataset, passing in and executing the encapsulated aggregator object, so that the calculation of each data is processed according to the logic defined by the function method, and generating an elastic distributed dataset operator.
[0075] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for improving big data computing efficiency based on Spark, characterized in that: The following steps are involved: A key calculation method is defined, wherein the key calculation method includes a first function method for processing the first data, a second function method for processing the second and subsequent data, and a third function method for processing data merging logic after reorganizing data across partitions; Encapsulate the first function method, the second function method, and the third function method into an aggregator object, where the aggregator object is an aggregation operator provided by Spark; the aggregator is used to implement a custom aggregation operation; Inject the encapsulated aggregator object into the Spark framework's cross-partition data reorganization process; Use the partition-level processing function to iterate each partition of the elastic distributed dataset, pass in the encapsulated aggregator object and execute it, so that the calculation of each data is processed according to the logic defined by the function method, and generate an elastic distributed dataset operator.
2. The method for improving big data computing efficiency based on Spark according to claim 1, characterized in that: The first function method is used to initialize the first data according to data processing requirements and determine its initial calculation method and storage form.
3. The method for improving big data computing efficiency based on Spark according to claim 1, characterized in that: The second function method is used to merge or perform calculation operations on the second and subsequent data with the processed data according to the data processing logic, so as to realize the step-by-step processing and integration of the data.
4. The method for improving big data computing efficiency based on Spark according to claim 1, characterized in that: After the cross-partition data reorganization operation is completed, the third function method merges and calculates the data of different partitions to ensure that the merged data meets the final calculation requirements.
5. The method for improving big data computing efficiency based on Spark according to any one of claims 1 to 4, characterized in that: The method further includes: When using partition-level processing functions to iterate over each partition of an elastic distributed dataset, the data in each partition is traversed through an interruptible iterator based on the task context, and methods in the aggregator object are called to process the data.
6. The method for improving big data computing efficiency based on Spark according to claim 5, characterized in that: When using a partition-level processing function to iterate over each partition of an elastic distributed dataset, the following steps are used to traverse the data in each partition using an interruptible iterator based on the task context and call methods in the aggregator object to process the data: Automatically initialize the task context when the Spark task starts; the task context is a context provided during task execution, including the task ID and stage ID; When the partition-level processing function starts, create an interruptible iterator for the data of the current partition; The interruptible iterator begins to access the data elements in the partition one by one and calls the methods in the aggregator object to process the data; At each iteration, the interruptible iterator will check whether there is an interrupt signal. If there is an interrupt signal, the traversal will be stopped; When the interruptible iterator traverses all data elements in the partition, the partition data processing process ends. At this time, the aggregator object stores the intermediate aggregation results of the partition; Returns the partitioned results processed by the aggregator.
7. The method for improving big data computing efficiency based on Spark according to claim 6, characterized in that: The steps of calling methods in the aggregator object to process data include: For the first data element in the partition, the first function method in the aggregator object needs to be called to perform the initialization operation; For other data elements except the first data element in the partition, the interruptible iterator sequentially passes these elements to the second function method of the aggregator object for merging data; In the second function approach, the aggregator processes the new data elements according to the business logic.
8. A device for improving big data computing efficiency based on Spark, characterized in that: It includes method definition module, encapsulation module, object injection module and iterative operation calculation module; A method definition module is used to define a key calculation method, wherein the key calculation method includes a first function method for processing the first data, a second function method for processing the second and subsequent data, and a third function method for processing data merging logic after reorganizing data across partitions; An encapsulation module, configured to encapsulate the first function method, the second function method, and the third function method into an aggregator object, where the aggregator object is an aggregation operator provided by Spark; the aggregator is configured to implement a custom aggregation operation; The object injection module is used to inject the encapsulated aggregator object into the cross-partition data reorganization process of the Spark framework; The iterative operation calculation module is used to use the partition-level processing function to perform iterative operations on each partition of the elastic distributed dataset, pass in the encapsulated aggregator object and execute it, so that the calculation of each data is processed according to the logic defined by the function method, and generate an elastic distributed dataset operator.
9. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the method for improving big data computing efficiency based on Spark as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the method for improving big data computing efficiency based on Spark as described in any one of claims 1 to 7.