Distributed big data key-value aggregation methods, devices, systems, and storage media

Abstract

This invention discloses a distributed big data key-value aggregation method, apparatus, system, and storage medium, comprising: when a user submits a Spark application, SparkContext is initialized and connected to the Spark cluster, creating Driver and Executor processes; the Driver decomposes the task into multiple Stages, and the Tasks of each Stage are assigned to different Executors for parallel execution; the Executors perform efficient packet processing through DPDK; after the DPDK generates a message, it writes the type of the current message; the smart network interface card of the aggregation server will prioritize on-network aggregation when it receives a message of the specified type; after the task is completed, the Executor returns the result to the Driver, the Driver summarizes the results, and SparkContext cleans up resources and shuts down the application.

Description

Technical Field

[0001] This invention belongs to the field of big data network processing technology, and in particular relates to a distributed big data key-value aggregation method, apparatus, system and storage medium. Background Technology

[0002] In recent years, with the rapid development of the Internet and the Internet of Things (IoT), the global data volume has exploded. A report by International Data Corporation (IDC) indicates that by 2025, the global data volume will reach 175 ZB, placing higher performance demands on data processing technologies. Traditional centralized data processing methods, due to their concentrated computing resources and poor scalability, are no longer sufficient to meet the current real-time and high-efficiency requirements of data processing. Therefore, distributed computing frameworks such as Hadoop and Spark have emerged. Existing distributed processing technologies employ software-based key-value aggregation and utilize network switch hardware to implement aggregation computation.

[0003] 1. Software-based distributed key-value aggregation: It is characterized by large memory capacity, but requires multiple round-trip latency processing, resulting in slow aggregation speed; throughput is limited by network and worker servers, and the shuffle stage will cause a large amount of memory and network I / O, which will seriously consume server resources, requiring high server costs and taking a long time.

[0004] 2. Switch-based intranet aggregation service: In the current switch-based intranet aggregation service, due to hardware memory limitations, aggregation may fail due to the length of the aggregation key or hash collisions. In order to ensure the correctness of the aggregation, the conflicting aggregation message will fall back to the server for processing. It cannot achieve complete on-network computing. It can achieve high efficiency, but there are flaws. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a distributed big data key-value aggregation method, apparatus, system and storage medium.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A distributed big data key-value aggregation method, comprising:

[0008] Step S1: When a user submits a Spark application, the SparkContext is initialized and connected to the Spark cluster, creating the Driver process and the Executor process.

[0009] Step S2: The Driver process decomposes the task into multiple Stages, each Stage contains multiple Tasks, and the Tasks are assigned to different Executor processes for parallel execution. The Executor processes perform efficient packet processing through DPDK.

[0010] Step S3: After the DPDK generates the message, write the type of the current message: query, aggregation, retransmission. When the smart network card of the aggregation server receives a message of the specified type, it will prioritize on-network aggregation. After the task is completed, the Executor process returns the result to the Driver process. The Driver process summarizes the results, and SparkContext cleans up resources and shuts down the application.

[0011] As a preferred option, the smart network card is based on an FPGA and RISCV computing unit fusion architecture. It utilizes the FPGA pipeline to quickly and efficiently aggregate key-value data, while the RISCV computing unit performs software-level aggregation processing through an intranet asynchronous aggregation protocol. At the same time, it adopts a dynamic hot and cold key separation mechanism, which uses a counter array and a count-minimum sketch to count the access frequency of keys in real time. Hot keys are preferentially processed by the FPGA pipeline, while cold keys are transferred to the RISCV computing unit for processing.

[0012] The present invention also provides a distributed big data key-value aggregation device, comprising:

[0013] The first processing module is used to initialize the SparkContext and connect to the Spark cluster, and create the Driver process and Executor process when a user submits a Spark application.

[0014] The second processing module is used by the Driver process to decompose the task into multiple Stages. Each Stage contains multiple Tasks, which are assigned to different Executor processes for parallel execution. The Executor processes perform efficient packet processing through DPDK.

[0015] The third processing module is used to write the type of the current message after the DPDK generates the message: query, aggregation, retransmission. When the smart network card of the aggregation server receives a message of the specified type, it will prioritize on-network aggregation. After the task is completed, the Executor process returns the result to the Driver process, the Driver summarizes the result, and the SparkContext process cleans up resources and shuts down the application.

[0016] As a preferred option, the smart network card is based on an FPGA and RISCV computing unit fusion architecture. It utilizes the FPGA pipeline to quickly and efficiently aggregate key-value data, while the RISCV computing unit performs software-level aggregation processing through an intranet asynchronous aggregation protocol. At the same time, it adopts a dynamic hot and cold key separation mechanism, which uses a counter array and a count-minimum sketch to count the access frequency of keys in real time. Hot keys are preferentially processed by the FPGA pipeline, while cold keys are transferred to the RISCV computing unit for processing.

[0017] The present invention also provides a distributed big data key-value aggregation system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a distributed big data key-value aggregation method when executed by the processor.

[0018] The present invention also provides a storage medium storing a computer program, which executes a distributed big data key-value aggregation method during runtime.

[0019] This invention is based on a smart network interface card (NIC) architecture integrating FPGA and RISC-CV computing units. It utilizes the FPGA pipeline to rapidly and efficiently aggregate key-value data. For hash collisions or key-length exceeding limits that the pipeline struggles to handle, the RISC-CV computing unit performs software-level aggregation. An intra-network asynchronous aggregation protocol is designed to improve data processing efficiency and reduce network latency asynchronously. Simultaneously, a dynamic hot / cold key separation mechanism is constructed. By using a counter array and a count-minimum sketch to monitor key access frequency in real time, hot keys are preferentially processed by the FPGA pipeline, while cold keys are handled by the RISC-CV computing unit. This invention fully leverages the advantages of hardware and software integration, reducing network data transmission volume and CPU load, lowering memory usage, improving the efficiency of aggregation operations and overall system throughput, and significantly alleviating bottlenecks such as high CPU load and large network latency in the Shuffle stage. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a flowchart of the distributed big data key-value aggregation method according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a smart network interface card (NIC).

[0023] Figure 3This is a schematic diagram of a pipelined hardware architecture FPGA structure.

[0024] Figure 4 This is a schematic diagram of the hardware pipeline process. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1:

[0028] like Figure 1 As shown, this embodiment of the invention provides a distributed big data key-value aggregation method, including:

[0029] Step S1: When a user submits a Spark application, the SparkContext is initialized and connected to the Spark cluster, creating the Driver process and the Executor process.

[0030] Step S2: The Driver process decomposes the task into multiple Stages, each containing multiple Tasks. These Tasks are assigned to different Executor processes for parallel execution. The Executor processes perform efficient packet processing using DPDK. Spark writes the data to be processed into a shared memory area, and DPDK reads the data directly from the shared memory for processing, avoiding data copying between user space and kernel space and improving processing efficiency. Control information between Spark and DPDK (such as task status, processing results, etc.) can be transmitted through message queues to ensure the orderly transmission and reliable delivery of control information. DPDK replaces the interrupt mode with a polling mode, uses large page memory and memory pool technology to reduce memory access overhead, and utilizes CPU affinity to bind processing threads to specific cores to achieve zero-copy and batch processing, thereby significantly improving the performance of network packet processing.

[0031] In step S3, after the dpdk generates the message, it will write the type of the current message: query, aggregation, retransmission. When the smart network card of the aggregation server receives a message of the specified type, it will prioritize on-network aggregation. After the task is completed, the Executor process will return the result to the Driver process. The Driver process will summarize the result. Finally, SparkContext will clean up resources and shut down the application.

[0032] As shown in Figure 2, the smart NIC includes a pipelined hardware architecture FPGA and a RISC-CV computing unit. However, the speed of the FPGA is much greater than that of the RISC-CV computing unit. The smart NIC first performs normal key-value aggregation on the pipelined FPGA. If key-value conflicts or hot / cold key alternation occur, the packet type will be rewritten by the hardware architecture, but the packet payload will not change until aggregation is complete. In the traffic splitter, after discrimination, conflicting packets are stored in shared memory, waiting for asynchronous execution by the RISC-CV computing unit. The execution result is also placed in shared memory, waiting for the information collection unit to retrieve it.

[0033] The specific process is as follows: The message first enters the hardware pipeline architecture, where most of the key-value aggregation operations are performed. After execution, if there is no hash collision, the message content to be processed is empty and it will be directly sent to the host. If there is a collision, the message will be copied and written to both the read memory and the hotkey statistics module. When the hotkey statistics module determines that the key-value pair of the current message has a hotkey, it will report it to RISCV. RISCV will compare the current hotkey with the key in the memory at a specified location in the hardware architecture. If it finds that the current key appears more times than the stored value, it will perform a replacement operation. On the other hand, if it is not a hotkey, after being written to the write memory, RISCV will poll the memory, aggregate the key-value pairs stored in the memory, assemble them into a message, store it in the read memory, and finally send it to the host. This achieves the function of decoupling the hardware and RISCV operations.

[0034] As shown in Figure 3, this pipeline-based hardware architecture first receives data packets, quickly parses and confirms their legality through security checks and preprocessing units. Subsequently, the keyword extraction module concurrently extracts packet keywords and calculates hash values ​​to determine the processing path for subsequent aggregation tasks. The ASK task allocation module dynamically allocates tasks to the corresponding aggregation units based on the hash index obtained from coroutine processing, thereby realizing pipelined aggregation. During the aggregation execution phase, the packets coroutinely complete the aggregation calculation, and the aggregation results are coroutinely written into the key-value field. Based on the coroutine decision, the data is retransmitted, broadcast, or discarded. Finally, the processed packets are coroutinely sent out of the pipeline through the forwarding unit, ensuring efficient aggregation and low-latency processing of packets.

[0035] The specific process is as follows: First, the packet enters the SPF (Security Preprocessing Filtering Unit), which filters out all packets except those with a specified VLAN ID and UDP protocol. Then, it enters the PPA (Programmable Protocol Independent Parsing Unit), which parses the key-value pairs carried in the packet and stores the units that need to be processed by the pipeline architecture as phv (Payload Header Vector). For the keyword extraction unit in each stage, the required key-value pair is extracted from the phv. The hash unit hashes the key to generate an index, finds the specified position in the databuffer, and extracts the value of the stored key. In the aggregation state execution unit: if two keys are the same, the current value is directly added to memory; if they are different, a hash collision occurs, which is not handled and is skipped, and the packet is handed over to RISCV for processing. Finally, the aggregator unit (value field) performs the corresponding operation on the packet according to the state provided by ASA and modifies the result content in the packet. Each stage processes different positions of the phv, thus forming a pipeline effect.

[0036] As shown in Figure 4, this invention proposes a best-effort solution for hot and cold key separation based on a hybrid architecture of smart network interface cards (NICs). This ensures that most hot keys are not overtaken by cold keys in the high-speed pipeline aggregator due to the unknown last occurrence time of asynchronous aggregation. Instead, cold keys are aggregated by low-speed RISCV computing units, while hot keys are aggregated by the high-speed pipeline architecture. In the hardware pipeline, a counter array and a count-min sketch are constructed using register arrays. The counter array counts the access count of each key-value pair aggregated in the pipeline; the count-min sketch approximates the access count of key-value pairs not aggregated in the pipeline. When the access count of a key-value pair reaches a certain threshold, the system reports it to the RISCV computing unit to maintain the approximate frequency of keys not aggregated in the pipeline. The RISCV computing unit then atomically replaces the cold keys in the aggregator.

[0037] Traditional software-based aggregation processing consumes significant server resources, including high-performance CPUs, large amounts of memory, and storage devices; while switch-based methods require costly programmable switches with high upgrade costs. This invention utilizes existing server infrastructure and offloads aggregation tasks through smart network interface cards (NICs), effectively reducing hardware upgrade and maintenance costs and offering a significant cost advantage.

[0038] From an efficiency perspective, this invention combines an FPGA pipeline with a RISC-CV computing unit, leveraging the high processing speed of FPGAs and the flexibility of RISC-CV in handling complex tasks to significantly improve the efficiency of key-value aggregation processing. Simultaneously, the asynchronous aggregation protocol and dynamic hot / cold key separation strategy effectively reduce network latency and resource waste, significantly increasing overall data processing throughput. Compared to pure software and pure switch solutions, the efficiency is greatly improved.

[0039] From a completeness perspective, in key-value aggregation, the keys in each word packet sent by each worker cannot be guaranteed to be distinct. The network interface card (NIC) is unaware of the last occurrence of a key or the completion status of key aggregation. The system needs to perform vectorized processing on the tuple aggregation of a data packet within a single switch pipeline, hashing the keys to be aggregated and distributing them to different pipeline stages to improve system throughput. However, the programmability and memory access patterns of the pipeline architecture limit vectorized processing. Due to the limited physical memory of the programmable switch, when the number of aggregated keys exceeds the programmable switch's predetermined capacity, key-value aggregation will fall back to the aggregation server. Furthermore, the actual packet may contain keys longer than the aggregator's length. To address this, the programmable switch can use multiple aggregators to store key-value tuples, but this wastes aggregator resources, only aggregating single key-value pairs. More seriously, when the key length exceeds the total size of all aggregators in a single stage, the aggregation service cannot be completed within the switch and must fall back to the receiving server, failing to achieve complete on-network aggregation. By leveraging the hybrid architecture of the RISCV computing unit, the smart network interface card (NIC) portion of this invention can achieve efficient hash collision avoidance: First, corresponding memory space is allocated for the RISCV computing unit dedicated to the smart NIC to implement a HashMap structure. If a hash collision occurs in the pipeline structure due to memory usage, the conflicting packets will be sent to the RISCV computing unit for aggregation. Since the execution efficiency of the RISCV computing unit is much lower than that of the pipeline hardware architecture, it can be used to handle small traffic volumes that cannot be aggregated by the hardware pipeline due to hash collisions.

[0040] The innovations of this invention are as follows:

[0041] 1 Design of a Distributed Key-Value Aggregation Framework Based on Smart NICs

[0042] This invention integrates hardware architecture into a distributed computing engine aggregation architecture, using Spark as an example, to build a distributed key-value aggregation framework based on smart network interface cards (NICs). Hardware-level on-network key-value aggregation frees up node CPU resources, eliminating the network bottleneck of traditional key-value aggregation. Secondly, a novel intra-network asynchronous aggregation protocol is designed to provide intra-network key-value aggregation services for various distributed applications. In addition, it features a reliability mechanism to ensure efficient aggregation even in complex networks such as retransmissions. The NIC increases network bandwidth, reduces intra-network CPU resource consumption and latency, and guarantees efficient aggregation prioritizing hot keys.

[0043] 2. Optimization Scheme for Distributed Key-Value Aggregation Conflicts Based on Hybrid Architecture

[0044] This invention addresses the limitations of the programmable switch's internal hardware, which forces aggregation to back off, thus limiting on-network computing capabilities. It proposes a distributed key-value aggregation conflict optimization scheme based on a hybrid architecture. This scheme allocates corresponding memory space for the smart network card's dedicated RISCV computing unit, implementing a HashMap structure. When a hash collision occurs in the pipeline structure due to memory usage, the conflicting message is passed to the RISCV computing unit for aggregation, significantly reducing the impact of backoff caused by memory resource limitations.

[0045] 3. Hot and cold bond aggregation optimization based on hybrid architecture

[0046] This invention, based on a hybrid architecture of smart network interface cards (NICs), proposes a best-effort solution for hot and cold key separation. This ensures that most hot keys are not overtaken by cold keys in the high-speed pipeline aggregator due to the unknown last occurrence time of asynchronous aggregation. Instead, cold keys are aggregated by low-speed RISC-CV computing units, while hot keys are aggregated by the high-speed pipeline architecture. In the hardware pipeline, a counter array and a count-min sketch are constructed using register arrays. The counter array counts the access frequency of each key-value pair aggregated in the pipeline; the count-min sketch approximates the access frequency of key-value pairs not aggregated in the pipeline. When the access frequency of a key-value pair reaches a certain threshold, the system reports it to the RISC-CV computing unit to maintain an approximate frequency of keys not aggregated in the pipeline. The RISC-CV computing unit then atomically replaces the cold keys in the aggregator.

[0047] Example 2:

[0048] This invention also provides a distributed big data key-value aggregation device, comprising:

[0049] The first processing module is used to initialize the SparkContext and connect to the Spark cluster, and create Driver and Executor processes when a user submits a Spark application.

[0050] The second processing module is used by the Driver to decompose the task into multiple Stages. Each Stage contains multiple Tasks, which are assigned to different Executors for parallel execution. The Executors use DPDK for efficient packet processing.

[0051] The third processing module is used to write the type of the current message after the DPDK generates the message: query, aggregation, retransmission. When the smart network card of the aggregation server receives a message of the specified type, it will prioritize on-network aggregation. After the task is completed, the Executor returns the result to the Driver, the Driver summarizes the result, and the SparkContext cleans up resources and shuts down the application.

[0052] As a consistent implementation of the present invention, the smart network card is based on an FPGA and RISCV computing unit fusion architecture. It utilizes the FPGA pipeline to quickly and efficiently aggregate key-value data, while the RISCV computing unit performs software-level aggregation processing through an intranet asynchronous aggregation protocol. At the same time, a dynamic hot and cold key separation mechanism is adopted. The access frequency of keys is counted in real time through a counter array and a count-minimum sketch. Hot keys are preferentially processed by the FPGA pipeline, while cold keys are transferred to the RISCV unit for processing.

[0053] Example 3:

[0054] This invention also provides a distributed big data key-value aggregation system, including: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a distributed big data key-value aggregation method when run by the processor.

[0055] Example 4:

[0056] This invention also provides a storage medium storing a computer program that executes a distributed big data key-value aggregation method during runtime.

[0057] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A distributed big data key-value aggregation method, characterized in that, include: Step S1: When a user submits a Spark application, the SparkContext is initialized and connected to the Spark cluster, creating the Driver process and the Executor process. Step S2: The Driver process decomposes the task into multiple Stages, each Stage contains multiple Tasks, and the Tasks are assigned to different Executor processes for parallel execution. The Executor processes perform efficient packet processing through DPDK. Step S3: After DPDK generates a message, write the type of the current message: query, aggregation, retransmission. When the smart network card of the aggregation server receives a message of the specified type, it will prioritize on-network aggregation. After the task is completed, the Executor process returns the result to the Driver process. The Driver process summarizes the results, and SparkContext cleans up resources and shuts down the application. The smart NIC is based on an architecture that integrates FPGA and RISCV computing units. It utilizes the FPGA pipeline to quickly and efficiently aggregate key-value data, while the RISCV computing unit performs software-level aggregation processing through an intranet asynchronous aggregation protocol. At the same time, it adopts a dynamic hot and cold key separation mechanism, which uses a counter array and a count-minimum sketch to count the access frequency of keys in real time. Hot keys are preferentially processed by the FPGA pipeline, while cold keys are transferred to the RISCV computing unit for processing.

2. A distributed big data key-value aggregation device, characterized in that, include: The first processing module is used to initialize the SparkContext and connect to the Spark cluster, and create the Driver process and Executor process when a user submits a Spark application. The second processing module is used by the Driver process to decompose the task into multiple Stages. Each Stage contains multiple Tasks, which are assigned to different Executor processes for parallel execution. The Executor processes perform efficient packet processing through DPDK. The third processing module is used to write the type of the current message after DPDK generates the message: query, aggregation, retransmission. When the smart network card of the aggregation server receives a message of the specified type, it will prioritize on-network aggregation. After the task is completed, the Executor process returns the result to the Driver process, the Driver process summarizes the result, and SparkContext cleans up resources and shuts down the application. The smart NIC is based on an architecture that integrates FPGA and RISCV computing units. It utilizes the FPGA pipeline to quickly and efficiently aggregate key-value data, while the RISCV computing unit performs software-level aggregation processing through an intranet asynchronous aggregation protocol. At the same time, it adopts a dynamic hot and cold key separation mechanism, which uses a counter array and a count-minimum sketch to count the access frequency of keys in real time. Hot keys are preferentially processed by the FPGA pipeline, while cold keys are transferred to the RISCV computing unit for processing.

3. A distributed big data key-value aggregation system, characterized in that, include: The system includes a memory and a processor, wherein the memory stores a computer program that is executed by the processor, and the computer program, when executed by the processor, performs the distributed big data key-value aggregation method as described in claim 1.

4. A storage medium, characterized in that, The storage medium stores a computer program, which executes the distributed big data key-value aggregation method as described in claim 1 when it runs.

Images