Dynamic load balancing method and device for big data stream processing system

Abstract

The application provides a dynamic load balancing method and device for a big data stream processing system, the method comprising: collecting message primary key distribution on a computing node and data volume corresponding to each message primary key; analyzing distribution of stream data according to the message primary key distribution and the data volume corresponding to each message primary key; and selecting a load distribution strategy based on the distribution of stream data to distribute unassigned load. The application can better balance the load of the computing node.

Description

Technical Field

[0001] This invention belongs to the field of software technology, and in particular relates to a dynamic load balancing method and apparatus for big data streaming systems. Background Technology

[0002] With the rapid development of the internet, especially the widespread application of technologies such as mobile internet, e-commerce, and social networks, the scale of data has expanded rapidly, ushering in the era of big data in modern computing. How to quickly process massive amounts of data to extract valuable information has become a pressing problem. Modern big data processing systems often utilize horizontally scalable clusters for parallel processing. For example, the streaming big data processing system Apache Flink uses the KeyBy operator to aggregate data with the same primary key and distributes the data across computing nodes for parallel processing.

[0003] Generally, data can be divided into batch data and streaming data. Batch data refers to fixed-bounded datasets, typically processed by batch big data processing systems. Streaming data refers to continuously generated, unbounded data streams, such as ad click events on a webpage, which are a continuous data stream. Streaming data is typically processed by streaming big data processing systems. For example, Apache Flink can be used to update recommendation models online in real time based on ad click-through rates. Streaming data more accurately depicts the continuously generated data in the real world. Streaming big data processing systems process data and provide responses more quickly through continuous computational tasks. For example, in tasks such as daily stock trading volume statistics, streaming processing can perform real-time incremental processing on each transaction, resulting in a faster response time.

[0004] Modern streaming big data processing systems, such as Apache Flink, horizontally scale computing power by parallelizing computational tasks on computing nodes, but new challenges arise as well. For example, uneven load on computing nodes can lead to single-point performance bottlenecks and limited overall throughput. Existing research (reference: Zeuch S, Monte BD, Karimov J, et al. Analyzing efficient stream processing on modern hardware[J]. Proceedings of the VLDB Endowment, 2019, 12(5): 516-530.) indicates that the horizontal scalability of Flink streaming data analysis applications, such as Yahoo Streaming Benchmark, is very poor. Scaling from 1 node to 4 nodes only doubles the throughput, and scaling from 4 nodes to 16 nodes does not even improve the throughput.

[0005] With an auto-incrementing numeric ID as the message primary key and a parallelism of 60, ideally, all 60 compute nodes should handle approximately 100 / 60 = 1.67% of the load. However, experimental results show that 13.3% of the subtasks bear more than 2.17% of the load, and the highest-loaded node bears an additional 111.0% of the load compared to the lowest-loaded node. This causes the heavily loaded nodes to become the performance bottleneck of the system, limiting the overall throughput. KeyBy uses murmurHash as the basis for load distribution, allocating the load across compute nodes. This method leads to uneven load distribution, resulting in poor horizontal scalability and limited throughput of the streaming processing system. Summary of the Invention

[0006] To address the load imbalance problem in streaming big data processing systems, such as Apache Flink, this invention aims to provide a dynamic load balancing method and apparatus for big data streaming processing systems. This method extends existing node load distribution strategies by sampling and summarizing streaming data to obtain data distribution information, which is then passed to an optimal strategy analyzer. During runtime, the load distribution strategy is dynamically adjusted to better balance node load.

[0007] The technical solution of the present invention is as follows:

[0008] A method for dynamic node load balancing in a big data streaming system, the method comprising:

[0009] Collect the distribution of message primary keys on the computing nodes and the amount of data corresponding to each message primary key;

[0010] Based on the distribution of the message primary keys and the amount of data corresponding to each message primary key, the distribution of streaming data is analyzed; wherein, the distribution of streaming data includes: message primary keys are evenly distributed and data is evenly distributed, message primary keys are not evenly distributed but data is evenly distributed, and data is not evenly distributed.

[0011] A load distribution strategy is selected based on the distribution of the streaming data to allocate unassigned load.

[0012] Furthermore, the load distribution strategies include: Modulo load distribution strategy, LeastKey load distribution strategy, and LeastCount load distribution strategy; wherein, the Modulo load distribution strategy allocates unallocated load based on the result of modulo the message primary keys on the compute nodes according to the degree of parallelism, the LeastKey load distribution strategy allocates unallocated load based on the number of primary keys on the compute nodes, and the LeastCount load distribution strategy allocates unallocated load based on the amount of data on the compute nodes.

[0013] Furthermore, a load allocation strategy is selected based on the distribution of the streaming data to allocate unallocated load, including:

[0014] When the distribution of streaming data is such that the message primary key is uniformly distributed and the data is uniformly distributed, the node number of the computing node is obtained by taking the modulo of the message primary key on the computing node according to the degree of parallelism.

[0015] Assign the unassigned load to the compute node with that node number.

[0016] Furthermore, a load allocation strategy is selected based on the distribution of the streaming data to allocate unallocated load, including:

[0017] In the case where the distribution of the streaming data is such that the message primary key is not uniformly distributed but the data is uniformly distributed, the message primary key of the unallocated load is obtained and the historical allocation record is retrieved.

[0018] If the message primary key of the unallocated load exists in the historical allocation record, the unallocated load will be allocated to a computing node with the same message primary key.

[0019] If the message primary key of the unallocated load does not exist in the historical allocation record, calculate the number of message primary keys on each computing node, and allocate the unallocated load to the computing node with the fewest message primary keys.

[0020] Furthermore, a load allocation strategy is selected based on the distribution of the streaming data to allocate unallocated load, including:

[0021] When the distribution of the streaming data is uneven, obtain the primary key of the message with unallocated load and retrieve historical allocation records;

[0022] If the message primary key of the unallocated load exists in the historical allocation record, the unallocated load will be allocated to a computing node with the same message primary key.

[0023] If the message primary key of the unallocated load does not exist in the historical allocation record, the amount of data on each computing node is calculated, and the unallocated load is allocated to the computing node with the least amount of data.

[0024] Furthermore, the method also includes:

[0025] When a triggering condition is met in the big data streaming system, a state exchange instruction is generated; wherein, the state exchange instruction is used to trigger state exchange between computing nodes, and the triggering condition includes: reaching a configured specified interval, the amount of data in the big data streaming system being greater than a first set threshold, or the load balancing degree of the big data streaming system being lower than a second set threshold.

[0026] For state switching instructions, an optimal load distribution strategy is selected based on the load balancing of the big data streaming processing system.

[0027] The state exchange of computing nodes is performed based on the optimal load distribution strategy.

[0028] Furthermore, the state switching of computing nodes based on the optimal load allocation strategy includes:

[0029] The streaming computing tasks of a big data streaming processing system are abstracted into a task graph, where the vertices of the task graph are data source operators and the edges are data streams.

[0030] Generate a barrier marker, which is used to distinguish between the current snapshot and the next snapshot;

[0031] The barrier marker flows from the data source of the task graph along the data stream, and when any data source operator receives the barrier marker, the corresponding data stream is blocked, and the barrier continues to be broadcast to downstream data source operators until all corresponding data source operators in the task graph receive the barrier marker, and the current snapshot is generated.

[0032] After the compute node obtains the state corresponding to the required message primary key from the current snapshot, it performs state exchange between the compute node and the current snapshot based on the previous snapshot, according to the incremental snapshot technology and the optimal load distribution strategy.

[0033] The compute node unblocks all channels and returns to allow the barrier marker to flow from the data source of the task graph along the data stream to perform a state exchange between the compute node and the next snapshot.

[0034] The state after the computing nodes are swapped is obtained only after specific requirements are met.

[0035] A node load balancing device for big data streaming processing systems, the device comprising:

[0036] The data collection module is used to collect the distribution of message primary keys on the computing nodes and the amount of data corresponding to each message primary key;

[0037] The data analysis module is used to analyze the distribution of streaming data based on the distribution of message primary keys and the amount of data corresponding to each message primary key; wherein the distribution of streaming data includes: uniform distribution of message primary keys and uniform distribution of data, non-uniform distribution of message primary keys but uniform distribution of data, and non-uniform distribution of data.

[0038] The load distribution module is used to select a load distribution strategy based on the distribution of the streaming data in order to distribute the unallocated load.

[0039] A computer device, characterized in that the computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the node load dynamic balancing method for big data streaming processing systems as described above.

[0040] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the node load dynamic balancing method for big data streaming processing systems described above.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) This invention supports three load distribution strategies. Compared with only supporting hash-based load distribution strategies, it can better balance the load of computing nodes under different streaming data distribution conditions.

[0043] (2) By sampling streaming data, this invention can analyze the distribution of streaming data and the optimal load allocation strategy under such conditions. Compared with selecting a fixed load allocation strategy, it can better adapt to the unknown or dynamically changing distribution of streaming data.

[0044] (3) This invention provides a method for dynamically adjusting the load distribution strategy at runtime through dynamic configuration changes and inter-node state exchange based on asynchronous barrier algorithm, so that the streaming big data processing system can balance the node load in a timely and efficient manner when the load is unbalanced. Attached Figure Description

[0045] Figure 1 This is a framework diagram of the node load dynamic balancing device for big data streaming systems according to the present invention.

[0046] Figure 2 This is a flowchart of the dynamic load balancing based on sampled data according to the present invention.

[0047] Figure 3 This is a data flow diagram of low voltage detection in a specific embodiment of the present invention.

[0048] Figure 4 These are the load balancing test results for various load distribution strategies under different data distribution conditions.

[0049] Figure 5 This is a flowchart of the process for generating a state snapshot based on the asynchronous barrier snapshot algorithm of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings.

[0051] The dynamic load balancing method for big data streaming systems disclosed in this invention first samples and summarizes the data flowing through the streaming big data processing system, then analyzes the current distribution of the streaming data based on the collected information, then selects an appropriate load distribution strategy based on the distribution of the streaming data, and finally dynamically adjusts the distribution strategy during runtime to better balance the node load and improve the overall throughput of the streaming processing system.

[0052] like Figure 1 and Figure 2 As shown, this invention mainly includes the following five functional modules: load distribution strategy extension (Strategy), streaming data sampler (Profiler), data distribution and optimal strategy analyzer (Analyzer), load distribution strategy modifier (Modifier), and configuration manager (Manager).

[0053] 1. Load distribution strategy extension

[0054] In the method of this invention, the load distribution strategy extension, while maintaining compatibility with hash-based load distribution strategies, extends the following three load distribution strategies:

[0055] (1) Modulo: Calculate the node number by taking the modulo of the message primary key according to the degree of parallelism, and distribute the load to the node.

[0056] (2) LeastKey: Distributes unallocated load to the node with the lowest current load. When allocating compute nodes corresponding to primary keys, historical allocation records are first retrieved. If a message primary key with unallocated load exists in the historical allocation record, the unallocated load is allocated to compute nodes with the same message primary key; if no message primary key with unallocated load exists in the historical allocation record, the number of message primary keys on each compute node is calculated, and the unallocated load is allocated to the compute node with the fewest message primary keys. Then, the load function and historical allocation records are updated. The node load function of LeastKey is the sum of the number of primary keys allocated to the nodes, and the load of the corresponding node is incremented by one during the update. This calculation is performed by the allocation strategy coordinator on the management node, and the results are distributed to the compute nodes.

[0057] (3) LeastCount allocates unallocated load to the node with the lowest current load. Its algorithm is the same as that described by the LeastKey strategy, requiring the retrieval of historical allocation records. The difference lies in that the node load function of LeastCount is the sum of the historical data volume corresponding to the primary key allocated to the node; during updates, the corresponding node load is added to the historical data volume corresponding to the primary key.

[0058] 2. Streaming data sampler

[0059] In this invention, the streaming data sampler consists of two parts: a data sampler and a data aggregator. The data sampler performs asynchronous information collection based on a configured sampling rate. A higher sampling rate means a more accurate reflection of the data stream, but it may also lead to higher sampling costs. Although asynchronous information collection does not block data processing, it still consumes CPU resources. When the streaming application is first started, system administrators may lack information about the data distribution and can only use the default strategy with a high sampling rate. After a period of observation, administrators can choose a better strategy and a lower sampling rate based on monitoring information.

[0060] Among them, the data sampler is not only responsible for sampling on the computing nodes to collect the distribution of data primary keys and the amount of data corresponding to each primary key, but also responsible for collecting the load distribution of streaming data under different load distribution strategies through sampling on the computing nodes, thereby visualizing the system load situation or implementing other technical solutions that require load distribution information.

[0061] The data aggregator is responsible for collecting information from the data samplers on each computing node, including the distribution of primary keys and the amount of data corresponding to each primary key, as well as the load distribution of streaming data under different load distribution strategies. This information is then aggregated and stored in the key-value store.

[0062] 3. Data distribution and optimal strategy analyzer

[0063] In the method of this invention, the data distribution and optimal strategy analyzer are divided into a data distribution analyzer and an optimal strategy analyzer. The data distribution analyzer is responsible for analyzing the current streaming data distribution based on the primary key distribution and the amount of data corresponding to each primary key. Data distribution refers to whether the primary keys are evenly distributed (hereinafter referred to as uniform primary key distribution) and whether the amount of data corresponding to each primary key is evenly distributed (hereinafter referred to as uniform data distribution), including:

[0064] (1) The primary key is evenly distributed and the data is evenly distributed;

[0065] (2) The primary key is not uniformly distributed, but the data is uniformly distributed;

[0066] (3) The primary key is uniformly distributed, but the data is not uniformly distributed;

[0067] (4) The primary key is not uniformly distributed and the data is not uniformly distributed.

[0068] The optimal strategy analyzer selects the optimal load distribution strategy based on the load distribution of streaming data under different load distribution strategies and the optimal strategy selection rules, minimizing the difference between the load of the highest and lowest nodes. This is achieved through testing under different data distribution conditions. The results are attached. Figure 3 As shown, the following three rules are proposed:

[0069] Rule 1: When the primary key and data are evenly distributed, choose Modulo.

[0070] Explanation 1: When the primary key and data are evenly distributed, taking the modulo of the message primary key according to the degree of parallelism can evenly distribute the data to the computing nodes.

[0071] Rule 2: When the primary key is not uniformly distributed but the data is uniformly distributed, select LeastKey.

[0072] Explanation 2: When the primary keys are not evenly distributed but the data is evenly distributed, since LeastKey distributes the unallocated load to the node with the lowest current load, the node load is calculated by the number of primary keys allocated to the node, which can evenly distribute the data to the computing nodes.

[0073] Rule 3: When the data is unevenly distributed, choose LeastCount.

[0074] Explanation 3: When data is unevenly distributed, other strategies do not consider the amount of data corresponding to different primary keys, making it difficult to achieve load balancing. LeastCount allocates the unallocated load to the node with the lowest current load. The node load is calculated by the amount of data allocated to the node, which can evenly distribute the data to the computing nodes.

[0075] 4. Load Distribution Strategy Modifier

[0076] In the method of this invention, the load distribution strategy modifier is divided into a strategy modification trigger and a node state exchanger. The strategy modification trigger triggers strategy modification according to the configured strategy optimization mode. The triggering modes include:

[0077] (1) Periodic: Triggered periodically according to the specified interval configured, suitable for scenarios with stable data source flow rate;

[0078] (2) Quantitative, triggered when the sampled data reaches the configured quantity, suitable for scenarios where the data source flow rate is unstable;

[0079] (3) Threshold: This is triggered when the load balancing of the current load distribution strategy is lower than the configured threshold. It is suitable for scenarios where you want to minimize strategy switching.

[0080] The node state exchanger is responsible for exchanging states between nodes after switching load balancing strategies. Due to the strategy change, the compute node corresponding to the primary key may change, requiring the state corresponding to the primary key to be transferred to the correct compute node. The node state exchanger includes a state snapshot coordinator located on the management node and a state snapshot generator located on the compute nodes. The node state exchanger uses a variant of the Chandy-Lamport algorithm—the asynchronous barrier snapshot algorithm—to generate state snapshots, thereby facilitating state exchange between nodes. Streaming computation tasks can be abstracted as a graph, where vertices are operators and edges are data flows, corresponding to the graph model of the Chandy-Lamport algorithm. The asynchronous barrier snapshot algorithm uses special barrier markers to divide the data flow temporally. The barrier serves as the boundary of the snapshot, distinguishing the data in the current snapshot from the data in the next snapshot. Barriers flow from the data source along the data flow. When an operator receives a barrier from its input data flow, it blocks that input data flow and continues broadcasting the barrier to downstream operators. A state snapshot is generated when all input data flows have received barriers. After the snapshot is generated, the blocking of the data flow is lifted, and computation continues. The node state exchange is based on incremental snapshot technology, which only generates state changes compared to the previous snapshot, thus improving snapshot generation efficiency.

[0081] 5. Configuration Manager

[0082] In the method of this invention, the configuration manager is divided into a dynamic configuration center and a local configuration manager. The dynamic configuration center is responsible for the configuration management and dynamic configuration publishing of the distributed system, while the local configuration manager is responsible for subscribing to the configuration publishing of the dynamic configuration center and maintaining the local configuration.

[0083] The following example uses LowVoltage Detection, an event-driven application in the Flink streaming big data processing system. The application uses a state machine to detect the voltage state. The input data is voltage data, and the structure includes the measurement point (mpoint), measurement time (mptime), and three-phase voltages (Va, Vb, Vc), where the measurement point is the primary key of the data.

[0084] Flink generates the logical processing flow and physical execution plan for low voltage detection. According to the processing flow, Flink first reads data from the data source, aggregates data with the same primary key using the KeyBy operator, and distributes the data to compute nodes for computation. The data processing operators on the compute nodes store a copy of the state information for each primary key. The tool of this invention then begins to work.

[0085] 1. Streaming data samplers collect and summarize relevant information.

[0086] (1) Under the configuration of sampling rate of 100, the distribution of primary keys and the amount of data corresponding to each primary key were collected: the primary keys are 1-10000, and each primary key corresponds to 100 data;

[0087] (2) Collect streaming data to determine the load distribution under different load allocation strategies:

[0088] Load distribution strategy Highest node load Minimum node load Load balancing Hash 26800 12700 0.474 Modulo 16700 16600 0.994 LeastKey 16700 16600 0.994 LeastCount 16700 16600 0.994

[0089] 2. Data Distribution and Optimal Strategy Analyzer analyzes the current streaming data distribution based on the summarized primary key distribution and the amount of data corresponding to each primary key. The distribution is as follows: (1) Primary keys are evenly distributed and data is evenly distributed. In this scenario, the optimal strategy analyzer selects Modulo as the optimal load distribution strategy according to rule 1.

[0090] 3. A load balancing strategy modifier configured in threshold mode (threshold value of 0.8) triggers a strategy switch when the load balancing score is found to be 0.474, which is lower than 0.8. Based on the results of the optimal strategy analyzer, the load balancing strategy is modified to Modulo, and inter-node state exchange is performed through the node state exchanger. Figure 4 and Figure 5 As shown, the specific stages and steps of state switching are as follows:

[0091] (1) The state snapshot coordinator issues a state exchange instruction;

[0092] (2) When the data source operator receives the state exchange instruction, it issues a special barrier marker to distinguish the interval between two snapshots;

[0093] (3) The barrier marker flows from input channel 1 to the KeyBy operator along with the data stream. After receiving the marker, the KeyBy operator blocks the input channel until the markers of all channels arrive. At the same time, the KeyBy operator broadcasts the barrier marker to the subsequent data analysis operators.

[0094] (4) After receiving the marker, the data analysis operator blocks the corresponding input channel until all channels have received their markers, then generates a state snapshot. Since the local state is marked with whether it has been modified since the last snapshot, only an incremental snapshot needs to be generated. Compute node 1 adds the state information with primary key 3 to the incremental snapshot. Under the Modulo load balancing strategy, the data with primary key 3 will subsequently be processed by compute node 3. After the snapshot is generated, the compute node retrieves the required state corresponding to the primary key from the snapshot and unblocks all channels.

[0095] Compute node 3 obtains the state information of primary key 3 through state snapshot.

[0096] After the low-voltage detection application switches the load distribution strategy to Modulo, subsequent load distribution is based on Modulo, and the load on the computing nodes gradually returns to balance. This means that the system throughput is no longer limited by the heavily loaded nodes, and the resource utilization of the lightly loaded nodes is also improved.

[0097] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0098] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0099] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for dynamic load balancing of nodes in a big data streaming system, characterized in that, The method includes: Collect the distribution of message primary keys on the computing nodes and the amount of data corresponding to each message primary key; Based on the distribution of the message primary keys and the amount of data corresponding to each message primary key, the distribution of streaming data is analyzed; wherein, the distribution of streaming data includes: message primary keys are evenly distributed and data is evenly distributed, message primary keys are not evenly distributed but data is evenly distributed, and data is not evenly distributed. A load allocation strategy is selected based on the distribution of the streaming data to allocate unallocated load. The load distribution strategies include: Modulo load distribution strategy, LeastKey load distribution strategy, and LeastCount load distribution strategy. The Modulo load distribution strategy allocates unallocated load based on the modulo result of the message primary keys on the compute nodes according to the degree of parallelism. The LeastKey load distribution strategy allocates unallocated load based on the number of primary keys on the compute nodes. The LeastCount load distribution strategy allocates unallocated load based on the amount of data on the compute nodes. The Modulo load distribution strategy is selected when the message primary keys and data are evenly distributed in the streaming data distribution. The LeastKey load distribution strategy is selected when the message primary keys are not evenly distributed but the data is evenly distributed. The LeastCount load distribution strategy is selected when the data is not evenly distributed in the streaming data distribution.

2. The method as described in claim 1, characterized in that, Based on the distribution of the streaming data, a load allocation strategy is selected to allocate unallocated load, including: When the distribution of streaming data is such that the message primary key is uniformly distributed and the data is uniformly distributed, the node number of the computing node is obtained by taking the modulo of the message primary key on the computing node according to the degree of parallelism. Assign the unassigned load to the compute node with that node number.

3. The method as described in claim 1, characterized in that, Based on the distribution of the streaming data, a load allocation strategy is selected to allocate unallocated load, including: In the case where the distribution of the streaming data is such that the message primary key is not uniformly distributed but the data is uniformly distributed, the message primary key of the unallocated load is obtained and the historical allocation record is retrieved. If the message primary key of the unallocated load exists in the historical allocation record, the unallocated load will be allocated to a computing node with the same message primary key. If the message primary key of the unallocated load does not exist in the historical allocation record, calculate the number of message primary keys on each computing node, and allocate the unallocated load to the computing node with the fewest message primary keys.

4. The method as described in claim 1, characterized in that, Based on the distribution of the streaming data, a load allocation strategy is selected to allocate unallocated load, including: When the distribution of the streaming data is uneven, obtain the primary key of the message with unallocated load and retrieve historical allocation records; If the message primary key of the unallocated load exists in the historical allocation record, the unallocated load will be allocated to a computing node with the same message primary key. If the message primary key of the unallocated load does not exist in the historical allocation record, the amount of data on each computing node is calculated, and the unallocated load is allocated to the computing node with the least amount of data.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When a triggering condition is met in the big data streaming system, a state exchange instruction is generated; wherein, the state exchange instruction is used to trigger state exchange between computing nodes, and the triggering condition includes: reaching a configured specified interval, the amount of data in the big data streaming system being greater than a first set threshold, or the load balancing degree of the big data streaming system being lower than a second set threshold. For state switching instructions, an optimal load distribution strategy is selected based on the load balancing of the big data streaming processing system. The state exchange of computing nodes is performed based on the optimal load distribution strategy.

6. The method as described in claim 5, characterized in that, Based on the optimal load distribution strategy, the state switching of computing nodes includes: The streaming computing tasks of a big data streaming processing system are abstracted into a task graph, where the vertices of the task graph are data source operators and the edges are data streams. Generate a barrier marker, which is used to distinguish between the current snapshot and the next snapshot; The barrier marker flows from the data source of the task graph along the data stream, and when any data source operator receives the barrier marker, the corresponding data stream is blocked, and the barrier continues to be broadcast to downstream data source operators until all corresponding data source operators in the task graph receive the barrier marker, and the current snapshot is generated. After the compute node obtains the state corresponding to the required message primary key from the current snapshot, it performs state exchange between the compute node and the current snapshot based on the previous snapshot, according to the incremental snapshot technology and the optimal load distribution strategy. The compute node unblocks all channels and returns to allow the barrier marker to flow from the data source of the task graph along the data stream to perform a state exchange between the compute node and the next snapshot. The state after the computing nodes are swapped is obtained only after specific requirements are met.

7. A node load dynamic balancing device for big data streaming processing systems, characterized in that, The device includes: The data collection module is used to collect the distribution of message primary keys on the computing nodes and the amount of data corresponding to each message primary key; The data analysis module is used to analyze the distribution of streaming data based on the distribution of message primary keys and the amount of data corresponding to each message primary key; wherein the distribution of streaming data includes: uniform distribution of message primary keys and uniform distribution of data, non-uniform distribution of message primary keys but uniform distribution of data, and non-uniform distribution of data. The load distribution module is used to select a load distribution strategy based on the distribution of the streaming data in order to distribute the unallocated load. The load distribution strategies include: Modulo load distribution strategy, LeastKey load distribution strategy, and LeastCount load distribution strategy. The Modulo load distribution strategy allocates unallocated load based on the modulo result of the message primary keys on the compute nodes according to the degree of parallelism. The LeastKey load distribution strategy allocates unallocated load based on the number of primary keys on the compute nodes. The LeastCount load distribution strategy allocates unallocated load based on the amount of data on the compute nodes. The Modulo load distribution strategy is selected when the message primary keys and data are evenly distributed in the streaming data distribution. The LeastKey load distribution strategy is selected when the message primary keys are not evenly distributed but the data is evenly distributed. The LeastCount load distribution strategy is selected when the data is not evenly distributed in the streaming data distribution.

8. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the node load dynamic balancing method for big data streaming systems as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the node load dynamic balancing method for big data streaming systems as described in any one of claims 1-6.

Images