System load balancing processing method and device and related equipment

Abstract

The invention provides a load balancing processing method and device and related equipment, and the method comprises the steps: monitoring physical computing nodes for data processing in a system, and determining the physical computing nodes with data hotspots; adding a new virtual node in the target idle physical computing node; generating an adjusted physical execution graph according to the new virtual node; and according to the physical execution graph, transferring a data processing task on the physical computing node with the data hot spot to the corresponding virtual node for processing. According to the load balancing method provided by the invention, the data processing task on the physical computing node where the hotspot data appears is transferred to the newly added virtual node on the idle physical computing node for processing, so that a streaming data processing scene taking an event as a unit, such as invoice data processing, can be dealt with in time; the problem of data skew in the Flink engine is effectively solved, the complexity of a program is low, and the implementation cost is low.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a system load balancing processing method, apparatus and related equipment. Background Technology

[0002] With the increasing activity of the social economy and the rapid development of big data technology, the requirements for the speed and stability of data processing computing engines, such as in the field of tax data processing technology, are also increasing. In accordance with the State Taxation Administration's big data platform construction specifications, all provinces are utilizing big data and artificial intelligence technologies to integrate and merge internal and external data resources of the tax bureau, forming a so-called provincial-level comprehensive data lake, to provide data processing, application, sharing, and innovation services to all levels of the tax system.

[0003] In building a tax big data platform, adopting a unified batch and stream processing approach to handle both streaming and batch business processes is of significant value for the fusion and processing of diverse data types and raw record data in the data lake. Invoice data is the core and largest component of tax big data. However, since invoice data primarily originates from invoicing activities by businesses or individuals, data surges occur periodically, leading to data congestion or loss if processing is not timely. To address this, using the Flink computing framework as a unified batch and stream processing engine has become the mainstream choice in the industry. Flink can meet most stream processing scenarios, providing rich high-order functions and corresponding APIs for batch processing, possessing the computational capability to process batch data based on a streaming computing engine, truly achieving batch and stream unification. However, when processing massive amounts of data, Flink is prone to situations where some physical computing nodes become overheated and suspended, leading to task failures or restarts. To mitigate this, the industry typically uses system sampling to guide the allocation of processing resources for a batch of data to be processed in subsequent processing times. However, this approach requires adding an intermediate layer to the system's operator layer, complicating the program. Furthermore, this batch-based load balancing method is not suitable.

[0004] [HZTC-HXZSFGS241001][HS2410498CCN] Flink streaming data processing, which uses events as the processing unit, struggles to effectively address the data skew problem within the Flink engine. Therefore, providing a load balancing method with low program complexity that is suitable for Flink streaming programs that use events as the processing unit has become a pressing technical problem in the industry. Summary of the Invention

[0005] In view of this, embodiments of this application provide a load balancing processing method, apparatus, and related equipment to at least partially solve the above-mentioned problems.

[0006] In a first aspect, embodiments of this application provide a method for verifying load balancing processing, including:

[0007] Monitor the physical computing nodes used for data processing in the system to identify physical computing nodes where data hotspots occur;

[0008] Add a new virtual node to the target idle physical computing node;

[0009] Based on the new virtual nodes, an adjusted physical execution graph is generated;

[0010] According to the physical execution graph, the data processing tasks on the physical computing nodes where data hotspots occur are transferred to the corresponding virtual nodes for processing.

[0011] Optionally, in one embodiment of this application, adding a new virtual node to the target idle physical computing node includes:

[0012] A consistent hashing partitioning algorithm is used to determine the mapping relationship between all physical computing nodes and virtual nodes in the system;

[0013] Based on the hash value of the access key of the physical computing node where the data hotspot appears, and based on the mapping relationship, physical computing nodes with a data processing task volume less than a preset threshold are selected as target idle physical computing nodes.

[0014] Add a new virtual node to the target idle physical computing node and establish connections between the new virtual node and other connected nodes upstream and downstream.

[0015] Optionally, in one embodiment of this application, the step of using a consistent hashing partitioning algorithm to determine the mapping relationship between all physical computing nodes and virtual nodes in the system includes:

[0016] Number all physical computing nodes and virtual nodes in the system, and determine the physical nodes numbered.

[0017] [HZTC-HXZSFGS241001][HS2410498CCN] Computation nodes S1, S2, S3…S M And correspondingly create M empty stacks, numbered B1, B2, B3…B, to accommodate virtual nodes. M ;

[0018] Based on the range of hash ring values, virtual nodes V1, V2, V3...V are of number N. N They are mapped sequentially onto the hash ring, where N is greater than M;

[0019] Based on the value distribution of the hash ring, the number of virtual nodes V1, V2, V3...V... is N. N With the empty stacks B1, B2, B3...B M Sequentially bind them to establish the physical computing nodes S1, S2, S3…S in the system. M and the virtual nodes V1, V2, V3...V N The mapping relationship between them.

[0020] Optionally, in one embodiment of this application, transferring the data processing tasks on the physical computing nodes where data hotspots occur to the corresponding virtual nodes for processing includes:

[0021] Hash calculations are performed on the data processing tasks on the physical computing nodes where data hotspots occur to determine the data processing tasks to be processed.

[0022] The data processing task to be processed is transferred to the corresponding virtual node for processing.

[0023] Optionally, in one embodiment of this application, transferring the data processing task on the physical computing node where the data hotspot occurs to the corresponding virtual node for processing includes:

[0024] At least half of the data processing tasks on the physical computing nodes where data hotspots occur will be transferred to the corresponding virtual nodes for processing.

[0025] Optionally, in one embodiment of this application, after transferring the data processing tasks on the physical computing node where the data hotspot occurs to the corresponding virtual node for processing, the method further includes: monitoring the amount of data to be processed on the physical computing node where the data processing tasks are transferred according to preset polling conditions; when the monitoring result exceeds a preset task processing threshold, re-determining the physical computing node as the physical computing node where the data hotspot occurs, so as to perform load balancing processing.

[0026] Optionally, in one embodiment of this application, the access key value of the physical computing node is determined based on the invoice information contained in the data processing task.

[0027] Secondly, embodiments of this application also provide a load balancing processing device, comprising:

[0028] The monitoring module is used to monitor the physical computing nodes in the system that are used for data processing and to identify the physical computing nodes where data hotspots occur.

[0029] The determination module is used to add new virtual nodes to the target idle physical computing nodes;

[0030] [HZTC-HXZSFGS241001][HS2410498CCN]

[0031] The adjustment module is used to generate an adjusted physical execution graph based on the new virtual nodes;

[0032] The processing module is used to transfer data processing tasks on physical computing nodes where data hotspots occur to the corresponding virtual nodes for processing, based on the physical execution graph.

[0033] Thirdly, embodiments of this application also provide a storage medium on which computer executable instructions are stored, and when the computer executable instructions are executed, any of the load balancing processing methods described in the first aspect of this application are performed.

[0034] Fourthly, embodiments of this application also provide an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0035] The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to any of the load balancing processing methods described in the first aspect of this application.

[0036] This application provides a load balancing method, apparatus, and related equipment. By monitoring the physical computing nodes used for data processing in the system, it identifies physical computing nodes with data hotspots; adds new virtual nodes to target idle physical computing nodes; generates an adjusted physical execution graph based on the new virtual nodes; and transfers data processing tasks from the physical computing nodes with data hotspots to the corresponding virtual nodes according to the physical execution graph. This load balancing method, by transferring data processing tasks from the physical computing nodes with hot data to newly added virtual nodes on relatively idle physical computing nodes, can promptly address event-based streaming data processing scenarios such as invoice data processing, effectively solving the data skew problem in the Flink engine. It also features low program complexity and low implementation cost. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0038] Figure 1 A schematic diagram illustrating the workflow of a load balancing processing method provided in an embodiment of this application;

[0039] [HZTC-HXZSFGS241001][HS2410498CCN]

[0040] Figure 2 This is a schematic diagram of a load balancing processing device provided in an embodiment of this application.

[0041] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0043] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0044] Example 1

[0045] This application provides a load balancing processing method, such as... Figure 1 As shown, Figure 1 A schematic diagram of the workflow of a load balancing processing method provided in this application embodiment includes:

[0046] Step S101: Monitor the physical computing nodes used for data processing in the system and identify the physical computing nodes where data hotspots occur. In this embodiment, the physical computing node can be a single data processing server in a data processing server cluster, used for centralized processing of data in a preset service area. In this embodiment, a data hotspot occurs when the utilization rate of the processing or data transmission resources of the physical computing node exceeds a preset threshold, resulting in a data skew problem in the system. When data skew occurs, some service nodes in the system process far more data than other service nodes, which can easily lead to excessive service pressure on the service node, ultimately resulting in operational failure and the failure of the overall system task. For example, in a possible practical application scenario of this embodiment, the servers in the Flink real-time computing framework system used for real-time processing of invoice stream data, i.e., physical computing nodes, often run in a cluster. In the Flink real-time computing framework system, a physical computing node refers to the actual worker node running a Flink Job, which is the machine that actually performs data processing or computation, such as a physical...

[0047] [HZTC-HXZSFGS241001][HS2410498CCN] Servers or cloud instances. Virtual nodes refer to logical partitions within a data stream, used for parallel processing. Each physical compute node has a set of virtual task nodes corresponding to the execution of a Flink Job. This is equivalent to launching multiple containers or virtual machines on a physical server to perform actual data or task processing. Theoretically, as long as there are enough virtual task nodes, the system can handle a large amount of data. However, when faced with massive amounts of pending data processing tasks, such as a Flink job package typically involving 200 virtual task nodes executing a Flink Job on a physical compute node, where 199 virtual nodes can complete data processing in a short time, but one virtual node takes significantly longer than the others, and as the data volume continues to increase, the worker physical compute node will be suspended due to data overheating. The most direct manifestation of data skew in Flink tasks is frequent backpressure on the virtual nodes actually processing the data tasks, with some nodes experiencing OutOfMemoryError (OOM) exceptions. This phenomenon mainly occurs because a large amount of data is concentrated on a single node, causing that node's memory to run out, leading to task failures and restarts in the system. Traditional methods of manually increasing task parallelism cannot solve this problem. This embodiment addresses this issue by real-time monitoring of each physical computing node (Worker node) in the system to promptly identify nodes with hot data, thereby adjusting the pending tasks on those nodes and improving the stability of system data processing. Specifically, in this embodiment, the monitoring can use metrics such as the amount of tasks currently being processed by each physical computing node, the amount of bandwidth consumed, and hardware resource utilization. Other metrics, such as the real-time temperature of hardware devices, can also be used; this embodiment does not limit the specific metrics used.

[0048] Specifically, in a practical application scenario of this application embodiment, the load balancing function in the Flink system application mainly includes three types of nodes: terminal nodes, master nodes, and worker nodes. Terminal nodes primarily support interactive operations between the system and the user. Master nodes are mainly responsible for parsing instructions that increase the parallelism of computational operators used for data processing. Worker nodes (i.e., TaskManagers) are primarily responsible for the actual operation of the physical computing nodes. The dynamic expansion of physical computing nodes occurs within the worker nodes. In an exemplary application scenario of this application embodiment, the terminal nodes display the amount of data received by each physical computing node in real time on the system's human-computer interaction interface, based on the monitoring data obtained from the system. Once the amount of data received by a physical computing node exceeds a preset threshold, it is considered that the physical computing node has become a data hotspot.

[0049] [HZTC-HXZSFGS241001][HS2410498CCN] In the event of a data skew or other event, the terminal node will promptly generate an alarm message and send it to the user. The user can then add parallelism to the physical computing nodes where data hotspots occur. After receiving the user's instruction to add parallelism, the terminal node will, for example, pass the instruction to the Master node via REST. Upon receiving the instruction, the Master node will modify the ExecutionGraph to add virtual nodes to the selected physical computing nodes, while maintaining the data transmission channels between the newly added virtual nodes and other upstream and downstream nodes. The affected upstream and downstream nodes will add edge objects for receiving and sending data, and the newly added Task virtual node object will be submitted to the Worker node as a new pending task, i.e., the target idle physical computing node. This simple and orderly method of adding virtual nodes is relatively easy to implement, has low complexity, does not consume too many system resources, and has strong timeliness for processing Flink streaming data with events as the processing unit.

[0050] Step S102: Add a new virtual node to the target idle physical computing node. In this embodiment, when a physical computing node with a data hotspot is detected in the system, a non-data hotspot physical computing node with low current task processing volume is selected from the other physical computing nodes in the system as the target idle physical computing node, and a new virtual node is added to it. This embodiment of the application dynamically expands virtual nodes in this way, ensuring the stability of program operation and avoiding the probability of problems such as unreasonable task parallelism settings and data processing task interruptions leading to system service pauses caused by direct manual intervention in traditional load balancing methods.

[0051] Step S103: Generate an adjusted physical execution graph based on the new virtual nodes. In the actual application scenario of this application embodiment, specifically, the Flink system maps the executing program into a data flow operator graph (StreamGraph). Before submitting it to the data service processing cluster, it optimizes the data processing operators involved in the StreamGraph to form a job graph (JobGraph) to reduce the latency and resource consumption of data exchange between operators. The JobManager (system job manager) combines the JobGraph with the set concurrency level to generate an execution graph (ExecutionGraph), and then schedules and deploys it to the TaskManager to form a physical execution graph that ultimately guides the data processing flow. In this application embodiment, after adding a new virtual node to the target idle physical computing node, the corresponding physical execution graph is optimized and adjusted to better ensure the system stability and orderly data processing during the execution of data processing tasks.

[0052] Step S104: According to the physical execution graph, the physical computing nodes where data hotspots occur...

[0053] [HZTC-HXZSFGS241001][HS2410498CCN] Data processing tasks are transferred to the corresponding virtual nodes for processing. This embodiment of the application uses this method to quickly and easily transfer pending tasks from physical computing nodes containing hot data to newly added virtual nodes for data processing, thereby improving the system's operational stability.

[0054] Optionally, in one embodiment of this application, adding a new virtual node to the target idle physical computing node includes: using a consistent hashing partitioning algorithm to determine the mapping relationship between all physical computing nodes and virtual nodes in the system; based on the hash value of the access key of the physical computing node with data hotspots, selecting physical computing nodes with data processing task volume less than a preset threshold as target idle physical computing nodes based on the mapping relationship; adding a new virtual node to the target idle physical computing node; and establishing connections between the upstream and downstream connected nodes of the new virtual node. In this embodiment of the application, the entire hash space is divided into a hash ring with its head and tail connected by a consistent hashing partitioning algorithm. Each physical computing node in the system is assigned a corresponding position on this ring, and each physical computing node is configured with a corresponding virtual node. When it is necessary to read or write certain data in the system, a hash operation is first performed to obtain a hash value. Then, based on this hash value, the corresponding physical computing node and the corresponding virtual node are found on the ring. The task to be processed is then assigned and processed to complete the data read / write task. This embodiment of the application uses a consistent hash partitioning algorithm to adjust the mapping between physical computing nodes and virtual nodes, which can make the data processing distribution of the system as balanced as possible. Furthermore, when the number of virtual nodes is much greater than that of physical computing nodes, the system balance is better achieved, reducing the possibility of data hotspots.

[0055] Optionally, in a preferred implementation of this application, the step of using a consistent hashing partitioning algorithm to determine the mapping relationship between all physical computing nodes and virtual nodes in the system includes:

[0056] Step 201: Number all physical computing nodes and virtual nodes in the system, and determine the numbered physical computing nodes S1, S2, S3…S… M And correspondingly create M empty stacks, numbered B1, B2, B3…B, to accommodate virtual nodes. M ;

[0057] Step 202: Based on the hash ring value range, assign N virtual nodes V1, V2, V3...V N They are mapped sequentially onto the hash ring, where N is greater than M;

[0058] Step 203: Based on the value distribution of the hash ring, assign the number of virtual nodes V1, V2, V3...V... to N. N With the empty stacks B1, B2, B3...B M Sequentially bind them to establish the physical computing nodes S1, S2, S3…S in the system. M and the virtual nodes V1, V2, V3...V N The mapping relationship between them.

[0059] This application embodiment, through the above algorithm, ensures that the number of virtual nodes in the stack corresponding to each physical computing node is an integer between [N / M] and [N / M]+1, meaning the difference between the number of virtual nodes corresponding to different physical computing nodes does not exceed 1. Even if the total number of virtual nodes in the system is less than the total number of physical computing nodes M, the above algorithm can still ensure that the difference in hash ring space between different physical computing nodes is controlled within 1%. This allows for better data processing and maintains a more balanced system state during data processing, significantly reducing the possibility of data skew.

[0060] This embodiment of the application illustrates the above algorithm: Assuming the number of physical computing nodes M is 4 and there are 30 virtual nodes N, the mapping relationship between physical computing nodes and virtual nodes is shown in the table below:

[0061] Stack number Physical computing nodes Virtual Node <![CDATA[B1]]> <![CDATA[S1]]> <![CDATA[V1,V5,V9,V 13 ,V 17 ,V 21 ,V 25 ,V 29 ]]> <![CDATA[B2]]> <![CDATA[S2]]> <![CDATA[V2,V6,V 10 ,V 14 ,V 18 ,V 22 ,V 26 ,V 30 ]]> <![CDATA[B3]]> <![CDATA[S3]]> <![CDATA[V3,V7,V 11 ,V 15 ,V 19 ,V 23 ,V 27 ]]> <![CDATA[B4]]> <![CDATA[S4]]> <![CDATA[V4,V8,V 12 ,V 16 ,V 20 ,V 24 ,V 28 ]]>

[0062] This embodiment of the application introduces the Consistent Haas algorithm into the Flink computing engine to virtually partition the physical computing in the system, mapping one physical computing node to multiple virtual nodes to support data processing tasks. This improves the load balancing of each node in the system after the data processing tasks are allocated. Even when the number of virtual nodes is much larger than the actual number of physical computing nodes, the system can still maintain near-perfect balance, improving the performance of data stream processing and the stability of the system. It also effectively solves the problem that the Flink engine cannot dynamically expand nodes when data skew occurs, while avoiding the risk of data inconsistency caused by manually increasing parallelism.

[0063] Optionally, in one implementation of this application embodiment, transferring the data processing tasks on the physical computing node where the data hotspot occurs to the corresponding virtual node for processing includes: performing hash calculations on the data processing tasks on the physical computing node where the data hotspot occurs to determine the data processing tasks to be processed, and transferring the data processing tasks to be processed to the corresponding virtual node.

[0064] Processing is performed on the [HZTC-HXZSFGS241001][HS2410498CCN] points. This application embodiment determines the virtual nodes used for corresponding data processing tasks through hash calculation, thereby providing a relatively evenly distributed data transfer method to achieve load balancing. It is easy to implement, has a unified standard, and is readily applicable.

[0065] Optionally, in one embodiment of this application, transferring the data processing tasks on the physical computing node where the data hotspot occurs to the corresponding virtual node includes: transferring at least half of the data processing tasks on the physical computing node where the data hotspot occurs to the corresponding virtual node. This allows the system load processed by the load balancing method described in this application to reach load balance as quickly as possible, rapidly reducing the degree of data skew.

[0066] Optionally, in one embodiment of this application, after transferring the data processing tasks on the physical computing node exhibiting the data hotspot to the corresponding virtual node for processing, the method further includes: monitoring the amount of data to be processed on the physical computing node from which the data processing tasks are transferred according to preset polling conditions; when the monitoring result exceeds a preset task processing threshold, re-identifying the physical computing node as the physical computing node exhibiting the data hotspot for load balancing. This embodiment of the application monitors the data to be processed on physical computing nodes by setting polling conditions, for example, monitoring the physical computing nodes in the system again after a certain time interval. This efficiently determines whether the hotspot phenomenon still exists or whether it causes data skew in the system from the physical computing nodes identified as exhibiting the data hotspot phenomenon at the previous processing time, saving system resources while performing better and faster load balancing.

[0067] Optionally, in one embodiment of this application, the access key value of the physical computing node is determined based on the invoice information included in the data processing task, wherein the invoice information includes, but is not limited to, the enterprise tax number or invoice number. This embodiment of the application, through this limitation method and combined with actual application scenarios, provides a unified hash calculation basis for subsequent hashes, facilitating the rapid allocation of corresponding physical computing nodes in the Flink system's data service cluster when processing data, and enabling better rapid summarization and organization of processed data for easier data storage.

[0068] This application provides a load balancing method that involves monitoring physical computing nodes used for data processing in a system to identify physical computing nodes experiencing data hotspots; adding new virtual nodes to target idle physical computing nodes; generating an adjusted physical execution graph based on the new virtual nodes; and then, according to the physical execution graph, balancing the physical computing nodes experiencing data hotspots.

[0069] Data processing tasks on [HZTC-HXZSFGS241001][HS2410498CCN] are transferred to the corresponding virtual nodes for processing. This load balancing method provided in this application, by transferring data processing tasks from physical computing nodes experiencing hot data to newly added virtual nodes on relatively idle physical computing nodes, can promptly address event-based streaming data processing scenarios, such as invoice data processing, effectively solving the data skew problem in the Flink engine. It also features low program complexity and low implementation cost.

[0070] Example 2

[0071] Based on the load balancing method provided in Embodiment 1 of this application, this embodiment also provides a load balancing device, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of a load balancing processing device 20 provided in an embodiment of this application. The load balancing processing device 20 includes:

[0072] Monitoring module 201 is used to monitor the physical computing nodes in the system that are used for data processing and to identify the physical computing nodes where data hotspots occur.

[0073] Module 202 is used to add new virtual nodes to the target idle physical computing nodes;

[0074] The adjustment module 203 is used to generate an adjusted physical execution graph based on the new virtual nodes;

[0075] The processing module 204 is used to transfer the data processing tasks on the physical computing nodes where data hotspots occur to the corresponding virtual nodes for processing, according to the physical execution graph.

[0076] Optionally, in one implementation of this application embodiment, the determining module 202 is further configured to:

[0077] A consistent hashing partitioning algorithm is used to determine the mapping relationship between all physical computing nodes and virtual nodes in the system;

[0078] Based on the hash value of the access key of the physical computing node where the data hotspot appears, and based on the mapping relationship, physical computing nodes with a data processing task volume less than a preset threshold are selected as target idle physical computing nodes.

[0079] Add a new virtual node to the target idle physical computing node and establish connections between the new virtual node and other connected nodes upstream and downstream.

[0080] Optionally, in one implementation of this application embodiment, the determining module 202 is further configured to:

[0081] Number all physical and virtual computing nodes in the system, and determine the numbered physical computing nodes S1, S2, S3…S M And correspondingly create M empty stacks, numbered B1, B2, B3…B, to accommodate virtual nodes. M ;

[0082] [HZTC-HXZSFGS241001][HS2410498CCN]

[0083] Based on the range of hash ring values, virtual nodes V1, V2, V3...V are of number N. N They are mapped sequentially onto the hash ring, where N is greater than M;

[0084] Based on the value distribution of the hash ring, the number of virtual nodes V1, V2, V3...V... is N. N With the empty stacks B1, B2, B3...B M Sequentially bind them to establish the physical computing nodes S1, S2, S3…S in the system. M and the virtual nodes V1, V2, V3...V N The mapping relationship between them.

[0085] Optionally, in one implementation of this application embodiment, the processing module 204 is further configured to:

[0086] Hash calculations are performed on the data processing tasks on the physical computing nodes where data hotspots occur to determine the data processing tasks to be processed.

[0087] The data processing task to be processed is transferred to the corresponding virtual node for processing.

[0088] Optionally, in one implementation of this application embodiment, the processing module 204 is further configured to:

[0089] At least half of the data processing tasks on the physical computing nodes where data hotspots occur will be transferred to the corresponding virtual nodes for processing.

[0090] Optionally, in one implementation of this application embodiment, the device further includes a polling module (not shown in the figures). The polling module is used to monitor the amount of data to be processed on the physical computing node where the data processing task is transferred to the corresponding virtual node after the data processing task on the physical computing node where the data processing task is transferred is transferred, according to preset polling conditions. When the monitoring result exceeds a preset task processing threshold, the physical computing node is re-determined as the physical computing node where the data hotspot occurs, so as to perform load balancing processing.

[0091] Optionally, in one implementation of this application embodiment, the access key value of the physical computing node is determined based on the invoice information contained in the data processing task.

[0092] This application provides a load balancing processing device. A monitoring module monitors the physical computing nodes used for data processing in the system to identify physical computing nodes experiencing data hotspots. A determination module adds new virtual nodes to the target idle physical computing nodes. An adjustment module generates an adjusted physical execution graph based on the new virtual nodes. A processing module, based on the physical execution graph, transfers data processing tasks from the physical computing nodes experiencing data hotspots to the corresponding virtual nodes. This load balancing device, by transferring data processing tasks from the physical computing nodes experiencing hotspots to newly added virtual nodes on relatively idle physical computing nodes, can respond promptly to data hotspots.

[0093] [HZTC-HXZSFGS241001][HS2410498CCN] For example, in scenarios involving event-based streaming data processing of invoice data, this effectively solves the data skew problem that occurs in the Flink engine, with low program complexity and low implementation cost.

[0094] Example 3

[0095] This application also provides a storage medium storing a computer program that, when executed by a processor, implements any of the load balancing processing methods described in Embodiment 1 of this application.

[0096] Example 4

[0097] This application also provides an electronic device, such as... Figure 3 As shown, Figure 3 This application provides a schematic diagram of the structure of an electronic device 30, which includes:

[0098] One or more processors 301;

[0099] Memory 303 is used to store one or more programs;

[0100] When the one or more programs are executed by the one or more processors 303, the one or more processors perform the operations corresponding to any of the vehicle-to-cloud service proxy verification methods described in Embodiment 1 of this application.

[0101] This application has now described specific embodiments of the subject matter. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0102] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit, whose logic functions are defined by...

[0103] [HZTC-HXZSFGS241001][HS2410498CCN] The device is programmed by the user. Designers can program a digital system layer onto a single PLD themselves, eliminating the need for chip manufacturers to design and fabricate dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0104] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0105] [HZTC-HXZSFGS241001][HS2410498CCN]

[0106] The system layers, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0107] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0108] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, system-level, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific transactions or implement specific abstract data types. This application can also be practiced in distributed computing environments where transactions are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system-level embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0112] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A load balancing processing method, characterized by, The method comprises the following steps: monitoring physical computing nodes in a system for data processing, and determining a physical computing node with a data hotspot; adding a new virtual node in a target idle physical computing node; generating an adjusted physical execution graph according to the new virtual node; transferring data processing tasks on the physical computing node with the data hotspot to the corresponding virtual node for processing according to the physical execution graph.

2. The load balancing processing method according to claim 1, wherein, The step of adding a new virtual node in a target idle physical computing node comprises the following steps: determining a mapping relationship between all physical computing nodes and virtual nodes in the system by using a consistent hash partitioning algorithm; selecting a physical computing node with a data processing task amount less than a preset threshold as a target idle physical computing node based on the mapping relationship according to a hash value of an access key of the physical computing node with the data hotspot; adding a new virtual node in the target idle physical computing node, and establishing a connection between the new virtual node and other connected nodes upstream and downstream of the new virtual node.

3. The load balancing processing method according to claim 2, wherein, The step of determining a mapping relationship between all physical computing nodes and virtual nodes in the system by using a consistent hash partitioning algorithm comprises the following steps: All physical and virtual nodes contained in the system are numbered, and the numbered physical computing nodes S1, S2, S3…S M , and the corresponding M empty stacks accommodating virtual nodes B1, B2, B3…B M are established. Based on the hash ring value range, the virtual nodes V1, V2, V3…V N are sequentially mapped to the hash ring described by [HZTC-HXZSFGS241001][HS2410498CCN], and N is greater than M. Based on the value distribution of the hash ring, the number N of virtual nodes V1, V2, V3…V N With the empty stack B1, B2, B3…B M In turn, to establish the mapping relationship between the physical computing nodes S1, S2, S3…S M And the virtual nodes V1, V2, V3…V N Between the system.

4. The load balancing processing method according to claim 1, wherein, The step of transferring data processing tasks on the physical computing node with the data hotspot to the corresponding virtual node for processing comprises the following steps: performing hash calculation on the data processing tasks on the physical computing node with the data hotspot to determine data processing tasks to be processed; transferring the data processing tasks to be processed to the corresponding virtual node for processing.

5. The load balancing processing method according to claim 1, wherein, The step of transferring data processing tasks on the physical computing node with the data hotspot to the corresponding virtual node for processing comprises the following steps: transferring at least half of the data processing tasks on the physical computing node with the data hotspot to the corresponding virtual node for processing.

6. The load balancing processing method according to claim 1, wherein, After transferring the data processing tasks on the physical computing node with the data hotspot to the corresponding virtual node for processing, the method further comprises the following steps: monitoring the amount of data to be processed on the physical computing node for data processing task transfer according to a preset polling condition, and re-determining the physical computing node as a physical computing node with a data hotspot when the monitoring result exceeds a preset task processing threshold, so as to perform load balancing processing.

7. The load balancing processing method of claim 1, wherein, The access key value of the physical computing node is determined based on invoice information contained in a data processing task.

8. A load balancing processing device, characterized by, The method comprises the following steps: monitoring physical computing nodes in a system for data processing, determining a physical computing node with a data hotspot; adding a new virtual node in a target idle physical computing node; generating an adjusted physical execution graph according to the new virtual node; transferring data processing tasks on the physical computing node with the data hotspot to the corresponding virtual node for processing according to the physical execution graph.

9. A computer storage medium, characterized in that The computer storage medium stores computer executable instructions, and the computer executable instructions perform the load balancing processing method in any one of claims 1-6 when executed.

10. An electronic device, comprising: Comprise: A processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete mutual communication through the communication bus; The memory is used for depositing at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the load balancing processing method in any one of claims 1-6.

Images