Load balancing-based data processing system and method, device, and medium

Abstract

A load balancing-based data processing system and method, a device, and a medium. The method comprises: by means of an operation module, acquiring a task list from a central node, and forwarding the task list to parallel task modules; by means of the parallel task modules, receiving the task list, and on the basis of the task list, performing data transmission with a plurality of working nodes; by means of the working nodes, acquiring health load information, and transmitting the health load information to the central node; and by means of the central node, receiving the health load information, and on the basis of the health load information, performing load updating on a load list, wherein the load list is used for recording an overall load of the working nodes in the current state, and the task list is used for representing a set of working nodes capable of performing data transmission with the parallel task modules.

Description

Data processing system, method, device and medium based on load balancing

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202410361905.5, filed on March 27, 2024, entitled “A data processing system, method, device and medium based on load balancing”, the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of big data processing technology, and in particular to a data processing system, method, device and medium based on load balancing. Background Art

[0004] With the continuous development of science and technology, the scale of data generated by human activities is becoming increasingly large, and ultra-large-scale data processing has become a current research hotspot.

[0005] Currently, the traditional approach is to deploy a cluster using Spark on Yarn, while also deploying an external Shuffle Service to store intermediate data on disk to assist the Spark big data processing framework in file data processing. This traditional approach involves a large number of random disk reads and writes during the Shuffle process, placing significant pressure on the disk I / O load. This significantly increases the processing time of the Spark big data processing framework, resulting in a significant waste of computing resources and severely impacting the stability of data processing tasks.

[0006] In summary, the technical problems existing in the above-mentioned related technologies need to be improved. Summary of the Invention

[0007] The purpose of this application is to solve the above-mentioned technical problems existing in the related art.

[0008] The main purpose of the embodiments of the present application is to propose a data processing system, method, device and medium based on load balancing, wherein the data processing method can effectively reduce the load pressure on disk I / O during the Shuffle period, reduce the time and computing resources required for data processing, and improve the stability of data processing task operation.

[0009] To achieve the above objectives, the present application proposes a data processing system based on load balancing, which includes: an operation module, a parallel task module, a plurality of working nodes and a central node;

[0010] The running module is used to obtain the task list from the central node and forward the task list to the parallel task module;

[0011] The parallel task module is used to perform data transmission with a plurality of the working nodes according to the task list;

[0012] The working node is configured to obtain healthy load information and transmit the healthy load information to the central node;

[0013] The central node is configured to update the load list according to the healthy load information;

[0014] The load list is used to record the overall load of several working nodes in their current state, and the task list is used to represent the set of working nodes that can perform data transmission with the parallel task module.

[0015] In some embodiments, the parallel task module includes a write component and a read component;

[0016] The writing component is used to transmit and push data to a plurality of the working nodes according to the task list;

[0017] The reading component is used to obtain the merged data in the plurality of working nodes according to the task list.

[0018] In some embodiments, each of the working nodes is further configured to perform data merging processing on the pushed data to obtain the merged data.

[0019] In some embodiments, the parallel task module is further configured to send node request information to the execution module;

[0020] The operation module is further configured to forward the node request information to the central node;

[0021] The central node is further configured to perform node screening processing on the load list according to the node request information to obtain the task list.

[0022] In some embodiments, the operation module is further configured to obtain a central registration request and send the central registration request to the central node;

[0023] The central node is further configured to perform a first registration process on the central registration request, obtain a central registration result, and send the central registration result to the operation module;

[0024] or,

[0025] The operation module is further configured to obtain a work registration request and send the work registration request to the work node;

[0026] The working node is further configured to perform a second registration process on the work registration request, obtain a work registration result, and send the work registration result to the running module.

[0027] In some embodiments, the operation module is further configured to obtain a central deregistration request and send the central deregistration request to the central node;

[0028] The central node is further configured to receive and perform data deregistration processing on the load list according to the central deregistration request, obtain a data deregistration result, and send the data deregistration result to the operation module.

[0029] To achieve the above objectives, another aspect of the present invention provides a data processing method based on load balancing, including:

[0030] Obtaining a task list from a central node through a running module, and forwarding the task list to a parallel task module;

[0031] Receiving the task list through the parallel task module, and performing data transmission with a plurality of working nodes according to the task list;

[0032] Acquire healthy load information through the working node and transmit the healthy load information to the central node;

[0033] receiving the healthy load information through the central node, and updating the load list according to the healthy load information;

[0034] The load list is used to record the overall load of several working nodes in their current state, and the task list is used to represent the set of working nodes that can perform data transmission with the parallel task module.

[0035] In some embodiments, the parallel task module includes a writing component and a reading component, and receiving the task list through the parallel task module and performing data transmission with the plurality of working nodes according to the task list includes:

[0036] The write component receives and transmits push data to several working nodes according to the task list.

[0037] In some embodiments, the parallel task module includes a writing component and a reading component, and receiving the task list through the parallel task module and performing data transmission with the plurality of working nodes according to the task list includes:

[0038] The merged data in the plurality of working nodes is received and acquired according to the task list through the reading component.

[0039] In some embodiments, the task list is obtained by the following steps:

[0040] Sending node request information to the operation module through the parallel task module, and forwarding the node request information to the central node through the operation module;

[0041] The central node performs node screening processing on the load list according to the node request information to obtain the task list.

[0042] In some embodiments, the data processing method further includes:

[0043] The operation module obtains a central registration request and sends the central registration request to the central node; the central node performs a first registration process on the central registration request to obtain a central registration result, and sends the central registration result to the operation module.

[0044] In some embodiments, the data processing method further includes:

[0045] The operation module obtains a work registration request and sends the work registration request to the work node; the work node performs a second registration process on the work registration request to obtain a work registration result, and sends the work registration result to the operation module.

[0046] In some embodiments, the data processing method further includes:

[0047] Obtaining a central deregistration request through the operation module, and sending the central deregistration request to the central node;

[0048] The central deregistration request is received through the central node, and data deregistration processing is performed on the load list according to the central deregistration request to obtain a data deregistration result, and the data deregistration result is sent to the operation module.

[0049] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0050] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0051] The embodiments of the present application include at least the following beneficial effects:

[0052] The present application provides a data processing system, method, device and medium based on load balancing, wherein the data processing method obtains a task list from a central node through a running module and forwards the task list to a parallel task module; the parallel task module receives and transmits data with several working nodes according to the task list; obtains health load information through the working nodes and transmits the health load information to the central node; receives and updates the load list according to the health load information through the central node; wherein the load list is used to record the overall load of several working nodes in the current state, and the task list is used to characterize the set of working nodes that can transmit data with the parallel task module. The data processing method updates the load list through the central node and provides the task list for the parallel task module, which can realize the management and control of each node and the scheduling of tasks, thereby reducing the number of random reads and writes of the Spark architecture to the disk during shuffle, reducing the load pressure on the disk I / O, reducing the time and computing resources required for data processing, and effectively improving the stability of the data processing task operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be derived from these drawings without inventive effort.

[0054] FIG1 is a simplified schematic diagram of a data processing system based on load balancing according to an embodiment of the present application.

[0055] FIG2 is a simplified framework diagram of a parallel task module provided in one embodiment of the present application.

[0056] FIG3 is a flow chart of a data processing method based on load balancing provided in one embodiment of the present application.

[0057] FIG4 is a timing flow chart of step S210 according to an embodiment of the present application.

[0058] FIG5 is a timing flow chart of step S210 according to another embodiment of the present application.

[0059] FIG6 is a flow chart of a data processing method based on load balancing provided in another embodiment of the present application.

[0060] FIG7 is a flowchart of a task list provided in an embodiment of the present application.

[0061] FIG8 is a schematic diagram of an optional flow chart of a data processing method based on load balancing provided in an embodiment of the present application.

[0062] FIG9 is an optional flowchart of a data processing method based on load balancing provided in another embodiment of the present application.

[0063] FIG10 is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices / equipment and methods that are consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0065] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0066] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0068] The following is an introduction to several terms involved in this application:

[0069] Shuffle: refers to the process of redistributing and reorganizing large-scale data sets in a distributed computing environment.

[0070] Shuffle Service: A component of the Spark big data processing framework and an important part of the original Shuffle framework. It is deeply bound to the NodeManager and is responsible for interacting with the machine's local disk.

[0071] NodeManager: This is the NodeManager service in the Hadoop cluster and is one of the essential services of the Hadoop cluster. In the original Spark Shuffle architecture, the Shuffle Service is bundled with it.

[0072] MapTask / ReduceTask: Specifically refers to the two core task types used to implement mapping and aggregation operations in distributed computing in the Spark Shuffle architecture. Generally, ReduceTask reads the intermediate data generated by MapTask.

[0073] All-to-all: specifically refers to the full connection in the original Spark Shuffle architecture, that is, the ReduceTask will read the data generated by each MapTask.

[0074] Cold start: This refers specifically to the initial stage of a Spark job. The amount of resources requested is small, causing the Spark job to be unaware of the number of machines in its cluster. In other words, a single Spark job does not have a view of the entire cluster.

[0075] Currently, the traditional approach is to deploy a cluster using Spark on Yarn, while also deploying an external Shuffle Service to store intermediate data on disk to assist the Spark big data processing framework in file data processing. In the original Spark Shuffle architecture (also known as the All-to-All architecture), this traditional approach generates a large number of random, small file blocks during the shuffle process, requiring a large number of random disk read and write operations, which places a heavy strain on the disk I / O load. This significantly increases the time it takes for the Spark big data processing framework to process data, resulting in a significant waste of computing resources. This is particularly noticeable when processing very large amounts of data, making tasks extremely unstable and prone to failure, seriously impacting the stability of data processing tasks.

[0076] Furthermore, this traditional approach, when pushing MapTask-generated data to intermediate nodes, is subject to cold start conditions. Spark's Shuffle Service lacks a comprehensive view of the cluster, so it only obtains information about a subset of nodes. This results in extremely large amounts of data being pushed to a small number of intermediate nodes during the shuffle, increasing the disk I / O load on these nodes and risking insufficient storage capacity. Furthermore, the selection of these intermediate nodes is random, so if a machine with significant load is randomly selected, the shuffle efficiency will be low.

[0077] In view of this, the present application provides a data processing system, method, device and medium based on load balancing. Among them, the data processing method updates the load list through the central node, and provides a task list for the parallel task module, so that the central node can have a global perspective and can evenly select or blacklist the working nodes according to the load situation, thereby realizing the management and control of each node and the scheduling of tasks, reducing the number of random reads and writes of the disk by the ReduceTask during the shuffle period, reducing the load pressure on the disk I / O, reducing the time and computing resources required for data processing, and effectively improving the stability of the data processing task operation.

[0078] The load-balancing data processing method provided in the embodiments of the present application can be applied in database application scenarios. In database application scenarios, database service providers can use the load-balancing data processing method provided in the embodiments of the present application to process ultra-large-scale data, reduce the number of random disk reads and writes, reduce the load pressure on disk I / O, reduce the time and computing resources required for data processing, and effectively improve the stability of data processing tasks.

[0079] The data processing method based on load balancing provided in the embodiments of the present application can be applied to a terminal or a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this. The server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers. It can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the server can also be a node server in a blockchain network; the software can be an application that implements the method, etc., but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0081] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0082] FIG1 is a schematic diagram of a framework of a data processing system based on load balancing provided in an embodiment of the present application. As shown in FIG1 , the data processing system includes: an operation module, a parallel task module, several working nodes and a central node.

[0083] The running module is used to obtain the task list from the central node and forward the task list to the parallel task module.

[0084] The parallel task module is used to perform data transmission with a plurality of the working nodes according to the task list.

[0085] The working node is used to obtain healthy load information and transmit the healthy load information to the central node.

[0086] The central node is used to update the load list according to the healthy load information.

[0087] The load list is used to record the overall load of several working nodes in their current state; and the task list is used to represent the set of working nodes that can perform data transmission with the parallel task module.

[0088] In the embodiment of the present application, the data processing system is optimized based on the original Spark architecture. The specific optimization is to add a central node and a worker node, and adjust the execution tasks of the running module, parallel task module, worker node and central node, so that the data processing system can be deployed in a mixed manner with the Hadoop cluster, and the idle resources of the existing Hadoop cluster can be more fully utilized. These resources include: disk I / O, disk storage, CPU, etc., and Spark Shuffle can mobilize all machines in the cluster to work simultaneously, and can complete the optimization of job operation without introducing additional machine resources.

[0089] In an embodiment of the present application, the central node (Master) updates the load list based on the healthy load information, and can abstract the Shuffle Service into a working node (Worker). In this way, the completely independent Shuffle Services in the original Spark Shuffle architecture are associated through the central node (Master), so that the optimized data processing system (i.e., the optimized Spark Shuffle architecture) can perceive any Shuffle Service and can actively select a suitable Shuffle Service as the working node (Worker) of this Shuffle process.

[0090] For example, the running module can be implemented through the Spark Driver in the Spark Shuffle architecture, and is mainly used to obtain the task list from the central node. The acquisition method can be to send a list request to the central node through the running module. After the central node receives the corresponding list request, it returns the task list to the running module according to the list request, and the running module forwards the task list to the parallel task module. In addition, the parallel task module can be implemented through the Spark MapTask and Spark ReduceTask in the Spark Shuffle architecture. It is mainly used to determine the working nodes designated by the central node based on the task list, that is, based on the task list, determine all working nodes that can transmit data with the parallel task module, and then the parallel task module transmits data with these working nodes.

[0091] It is understandable that the working node in the embodiment of the present application can be implemented by adding a new Java service DCI Shuffle Worker (abbreviated as Worker). In order not to affect the normal operation of the original NodeManager service of the Hadoop cluster, the working node (Worker) is decoupled from the NodeManager while taking on the Shuffle Service task in the original architecture of Spark Shuffle. The working node (Worker) is normally connected to the NodeManager through the network heartbeat. In addition, the health load information in the embodiment of the present application is used to characterize the current overall load of the working node, which can be a performance indicator of the working node itself. The health load information includes core indicators and service indicators. Among them, the core indicators may include at least one of the disk I / O load, the available remaining capacity of the disk, whether the disk is bad, the CPU load, etc.; and the service indicators may include the number of interface queries per minute, the number of data received per minute, the amount of data received per minute, the number of transactions processed per minute, etc. The examples in this application are for illustration only and are not intended to limit this application.

[0092] It's important to note that after a worker node obtains health load information, it can transmit this information to the central node. This central node can also be implemented through the newly added Java service DCI Shuffle Master (Master). Furthermore, the newly added central node (Master) is primarily responsible for managing and controlling the worker nodes (Workers) and scheduling tasks, which determines which worker node (Worker) will execute the task.

[0093] It should be noted that in the data processing system in the embodiment of the present application, each worker node (Worker) can send health load information to the central node (Master) via network heartbeat at a fixed frequency, so that the central node (Master) has all core information (i.e., the core indicators and service indicators of all worker nodes (Worker)) globally. The task list received by the running module can effectively reduce the situation where large-scale data is pushed to a small number of intermediate nodes, reduce the disk I / O load of the worker node, effectively reduce the risk of insufficient storage capacity, and effectively improve the processing efficiency during the shuffle.

[0094] It should be noted that when the running module obtains the task list, the central node (Master) can filter the working nodes in the load list based on the node request information such as the amount of Shuffle data, the number of MapTasks, the number of ReduceTasks, and the number of concurrent jobs included in the list request provided by the running module, thereby screening out a set of working nodes that transmit data with the parallel task module. The number of working nodes in this set can be greater than or equal to 1, and the working nodes determined by this set can reduce the number of random reads and writes to the disk by the ReduceTask during the shuffle period, thereby reducing the load pressure on the disk I / O, reducing the time and computing resources required for data processing, and effectively improving the stability of the data processing task operation. For example, when the number of working nodes in the task list is equal to 1, the central node can preliminarily select a number of working nodes that meet the list request based on the list request provided by the running module, and then use at least one of the disk I / O remaining load, disk available remaining capacity, CPU remaining load, data processing time, etc. as the optimization index of load balancing, through optimization methods such as optimization algorithms, determine the final working node, and obtain the task list based on the final working node; or, when the number of working nodes in the task list is greater than or equal to 2, the central node can preliminarily screen out working nodes that meet the remaining indicators of the list request and are greater than the number of working nodes based on the number of working nodes applied for by the running module, and then use at least one of the disk I / O remaining load, disk available remaining capacity, CPU remaining load, data processing time, etc. as the optimization index of load balancing, through optimization methods such as optimization algorithms, determine working nodes equal to the number of working nodes, and integrate these working nodes to obtain the task list. The examples of this application are for illustration only and do not limit this application in any way.

[0095] It is also worth noting that for the load update of the central node load list, the central node can update all the original healthy load information in the load list based on all the current healthy load information received. Specifically, all the original healthy load information in the load list can be replaced with all the current healthy load information to obtain the updated load list, where each healthy load information corresponds to a working node.

[0096] In addition, in an embodiment of the present application, the load update may also include the central node blacklisting several working nodes in the load list after the load update according to the blacklisting strategy, thereby effectively reducing the situation where some working nodes are still selected for data processing when the pressure on some working nodes themselves is high, thereby avoiding busy working nodes and allowing idle working nodes to participate in the Shuffle process, which can further improve the efficiency during the Shuffle. Specifically, the blacklisting strategy in the embodiment of the present application may first be to obtain a preset disk threshold and a threshold number, and the disk threshold may be used to characterize at least one of the disk I / O load threshold, the disk used capacity threshold, the CPU load threshold, the interface query number threshold per minute, the number of data received per minute threshold, and the amount of data received per minute threshold, etc., and then, based on the disk threshold and the threshold number, the current healthy load information in the load list is screened, thereby screening out the healthy load information with a specific indicator value greater than or equal to the corresponding disk threshold and the total number of specific indicators greater than or equal to the threshold number, and then determining the corresponding working node and blacklisting it, thereby completing the load update of the load list. Alternatively, the blacklisting strategy in the embodiment of the present application can also be to obtain a preset disk threshold, threshold number and threshold time, and judge whether each working node has a threshold load greater than or equal to the threshold number within the threshold time, and the threshold load is greater than or equal to the corresponding disk threshold. If so, the corresponding working node will be blacklisted, otherwise no operation will be performed, thereby completing the load update of the load list.

[0097] For example, the working node in the embodiment of the present application can be implemented by deploying a DCI Worker service process on a physical machine, while the central node can be implemented by deploying a DCI Master service process on a physical machine. In some embodiments, the central node and the working node are deployed on different physical machines. In addition, the working node may include a heartbeat thread, a health check thread, and a message queue thread pool. Among them, the heartbeat thread can specifically use Netty RPC to communicate with the central node and send heartbeat information. The health check thread can be used to collect health load information of the working node, specifically including core indicators and service indicators. The message queue thread pool can be used to process requests from the running module and forwarded by the parallel task module, which include node registration, pushing data, and pulling data. The central node is used to maintain the data structure WorkerInfo that stores load list information in memory. After receiving heartbeat information from different working nodes through network heartbeat, it deserializes the heartbeat information and updates it to the data structure WorkerInfo, thereby completing the load update of the load list. It should be noted that although the WorkerInfo data structure is a large data structure stored in memory, which maintains the health load information of all worker nodes, the stored health load information can be very concise. The size of the health load information of a single worker node in the WorkerInfo data structure does not exceed 50KB. Therefore, the computing resources and time required to filter the task list based on the load list are very low.

[0098] 2 , in some embodiments, the parallel task module includes a writing component and a reading component.

[0099] The writing component is used to transmit and push data to a plurality of the working nodes according to the task list.

[0100] The reading component is used to obtain the merged data in the plurality of working nodes according to the task list.

[0101] In an embodiment of the present application, a parallel task module may include a write component and a read component, wherein the write component may send a task list request to the run module, which forwards the task list request to the central node, and ultimately causes the write component to receive the task list forwarded by the run module, which is used to determine the target working node to which the write component transmits and pushes data. Similarly to the aforementioned write component, the read component may also send a task list request to the run module, and then obtain a task list forwarded by the run module. For a certain data, the task list can be used to find the target working node to which the write component transmits the data.

[0102] It can be understood that after obtaining the specified target working node, the write component can transmit push data to the target working node, and the read component is used to obtain merged data from the target working node. The write component and the read component jointly realize the data transmission between the parallel task module and the working node.

[0103] In some embodiments, each of the working nodes is further configured to perform data merging processing on the pushed data to obtain the merged data.

[0104] It is understandable that due to the different amounts of data read by a Spark job, its job concurrency is often different. In actual applications, it is a relatively rare situation that the number of worker nodes required is equal to 1, and it is often more common to require a number of worker nodes greater than or equal to 2. Therefore, in the embodiment of the present application, the several worker nodes in the embodiment of the present application can refer to 1 worker node, or can refer to greater than or equal to 2 worker nodes, so that the data generated by the parallel task module can be pushed to several specified target worker nodes, and then in the target worker nodes, the scattered intermediate data is merged to splice it from a large number of random small files into a small number of continuous large files, so that the ReduceTask only needs to read a small number of large file blocks continuously, thereby further reducing a large number of random read and write disk operations, reducing the pressure of disk I / O load, and reducing the time to process data.

[0105] An embodiment of the present application further provides a data processing method based on load balancing. As shown in FIG3 , the data processing method performs data processing based on the aforementioned data processing system based on load balancing. The data processing method includes the following steps S210 to S240 .

[0106] Step S210: Obtain a task list from the central node through the running module, and forward the task list to the parallel task module.

[0107] In an embodiment of the present application, when transmitting and pushing data in the Shuffle stage, the running module can send a task list request to the central node, and the task list request includes information such as the amount of Shuffle data, the number of MapTasks, the number of ReduceTasks, and the number of concurrent jobs; then, the central node filters the working nodes in the load list according to the task list request, thereby filtering out a set of working nodes that can perform data transmission with the parallel task module; then, the central node can also notify the working node to logically reserve a certain amount of disk space for the Shuffle job corresponding to the task list request, and receive the reservation success information and / or reservation failure information returned by the working node; then, based on the reservation success information and / or reservation failure information, the set of working nodes that perform data transmission on the parallel task module is secondary screened, and the final task list is obtained based on the working nodes that have successfully reserved; finally, as shown in Figure 4, the central node returns the task list to the running module, and the running module schedules the parallel task module to start Shuffle writing data, and sends the working node address in the task list to the parallel task module.

[0108] It is understandable that when pulling data in the Shuffle phase, the parallel task module can obtain the work node address of the data in the data partition from the task list provided by the execution module based on the partition information it needs to read. It is also understandable that, as shown in Figure 5, since the task list has been transmitted to the execution module during the data transmission and pushing stage of the Shuffle phase, the execution module can also push the work node address recorded in the task list to the parallel task module when pulling data in the Shuffle phase.

[0109] Step S220: receiving the task list through the parallel task module, and performing data transmission with a plurality of working nodes according to the task list.

[0110] As shown in Figure 6, in some embodiments, the parallel task module includes a writing component and a reading component, and the step of receiving the task list through the parallel task module and transmitting data with several working nodes according to the task list may include the following steps S221 or S222.

[0111] Step S221: receiving the task list through the writing component, and transmitting push data to a plurality of the working nodes according to the task list; or

[0112] Step S222: Receive the task list through the reading component, and obtain the merged data in the plurality of working nodes according to the task list.

[0113] In this embodiment of the present application, the data push during the shuffle phase is implemented by the write component (Shuffle MapTask), while the data pull during the shuffle phase is implemented by the read component (Shuffle ReduceTask). The specific technical content of step S220 is similar to the aforementioned content regarding the data processing system and can be simply deduced by analogy. This application will not elaborate on this in detail.

[0114] Exemplarily, the merged data in the present application can be the logical merging of continuous partitioned data. For example, the partitioned data of logical partitions 1 to 5, the parallel task module can push these partitioned data to the working node A; the partitioned data of partitions 6 to 10, the parallel task module can push these partitioned data to the working node B, and then the working node merges the received partitioned data to obtain merged data.

[0115] Step S230: Obtain healthy load information through the working node and transmit the healthy load information to the central node.

[0116] Step S240: Receive the healthy load information through the central node, and update the load list according to the healthy load information.

[0117] The load list is used to record the overall load of several working nodes in their current state; and the task list is used to represent the set of working nodes that can perform data transmission with the parallel task module.

[0118] 7 , in some embodiments, the task list is obtained through the following steps S250 to S260 .

[0119] Step S250: Sending node request information to the operation module through the parallel task module, and forwarding the node request information to the central node through the operation module.

[0120] Step S260: The central node performs node screening processing on the load list according to the node request information to obtain the task list.

[0121] In the embodiment of the present application, the specific technical contents of steps S230 to S260 are similar to the aforementioned contents about the data processing system and can be simply deduced by analogy, so the present application will not elaborate on them here.

[0122] 8 , in some embodiments, the data processing method further includes the following steps S270 or S280 .

[0123] Step S270: obtaining a central registration request through the operation module and sending the central registration request to the central node; performing a first registration process on the central registration request through the central node to obtain a central registration result, and sending the central registration result to the operation module; or

[0124] Step S280: Obtain a work registration request through the operation module and send the work registration request to the work node; perform a second registration process on the work registration request through the work node to obtain a work registration result, and send the work registration result to the operation module.

[0125] It is understandable that before the Shuffle task starts, the running module needs to be registered with the central node and the worker nodes. Specifically, the central registration request is used to register the current Spark task ID (i.e., Application ID) with the central node; the worker registration request is used to register the current Application ID and the Shuffle task ID (i.e., Shuffle ID) with the worker node. When the central registration result is successful, it indicates that the central node has successfully confirmed the Application ID; and when the worker registration result is successful, it indicates that the worker node has created a storage directory for the Shuffle intermediate data for the Shuffle task and granted the corresponding permissions.

[0126] 9 , in some embodiments, the data processing method further includes the following steps S290 to S300 .

[0127] Step S290: Obtain a central deregistration request through the operation module, and send the central deregistration request to the central node.

[0128] Step S300: receiving the central deregistration request through the central node, performing data deregistration processing on the load list according to the central deregistration request, obtaining a data deregistration result, and sending the data deregistration result to the operation module.

[0129] In an embodiment of the present application, when a shuffle task ends, the execution module may generate a central deregistration request, which is used to deregister the first intermediate data generated by the current shuffle task. The first intermediate data may specifically include the ID of the current shuffle task and the health load information of each working node. The data deregistration process may be to delete the first intermediate data in the load list. After the deletion is completed, a data deregistration result indicating a successful deletion is obtained.

[0130] It is understood that the present application can also obtain a work deregistration request through the operation module and send the work deregistration request to the work node. The work node then receives and performs a second data deregistration process on the load list based on the work deregistration request to obtain a second data deregistration result, and sends the second data deregistration result to the operation module. The work deregistration request is used to deregister the second intermediate data stored by the work node. The rest of the content is similar to the data deregistration process of the central node described above and can be simply deduced by analogy. The present application will not elaborate on this in detail.

[0131] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any smart terminal such as a tablet computer or an in-vehicle computer.

[0132] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] Please refer to FIG. 10 . In this embodiment, FIG. 10 shows a hardware structure of an electronic device, including a processor 901 , a memory 902 , an input / output interface 903 , a communication interface 904 , and a bus 905 .

[0134] The processor 901 can be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0135] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the methods of the embodiments of this application.

[0136] The input / output interface 903 is used to implement information input and output.

[0137] The communication interface 904 is used to realize communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0138] The bus 905 is used to transmit information between various components of the device (eg, the processor 901 , the memory 902 , the input / output interface 903 , and the communication interface 904 ).

[0139] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0140] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above method is implemented.

[0141] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] The memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0143] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0144] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0146] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0147] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0148] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0149] In the several embodiments provided in this application, it should be understood that the provided devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0153] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0154] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing system based on load balancing, comprising: Running module, parallel task module, several working nodes and central node; The running module is used to obtain the task list from the central node and forward the task list to the parallel task module; The parallel task module is used to perform data transmission with a plurality of the working nodes according to the task list; The working node is configured to obtain healthy load information and transmit the healthy load information to the central node; The central node is configured to update the load list according to the healthy load information; The load list is used to record the overall load of several working nodes in their current state, and the task list is used to represent the set of working nodes that can perform data transmission with the parallel task module.

2. The data processing system based on load balancing according to claim 1, wherein the parallel task module includes a writing component and a reading component; The writing component is used to transmit and push data to a plurality of the working nodes according to the task list; The reading component is used to obtain the merged data in the plurality of working nodes according to the task list.

3. The load balancing-based data processing system according to claim 2, wherein each of the working nodes is further configured to perform data merging processing on the pushed data to obtain the merged data.

4. The data processing system based on load balancing according to claim 1, wherein the parallel task module is further configured to send node request information to the operation module; The operation module is further configured to forward the node request information to the central node; The central node is further configured to perform node screening processing on the load list according to the node request information to obtain the task list.

5. The data processing system based on load balancing according to claim 1, wherein the operation module is further configured to obtain a central registration request and send the central registration request to the central node; The central node is further configured to perform a first registration process on the central registration request, obtain a central registration result, and send the central registration result to the operation module; or The operation module is further configured to obtain a work registration request and send the work registration request to the work node; The working node is further configured to perform a second registration process on the work registration request, obtain a work registration result, and send the work registration result to the running module.

6. The data processing system based on load balancing according to claim 5, wherein the operation module is further configured to obtain a central deregistration request and send the central deregistration request to the central node; The central node is further configured to receive and perform data deregistration processing on the load list according to the central deregistration request, obtain a data deregistration result, and send the data deregistration result to the operation module.

7. A data processing method based on load balancing, comprising: Obtaining a task list from a central node through a running module, and forwarding the task list to a parallel task module; Receiving the task list through the parallel task module, and performing data transmission with a plurality of working nodes according to the task list; Acquire healthy load information through the working node and transmit the healthy load information to the central node; receiving the healthy load information through the central node, and updating the load list according to the healthy load information; The load list is used to record the overall load of several working nodes in their current state, and the task list is used to represent the set of working nodes that can perform data transmission with the parallel task module.

8. The data processing method based on load balancing according to claim 7, wherein the parallel task module includes a writing component and a reading component, and receiving the task list through the parallel task module and performing data transmission with the plurality of working nodes according to the task list comprises: The write component receives and transmits push data to several working nodes according to the task list.

9. The data processing method based on load balancing according to claim 7, wherein the parallel task module includes a writing component and a reading component, and receiving the task list through the parallel task module and performing data transmission with the plurality of working nodes according to the task list comprises: The merged data in the plurality of working nodes is received and acquired according to the task list through the reading component.

10. The data processing method based on load balancing according to claim 7, wherein the task list is obtained by the following steps: Sending node request information to the operation module through the parallel task module, and forwarding the node request information to the central node through the operation module; The central node performs node screening processing on the load list according to the node request information to obtain the task list.

11. The data processing method based on load balancing according to claim 7, further comprising: Obtaining a central registration request through the operation module, and sending the central registration request to the central node; The central node performs a first registration process on the central registration request to obtain a central registration result, and sends the central registration result to the operation module.

12. The data processing method based on load balancing according to claim 7, further comprising: Obtaining a work registration request through the operation module, and sending the work registration request to the work node; The work registration request is subjected to a second registration process by the work node to obtain a work registration result, and the work registration result is sent to the operation module.

13. The data processing method based on load balancing according to claim 11 or 12, further comprising: Obtaining a central deregistration request through the operation module, and sending the central deregistration request to the central node; The central deregistration request is received through the central node, and data deregistration processing is performed on the load list according to the central deregistration request to obtain a data deregistration result, and the data deregistration result is sent to the operation module.

14. An electronic device comprising: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 7 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 7 to 13 is implemented.

Images