Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and can be applied to the technical field of databases. The method comprises the steps that in response to a loading task, the number of to-be-processed tasks of each physical machine is determined according to the number of to-be-processed sub-tasks of multiple execution nodes in each physical machine and the corresponding relation, the to-be-processed tasks comprise the to-be-processed sub-tasks, and the corresponding relation represents the relation between the physical machine and the execution nodes; based on an equalization algorithm, according to the number of the to-be-processed sub-tasks, the number of the to-be-processed tasks and the number of the target execution nodes, at least one target physical machine is determined from the multiple physical machines, and the target physical machine comprises at least one target execution node; and loading the target data according to the loading task based on the at least one target execution node.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed databases, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In the database cluster system, the physical machine needs to call the node to assign the execution node to the task. After the execution node has loaded all the loaded tasks, the number of unprocessed tasks is updated. The execution node stores the acquired data in the database shard.

[0003] However, when allocating load tasks to execution nodes, focusing on task balance between execution nodes will lead to load task skew between physical machines, resulting in an uneven distribution of the number of tasks on physical machines, which in turn leads to a decrease in database performance. Summary of the invention

[0004] In view of the above problems, the present invention provides a data processing method, apparatus, device and storage medium.

[0005] According to a first aspect of the present invention, a data processing method is provided, which is applied to a cluster system, wherein the cluster system includes multiple physical machines, each of which has multiple execution nodes deployed thereon, including:

[0006] In response to the loading task, the number of pending tasks of each physical machine is determined according to the number of pending subtasks and corresponding relationships of multiple execution nodes in each physical machine, wherein the pending tasks include pending subtasks, and the corresponding relationships characterize the relationship between the physical machine and the execution nodes; based on the balancing algorithm, at least one target physical machine is determined from the multiple physical machines according to the number of pending subtasks, the number of pending tasks and the number of target execution nodes, and the target physical machine includes at least one target execution node; based on the at least one target execution node, the target data is loaded according to the loading task.

[0007] Optionally, based on a balancing algorithm, according to the number of pending subtasks, the number of pending tasks, and the number of target execution nodes, a target physical machine is determined from multiple physical machines, and the target physical machine includes at least one target execution node, including:

[0008] According to the number of pending tasks on each physical machine, multiple physical machines are arranged in ascending order to obtain a first allocation sequence; according to the ascending arrangement order represented by the first allocation sequence, the allocation order among the multiple physical machines is determined; based on a balancing algorithm, according to the allocation order of the multiple physical machines, the number of execution nodes of each physical machine and the number of target execution nodes, at least one target physical machine is determined from the multiple physical machines; according to the number of pending subtasks corresponding to each of the multiple execution nodes on the target physical machine and the number of target execution nodes allocated to the target physical machine, at least one target execution node is determined from the multiple execution nodes on the target physical machine.

[0009] Optionally, based on a balancing algorithm, according to the allocation order of the multiple physical machines, the number of execution nodes of each physical machine, and the number of target execution nodes, determining at least one target physical machine from the multiple physical machines includes:

[0010] When the number of execution nodes of a physical machine is greater than or equal to the target number of execution nodes, a target physical machine is determined from multiple physical machines based on a balancing algorithm according to the allocation order of the multiple physical machines, wherein the number of execution nodes of each physical machine is the same; when the number of execution nodes of a physical machine is less than the target number of execution nodes, multiple target physical machines are determined from multiple physical machines based on a balancing algorithm according to the allocation order of the multiple physical machines.

[0011] Optionally, at least one target execution node is determined from the multiple execution nodes on the target physical machine according to the number of pending subtasks corresponding to each of the multiple execution nodes on the target physical machine and the number of target execution nodes allocated to the target physical machine, including: arranging the multiple execution nodes in ascending order according to the number of pending subtasks corresponding to each of the multiple execution nodes on the target physical machine to obtain a second allocation sequence; determining an allocation order among the multiple execution nodes according to the ascending arrangement order represented by the second allocation sequence; and determining at least one target execution node based on a balancing algorithm according to the allocation order of the multiple execution nodes and the number of target execution nodes allocated to the target physical machine.

[0012] Optionally, the target number of execution nodes is determined based on the following operations: calling a scheduling node to evaluate the loading task and determine the occupied resources of the target data; determining the target number of execution nodes based on the occupied resources of the target data and a preset loading threshold of the execution node.

[0013] Optionally, the corresponding relationship is stored in a key-value format on the recording node; wherein obtaining the corresponding relationship includes: calling the scheduling node to parse the storage file in the recording node to obtain the key-value content and the parsing verification value; comparing the parsing verification value and the cache verification value to obtain a comparison result; when the comparison results are inconsistent, determining the key-value content as the corresponding relationship.

[0014] Optionally, the data processing method also includes: calling the scheduling node, allocating the loading task to the target execution node, and updating the number of subtasks to be processed of the target execution node; loading the target data, calling the callback function, and updating the number of subtasks to be processed of the target execution node.

[0015] A second aspect of the present invention provides a data processing device, comprising:

[0016] A response module, configured to respond to the loading task and determine the number of pending tasks of each physical machine according to the number of pending subtasks and corresponding relationships of the plurality of execution nodes in each physical machine, wherein the pending tasks include pending subtasks, and the corresponding relationships represent the relationship between the physical machine and the execution nodes;

[0017] A determination module, configured to determine at least one target physical machine from a plurality of physical machines based on a balancing algorithm according to the number of pending subtasks, the number of pending tasks, and the number of target execution nodes, wherein the target physical machine includes at least one target execution node;

[0018] The loading module is used to load target data according to the loading task based on at least one target execution node.

[0019] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data processing method.

[0020] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned data processing method.

[0021] According to the data processing method, device, equipment and storage medium provided by the present invention, the number of pending tasks of each physical machine is determined according to the number of pending subtasks of each execution node and the corresponding relationship; thus, based on the balancing algorithm, the number of pending tasks of each physical machine is first considered, and the loading task is allocated to the physical machine; then, according to the number of pending subtasks of each execution node on the physical machine, the loading task is specifically allocated to the target execution node. Since the physical machine with a small number of pending tasks is preferentially selected, and then the execution node with a small number of pending subtasks is selected as the target execution node within the target physical machine, the task load balancing of both the physical machine and the execution node is comprehensively considered based on the corresponding relationship and the balancing algorithm, global balancing is achieved, and the overall performance of the cluster database is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0023] Figure 1 An application scenario of the data processing method according to an embodiment of the present invention is shown;

[0024] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown;

[0025] Figure 3 A flowchart of determining a target physical machine according to an embodiment of the present invention is shown;

[0026] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present invention is shown;

[0027] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0030] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] In the process of implementing the present invention, it is found that in the database cluster system, the physical machine needs to call the node to assign the execution node to the task, and after the execution node has loaded all the loading tasks, the number of unprocessed tasks is updated. The execution node stores the acquired data in the database shard. However, when assigning loading tasks to the execution node, it is important to consider that the task balance between the execution nodes will cause the loading tasks between the physical machines to be tilted, resulting in an uneven distribution of the number of tasks on the physical machines, which in turn leads to a decrease in database performance.

[0033] In view of this, an embodiment of the present invention provides a data processing method, device, equipment and storage medium. The method includes: in response to a loading task, according to the number of to-be-processed subtasks and corresponding relationships of multiple execution nodes in each physical machine, determining the number of to-be-processed tasks of each physical machine, wherein the to-be-processed tasks include to-be-processed subtasks, and the corresponding relationship represents the relationship between the physical machine and the execution node; based on a balancing algorithm, according to the number of to-be-processed subtasks, the number of to-be-processed tasks and the number of target execution nodes, determining at least one target physical machine from multiple physical machines, the target physical machine including at least one target execution node; based on at least one target execution node, loading target data according to the loading task.

[0034] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] Figure 1 The application scenario of the data processing method according to the embodiment of the present invention is shown.

[0036] like Figure 1 As shown, the business system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0037] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server execution system through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can generally be set in the server 105. The data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0041] Alternatively, the data processing method provided in the embodiment of the present invention may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the data processing apparatus provided in the embodiment of the present invention may also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0042] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0044] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown.

[0045] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0046] In operation S210 , in response to loading a task, the number of tasks to be processed of each physical machine is determined according to the number of subtasks to be processed and the corresponding relationship of each of the plurality of execution nodes in each physical machine.

[0047] Optionally, the data processing method is applied to a cluster system, the cluster system includes multiple physical machines, each of which has multiple execution nodes deployed. The loading task represents a task instruction to load target data from a target address, the loading task can be a structured query statement, and the target address can be a web page address, etc.

[0048] Optionally, the pending task includes pending subtasks. The number of pending subtasks represents the number of tasks waiting to be executed on the execution node. The number of pending tasks represents the total number of tasks waiting to be executed on multiple execution nodes on the physical machine.

[0049] Optionally, the corresponding relationship represents the relationship between the physical machine and the execution node. According to the corresponding relationship, multiple execution nodes on the physical machine are determined, and the number of pending subtasks on each execution node is counted, so as to determine the number of pending tasks corresponding to the physical machine.

[0050] Optionally, the corresponding relationship records the name of the physical machine and the address of the execution node included in the physical machine. For example, the corresponding relationship "phy_1": "instance_1_ip,instance_2_ip" indicates that the physical machine phy_1 includes the execution node instance_1 and the execution node instance_2.

[0051] Optionally, the number of pending tasks corresponding to the physical machine phy_1 is determined according to the number of pending subtasks of the execution node instance_1 and the number of pending subtasks of the execution node instance_2.

[0052] In operation S220 , based on a balancing algorithm, at least one target physical machine is determined from a plurality of physical machines according to the number of to-be-processed subtasks, the number of to-be-processed tasks, and the number of target execution nodes.

[0053] Optionally, the target physical machine includes at least one target execution node.

[0054] Optionally, based on a balancing algorithm, according to the number of subtasks to be processed, the number of tasks to be processed and the number of target execution nodes, at least one target physical machine is first determined from multiple physical machines, and then at least one target execution node is determined from the target physical machine.

[0055] Optionally, the physical machine further includes a scheduling node, which evaluates and divides the loading task in response to the received execution loading task, and the target execution node is used to execute the divided loading task. If the loading task is divided into n, n target execution nodes are required.

[0056] For example, the cluster system includes 4 physical machines, each physical machine includes 3 execution nodes, and the target number of execution nodes is 4. Based on the balancing algorithm, according to the number of pending subtasks of the execution nodes on each physical machine, the number of pending tasks of each physical machine and the target number of execution nodes, physical machine 1 and physical machine 2 can be determined as the target physical machines, among which the 3 execution nodes in physical machine 1 are all target execution nodes, and the 1 execution node in physical machine 2 is the target execution node.

[0057] In operation S230, target data is loaded according to the loading task based on at least one target execution node.

[0058] Optionally, when the loading task does not need to be divided, the target data is loaded according to the loading task based on one target execution node; when the loading task needs to be divided, the target data is loaded according to the loading task based on multiple target execution nodes.

[0059] Optionally, the scheduling node assigns a target execution node, the target execution node executes the loading task, and loading the target data according to the loading task may include: loading the data according to the target address in the loading task. After the target execution node loads the target data, it sends the target data to the data node for storage.

[0060] Optionally, the number of pending tasks of each physical machine is determined based on the number of pending subtasks of each execution node and the corresponding relationship; thus, based on the balancing algorithm, the number of pending tasks of each physical machine is first considered, and the loading tasks are allocated to the physical machines; then, based on the number of pending subtasks of each execution node on the physical machine, the loading tasks are specifically allocated to the target execution nodes, thereby comprehensively considering the task load balancing of both physical machines and execution nodes based on the corresponding relationship and the balancing algorithm, achieving global balance and improving the overall performance of the cluster database.

[0061] Figure 3 A flowchart for determining a target physical machine according to an embodiment of the present invention is shown.

[0062] like Figure 3 As shown, based on the balancing algorithm, according to the number of to-be-processed subtasks, the number of to-be-processed tasks and the number of target execution nodes, determining a target physical machine from multiple physical machines includes operations S310 to S340.

[0063] In operation S310 , multiple physical machines are arranged in ascending order according to the number of tasks to be processed on each physical machine to obtain a first allocation sequence.

[0064] In operation S320 , an allocation order among the plurality of physical machines is determined according to the ascending order represented by the first allocation sequence.

[0065] In operation S330 , based on a balancing algorithm, at least one target physical machine is determined from the multiple physical machines according to the allocation order of the multiple physical machines, the number of execution nodes of each physical machine, and the target number of execution nodes.

[0066] In operation S340, at least one target execution node is determined from the multiple execution nodes on the target physical machine according to the number of to-be-processed subtasks corresponding to the multiple execution nodes on the target physical machine and the number of target execution nodes allocated to the target physical machine.

[0067] Optionally, multiple physical machines are arranged in ascending order according to the number of pending tasks of each physical machine to obtain a first allocation sequence. For example, if the number of pending tasks of physical machine 1 is 4, the number of pending tasks of physical machine 2 is 3, and the number of pending tasks of physical machine 3 is 2, then the first allocation sequence is: physical machine 3, physical machine 2, physical machine 1.

[0068] For example, taking the first allocation sequence as: physical machine 3, physical machine 2, physical machine 1, the allocation sequence among the multiple physical machines is determined as: physical machine 3, physical machine 2, physical machine 1. The target physical machine is determined according to the allocation sequence among the multiple physical machines.

[0069] Optionally, the number of execution nodes of each physical machine may be the same, and the target number of execution nodes and the number of execution nodes of each physical machine are compared to determine the number of target physical machines. Based on the balancing algorithm, the target physical machine is determined from the multiple physical machines according to the allocation order of the multiple physical machines and the number of target physical machines.

[0070] Optionally, the number of target execution nodes assigned to the target physical machine represents the number of target execution nodes required on the target physical machine for the divided loading task. The scheduling node assigns the divided loading task to the target physical machine. For example, the loading task is divided into 4 subtasks. After determining the target physical machine 1 and the target physical machine 2, the scheduling node assigns 3 subtasks and 1 subtask to the target physical machine 1 and the target physical machine 2, respectively. The 3 target execution nodes of the target physical machine 1 are required to execute 3 subtasks respectively, and the 1 target execution node of the target physical machine 2 is required to execute 1 subtask. Then, the number of target execution nodes assigned to the target physical machine 1 and the target physical machine 2 are 3 and 1 respectively.

[0071] Optionally, for each target physical machine, when the number of execution nodes on the target physical machine is greater than the number of target execution node sub-tasks, at least one target execution node is determined from the multiple execution nodes on the target physical machine based on the number of to-be-processed sub-tasks corresponding to each of the multiple execution nodes on the target physical machine and the number of target execution node sub-tasks allocated to the target physical machine.

[0072] Optionally, for each target physical machine, when the number of execution nodes on the target physical machine is equal to the number of target execution node sub-nodes, all execution nodes on the target physical machine are directly determined as target execution nodes.

[0073] Optionally, first consider the balance of the physical machines according to the number of pending tasks on the physical machines, and evenly distribute the loading tasks to the physical machines. Then, based on the number of pending subtasks on each execution node on the physical machine, consider the balance of the execution nodes and specifically distribute the loading tasks to the target execution nodes, thereby achieving a global balance between the physical machines and the execution nodes and improving the overall performance of the cluster database.

[0074] Optionally, based on a balancing algorithm, according to the allocation order of the multiple physical machines, the number of execution nodes of each physical machine, and the number of target execution nodes, determining at least one target physical machine from the multiple physical machines includes:

[0075] When the number of execution nodes of a physical machine is greater than or equal to the target number of execution nodes, a target physical machine is determined from multiple physical machines based on a balancing algorithm according to the allocation order of the multiple physical machines, wherein the number of execution nodes of each physical machine is the same; when the number of execution nodes of a physical machine is less than the target number of execution nodes, multiple target physical machines are determined from multiple physical machines based on a balancing algorithm according to the allocation order of the multiple physical machines.

[0076] Optionally, the loading task is divided into multiple subtasks, and the scheduling node needs to determine a target execution node for executing each subtask and a target physical machine where the target execution node is located.

[0077] For example, taking the above allocation order as: physical machine 3, physical machine 2, physical machine 1 as an example, the number of execution nodes of each physical machine is 3, and the loading task is divided into 2 subtasks, then the target number of execution nodes is 2. The number of execution nodes of the physical machine is greater than the target number of execution nodes. Based on the balancing algorithm and the allocation order of multiple physical machines, physical machine 3 is determined as the target physical machine. When the scheduling node allocates each subtask, it allocates the subtask to physical machine 3.

[0078] For example, taking the above allocation order as: physical machine 3, physical machine 2, physical machine 1 as an example, the number of execution nodes of each physical machine is 3, and the loading task is divided into 4 subtasks, then the target number of execution nodes is 4. The number of execution nodes of the physical machine is less than the target number of execution nodes. Based on the balancing algorithm and the allocation order of multiple physical machines, physical machine 3 and physical machine 2 are determined as the target physical machines. When the scheduling node allocates each subtask, the first three subtasks are allocated to physical machine 3 in turn, and then the last subtask is allocated to physical machine 2.

[0079] Optionally, when there are physical machines with the same number of tasks to be processed, the allocation order of the physical machines is the same, and when allocating each subtask, the target physical machine can be randomly determined based on a balancing algorithm.

[0080] Optionally, the scheduling node determines the allocation order between multiple physical machines. When allocating the next subtask, it is necessary to determine whether the target physical machine assigned to the subtask can continue to accept the subtask. If it can, otherwise remove the target physical machine from the physical machine set, and repeat the above steps until all subtasks are allocated. The target physical machine can receive the subtask, indicating that there is at least one execution node on the target physical machine that has not yet been assigned a subtask.

[0081] Optionally, the target physical machine does not receive the subtask, indicating that all execution nodes on the target physical machine have been assigned to the subtask. For example, when assigning the 4th subtask, the 3 execution nodes on physical machine 3 have been assigned to the subtask, and the 4th subtask is assigned to physical machine 2. Both physical machine 3 and physical machine 2 are target physical machines, that is, when the number of execution nodes of the physical machine is less than the target number of execution nodes, multiple target physical machines are determined.

[0082] Optionally, determining at least one target execution node from the multiple execution nodes on the target physical machine according to the number of to-be-processed subtasks corresponding to each of the multiple execution nodes on the target physical machine and the number of target execution nodes allocated to the target physical machine includes:

[0083] According to the number of to-be-processed subtasks corresponding to each of the multiple execution nodes on the target physical machine, the multiple execution nodes are arranged in ascending order to obtain a second allocation sequence; according to the ascending arrangement order represented by the second allocation sequence, the allocation order among the multiple execution nodes is determined; based on a balancing algorithm, at least one target execution node is determined according to the allocation order of the multiple execution nodes and the number of target execution nodes allocated to the target physical machine.

[0084] Optionally, multiple execution nodes are arranged in ascending order according to the number of subtasks to be processed of each execution node to obtain a second allocation sequence. For example, if the number of subtasks to be processed of execution node 1 is 1, the number of subtasks to be processed of execution node 2 is 3, and the number of subtasks to be processed of physical machine 3 is 2, then the second allocation sequence is: execution node 1, execution node 3, execution node 2.

[0085] For example, taking the above second allocation sequence of: execution node 1, execution node 3, execution node 2 as an example, the allocation order between multiple execution nodes on the target physical machine is: execution node 1, execution node 3, execution node 2.

[0086] Optionally, when the number of target execution nodes allocated to the target physical machine is the same as the number of execution nodes on the target physical machine, each subtask is allocated to the execution node on the target physical machine in turn based on the allocation order of multiple execution nodes based on a balancing algorithm, and it is determined that all execution nodes on the target physical machine are target execution nodes.

[0087] For example, taking the above allocation order: execution node 1, execution node 3, execution node 2 as an example, the target physical machine is allocated to 3 subtasks, the first subtask is allocated to execution node 1, each execution node processes one subtask, the second subtask is allocated to execution node 3, and the third subtask is allocated to execution node 2. Execution node 1, execution node 3, and execution node 2 are all target execution nodes.

[0088] Optionally, when the number of target execution nodes allocated to the target physical machine is less than the number of execution nodes on the target physical machine, each subtask is allocated to some execution nodes on the target physical machine in turn based on a balancing algorithm according to the allocation order of multiple execution nodes, and some execution nodes on the target physical machine are determined as target execution nodes.

[0089] For example, taking the above allocation order of: execution node 1, execution node 3, execution node 2 as an example, the target physical machine is allocated 1 subtask, the subtask is allocated to execution node 1, and execution node 1 is the target execution node.

[0090] Optionally, give priority to physical machines with fewer tasks to be processed, and then select execution nodes with fewer subtasks to be processed within the target physical machine as target execution nodes. If multiple physical machines have the same number of tasks to be processed, they are randomly selected. If the number of subtasks to be processed is the same, they are also randomly selected, ensuring the number of tasks on the physical machines and execution nodes.

[0091] Optionally, the target number of execution nodes is determined based on the following operations:

[0092] The scheduling node is called to evaluate the loading task and determine the occupied resources of the target data; the number of target execution nodes is determined based on the occupied resources of the target data and the preset loading threshold of the execution node.

[0093] Optionally, the calling scheduling node evaluates the loading task in response to receiving the loading task to determine the occupied resources of the target data. The occupied resources of the target data represent the computing resources consumed by loading the target data.

[0094] Optionally, the preset loading threshold of the execution node represents the upper limit of resources consumed by the execution node in processing the loading task.

[0095] For example, if the occupied resources of the target data are 100 and the preset loading threshold of the execution node is 25, the loading task is divided into 4 subtasks, and the number of target execution nodes is determined to be 4.

[0096] Optionally, the corresponding relationship is stored in a key-value manner on the recording node; wherein obtaining the corresponding relationship includes:

[0097] The scheduling node is called to parse the storage file in the recording node to obtain the key value content and the parsing check value; the parsing check value and the cache check value are compared to obtain the comparison result; when the comparison result is inconsistent, the key value content is determined as a corresponding relationship.

[0098] Optionally, the physical machine also includes a recording node, and the corresponding relationship is stored in a key-value manner on the recording node. After the cluster system is installed, expanded or reduced, and the node is replaced, the corresponding relationship between the physical machine and the execution node is automatically updated by enumerating the folder structure of the storage file on the recording node.

[0099] Optionally, when obtaining the corresponding relationship, call the scheduling node to parse the storage file in the record node to obtain the key value content and parsing check value of the corresponding relationship. Use the key value consistency of the record node to store the record correspondence. When reading or writing from the record node, the corresponding key is specified as phy-inst, and the value uses the parsing check value format. For example, CRC32{"phy_1":"instance_1_ip,instance_2_ip","phy_2":"instance_3_ip,instance_4_ip"}, where CRC32 is the parsing check value, the execution node addresses on the physical machine phy_1 are instance_1_ip and instance_2_ip, and the execution node addresses on the physical machine phy_2 are instance_3_ip and instance_4_ip.

[0100] Optionally, the recording node caches the key value content locally, and when a write operation is performed on the key value content, the local cache is automatically updated. The key value content is obtained from the local cache, a cache check value is obtained according to the value in the key value content, and the parsed check value is compared with the cache check value to obtain a comparison result.

[0101] Optionally, when the comparison result is inconsistent, the corresponding relationship in the representation record node has been updated, and the parsed key value content is determined as the corresponding relationship; when the comparison result is consistent, the corresponding relationship in the representation record node has not been updated, and the corresponding relationship in the local cache can be used.

[0102] Optionally, the correspondence between the physical machines and the execution nodes is automatically updated, and the number of pending tasks of the physical machine can be determined according to the number of pending subtasks of the execution node, so that the balance between the physical machines can be fully considered when allocating subtasks of the loading task.

[0103] Optionally, the data processing method further includes:

[0104] Call the scheduling node, assign the loading task to the target execution node, and update the number of pending subtasks of the target execution node; load the target data, call the callback function, and update the number of pending subtasks of the target execution node.

[0105] Optionally, calling the scheduling node to distribute the subtasks of the loading task to the target execution node, and updating the number of pending subtasks of the target execution node may include: adding 1 to the number of pending subtasks of the target execution node.

[0106] Optionally, after the target execution node has loaded the target data corresponding to the subtask, the callback function is called to reduce the number of pending subtasks at the target execution node by 1. The callback function may be a dispatch function.

[0107] Optionally, the number of pending subtasks of the target execution node is updated in real time according to the time when the subtask is received and the time when the subtask is loaded. This determines how the next subtask is allocated based on the current number of tasks, ensuring a dynamic balance in the number of tasks between multiple physical machines and between multiple execution nodes, and solving the problem of task skew caused by updating the number of tasks after all subtasks are loaded.

[0108] Based on the above data processing method, the present invention also provides a data processing device. Figure 4 The device is described in detail.

[0109] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present invention is shown.

[0110] like Figure 4 As shown, the data processing device 400 of this embodiment includes a response module 410 , a determination module 420 and a loading module 430 .

[0111] The response module 410 is used to respond to the loading task and determine the number of pending tasks of each physical machine according to the number of pending subtasks and the corresponding relationship of each of the multiple execution nodes in each physical machine, wherein the pending task includes the pending subtasks, and the corresponding relationship represents the relationship between the physical machine and the execution node. In one embodiment, the response module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0112] The determination module 420 is used to determine at least one target physical machine from multiple physical machines based on the balancing algorithm according to the number of pending subtasks, the number of pending tasks, and the number of target execution nodes, wherein the target physical machine includes at least one target execution node. In one embodiment, the determination module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0113] The loading module 430 is used to load the target data according to the loading task based on at least one target execution node. In one embodiment, the loading module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0114] Optionally, the determination module 420 includes a first determination submodule, a second determination submodule, a third determination submodule and a fourth determination submodule.

[0115] The first determining submodule is used to arrange the multiple physical machines in ascending order according to the number of tasks to be processed on each physical machine to obtain a first allocation sequence.

[0116] The second determining submodule is used to determine the allocation order among the multiple physical machines according to the ascending arrangement order represented by the first allocation sequence.

[0117] The third determination submodule is used to determine at least one target physical machine from the multiple physical machines based on a balancing algorithm, according to the allocation order of the multiple physical machines, the number of execution nodes of each physical machine, and the number of target execution nodes.

[0118] The fourth determination submodule is used to determine at least one target execution node from multiple execution nodes on the target physical machine according to the number of to-be-processed subtasks corresponding to each of the multiple execution nodes on the target physical machine and the number of target execution nodes allocated to the target physical machine.

[0119] Optionally, the third determining submodule includes a first determining unit and a second determining unit.

[0120] The first determination unit is used to determine a target physical machine from multiple physical machines based on a balancing algorithm according to the allocation order of multiple physical machines when the number of execution nodes of the physical machine is greater than or equal to the target number of execution nodes, wherein the number of execution nodes of each physical machine is the same.

[0121] The second determining unit is used to determine multiple target physical machines from the multiple physical machines based on a balancing algorithm according to the allocation order of the multiple physical machines when the number of execution nodes of the physical machine is less than the target number of execution nodes.

[0122] Optionally, the fourth determining submodule includes a third determining unit, a fourth determining unit and a fifth determining unit.

[0123] The third determining unit is used to arrange the multiple execution nodes in ascending order according to the number of to-be-processed subtasks corresponding to each of the multiple execution nodes on the target physical machine to obtain a second allocation sequence.

[0124] The fourth determining unit is used to determine the allocation order among the plurality of execution nodes according to the ascending arrangement order represented by the second allocation sequence.

[0125] The fifth determination unit is used to determine at least one target execution node based on a balancing algorithm, according to the allocation order of the multiple execution nodes and the number of target execution node sub-nodes allocated to the target physical machine.

[0126] Optionally, the response module 410 further includes a calling submodule, a comparing submodule and a generating submodule.

[0127] The calling submodule is used to call the scheduling node to parse the storage file in the recording node to obtain the key value content and the parsing check value.

[0128] The comparison submodule is used to compare the parsed check value and the cache check value to obtain a comparison result.

[0129] The generation submodule is used to determine the key-value contents as a corresponding relationship when the comparison results are inconsistent.

[0130] Optionally, the data processing device 400 further includes a first updating module and a second updating module.

[0131] The first updating module is used to call the scheduling node, allocate the loading task to the target execution node, and update the number of to-be-processed subtasks of the target execution node.

[0132] The second update module is used to load the target data, call the callback function, and update the number of pending subtasks of the target execution node.

[0133] Optionally, any multiple modules among the response module 410, the determination module 420 and the loading module 430 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the response module 410, the determination module 420 and the loading module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the response module 410, the determination module 420 and the loading module 430 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0134] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown.

[0135] Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0136] like Figure 5 As shown, the computer electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an on-board memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0137] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0138] Optionally, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0139] Optionally, the method flow according to the embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. Optionally, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0140] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the data processing method according to the embodiment of the present invention is implemented.

[0141] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0142] For example, optionally, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0143] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data processing method provided by the embodiment of the present invention.

[0144] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are performed. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0145] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0146] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0148] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a cluster system, the cluster system includes multiple physical machines, each of the physical machines is deployed with multiple execution nodes, and the method includes: In response to the loading task, the number of tasks to be processed of each physical machine is determined according to the number of subtasks to be processed and the corresponding relationship of each of the plurality of execution nodes in each physical machine, wherein the tasks to be processed include subtasks to be processed, and the corresponding relationship represents the relationship between the physical machine and the execution node; Based on a balancing algorithm, according to the number of the to-be-processed subtasks, the number of the to-be-processed tasks and the number of target execution nodes, at least one target physical machine is determined from the plurality of physical machines, wherein the target physical machine includes at least one target execution node; Based on the at least one target execution node, target data is loaded according to the loading task.

2. The method according to claim 1, characterized in that Based on a balancing algorithm, according to the number of the to-be-processed subtasks, the number of the to-be-processed tasks and the number of target execution nodes, a target physical machine is determined from the plurality of physical machines, the target physical machine including at least one target execution node, including: Arrange the plurality of physical machines in ascending order according to the number of tasks to be processed on each of the physical machines to obtain a first allocation sequence; Determine an allocation order among the multiple physical machines according to the ascending order represented by the first allocation sequence; Based on the balancing algorithm, according to the allocation order of the multiple physical machines, the number of execution nodes of each of the physical machines and the number of target execution nodes, determining at least one target physical machine from the multiple physical machines; At least one target execution node is determined from the multiple execution nodes on the target physical machine according to the number of to-be-processed subtasks corresponding to each of the multiple execution nodes on the target physical machine and the number of target execution nodes allocated to the target physical machine.

3. The method according to claim 2, characterized in that Based on the balancing algorithm, according to the allocation order of the multiple physical machines, the number of execution nodes of each of the physical machines and the target number of execution nodes, at least one target physical machine is determined from the multiple physical machines, including: In the case where the number of execution nodes of the physical machine is greater than or equal to the target number of execution nodes, determining one of the target physical machines from the multiple physical machines based on the balancing algorithm and the allocation order of the multiple physical machines, wherein the number of execution nodes of each of the physical machines is the same; In a case where the number of execution nodes of the physical machine is less than the target number of execution nodes, a plurality of target physical machines are determined from the plurality of physical machines based on the balancing algorithm according to the allocation order of the plurality of physical machines.

4. The method according to claim 2, characterized in that: According to the number of the to-be-processed subtasks corresponding to each of the plurality of the execution nodes on the target physical machine and the number of the target execution nodes allocated to the target physical machine, determining at least one target execution node from the plurality of the execution nodes on the target physical machine, comprising: Arrange the plurality of execution nodes in ascending order according to the number of the to-be-processed subtasks corresponding to each of the plurality of execution nodes on the target physical machine to obtain a second allocation sequence; Determining an allocation order among the plurality of execution nodes according to the ascending arrangement order represented by the second allocation sequence; Based on the balancing algorithm, at least one of the target execution nodes is determined according to the allocation order of the multiple execution nodes and the number of target execution nodes allocated to the target physical machine.

5. The method according to claim 1, characterized in that The target number of execution nodes is determined based on the following operations: Calling a scheduling node to evaluate the loading task and determine the occupied resources of the target data; The target number of execution nodes is determined according to the occupied resources of the target data and the preset loading threshold of the execution node.

6. The method according to claim 1, characterized in that The corresponding relationship is stored in the recording node in a key-value manner; Wherein, obtaining the corresponding relationship includes: Calling the scheduling node to parse the storage file in the recording node to obtain the key value content and the parsing check value; Compare the parsed check value and the cache check value to obtain a comparison result; When the comparison results are inconsistent, the key value content is determined as the corresponding relationship.

7. The method according to claim 1, characterized in that The method further comprises: Calling the scheduling node, allocating the loading task to the target execution node, and updating the number of subtasks to be processed of the target execution node; The target data is loaded, a callback function is called, and the number of subtasks to be processed of the target execution node is updated.

8. A data processing device, characterized in that: include: A response module, configured to determine the number of pending tasks of each physical machine in response to a loading task, according to the number of pending subtasks and corresponding relationships of the plurality of execution nodes in each physical machine, wherein the pending tasks include the pending subtasks, and the corresponding relationships represent the relationship between the physical machine and the execution nodes; A determination module, configured to determine at least one target physical machine from the plurality of physical machines based on a balancing algorithm according to the number of the to-be-processed subtasks, the number of the to-be-processed tasks and the number of target execution nodes, wherein the target physical machine includes at least one target execution node; A loading module is used to load target data according to the loading task based on the at least one target execution node.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images