Data processing method, device, medium and program product

Abstract

The application provides a data processing method, device, medium and program product, which can be applied to the technical field of databases. The data processing method comprises the following steps: determining the respective processor utilization rates of a plurality of working nodes within a predetermined historical period; in response to the presence of a first working node with a processor utilization rate greater than a preset threshold in the plurality of working nodes, screening a plurality of historical tasks executed by the first working node within the predetermined historical period according to the respective historical processor utilization rates of the plurality of historical tasks to obtain a target historical task; storing a target data set required for processing the target historical task to a first backup node, the first backup node being determined from a plurality of idle backup nodes of a cluster; and in response to receiving a new task, sending the new task to the first backup node to process the new task by using the first backup node in the case of determining that the new task is used to execute a target operation on the target data set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and more particularly to a data processing method, device, medium and program product. BACKGROUND

[0002] In a database cluster, in order to ensure service stability, backup nodes are usually set. When a working node fails, the backup node is usually used to replace the failed working node to process data.

[0003] However, when the number of failed working nodes in the cluster is small, the backup nodes will be in an idle state, resulting in resource waste. SUMMARY

[0004] In view of the above problems, the present application provides a data processing method, device, medium and program product.

[0005] According to a first aspect of the present application, a data processing method is provided, comprising: analyzing the working state of each of a plurality of working nodes in a cluster within a predetermined historical period, and determining the processor utilization of each of the plurality of working nodes within the predetermined historical period; in response to the existence of a first working node with processor utilization greater than a preset threshold in the plurality of working nodes, screening a plurality of historical tasks executed by the first working node within the predetermined historical period according to the historical processor utilization of each of the plurality of historical tasks, to obtain a target historical task; storing a target data set required for processing the target historical task to a first backup node, the first backup node being determined from a plurality of idle backup nodes of the cluster; in response to receiving a new task, in a case where it is determined that the new task is used to perform a target operation on the target data set, sending the new task to the first backup node to process the new task by using the first backup node.

[0006] The second aspect of the present application provides a data processing apparatus, comprising: an analysis module configured to analyze respective working states of a plurality of worker nodes in a cluster within a predetermined historical period, and determine respective processor utilization rates of the plurality of worker nodes within the predetermined historical period; a screening module configured to, in response to a first worker node with a processor utilization rate greater than a preset threshold existing in the plurality of worker nodes, screen a plurality of historical tasks executed by the first worker node within the predetermined historical period according to respective historical processor utilization rates of the plurality of historical tasks, to obtain a target historical task; a storage module configured to store a target data set required for processing the target historical task to a first backup node, the first backup node being determined from a plurality of idle backup nodes of the cluster; and a sending module configured to, in response to receiving a new task, determine that the new task is used to perform a target operation on the target data set, and send the new task to the first backup node, so as to process the new task by using the first backup node.

[0007] The third aspect of the present application provides an electronic device, comprising: one or more processors; and a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement steps of the method.

[0008] The fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement steps of the method.

[0009] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement steps of the method.

[0010] According to the embodiments of the present application, in the case that the processor utilization rate of the first worker node is high, the historical tasks executed by the first worker node within the predetermined historical period are screened, the target historical task causing the high processor utilization rate of the first worker node is determined, and the target data set required for executing the target historical task is stored to the first backup node, the new task capable of performing the target operation on the target data set is processed by using the first backup node, thereby reducing the resource consumption of the first worker node while fully utilizing the idle backup node resources, and improving the processing efficiency and resource utilization rate of the cluster as a whole. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above content of the present application and other purposes, features and advantages will be more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings.

[0012] Figure 1An application scenario diagram of the data processing method, device, medium and program product according to the embodiments of the present application is shown.

[0013] Figure 2 A flowchart of the data processing method according to the embodiments of the present application is shown.

[0014] Figure 3 A schematic diagram of storing a distributed data table to a first backup node according to the embodiments of the present application is shown.

[0015] Figure 4 A flowchart of processing a second data processing task according to the embodiments of the present application is shown.

[0016] Figure 5 A schematic diagram of processing an added task according to the embodiments of the present application is shown.

[0017] Figure 6 A schematic diagram of the second backup node loading data according to the embodiments of the present application is shown.

[0018] Figure 7 A structural block diagram of the data processing apparatus according to the embodiments of the present application is shown.

[0019] Figure 8 A block diagram of an electronic device suitable for implementing the data processing method according to the embodiments of the present application is shown. DETAILED DESCRIPTION

[0020] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it is to be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that one or more embodiments can be practiced without these specific details. In other instances, well-known structures and

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so on, mean the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings that are consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0023] In the case of using expressions such as "at least one of A, B, and C", it generally means all of the following: "A alone", "B alone", "C alone", "A and B together", "A and C together", "B and C together", and "A and B and C together".

[0024] A small number of backup nodes are configured in the database cluster. When a working node in the cluster is down, the backup node automatically mounts the external storage volume of the down working node, and the backup node is converted into a working node to provide normal data read and write services. If all working nodes in the cluster are running normally, the backup node will not mount the external storage volume of any other working node, that is, the working node is always in an idle state without any task execution. The high-resource configuration device is always in an idle state, which is a great waste of resources. Moreover, even if other working nodes are very busy, the backup node will not participate in any task of the cluster, and has no help for the efficiency of the cluster.

[0025] Embodiments of the present application provide a data processing method, comprising: analyzing the working state of each of a plurality of working nodes in a cluster within a predetermined historical period, and determining the processor utilization of each of the plurality of working nodes within the predetermined historical period; in response to the existence of a first working node with processor utilization greater than a preset threshold in the plurality of working nodes, filtering a plurality of historical tasks executed by the first working node within the predetermined historical period according to the historical processor utilization of each of the plurality of historical tasks to obtain a target historical task; storing a target data set required for processing the target historical task to a first backup node, the first backup node being determined from a plurality of idle backup nodes of the cluster; in response to receiving a new task, in a case where it is determined that the new task is used to perform a target operation on the target data set, sending the new task to the first backup node to process the new task using the first backup node, thereby sharing the load pressure of the first working node, improving the task processing efficiency of the cluster, improving the resource utilization of the cluster, and further improving the overall performance of the cluster.

[0026] Figure 1 An application scenario diagram of the data processing method, device, medium and program product according to embodiments of the present application is shown.

[0027] As Figure 1As shown, the application scenarios according to this embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a database cluster 105. The database cluster 105 can include multiple nodes, such as a first node 105_1, a second node 105_2, and a third node 105_3. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the database cluster 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, and the like.

[0028] A user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the database cluster 105 through the network 104 to receive or send messages, and the like. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like (only as examples).

[0029] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, and the like.

[0030] The database cluster 105 can provide data processing services, such as supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103, analyzing and processing received user requests and the like, and feeding back processing results (such as data obtained or generated according to user requests) to terminal devices.

[0031] It should be noted that the data processing method provided by the embodiments of the present application can generally be executed by the database cluster 105. Correspondingly, the data processing apparatus provided by the embodiments of the present application can generally be arranged in the database cluster 105. The data processing method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the database cluster 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the database cluster 105. Correspondingly, the data processing apparatus provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the database cluster 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the database cluster 105.

[0032] It should be understood that, Figure 1The numbers of the first terminal device, the second terminal device, the third terminal device, the network, the database cluster, the first node, the second node and the third node in the foregoing scenario are merely illustrative. According to implementation needs, there can be any number of the first terminal device, the second terminal device, the third terminal device, the network, the database cluster, the first node, the second node and the third node.

[0033] The following will be based on Figure 1 The scenario described above, by Figures 2-6 The data processing method of the application embodiment is described in detail.

[0034] Figure 2 A flowchart of the data processing method according to the application embodiment is shown.

[0035] As Figure 2 shown, the data processing method of this embodiment includes operation S210 to operation S240.

[0036] In operation S210, the working states of the plurality of working nodes in the cluster in a predetermined historical period are analyzed respectively to determine the processor utilization rates of the plurality of working nodes in the predetermined historical period respectively.

[0037] In operation S220, in response to the existence of a first working node with a processor utilization rate greater than a preset threshold in the plurality of working nodes, a plurality of historical tasks executed by the first working node in the predetermined historical period are screened according to the historical processor utilization rates of the plurality of historical tasks respectively to obtain a target historical task.

[0038] In operation S230, a target data set required for processing the target historical task is stored to a first backup node.

[0039] In operation S240, in response to receiving a new task, in a case where it is determined that the new task is used to perform a target operation on the target data set, the new task is sent to the first backup node to process the new task by using the first backup node.

[0040] In the embodiment of the application, the cluster can be a database cluster, for example, a Scale-Out cluster, and the database cluster can include a plurality of nodes of different types, for example, working nodes, backup nodes and scheduling nodes. The working nodes can be used to provide data read-write services, the backup nodes can provide read-write services instead of the failed working nodes, and the scheduling nodes can schedule tasks received by the database cluster to the working nodes or the backup nodes providing read-write services instead of the failed working nodes for execution. In one specific embodiment, the data processing method of the application can be executed by the scheduling nodes.

[0041] Since there are no or few faulty worker nodes in the cluster, there are backup nodes in an idle state in the cluster, which cannot share the load of the worker nodes when the processor utilization of the worker nodes is high, and thus the resources of the backup nodes are wasted. Therefore, the idle backup nodes can be used to process the tasks originally processed by the worker nodes to reduce the processor utilization of the worker nodes.

[0042] To better utilize the idle backup nodes to reduce the processor utilization of the worker nodes, a first worker node with high processor utilization can be determined from the plurality of worker nodes, and the tasks to be processed by the first worker node are analyzed, and the tasks consuming more processor resources are allocated to the idle backup nodes for processing to share the tasks of the first worker node by the idle first backup nodes.

[0043] In some embodiments, the first backup node can be determined according to the resource configuration information of each of the plurality of idle backup nodes. For example, an idle backup node with high resource configuration can be selected as the first backup node.

[0044] In determining the first worker node, the working states of the plurality of worker nodes in the cluster in a predetermined historical period can be analyzed, such as statistical analysis of the processor utilization of the worker nodes at a plurality of predetermined historical moments in the predetermined historical period to obtain the processor utilization of each worker node in the predetermined historical period, and the worker node with processor utilization greater than a preset threshold is determined as the first worker node.

[0045] For example, the scheduling node can send a first acquisition instruction for acquiring the working state data of the predetermined historical period to each worker node, and the worker node returns the working state data of the predetermined historical period to the scheduling node in response to the first acquisition instruction, and the scheduling node analyzes the working states of the plurality of worker nodes according to the working state data returned by the worker nodes.

[0046] After the first worker node is determined, the target historical task causing the high processor utilization of the first worker node can be determined according to the historical tasks executed by the first worker node in the predetermined historical period.

[0047] For example, the scheduling node can send a second acquisition instruction for acquiring the log data of the predetermined historical period to the first worker node, and the first worker node returns the log data of the predetermined historical period to the scheduling node in response to the second acquisition instruction. Since the log data includes the historical processor utilization of each of the plurality of historical tasks executed by the first worker node in the predetermined historical period, the scheduling node can filter out the target historical task with high historical processor utilization according to the received log data.

[0048] In the embodiments of the present application, since the data set from multiple data tables can be stored in the worker node, the high historical processor utilization of the target historical task can be caused by a large amount of data or high data complexity of the target data set required for processing the target historical task, and therefore the target data set can be stored to the first backup node to execute the task for the target data set by using the first backup node.

[0049] Since the task for the target data set includes query operation, calculation operation and change operation on the target data set, and the processor resources consumed by the query operation and the calculation operation are large, the query operation and the calculation operation can be determined as the target operation, and the task for executing the target operation on the target data set can be assigned to the first backup node to execute, so that the first worker node does not need to execute the target operation for the target data set.

[0050] After the target data set is completely stored to the first backup node, the first backup node has the condition to execute the task for the target data set. In the case that the scheduling node determines that the new task is used to execute the target operation on the target data set, the new task can be directly sent to the first backup node to process the new task by using the first backup node.

[0051] According to the embodiments of the present application, in the case that the processor utilization of the first worker node is high, the historical tasks executed by the first worker node in a predetermined historical period are screened to determine the target historical task causing the high processor utilization of the first worker node, and the target data set required for executing the target historical task is stored to the first backup node, so that the first backup node can execute the new task for the target operation on the target data set, thereby reducing the resource consumption of the first worker node and improving the overall processing efficiency and resource utilization of the cluster while fully utilizing the idle backup node resources.

[0052] According to the embodiments of the present application, in the process of storing the target data set required for processing the target historical task to the first backup node, the data processing method further comprises: in response to receiving the first data processing task, parsing the query statement included in the first data processing task to determine the first operation type of the first to-be-executed operation and the first data identifier of the first to-be-operated data set to which the first to-be-executed operation is directed; based on the first data identifier and the first operation type, determining whether the first data processing task is a first data change task for executing a change operation on the target data set.

[0053] Since the first data change task for performing a change operation on the target data set can be received in the process of storing the target data set to the first backup node, the scheduling node cannot directly send the first data processing task to the working node after receiving the first data processing task, so as to avoid that the first working node directly performs a change operation on the target data set when the first data processing task is used to perform a change operation on the target data set, and cause the data inconsistency between the first backup node and the first working node. The change operation can include adding data, deleting data and modifying data.

[0054] In the identification of the first data processing task, since the user usually represents the operation to be performed through a query statement, the query statement in the first data processing task can be identified to determine the first operation type of the first to-be-executed operation and the first data identifier of the first to-be-operated data set to which the first to-be-executed operation is directed. For example, the query statement can be a structured query statement, and the first data identifier can be a table identifier of a data table corresponding to the first to-be-operated data set.

[0055] After the first operation type and the first data identifier are determined, it can be determined whether the first operation type is a change operation, and whether the first data identifier is the same as the data identifier of the target data table. In the case that the first operation type is a change operation and the first data identifier is the same as the data identifier of the target data table, it is determined that the first data processing task is the first data change task for performing a change operation on the target data set.

[0056] According to the embodiments of the present application, by identifying the received first data processing task in the process of storing the target data set required for processing the target historical task to the first backup node, it can be avoided that the target data set is changed in the process of storing the target data set to the first backup node, and the data consistency between the first backup node and the first working node is ensured.

[0057] According to the embodiments of the present application, the data processing method further comprises: in the process of storing the target data set required for processing the target historical task to the first backup node, in response to receiving the first data change task for performing a change operation on the target data set, updating the first data change task, generating a to-be-executed task added with identification information representing that the change operation is temporarily not performed; sending the to-be-executed task to the first working node, so that the first working node updates the first target data to be updated in the to-be-executed task based on the changed data in the to-be-executed task after the target data set is completely stored to the first backup node.

[0058] In the process of storing the target data set to the first backup node, the cluster can receive a data change task for performing a change operation on the target data set. Since performing the data change task can cause the data of the target data set in the first worker node to change, in order to ensure data consistency, the data change task can be temporarily not executed in the process of storing the target data set to the first backup node, and the target data set is changed after the target data set is completely stored to the first backup node, and the target data set stored in the first backup node is updated.

[0059] After receiving the first data change task, the scheduling node can update the first data change task. Specifically, the scheduling node can add an identifier representing temporary non-execution in the first data change task to obtain a to-be-executed task, and send the to-be-executed task to the first worker node, so that the first worker node temporarily does not execute the update to the target data set after receiving the to-be-executed task.

[0060] In an embodiment of the present application, the first worker node can send the target data set stored in the memory to the first backup node through the cluster heartbeat network to store the target data set in the memory of the first backup node. And the first worker node determines that the target data set has been completely stored to the first backup node, and then executes the to-be-executed task to update the first target data to be updated.

[0061] According to an embodiment of the present application, by temporarily not executing the first data change task for changing the target data set in the process of storing the target data set to the first backup node, and updating the first target data in the target data set after the target data set is completely stored to the first backup node, the data consistency between the first backup node and the first worker node can be ensured.

[0062] According to an embodiment of the present application, the data processing method further comprises: in response to having completely stored the target data set required for processing the target historical task to the first backup node, generating a second data change task for the first backup node based on the first data change task; sending the second data change task to the first backup node, so that the first backup node updates the first target data stored in the first backup node to the changed data in response to the second data change task.

[0063] After the target data set required for processing the target historical task has been completely stored to the first backup node, the first worker node can send a storage completion message to the scheduling node, and the scheduling node can generate a second data change task for changing the target data set in the first backup node based on the first data change task in response to the storage completion message.

[0064] According to the embodiment of the present application, the target data set in the first backup node is updated by generating a second data change task after the target data set is completely stored to the first backup node, which not only ensures the data consistency between the first backup node and the first worker node, but also ensures the timeliness of the data stored in the first backup node.

[0065] According to the embodiment of the present application, the data processing method further comprises: determining a distributed data table to which the target data set belongs; and storing the distributed data table to the first backup node according to a data table identifier of the distributed data table.

[0066] In the embodiment of the present application, the data table can be stored in a distributed manner in a plurality of worker nodes of the cluster. For example, the distributed data table can include a plurality of partitions, each partition can correspond to a data set, and each data set is stored in a different worker node. When the cluster receives a data processing task for the distributed data table, the scheduling node needs to send the data processing task to a plurality of worker nodes storing the data sets of the distributed data table at the same time, so that the plurality of worker nodes respectively execute the data processing task according to the stored data sets.

[0067] Since the reason why the processor utilization of the first worker node is high can be that the complexity of the target data set is high, such as many fields, which leads to high complexity of the distributed data table to which the target data set belongs, and in turn can lead to high processor utilization of the worker node storing the data set of the distributed data table, the data of the distributed data table can be stored to the first backup node to execute the target operation for the distributed data table by using the first backup node.

[0068] When determining the distributed data table to which the target data set belongs, the distributed data table corresponding to the table identifier corresponding to the target data set can be determined according to the table identifier corresponding to the target data set. Moreover, the scheduling node can determine other worker nodes storing the data set of the distributed data table according to the table identifier of the distributed data table, and send a copy instruction to the other worker nodes, so that the other worker nodes receive the copy instruction and send the data set of the distributed data table stored by themselves to the first backup node through the cluster heartbeat network.

[0069] According to the embodiment of the present application, by storing the distributed data table to which the target data set belongs to the first backup node, the first backup node can process the target operation for the distributed data table, so that the first backup node can process the tasks of the associated worker nodes of the first worker node, thereby improving the overall data processing efficiency of the cluster.

[0070] Figure 3 A schematic diagram of storing a distributed data table to a first backup node according to an embodiment of the present application is shown.

[0071] As Figure 3 shown, after analyzing the working status of each of the working node 1, the working node 2 and the working node 3 in a predetermined historical period, the first working node and the target data set are determined. Among them, the working node 1 stores the table A data set 1 and the table B data set 1, the working node 2 stores the table A data set 2 and the table C data set 1, and the working node 3 stores the table B data set 2 and the table C data set 2.

[0072] As Figure 3 shown, taking the working node 3 as the first working node and the table B data set 2 as the target data set as an example, the table B to which the table B data set 2 belongs needs to be stored to the first backup node, so the working node 1 needs to store the table B data set 1 to the first backup node through the node cluster heartbeat network, and the working node 3 needs to store the table B data set 2 to the first backup node through the cluster heartbeat network.

[0073] According to the embodiment of the present application, the data processing method further comprises: in the case that the target data set required for processing the target historical task has been completely stored to the first backup node, in response to receiving a second data processing task, parsing a query statement included in the second data processing task, determining a second operation type of a second to-be-executed operation and a second data identifier of a second to-be-operated data set to which the second to-be-executed operation is directed; based on the second data identifier and the second operation type, determining a recognition result for the second data processing task; in the case that the recognition result represents that the second data processing task is used to perform a change operation on the target data set, sending the second data processing task to the first working node and the first backup node, so as to process the second data processing task by using the first working node and the first backup node respectively.

[0074] In the case that the target data set required for processing the target historical task has been completely stored to the first backup node, at this time the first backup node already has the ability to perform the target operation on the target data set, therefore when receiving the second data processing task, the scheduling node can parse the query statement in the second data processing task to identify the second data identifier and the second operation type of the second data processing task. The second data identifier may, for example, be the table identifier of the data table to which the second data processing task is directed.

[0075] In the case that the second operation type is a change operation and the second data identifier is the same as the data table identifier of the distributed data table to which the target data set belongs, it can be determined that the recognition result represents that the second data processing task is used to perform a change operation on the target data set.

[0076] In the case that the identification result represents that the second data processing task is used for performing a change operation on the target data set, in order to ensure the data consistency and timeliness between the first backup node and the first working node, the second data processing task can be sent to the first working node and the first backup node, so that the first working node and the first backup node perform the second data processing task respectively, and the target data set is updated.

[0077] After the first working node and the first backup node complete the second data processing task, the execution results can be respectively sent to the scheduling node. In the case that the scheduling node determines that both execution results represent that the change is successful, it is determined that the second data processing task is successfully executed, and an execution result representing successful execution is returned to the system initiating the second data processing task.

[0078] According to the embodiments of the present application, in the case that the target data set required for processing the target historical task has been completely stored to the first backup node, by sending the second data processing task for performing a target operation on the target data set to the first backup node and the first working node respectively, the data consistency and timeliness of the first backup node and the first working node are ensured, and the data processing efficiency of the cluster is improved.

[0079] Figure 4 A flowchart of processing a second data processing task according to an embodiment of the present application is shown.

[0080] In operation S410, a second data processing task sent by an application system is received.

[0081] In operation S420, it is determined whether the target data set is synchronized to the first backup node. In the case that the target data set is synchronized to the first backup node, operation S430 is performed, otherwise operation S440 is performed.

[0082] In operation S430, the second data processing task is sent to the first working node and the first backup node respectively.

[0083] In operation S440, the second data processing task is sent to the first working node.

[0084] In operation S450, a modification success is sent to the application system.

[0085] According to the embodiments of the present application, the new task is sent to the first backup node to process the new task by the first backup node, which includes: sending the new task and system information of an application system initiating the new task to the first backup node, so that the first backup node establishes a connection with the application system according to the system information, and sends the processing result of the new task to the application system.

[0086] When the application system needs to utilize the cluster to process data, a connection can be initiated to the master node of the cluster, and the system information of the application system, such as the address of the application system, the connection username and password and the like, is included in the initiated connection. After the connection is established, the application system sends a new task to be executed to the master node. The master node parses the query statement of the new task, and in the case that the new task involves a distributed data table to which a target data set belongs, the master node sends the new task and the system information to the first backup node.

[0087] After receiving the system information and the new task, the first backup node can establish a connection with the application system according to the system information, so as to directly send the processing result of the new task to the application system.

[0088] According to the embodiments of the present application, by synchronously sending the system information and the new task to the first backup node, the first backup node directly establishes a connection with the application system, and after obtaining the execution result, the execution result does not need to be sent to the master node first and then returned to the application system through the master node, thereby reducing the occupation of the data transmission bandwidth between nodes and improving the processing efficiency of the cluster.

[0089] Figure 5 A schematic diagram of processing a new task according to an embodiment of the present application is shown.

[0090] As shown in Figure 5 , taking the working node 1 as the master node as an example, the application system establishes a connection with the working node 1 by sending system information to the working node 1, and sends a new task after the connection is established. After receiving the new task, the working node 1 sends the new task and the system information to the first backup node in the case that the new task needs to be executed by the first backup node.

[0091] After receiving the new task and the system information, the first backup node can execute the new task to obtain an execution result, and establish a connection with the application system according to the system information, so as to directly return the execution result of the new task to the application system without forwarding to the application system through the working node 1, thereby improving the task processing efficiency.

[0092] According to the embodiments of the present application, the data processing method further includes: analyzing the respective working states of the first working node and the first backup node in a target period to obtain a first target processor utilization rate of the first working node in the target period and a second target processor utilization rate of the first backup node in the target period; in the case that the difference between the first target processor utilization rate and the second target processor utilization rate is greater than a preset difference threshold, screening a plurality of tasks executed by the first working node in the target period according to the respective target processor utilization rates of the plurality of tasks to obtain a target task; storing a data set required for processing the target task to the first backup node, so as to utilize the first backup node to process a target operation on the data set.

[0093] In the embodiments of the present application, the working states of the first backup node and the first worker node can be continuously monitored after the first backup node is utilized to share the task of performing the target operation on the target data set.

[0094] In the case where the difference between the first target processor utilization and the second target processor is greater than the preset difference threshold, it indicates that the processor utilization of the first worker node is still high and the first backup node still has abundant processor resources, and thus the task performed by the first worker node in the target period can be analyzed, the target task is further screened out, and the task of performing the target operation on the data set is transferred to the first backup node for execution.

[0095] According to the embodiments of the present application, after the first backup node is utilized to process the task of performing the target operation on the target data set, the working states of the first backup node and the first worker node are monitored, and in the case where the difference between the first target processor utilization and the second target processor is greater than the preset difference threshold, the first backup node is continuously utilized to share the task required to be executed by the first worker node, thereby improving the overall resource utilization and processing efficiency of the cluster.

[0096] According to the embodiments of the present application, the data processing method further comprises: determining a plurality of idle backup nodes from the plurality of backup nodes according to the node states of the plurality of backup nodes; and determining the first backup node from the plurality of idle backup nodes according to the resource configuration information of the plurality of idle backup nodes.

[0097] In the determination of the first backup node, the plurality of backup nodes can be screened according to the node states and the resource configuration information of the plurality of backup nodes in the cluster. Since the priority of sharing the task with high processor utilization is low, the first backup node needs to be selected from the idle backup nodes according to the node states.

[0098] In the determination of the first backup node from the plurality of idle backup nodes, the idle backup node with higher resource configuration can be selected as the first backup node to avoid causing the processor utilization of the first backup node to be high.

[0099] According to the embodiments of the present application, by determining the idle backup node with higher resource configuration as the first backup node according to the node states and the resource configuration information, the overall resource utilization of the cluster can be improved.

[0100] According to an embodiment of the present application, the predetermined historical time period includes a plurality of predetermined historical time points; and the analysis of the respective working states of the plurality of working nodes in the cluster within the predetermined historical time period includes: obtaining the processor utilization of each working node at each of the plurality of predetermined historical time points within the predetermined historical time period; and determining the processor utilization of each working node within the predetermined historical time period based on an average of the processor utilization at each of the plurality of predetermined historical time points, to obtain the respective processor utilizations of the plurality of working nodes.

[0101] In the analysis of the working state of the working node within the predetermined historical time period, the processor utilization of the working node at each of the plurality of predetermined historical time points within the predetermined historical time period can be analyzed.

[0102] In an embodiment of the present application, for each working node, the processor utilization of the working node within the predetermined historical time period can be determined based on an average of the processor utilization of the working node at each of the plurality of predetermined historical time points.

[0103] According to an embodiment of the present application, by analyzing the processor utilization of the working node at each of the plurality of predetermined historical time points, the processor utilization of the working node within the predetermined historical time period can accurately represent the working state of the working node within the predetermined historical time period, and thus the task that needs to be shared by the first backup node can be more accurately determined, and the processing efficiency of the cluster can be improved.

[0104] According to an embodiment of the present application, the plurality of historical tasks are filtered based on the respective historical processor utilizations of the plurality of historical tasks executed by the first working node within the predetermined historical time period to obtain a target historical task, including: filtering historical tasks for performing a target operation from the plurality of historical tasks to obtain at least one candidate historical task, the target operation including at least one of a calculation operation and a query operation; and determining the target historical task from the at least one candidate historical task based on the respective historical processor utilizations of the at least one candidate historical task.

[0105] In the filtering of the plurality of historical tasks, since the target operation usually causes the processor utilization of the working node to be high, the plurality of historical tasks can be filtered based on the target operation and the historical processor utilization to filter out the target historical task.

[0106] In the filtering of the historical tasks using the target operation, it can be determined whether the historical task is for performing the target operation, and in the case where the historical task is for performing the target operation, the historical task is determined to be a candidate task.

[0107] After the at least one candidate task is determined, the at least one candidate task can be screened according to the historical processor utilization of each candidate task, and a candidate historical task with higher historical processor utilization is determined as the target historical task.

[0108] According to the embodiments of the present application, the screening of the historical task is implemented by using the target operation and the historical processor utilization, which can accurately determine the target historical task that causes the processor utilization of the first worker node to be high, and thus can improve the processing efficiency of the cluster when the first backup node is used to share the task related to the target historical task.

[0109] According to the embodiments of the present application, the target historical task is determined from the at least one candidate historical task according to the historical processor utilization of each candidate historical task, which includes: sorting the at least one candidate historical task in descending order according to the historical processor utilization of each candidate historical task to obtain a candidate historical task sequence; and determining the candidate historical task at a predetermined position in the candidate historical task sequence as the target historical task.

[0110] When the at least one candidate historical task is screened, the at least one candidate historical task can be sorted in descending order according to the historical processor utilization to obtain a candidate historical task sequence, and then the candidate historical task with a high ranking in the candidate historical task sequence can be selected as the target historical task.

[0111] According to the embodiments of the present application, the target historical task is selected from the candidate historical task sequence by sorting the at least one candidate historical task in descending order, which can improve the accuracy of determining the target historical task, and thus improve the processing efficiency of the cluster.

[0112] According to the embodiments of the present application, the data processing method further includes: in the case that there is a second worker node that has failed in the plurality of worker nodes, and there is no idle backup node in the cluster, determining a second backup node from the plurality of backup nodes; in response to the second worker node and the second backup node having the same target storage data, deleting, from the second backup node, storage data other than the target storage data; mounting an external storage volume of the second worker node to the second backup node, so that the second backup node loads, from the external storage volume, the storage data other than the target storage data stored in the second worker node, and switches a data processing task executed by the second worker node to the second backup node for execution.

[0113] Since the backup node is usually used to replace the failed worker node to execute the data processing task, in order to ensure the stability of the cluster, the priority of the backup node replacing the failed worker node to execute the data processing task should be higher than the priority of processing the task with high processor utilization.

[0114] In the presence of a malfunctioning second worker node, a backup node is needed to replace the second worker node to perform data processing tasks. In determining the backup node, it can be selected from a plurality of idle backup nodes.

[0115] In the absence of idle backup nodes in the cluster, the second backup node processing high processor utilization tasks is needed to replace the second worker node. Since the second backup node has stored data when used to share tasks of the worker node with high processor utilization, the data stored in the second backup node needs to be cleaned up before the second backup node is used to replace the second worker node.

[0116] In cleaning up the stored data in the second backup node, since the second backup node can be used to share tasks of the second worker node, the same target stored data can be stored in the second backup node and the second worker node, so the target stored data in the second backup node can be retained and other stored data in the second backup node can be deleted.

[0117] After cleaning up the second backup node, other stored data of the second worker node can be stored to the second backup node. Since the second worker node has failed, data can no longer be normally obtained from the memory of the second worker node, at which time the external storage volume of the second worker node can be mounted to the second backup node, so that the second backup node can load other stored data of the second worker node from the external storage volume, and thus the second backup node stores all data of the second worker node.

[0118] After the second backup node completes data loading, it can send a loading success instruction to the scheduling node, at which time the scheduling node can modify the node information of the second backup node, no longer share tasks with high processor utilization by the second backup node, but switch data processing tasks performed by the second worker node to the second backup node.

[0119] According to the embodiments of the present application, by retaining the target stored data common to the second worker node when storing data in the second backup node, the amount of data to be loaded is reduced, the loading time is reduced, and thus the processing efficiency of the cluster is improved.

[0120] Figure 6 A schematic diagram of the second backup node loading data according to an embodiment of the present application is shown.

[0121] As Figure 6As shown, taking the second working node that is faulty as an example, since both the second backup node and the working node 1 store table B dataset 1, when cleaning up the second backup node, table B dataset 1 stored in the second backup node can be retained.

[0122] like Figure 6 As shown, after the external storage volume of worker node 1 is mounted on the second backup node, the data set 1 of table A, which was originally stored on worker node 1, can be stored on the second backup node. At this time, the second backup node stores all the storage data that was originally stored on worker node 1.

[0123] Based on the above data processing method, this application also provides a data processing apparatus. The following will be combined with... Figure 7 The device is described in detail.

[0124] Figure 7 A structural block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0125] like Figure 7 As shown, the data processing device 700 of this embodiment includes an analysis module 710, a filtering module 720, a storage module 730, and a sending module 740.

[0126] The analysis module 710 is used to analyze the working status of multiple worker nodes in the cluster within a predetermined historical period, and to determine the processor utilization of each worker node within the predetermined historical period. In one embodiment, the analysis module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0127] The filtering module 720 is used to respond to the presence of a first working node among multiple working nodes whose processor utilization exceeds a preset threshold. Based on the historical processor utilization of multiple historical tasks executed by the first working node within a predetermined historical time period, the module filters multiple historical tasks to obtain a target historical task. In one embodiment, the filtering module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0128] Storage module 730 is used to store the target dataset required for processing the target historical task to a first backup node, which is determined from multiple idle backup nodes in the cluster. In one embodiment, storage module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0129] The sending module 740 is configured to, in response to receiving the new task, send the new task to the first backup node, so as to process the new task by using the first backup node, in a case where it is determined that the new task is used for performing the target operation on the target data set. In an embodiment, the sending module 740 can be configured to perform operation S240 described above, and details are not described herein again.

[0130] According to an embodiment of the present application, the data processing apparatus 700 further comprises a first generating module and a first changing module.

[0131] The first generating module is configured to, in a process of storing the target data set required for processing the target historical task to the first backup node, in response to receiving a first data changing task used for performing a changing operation on the target data set, update information of the first data changing task, and generate a to-be-executed task added with identification information representing that the to-be-executed task is temporarily not executed.

[0132] The first changing module is configured to send the to-be-executed task to the first working node, so as to update the first target data to be updated in the to-be-executed task based on the changed data in the to-be-executed task by the first working node after the target data set is completely stored to the first backup node.

[0133] According to an embodiment of the present application, the data processing apparatus 700 further comprises a first analyzing module and a changing determining module.

[0134] The first analyzing module is configured to, in response to receiving the first data processing task, analyze a query statement included in the first data processing task, and determine a first operation type of the first to-be-executed operation and a first data identifier of a first to-be-operated data set to which the first to-be-executed operation is directed.

[0135] The changing determining module is configured to determine whether the first data processing task is a first data changing task used for performing a changing operation on the target data set, based on the first data identifier and the first operation type.

[0136] According to an embodiment of the present application, the data processing apparatus 700 further comprises a second generating module and a second changing module.

[0137] The second generating module is configured to, in response to the target data set required for processing the target historical task being completely stored to the first backup node, generate a second data changing task for the first backup node based on the data changing task.

[0138] The second changing module is configured to send the second data changing task to the first backup node, so as to update the first target data stored in the first backup node to changed data by the first backup node in response to the second data changing task.

[0139] According to an embodiment of the present application, the sending module 740 comprises a sending sub-module.

[0140] The sending sub-module is configured to send the new task and system information of an application system initiating the new task to the first backup node, so that the first backup node establishes a connection with the application system according to the system information, and sends a processing result of the new task to the application system.

[0141] According to an embodiment of the present application, the data processing apparatus 700 further comprises a second analysis module, a task identification module and a change sending module.

[0142] The second analysis module is configured to, in a case where the target data set required for processing the target historical task has been completely stored to the first backup node, in response to receiving a second data processing task, analyze a query statement included in the second data processing task, determine a second operation type of a second to-be-executed operation and a second data identifier of a second to-be-operated data set to which the second to-be-executed operation is directed.

[0143] The task identification module is configured to determine an identification result for the second data processing task based on the second data identifier and the second operation type.

[0144] The change sending module is configured to, in a case where the identification result represents that the second data processing task is used to perform a change operation on the target data set, send the second data processing task to the first working node and the first backup node, so as to process the second data processing task by using the first working node and the first backup node respectively.

[0145] According to an embodiment of the present application, the data processing apparatus 700 further comprises a table determination module and a table storage module.

[0146] The table determination module is configured to determine a distributed data table to which the target data set belongs.

[0147] The table storage module is configured to store the distributed data table to the first backup node according to a data table identifier of the distributed data table.

[0148] According to an embodiment of the present application, the data processing apparatus 700 further comprises a backup determination module, a backup deletion module and a backup loading module.

[0149] The backup determination module is configured to, in a case where there is a second working node that has occurred a fault in the plurality of working nodes, and there is no idle backup node in the cluster, determine a second backup node from the plurality of backup nodes.

[0150] The backup deletion module is configured to, in response to the second working node and the second backup node having the same target stored data, delete, from the second backup node, stored data other than the target stored data.

[0151] The backup loading module is configured to mount the external storage volume of the second worker node to the second backup node, so that the second backup node loads other stored data of the second worker node except the target stored data from the external storage volume, and switches the data processing task executed by the second worker node to the second backup node.

[0152] According to an embodiment of the present application, the data processing apparatus 700 further comprises an idle determining module and a resource determining module.

[0153] The idle determining module is configured to determine a plurality of idle backup nodes from the plurality of backup nodes according to respective node states of the plurality of backup nodes.

[0154] The resource determining module is configured to determine the first backup node from the plurality of idle backup nodes according to respective resource configuration information of the plurality of idle backup nodes.

[0155] According to an embodiment of the present application, the predetermined historical period comprises a plurality of predetermined historical time points; and the analysis module 710 comprises an obtaining sub-module and an analysis sub-module.

[0156] The obtaining sub-module is configured to obtain processor utilization rates of the plurality of work nodes at the plurality of predetermined historical time points within the predetermined historical period.

[0157] The analysis sub-module is configured to determine the processor utilization rate of the work node within the predetermined historical period based on an average value of the processor utilization rates at the plurality of predetermined historical time points, to obtain respective processor utilization rates of the plurality of work nodes.

[0158] According to an embodiment of the present application, the screening module 720 comprises a first screening sub-module and a second screening sub-module.

[0159] The first screening sub-module is configured to screen historical tasks for executing target operations from the plurality of historical tasks to obtain at least one candidate historical task, the target operations comprising at least one of a calculation operation and a query operation.

[0160] The second screening sub-module is configured to determine the target historical task from the at least one candidate historical task according to respective historical processor utilization rates of the at least one candidate historical task.

[0161] According to an embodiment of the present application, the second screening sub-module comprises a sorting unit and a determining unit.

[0162] The sorting unit is configured to sort the at least one candidate historical task in descending order according to the respective historical processor utilization rates of the at least one candidate historical task to obtain a candidate historical task sequence.

[0163] The determining unit is configured to determine the candidate historical task at a predetermined position in the candidate historical task sequence as the target historical task.

[0164] According to an embodiment of the present application, any of the analysis module 710, the screening module 720, the storage module 730 and the sending module 740 can be combined in one module, or any of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of the other modules, and implemented in one module. According to an embodiment of the present application, at least one of the analysis module 710, the screening module 720, the storage module 730 and the sending module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware that can be integrated or packaged with a circuit, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the analysis module 710, the screening module 720, the storage module 730 and the sending module 740 can be at least partially implemented as a computer program module that can perform the corresponding function when it is run.

[0165] Figure 8 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present application is shown.

[0166] As shown in Figure 8 The electronic device 800 according to an embodiment of the present application includes a processor 801 that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 can include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset, and / or a special-purpose microprocessor (such as an application specific integrated circuit (ASIC)), and the like. The processor 801 can also include an on-board memory for cache use. The processor 801 can include a single processing unit or multiple processing units for performing different actions of the method processes according to an embodiment of the present application.

[0167] In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other via the bus 804. The processor 801 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 802 and / or the RAM 803. It should be noted that the programs can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0168] According to the embodiments of the present application, the electronic device 800 can further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 can further include one or more of the following components connected to the input / output (I / O) interface 805: an input part 806 including a keyboard, a mouse, and the like; an output part 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 808 including a hard disk, and the like; and a communication part 809 including a network interface card such as a LAN card, a modem, and the like. The communication part 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 810 as necessary, so that a computer program read therefrom is installed in the storage part 808 as necessary.

[0169] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0170] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include the ROM 802 and / or the RAM 803 described above and / or one or more memory other than the ROM 802 and the RAM 803.

[0171] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the data processing method provided by the embodiments of the present application.

[0172] The above functions defined in the system / device of the embodiments of the present application are performed when the computer program is executed by the processor 801. According to an embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0173] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of signals on a network medium, and be downloaded and installed through the communication part 809, and / or be installed from the detachable medium 811. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

[0174] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or be installed from the detachable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiments of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0175] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Program code can execute entirely on a user's computing device, partly on the user's device, as a stand-alone software package, partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0176] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0177] Those skilled in the art will understand that features recited in the various embodiments of the present application can be combined and / or integrated in various ways, even if such combinations or integrations are not expressly noted in the present application. In particular, features recited in the various embodiments of the present application can be combined and / or integrated in ways that are not expressly noted in the present application, without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

[0178] The embodiments of the present application have been described above. However, these embodiments are merely for the purpose of illustration, and are not intended to limit the scope of the present application. Although the embodiments are described separately above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present application, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A data processing method, characterized in that, The method includes: The working status of multiple worker nodes in the cluster is analyzed within a predetermined historical period to determine the processor utilization of each worker node within the predetermined historical period. In response to the existence of a first working node among the multiple working nodes with a processor utilization rate greater than a preset threshold, the multiple historical tasks are filtered according to the historical processor utilization rates of the multiple historical tasks executed by the first working node in the predetermined historical period to obtain a target historical task. The target dataset required for processing the target historical task is stored in the first backup node, which is determined from multiple idle backup nodes in the cluster; In response to receiving a new task, if it is determined that the new task is for performing a target operation on the target dataset, the new task is sent to the first backup node so that the first backup node can process the new task. During the process of storing the target dataset required for processing the target historical task to the first backup node, in response to receiving a first data change task for performing a change operation on the target dataset, the information of the first data change task is updated, and a task to be executed is generated with added identification information indicating that it will not be executed for the time being. The task to be executed is sent to the first working node so that after the target dataset is completely stored in the first backup node, the first working node updates the first target data to be updated in the task to be executed based on the changed data in the task to be executed.

2. The method according to claim 1, characterized in that, The method further includes: In response to receiving a first data processing task, the query statement included in the first data processing task is parsed to determine the first operation type of the first operation to be executed and the first data identifier of the first dataset to be operated on for the first operation to be executed. Based on the first data identifier and the first operation type, determine whether the first data processing task is a first data change task for performing change operations on the target dataset.

3. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the target dataset required for processing the target historical task has been completely stored in the first backup node, a second data change task is generated for the first backup node based on the first data change task. The second data change task is sent to the first backup node so that the first backup node responds to the second data change task and updates the first target data stored in the first backup node with the changed data.

4. The method according to claim 1, characterized in that, Sending the new task to the first backup node so that the first backup node can process the new task includes: The newly added task and the system information of the application system that initiated the newly added task are sent to the first backup node, so that the first backup node can establish a connection with the application system based on the system information and send the processing result of the newly added task to the application system.

5. The method according to claim 1, characterized in that, The method further includes: In the case that the target dataset required for processing the target historical task has been completely stored in the first backup node, in response to receiving the second data processing task, the query statement included in the second data processing task is parsed to determine the second operation type of the second operation to be executed and the second data identifier of the second data dataset to be operated on by the second operation to be executed. Based on the second data identifier and the second operation type, determine the identification result for the second data processing task; If the identification result indicates that the second data processing task is used to perform a change operation on the target dataset, the second data processing task is sent to the first working node and the first backup node so that the first working node and the first backup node can process the second data processing task respectively.

6. The method according to claim 1, characterized in that, The method further includes: Determine the distributed data table to which the target dataset belongs; Based on the data table identifier of the distributed data table, the distributed data table is stored in the first backup node.

7. The method according to claim 1, characterized in that, The method further includes: If a second working node fails among the plurality of working nodes and there is no idle backup node in the cluster, the second backup node is determined from the plurality of backup nodes. In response to the second working node and the second backup node having the same target storage data, other storage data besides the target storage data is deleted from the second backup node; The external storage volume of the second working node is mounted to the second backup node so that the second backup node loads other storage data stored in the second working node from the external storage volume, excluding the target storage data, and switches the data processing tasks executed through the second working node to the second backup node for execution.

8. The method according to claim 1, characterized in that, The method further includes: Based on the node status of each of the multiple backup nodes in the cluster, determine a number of idle backup nodes from the multiple backup nodes; The first backup node is determined from the plurality of idle backup nodes based on their respective resource configuration information.

9. The method according to claim 1, characterized in that, The predetermined historical period includes multiple predetermined historical moments; the analysis of the working status of multiple worker nodes in the cluster within the predetermined historical period, and the determination of the processor utilization of each of the multiple worker nodes within the predetermined historical period, includes: Obtain the processor utilization of the working node at multiple predetermined historical moments within the predetermined historical period; Based on the average of the processor utilization at multiple predetermined historical moments, the processor utilization of the working node within the predetermined historical period is determined, thereby obtaining the processor utilization of each of the multiple working nodes.

10. The method according to claim 1, characterized in that, The step of filtering multiple historical tasks based on the historical processor utilization of each of the multiple historical tasks executed by the first working node in the predetermined historical period to obtain a target historical task includes: From a plurality of historical tasks, at least one candidate historical task is obtained by filtering historical tasks for performing the target operation, wherein the target operation includes at least one of a calculation operation and a query operation; The target historical task is determined from at least one of the candidate historical tasks based on the historical processor utilization of each of the candidate historical tasks.

11. The method according to claim 10, characterized in that, The step of determining the target historical task from at least one candidate historical task based on the historical processor utilization of each of the at least one candidate historical task includes: Based on the historical processor utilization rate of each of the at least one candidate historical task, the at least one candidate historical task is sorted in descending order to obtain a candidate historical task sequence. The candidate historical task located at a predetermined position in the candidate historical task sequence is identified as the target historical task.

12. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images