Task processing methods, media, devices and computing equipment

Abstract

This disclosure provides a task processing method. The method includes: acquiring status data of a task being processed; and sending the status data to a second cluster, so that when the second cluster detects a cluster-level anomaly in the first cluster, it can continue executing the task being processed based on the status data. By synchronizing the status data of the task being processed from the first cluster to the second cluster, the second cluster can continue executing the task being processed even when the first cluster experiences an anomaly. This method avoids the problem of data loss due to cluster anomalies, thus preventing task processing failures and providing a better user experience. Furthermore, this disclosure also provides a medium, apparatus, and computing device.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically, embodiments of this disclosure relate to task processing methods, media, apparatus, and computing devices. Background Art

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] In the field of computer technology, tasks are categorized into real-time tasks and non-real-time tasks. Real-time tasks are those related to a quantitative expression of time; they are scheduled to complete all computational events involved in their execution within time constraints. The time constraint associated with a real-time task is a deadline, which the task must meet before. Therefore, in practical applications, to ensure that real-time tasks complete before their deadlines, it is necessary to allocate real-time computing resources appropriately.

[0004] In related technologies, real-time computing resources include Flink Kubernetes clusters and Flink Yarn clusters. To prevent real-time task processing failures when Flink Kubernetes clusters and / or Flink Yarn clusters experience cluster-level anomalies, the common practice is to restart the cluster experiencing the anomaly so that the restarted cluster can continue executing the real-time tasks that were not completed due to the anomaly. However, this approach may still result in the real-time task failing. Summary of the Invention

[0005] This disclosure provides a task processing method, medium, apparatus, and computing device to achieve successful processing of real-time tasks.

[0006] In a first aspect of the present disclosure, a task processing method is provided, applied to a first cluster, comprising: acquiring status data of a task being processed; and sending the status data to a second cluster, so that when the second cluster detects a cluster-level anomaly in the first cluster, it continues to execute the task being processed based on the status data.

[0007] In one embodiment of this disclosure, obtaining status data of a processing task includes: monitoring a status directory, which is used to store status data of the processing task; responding to the detection of newly added status data in the status directory and obtaining the newly added status data; and sending status data to a second cluster, including: sending the newly added status data to the second cluster.

[0008] In another embodiment of this disclosure, before obtaining the newly added state data, the method further includes: determining the path of the newly added state data in the state directory; determining the state file corresponding to the newly added state data in the state directory according to the path; performing serialization processing on the state file to obtain a serialized file; obtaining the newly added state data includes: obtaining the serialized file; and performing deserialization processing on the serialized file to obtain the newly added state data.

[0009] In another embodiment of this disclosure, before monitoring the status directory, the method further includes: obtaining the status directory address of the processing task before the processing task is executed; monitoring the status directory includes: monitoring the status directory based on the status directory address.

[0010] In another embodiment of this disclosure, before sending status data to the second cluster, the method further includes: obtaining a first number and a second number, wherein the first number is the number of status data items and the second number is the number of resource partitions in the first cluster; determining the ratio of the first number and the second number; and sending status data to the second cluster, including: responding to a ratio less than or equal to a running baseline value of the resource load of the resource partition, and controlling the resource partition to send status data to the second cluster, wherein the running baseline value is used to adjust the resource load of the resource partition.

[0011] In another embodiment of this disclosure, controlling a resource partition to send status data to a second cluster includes: splitting the status data according to the resource load of the resource partition to obtain multiple sets of data; storing the multiple sets of data in storage media corresponding to different resource partitions; controlling different resource partitions to read data from their corresponding storage media, and sending the read data and the corresponding storage address to the second cluster respectively, so that the storage addresses of each set of data in the first cluster and the second cluster are the same.

[0012] In yet another embodiment of this disclosure, the method further includes: in response to a ratio being greater than a running baseline value, requesting a target number of resource partitions, the target number being determined by a first number, a second number, and the running baseline value.

[0013] In another embodiment of this disclosure, after sending the status data to the second cluster, the method further includes: performing a consistency check on the status data in the first cluster and the status data in the second cluster.

[0014] In another embodiment of this disclosure, consistency verification of state data in the first cluster and state data in the second cluster includes: determining a first data size and a first hash value of the state data in the first cluster, and a second data size and a second hash value of the state data in the second cluster; and determining that the verification of state data in the first cluster and state data in the second cluster is successful in response to the first data size being equal to the second data size and the first hash value being equal to the second hash value.

[0015] In a second aspect of the present disclosure, a task processing method is provided, applied to a second cluster, comprising: monitoring whether a first cluster generates a cluster-level anomaly, wherein the first cluster is used to acquire status data of a task being processed and send the status data to the second cluster; and responding to the detection that a cluster-level anomaly has been generated in the first cluster, continuing to execute the task being processed based on the status data.

[0016] In a third aspect of the present disclosure, a task processing method is provided, comprising: a first cluster monitoring a status directory, the status directory being used to store status data of a processed task; the first cluster responding to the detection of newly added status data in the status directory and acquiring the newly added status data; the first cluster sending the newly added status data to a second cluster; the second cluster monitoring whether the first cluster has generated a cluster-level anomaly; and the second cluster responding to the detection of a cluster-level anomaly in the first cluster and continuing to execute the processed task based on the status data of the processed task in the first cluster, wherein the status data originates from the first cluster.

[0017] In a fourth aspect of the present disclosure, a medium is provided that stores computer execution instructions, which, when executed by a processor, are used to implement the task processing method of the first aspect.

[0018] In a fifth aspect of the present disclosure, a task processing apparatus is provided, applied to a first cluster, comprising: an acquisition module for acquiring status data of a task being processed; and a sending module for sending the status data to a second cluster, so that when the second cluster detects a cluster-level anomaly in the first cluster, it continues to execute the task being processed based on the status data.

[0019] In a sixth aspect of the present disclosure, a task processing apparatus is provided, applied to a second cluster, comprising: a monitoring module for monitoring whether a first cluster generates a cluster-level anomaly, wherein the first cluster acquires status data of a task being processed and sends the status data to the second cluster; and an execution module for responding to the detection of a cluster-level anomaly in the first cluster and, based on the status data, continuing to execute the task being processed.

[0020] In a seventh aspect of this disclosure, a task processing apparatus is provided, comprising: a first control module configured to control a first cluster to perform the following steps: monitoring a status directory, the status directory being used to store status data of processed tasks; in response to detecting newly added status data in the status directory, acquiring the newly added status data; and sending the newly added status data to a second cluster; and a second control module configured to control the second cluster to perform the following steps: monitoring whether the first cluster has generated a cluster-level anomaly; in response to detecting a cluster-level anomaly in the first cluster, continuing to execute the currently processed task based on the status data of the task being processed in the first cluster, wherein the status data originates from the first cluster.

[0021] In an eighth aspect of the present disclosure, a task processing system is provided, comprising: a first cluster for executing the task processing method of the first aspect; and a second cluster for executing the task processing method of the second aspect.

[0022] In a ninth aspect of the present disclosure, a computing device is provided, comprising: a processor and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement a task processing method as described in any one of the first to third aspects.

[0023] According to the task processing method, medium, apparatus, and computing device of this disclosure, by synchronizing the status data of the task being processed in the first cluster to the second cluster, the second cluster can continue to execute the task being processed when the first cluster encounters an anomaly. This avoids the problem of loss of status data of the task being processed when the cluster encounters an anomaly, thereby avoiding the failure of the task processing and bringing a better experience to the user. Attached Figure Description

[0024] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0025] Figure 1 A schematic diagram illustrating an application scenario of the task processing method provided in this embodiment of the disclosure;

[0026] Figure 2 A flowchart illustrating an embodiment of the task processing method provided in this disclosure;

[0027] Figure 3 A schematic diagram of a first embodiment of state data synchronization provided in this disclosure;

[0028] Figure 4 A schematic diagram of a second embodiment of state data synchronization provided in this disclosure;

[0029] Figure 5 A schematic diagram of Embodiment 3 of the status data synchronization provided in this disclosure;

[0030] Figure 6 A flowchart illustrating Embodiment 2 of the task processing method provided in this disclosure;

[0031] Figure 7 A flowchart illustrating Embodiment 3 of the task processing method provided in this disclosure;

[0032] Figure 8 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present disclosure;

[0033] Figure 9 This is a schematic diagram of the structure of a task processing device according to a first embodiment of the present disclosure;

[0034] Figure 10 This is a schematic diagram of the structure of a second embodiment of the task processing device according to the present disclosure;

[0035] Figure 11 This is a schematic diagram of the structure of a third embodiment of the task processing device according to the present disclosure;

[0036] Figure 12 This is a schematic diagram of the structure of a computing device according to an embodiment of the present disclosure.

[0037] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. Detailed Implementation

[0038] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0039] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0040] According to embodiments of this disclosure, a task processing method, medium, apparatus, and computing device are proposed.

[0041] In this article, it is important to understand that the terms used refer to the following:

[0042] Serialization: The process of converting a Java object into a sequence of bytes.

[0043] Deserialization: The process of converting a sequence of bytes into a Java object.

[0044] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0045] In addition, the data involved in this disclosure may be data authorized by the user or fully authorized by all parties. The collection, dissemination and use of the data shall comply with the requirements of relevant national laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.

[0046] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. SUMMARY OF THE INVENTION

[0048] The inventors have discovered that in practical applications, to ensure real-time tasks are completed before the deadline, it is necessary to allocate real-time computing resources reasonably for them. In related technologies, to avoid real-time task processing failures due to cluster-level anomalies in Flink Kubernetes clusters and / or Flink Yarn clusters, the cluster experiencing the anomaly is typically restarted so that the restarted cluster can continue executing the real-time tasks that were not completed due to the anomaly. However, the inventors have found that restarting a cluster experiencing a cluster-level anomaly can lead to partial or complete loss of the real-time task's state data. This means that even after restarting, the real-time task may still fail, and the failure of the real-time task will prevent the corresponding project or program from running, thus affecting the user experience.

[0049] Therefore, this disclosure provides a task processing method that synchronizes the status data of the tasks being processed in the first cluster to the second cluster for storage. This allows the second cluster to continue executing the tasks being processed when the first cluster encounters an anomaly, thereby preventing task processing failures and the loss of task status data. As a result, it provides a better user experience.

[0050] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0051] Application Scenarios Overview

[0052] First refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the task processing method provided in this embodiment of the disclosure. Figure 1This task processing method can be applied to a task processing system, which may include a first cluster and a second cluster. The first cluster may include two resource management clusters, which can be Flink Kubernetes clusters. The second cluster may include one resource management cluster, which can be a Flink YARN cluster. Using both Flink Kubernetes and Flink YARN clusters ensures the normal operation of more tasks. The two resource management clusters in the first cluster share the same data storage cluster, while the second cluster uses a separate data storage cluster, which can be a Hadoop cluster.

[0053] In this scenario, a task is typically run on one of two Flink Kubernetes clusters. While the task is being processed, the Flink Kubernetes cluster running the task also runs a state data synchronization program, which synchronizes the task's state data to the same directory in the other Flink Kubernetes cluster and / or Flink Yarn cluster in real time.

[0054] In this scenario, if a cluster-level anomaly occurs in the Flink Kubernetes cluster running the processing task, to prevent processing failure due to loss of state data, another Flink Kubernetes cluster or Flink YARN cluster can be restarted. This allows the other Flink Kubernetes cluster or Flink YARN cluster to continue executing the processing task based on the state data in the same directory. Since the other Flink Kubernetes cluster or Flink YARN cluster has already synchronized the state data of the processing task, the integrity of the state data can be guaranteed when the other Flink Kubernetes cluster or Flink YARN cluster continues to execute the processing task, thus ensuring the successful execution of the processing task and providing a better user experience.

[0055] Optionally, the second cluster can also run tasks on a daily basis. While running tasks on a daily basis, the second cluster can also synchronize the status data of the tasks to the first cluster.

[0056] Exemplary Methods

[0057] The following is combined Figure 1 Application scenarios, refer to Figure 2 , Figure 6 and Figure 7This document describes a task processing method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0058] Figure 2 A flowchart illustrating an embodiment of the task processing method provided in this disclosure is shown below. Figure 2 In this context, the task processing method can be applied to the first cluster, and the task processing method includes the following steps:

[0059] S201: Obtain status data for the task being processed.

[0060] Specifically, the first cluster can be a resource management cluster, such as a Flink Kubernetes cluster, and the tasks being processed can be real-time tasks. During the normal operation of the processing tasks within the first cluster, the first cluster can acquire real-time status data generated by the processing tasks.

[0061] Optionally, the status data of the processing task can be stored in real time in the data storage cluster used by the first cluster, such as a Hadoop cluster.

[0062] S202: Send status data to the second cluster so that when the second cluster detects a cluster-level anomaly in the first cluster, it can continue to execute the processing task based on the status data.

[0063] Specifically, the second cluster can also be a resource management cluster, such as a Flink Kubernetes cluster or a Flink Yarn cluster. While the first cluster is running tasks, it is also running a state data synchronization program. When a task successfully generates state data, the first cluster can synchronize this data to the second cluster via the synchronization program. If the first cluster experiences a cluster-level exception, it will be unable to continue running tasks, potentially leading to task failure and state data loss. For example, if a task is used for rendering, failure could result in rendering failure and negatively impact the user experience.

[0064] Specifically, in order to ensure that the processing task can run normally, when the first cluster generates a cluster-level anomaly, since the second cluster has already obtained the status data of the processing task, the second cluster can continue to execute the processing task based on the status data sent by the first cluster, thereby avoiding the loss of the status data of the processing task and realizing the normal operation of the processing task.

[0065] Optionally, the second cluster can store the received status data in real time in the data storage cluster used by the second cluster, such as a Hadoop cluster, which stores the received status data through HDFS backup.

[0066] The task processing method disclosed herein synchronizes the status data of the tasks being processed in the first cluster to the second cluster. This allows the second cluster to continue executing the tasks being processed even if the first cluster experiences an anomaly. This avoids the loss of status data of the tasks being processed when the first cluster experiences a cluster-level anomaly, thereby ensuring the normal operation of the tasks being processed and providing a better user experience.

[0067] In one embodiment of this disclosure, obtaining status data of a processing task includes: monitoring a status directory, which is used to store status data of the processing task; responding to the detection of newly added status data in the status directory and obtaining the newly added status data; and sending status data to a second cluster, including: sending the newly added status data to the second cluster.

[0068] Specifically, the status directory can be a directory in the data storage cluster of the first cluster used to store the status data of the processing tasks. All status data generated during the running of the processing tasks in the first cluster can be stored in this status directory.

[0069] Specifically, the first cluster can monitor the status directory in real time so that it can send newly generated status data of the processing task to the second cluster in real time. That is, when the first cluster detects that the processing task generates new status data and the status directory stores the new status data, it can obtain the new status data and then send the new status data to the second cluster. This enables the status data of the processing task to be synchronized to the second cluster in real time. Therefore, when the first cluster experiences a cluster-level anomaly, the second cluster can continue to execute the processing task, which can avoid the loss of status data of the processing task, ensure the successful execution of the processing task, and avoid affecting the user experience.

[0070] Specifically, the synchronization of state data between the first cluster and the second cluster can be as follows: Figure 3 As shown. Figure 3 This is a schematic diagram of a first embodiment of state data synchronization provided in this disclosure. Figure 3 In the first cluster, a state synchronization engine is configured to synchronize state data. This engine, implemented in Java, generates state data in real time as tasks run normally within the first cluster. This state data is stored in a state directory. The state synchronization engine monitors the state directory in real time and outputs the newly added state data to the second cluster when it detects any new data.

[0071] In another embodiment of this disclosure, before obtaining the newly added state data, the method further includes: determining the path of the newly added state data in the state directory; determining the state file corresponding to the newly added state data in the state directory according to the path; performing serialization processing on the state file to obtain a serialized file; obtaining the newly added state data includes: obtaining the serialized file; and performing deserialization processing on the serialized file to obtain the newly added state data.

[0072] Specifically, during the normal operation of the processing task in the first cluster, if a state is successfully generated, the corresponding state data will be stored in the state directory. After detecting the addition of new state data in the state directory, the first cluster can determine the path of the new state data in the state directory, read the path, and generate an array of file directories corresponding to the new state data. This array of file directories can also be called a state file. To facilitate the storage of the state file, it can be serialized to obtain a serialized file, which is then stored in the data storage cluster corresponding to the first cluster.

[0073] Specifically, the file directory array, also known as the state file, includes directories within the state directory containing a set of state data corresponding to the successful execution of a processing task. This file directory array represents HDFS block-level files to be copied to the second cluster. Serialized files can be stored in the data storage cluster of the first cluster. By synchronizing the state data of the processing tasks running in the first cluster to the second cluster, the second cluster can continue executing the processing tasks in the event of a cluster-level anomaly in the first cluster. This avoids the loss of state data for the processing tasks, ensuring successful execution and preventing any impact on the user experience.

[0074] Specifically, the serialized file is obtained by the first cluster after detecting new status data in the status directory, by serializing the status file corresponding to the new status data. Therefore, when obtaining new status data, the serialized file can be retrieved from the data storage cluster of the first cluster first, and then deserialized to obtain the new status data. This allows the status data generated by each processing task running in the first cluster to be synchronized to the second cluster. Thus, if a cluster-level anomaly occurs in the first cluster, the second cluster can continue executing the processing task, thereby preventing the loss of status data and ensuring the successful execution of the processing task, thus avoiding impact on the user experience.

[0075] Specifically, the first cluster can synchronize state data between the first cluster and the second cluster through a state synchronization engine. This state synchronization engine can include monitoring components and execution components, such as... Figure 4As shown, Figure 4 This is a schematic diagram of Embodiment 2 of the state data synchronization provided in this disclosure. Figure 4 In the process, after detecting new status data in the status directory, the monitoring component can read the path of the new status data in the status directory, generate a status file corresponding to the new status data, serialize the status file, store the resulting serialized file in the data storage cluster of the first cluster, and submit the storage path of the serialized file in the data storage cluster of the first cluster to the execution component, so that the execution component can obtain the serialized file according to the storage path of the serialized file, and then perform deserialization processing on the serialized file to obtain the new status data.

[0076] In another embodiment of this disclosure, before monitoring the status directory, the method further includes: obtaining the status directory address of the processing task before the processing task is executed; monitoring the status directory includes: monitoring the status directory based on the status directory address.

[0077] Specifically, the processing task, also known as a Flink task, includes the Flink state directory parameter `state.checkpoints.dir`, which is the address of the state directory for the processing task (the address of the state directory in the data storage cluster of the first cluster). Before the processing task is executed, the first cluster can obtain the address of the state directory for the processing task. During the normal operation of the processing task in the first cluster, the first cluster can monitor the state directory of the processing task through the address of the state directory for the processing task. When new state data is detected in the state directory, the first cluster can send the new state data to the second cluster. This allows all state data of the processing task to be synchronized to the second cluster in real time. Therefore, if a cluster-level anomaly occurs in the first cluster, the second cluster can continue to execute the processing task to avoid the loss of state data and ensure the successful execution of the processing task, thus avoiding any impact on the user experience.

[0078] In another embodiment of this disclosure, before sending status data to the second cluster, the method further includes: obtaining a first number and a second number, wherein the first number is the number of status data items and the second number is the number of resource partitions in the first cluster; determining the ratio of the first number and the second number; and sending status data to the second cluster, including: responding to a ratio less than or equal to a running baseline value of the resource load of the resource partition, and controlling the resource partition to send status data to the second cluster, wherein the running baseline value is used to adjust the resource load of the resource partition.

[0079] Specifically, to ensure the successful synchronization of the status data of tasks being processed in the first cluster to the second cluster, the resources of the first cluster need to be verified. This verification can be performed using the status data.

[0080] Specifically, after deserializing the serialized file to obtain state data, the newly added state data can exist in the form of a data list. Therefore, we can first obtain the list number (the first number) and the resource parallelism in the first cluster (the second number); then determine the ratio of the first and second numbers, and use this ratio as a judgment value for verifying the resources of the first cluster. By comparing this ratio with the running baseline value, we can determine whether the verification of the resources of the first cluster has passed. The running baseline value can be a configurable value for the first cluster, which can be used to adjust the resource load of a resource partition based on the number of files that need to be transferred in a single resource partition. Increasing the running baseline value increases the resource load of the resource partition, while decreasing it decreases it.

[0081] Specifically, if the ratio is less than or equal to the operating baseline value of the resource load of the resource partition, it can be determined that the resource verification of the first cluster has passed. Therefore, the first cluster can synchronize the status data of the processing task to the second cluster.

[0082] In yet another embodiment of this disclosure, the method further includes: in response to a ratio being greater than a running baseline value, requesting a target number of resource partitions, the target number being determined by a first number, a second number, and the running baseline value.

[0083] Specifically, if the ratio is greater than the baseline value of the resource load of the resource partition, it can be determined that the resource verification of the first cluster has failed. That is, the first cluster cannot call appropriate resources to synchronize the status data of the tasks being processed to the second cluster. Therefore, in order for the first cluster to successfully synchronize the status data of the tasks being processed to the second cluster, the first cluster needs to apply for the target number of resource partitions and synchronize the status data of the tasks being processed to the second cluster through the applied resource partitions.

[0084] Specifically, the target quantity can be determined by the first number, the second number, and the operating baseline value, and can be expressed by the following formula:

[0085] Target quantity = First number - Baseline value * Second number

[0086] In another embodiment of this disclosure, controlling a resource partition to send status data to a second cluster includes: splitting the status data according to the resource load of the resource partition to obtain multiple sets of data; storing the multiple sets of data in storage media corresponding to different resource partitions; controlling different resource partitions to read data from their corresponding storage media, and sending the read data and the corresponding storage address to the second cluster respectively, so that the storage addresses of each set of data in the first cluster and the second cluster are the same.

[0087] Specifically, in order to reduce the pressure on a certain resource partition in the first cluster, the status data can be grouped according to the resource load of each resource partition, so that a corresponding set of status data can be transmitted through multiple resource partitions.

[0088] Specifically, after deserializing the serialized file, the resulting state data exists in the form of a data list. Therefore, the state data is grouped to obtain multiple groups, each with its own array index. The first cluster can perform a modulo operation on the hash value of the partition number of each resource partition and the index of the corresponding group of data in the first cluster to obtain the directory address of the corresponding group of data in the data storage cluster of the first cluster. In this way, each resource partition can obtain the corresponding group of data according to the directory address and store the obtained data in the storage medium corresponding to that resource partition.

[0089] Specifically, when synchronizing status data to the second cluster, the first cluster can control each resource partition to read a corresponding set of data from its corresponding storage medium, and then control each resource partition to synchronize the corresponding set of data and its directory address in the first cluster's data storage cluster to the second cluster. After receiving the status data and corresponding directory address from each resource partition, the second cluster can store the status data in its own data storage cluster, ensuring that the address where the status data is stored in the second cluster's data storage cluster is the same as the address where the status data is stored in the first cluster's data storage cluster. Therefore, if a cluster-level anomaly occurs in the first cluster, the second cluster can read the status data through the same address and continue executing the processing task, ensuring the successful execution of the processing task.

[0090] Specifically, synchronizing state data between the first and second clusters through resource partitioning can be achieved as follows: Figure 5 As shown. Figure 5 This is a schematic diagram of Embodiment 3 of the state data synchronization provided in this disclosure. Figure 5 In the process, the execution component in the state synchronization engine first reads state data from the data storage cluster of the first cluster to verify the resources of the first cluster. After the verification is successful, it sends a signal to each resource partition to start executing the task. After receiving the signal, each resource partition reads the corresponding state data from the data storage cluster of the first cluster according to the directory address of its own set of state data, and transmits the read state data and the corresponding directory address to the second cluster. The second cluster then stores the received state data according to the directory address.

[0091] Optionally, the resource partition can transmit the read status data and the corresponding directory address to the second cluster via the network. The resource partition of the first cluster can also be a thread of the first cluster.

[0092] In another embodiment of this disclosure, after sending the status data to the second cluster, the method further includes: performing a consistency check on the status data in the first cluster and the status data in the second cluster.

[0093] Specifically, in order to ensure that the status data synchronized to the second cluster is consistent with the status data of the task being processed in the first cluster, after sending the status data to the second cluster, it is also necessary to perform a consistency check on the status data in the first cluster and the status data in the second cluster to ensure that the status data in the first cluster can be successfully synchronized to the second cluster.

[0094] In another embodiment of this disclosure, consistency verification of state data in the first cluster and state data in the second cluster includes: determining a first data size and a first hash value of the state data in the first cluster, and a second data size and a second hash value of the state data in the second cluster; and determining that the verification of state data in the first cluster and state data in the second cluster is successful in response to the first data size being equal to the second data size and the first hash value being equal to the second hash value.

[0095] Specifically, consistency checks can be performed on the state data in the first cluster and the state data in the second cluster using the data size and / or hash value of the state data in the two clusters. That is, if the data size and / or hash value of the state data in the two clusters are the same, then the state data in the first cluster and the state data in the second cluster are considered consistent, meaning the check on the state data in the first cluster and the state data in the second cluster is successful. If the data size and / or hash value of the state data in the two clusters are different, then the state data in the first cluster and the state data in the second cluster are considered inconsistent, meaning the check on the state data in the first cluster and the state data in the second cluster is unsuccessful. In this case, the state data in the first cluster needs to be resynchronized to the second cluster to ensure successful synchronization.

[0096] Figure 6 A flowchart illustrating Embodiment 2 of the task processing method provided in this disclosure is shown below. Figure 6 In this context, the task processing method can be applied to a second cluster, and the task processing method includes the following steps:

[0097] S601: Monitor whether the first cluster generates a cluster-level anomaly.

[0098] Specifically, the first cluster is used to obtain status data of the task being processed and send the status data to the second cluster.

[0099] Specifically, the first cluster can be a resource management cluster, such as a Flink Kubernetes cluster, and the tasks being processed can be real-time tasks. During the normal operation of the processing tasks in the first cluster, the first cluster can obtain the state data generated by the processing tasks in real time. The second cluster can also be a resource management cluster, such as a Flink Kubernetes cluster or a Flink YARN cluster. While running the processing tasks, the first cluster also runs a state data synchronization program. When the processing task successfully generates state data, the first cluster can synchronize the state data generated by the processing task to the second cluster by running the synchronization program.

[0100] Specifically, if the first cluster experiences a cluster-level anomaly, it will be unable to continue running the processing task, which may lead to the failure of the processing task and the loss of state data. For example, the processing task may be a task used for rendering. If the processing task fails to execute and the state data is lost, it may lead to rendering failure, thereby affecting the user experience. Therefore, the second cluster needs to monitor whether the first cluster experiences a cluster-level anomaly.

[0101] Optionally, the status data of the processing task can be stored in real time in the data storage cluster used by the first cluster, such as a Hadoop cluster.

[0102] S602: In response to the detection of a cluster-level anomaly in the first cluster, based on the status data, continue to execute the processing task.

[0103] Specifically, in order to ensure that the processing task can run normally, when the first cluster generates a cluster-level anomaly, since the second cluster has already obtained the status data of the processing task, the second cluster can continue to execute the processing task based on the status data sent by the first cluster, thereby avoiding the loss of the status data of the processing task and realizing the normal operation of the processing task.

[0104] Optionally, the second cluster can store the received status data in real time in the data storage cluster used by the second cluster, such as a Hadoop cluster, which stores the received status data through HDFS backup.

[0105] The task processing method disclosed herein synchronizes the status data of the tasks being processed in the first cluster to the second cluster. This allows the second cluster to continue executing the tasks being processed even if the first cluster experiences an anomaly. This avoids the loss of status data of the tasks being processed when the first cluster experiences a cluster-level anomaly, thereby ensuring the normal operation of the tasks being processed and providing a better user experience.

[0106] Figure 7 A flowchart illustrating Embodiment 3 of the task processing method provided in this disclosure is shown below. Figure 7 The task processing method includes the following steps:

[0107] S701: First cluster monitoring status directory.

[0108] Specifically, the status directory is used to store status data for processing tasks.

[0109] Specifically, the status directory can be a directory in the data storage cluster of the first cluster used to store the status data of the processing tasks. All status data generated during the running of the processing tasks in the first cluster can be stored in this status directory.

[0110] S702: The first cluster responded by detecting new status data in the status directory and retrieved the new status data.

[0111] Specifically, the first cluster can monitor the status directory in real time, enabling it to send newly generated status data of the processing task to the second cluster in real time. That is, when the first cluster detects new status data generated by the processing task and that the new status data is stored in the status directory, it can retrieve the new status data and then send it to the second cluster. This allows for real-time synchronization of the processing task's status data to the second cluster. Therefore, if a cluster-level anomaly occurs in the first cluster, the second cluster can continue executing the processing task, preventing the loss of status data and ensuring successful execution of the processing task, thus avoiding impact on user experience. Figure 3 As shown.

[0112] S703: The second cluster monitors whether the first cluster has generated cluster-level anomalies.

[0113] Specifically, if the first cluster experiences a cluster-level anomaly, it will be unable to continue running the processing task, which may lead to the failure of the processing task and the loss of state data. For example, the processing task may be a task used for rendering. If the processing task fails to execute and the state data is lost, it may lead to rendering failure, thereby affecting the user experience. Therefore, the second cluster needs to monitor whether the first cluster experiences a cluster-level anomaly.

[0114] S704: The second cluster responded by detecting a cluster-level anomaly in the first cluster and continued to execute the task being processed based on the status data of the task being processed in the first cluster.

[0115] Specifically, the status data comes from the first cluster.

[0116] Specifically, in order to ensure that the processing task can run normally, when the first cluster generates a cluster-level anomaly, since the second cluster has already obtained the status data of the processing task, the second cluster can continue to execute the processing task based on the status data sent by the first cluster, thereby avoiding the loss of the status data of the processing task and realizing the normal operation of the processing task.

[0117] Optionally, the second cluster can store the received status data in real time in the data storage cluster used by the second cluster, such as a Hadoop cluster, which stores the received status data through HDFS backup.

[0118] The task processing method disclosed herein synchronizes the status data of the tasks being processed in the first cluster to the second cluster. This allows the second cluster to continue executing the tasks being processed even if the first cluster experiences an anomaly. This avoids the loss of status data of the tasks being processed when the first cluster experiences a cluster-level anomaly, thereby ensuring the normal operation of the tasks being processed and providing a better user experience.

[0119] In one embodiment of this disclosure, Figure 7 The task processing method shown can be applied to a task processing system, which may include a first cluster and a second cluster. The first cluster is used to execute the aforementioned task processing method applied to the first cluster, and the second cluster is used to execute the aforementioned task processing method applied to the second cluster.

[0120] In one embodiment of this disclosure, a tag management and level definition specification can be pre-defined for the tasks being processed, i.e., real-time tasks. Tags can be used to determine different levels of the tasks being processed. For example, a tag of L1 indicates that the level of the task being processed is level 1; a tag of L2 indicates that the level of the task being processed is level 2; a tag of L3 indicates that the level of the task being processed is level 3; and a tag of L4 indicates that the level of the task being processed is level 4. The higher the level, the more important the task being processed, and different disaster recovery strategies correspond to different levels of tasks being processed.

[0121] Specifically, refer to Figure 1For L1 and L2 in-processing tasks, since they run in one of the Flink Kubernetes clusters, if the Flink Kubernetes cluster running the L1 and L2 in-processing tasks fails, it will switch to another Flink Kubernetes cluster. This allows the in-processing task to resume operation through a reused state directory. If the Hadoop cluster fails, it will notify the downstream data provider and wait for maintenance to restore it. For L3 in-processing tasks, the aforementioned task processing method is executed. For L4 in-processing tasks, since L4 in-processing tasks are highly sensitive and also affect business decisions, L4 in-processing tasks are routinely run in both Flink Kubernetes and Flink Yarn clusters with the same code but different downstream tables. The downstream data interface has already performed data checks when querying the status data of the in-processing task, so even if an exception occurs during the execution of the in-processing task, it will be imperceptible to the user interface.

[0122] Optionally, the disaster recovery strategies corresponding to the different levels of in-processing tasks mentioned above are only the optimal strategies. In actual use, the disaster recovery strategies used for in-processing tasks can be other disaster recovery strategies. For example, the disaster recovery strategy corresponding to L3 in-processing tasks is also applicable to other levels of in-processing tasks.

[0123] The task processing method disclosed herein, by designing a state synchronization engine for the first cluster, enables the task to be restarted quickly with zero data loss in different clusters, thereby ensuring data stability, making data users unaware of failures, optimizing the user experience, and improving data satisfaction.

[0124] Exemplary media

[0125] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 8 The storage medium of the exemplary embodiments of this disclosure will be described.

[0126] Figure 8 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present disclosure, with reference to... Figure 8 As shown, the storage medium 80 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes computer-executable instructions for causing a computing device to execute the task processing method provided in this disclosure. However, the program product of this disclosure is not limited thereto.

[0127] The program product may employ any combination of one or more readable media. A readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0128] A readable signal medium may include data signals propagated in baseband or as part of a carrier wave, carrying computer-executed instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0129] Computer-executable instructions for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer-executable instructions can be executed entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0130] Exemplary device

[0131] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figures 9 to 11 The task processing apparatus of exemplary embodiments of the present disclosure will be described, wherein, Figure 9 This is used to implement the method described in the method embodiment applied to the first cluster. Figure 10 This is used to implement the method described in the above embodiment applied to the second cluster. Figure 11 The methods used to implement any of the above method embodiments have similar implementation principles and technical effects, and will not be described again here.

[0132] Figure 9 This is a schematic diagram of the structure of a task processing device according to an embodiment of the present disclosure. Figure 9 In this context, the task processing device 900 includes:

[0133] Module 901 is used to acquire status data during task processing.

[0134] The sending module 902 is used to send status data to the second cluster so that when the second cluster detects a cluster-level anomaly in the first cluster, it can continue to execute the processing task based on the status data.

[0135] Optionally, when acquiring the status data of the processing task, the acquisition module 901 is specifically used to: monitor the status directory, which is used to store the status data of the processing task; respond to the detection that there is new status data in the status directory and acquire the new status data; and send the status data to the second cluster, including: sending the new status data to the second cluster.

[0136] Optionally, the acquisition module 901 is further configured to: determine the path of the newly added state data in the state directory before acquiring the newly added state data; determine the state file corresponding to the newly added state data in the state directory according to the path; perform serialization processing on the state file to obtain a serialized file; when acquiring the newly added state data, the acquisition module 901 is specifically configured to: acquire the serialized file; and perform deserialization processing on the serialized file to obtain the newly added state data.

[0137] Optionally, the acquisition module 901 is also used to: before monitoring the status directory, before executing the processing task, acquire the status directory address of the processing task; monitor the status directory, including: monitoring the status directory based on the status directory address.

[0138] Optionally, the sending module 902 is further configured to: obtain a first number and a second number before sending status data to the second cluster, wherein the first number is the number of status data and the second number is the number of resource partitions in the first cluster; determine the ratio of the first number and the second number; respond that the ratio is less than or equal to the operating baseline value of the resource load of the resource partition, and control the resource partition to send status data to the second cluster, wherein the operating baseline value is used to adjust the resource load of the resource partition.

[0139] Optionally, when the sending module 902 controls the resource partition to send status data to the second cluster, it specifically performs the following: splits the status data according to the resource load of the resource partition to obtain multiple sets of data; stores the multiple sets of data in the storage medium corresponding to different resource partitions; controls different resource partitions to read data in the corresponding storage medium, and sends the read data and the corresponding storage location to the second cluster respectively, so that the storage address of each set of data in the first cluster and the second cluster is the same.

[0140] Optionally, it also includes: an application module 903, used to apply for a target number of resource partitions in response to a ratio greater than a running baseline value, the target number being determined by a first number, a second number, and the running baseline value.

[0141] Optionally, the sending module 902 is further configured to: after sending status data to the second cluster, perform a consistency check on the status data in the first cluster and the status data in the second cluster.

[0142] Optionally, when the sending module 902 performs consistency verification on the status data in the first cluster and the status data in the second cluster, it is specifically used to: determine the first data size and the first hash value of the status data in the first cluster, and the second data size and the second hash value of the status data in the second cluster; in response to the first data size being equal to the second data size and the first hash value being equal to the second hash value, determine that the verification of the status data in the first cluster and the status data in the second cluster is successful.

[0143] Figure 10 This is a schematic diagram of the structure of a second embodiment of the task processing device according to the present disclosure. Figure 10 The task processing device 1000 includes:

[0144] Monitoring module 1001 is used to monitor whether the first cluster generates cluster-level anomalies. The first cluster is used to obtain status data of the task being processed and send status data to the second cluster.

[0145] Execution module 1002 is used to respond to the detection of a cluster-level anomaly in the first cluster and continue to execute the processing task based on the status data.

[0146] Figure 11 This is a schematic diagram of the structure of a third embodiment of the task processing device according to the present disclosure. Figure 11 The task processing device 1100 includes:

[0147] The first control module 1101 is used to control the first cluster to perform the following steps: monitor the status directory, which is used to store status data of processing tasks; respond to the detection of newly added status data in the status directory, obtain the newly added status data; and send the newly added status data to the second cluster.

[0148] The second control module 1102 is used to control the second cluster to perform the following steps: monitor whether the first cluster generates a cluster-level anomaly; in response to the detection of a cluster-level anomaly in the first cluster, continue to execute the task being processed based on the status data of the task being processed in the first cluster, where the status data comes from the first cluster.

[0149] Exemplary computing device

[0150] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 12 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0151] Figure 12 The computing device 1200 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0152] Figure 12 This is a schematic diagram of the structure of a computing device according to an embodiment of the present disclosure. Figure 12 In this embodiment, the computing device 1200 is presented as a general-purpose computing device. The components of the computing device 1200 may include, but are not limited to: at least one processing unit 1201, at least one storage unit 1202, and a bus 1203 connecting different system components (including the processing unit 1201 and the storage unit 1202). The at least one storage unit 1202 stores computer-executable instructions; the at least one processing unit 1201 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0153] Bus 1203 includes a data bus, a control bus, and an address bus.

[0154] Storage unit 1202 may include readable media in the form of volatile memory, such as random access memory (RAM) 1221 and / or cache memory 1222, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 1223.

[0155] Storage unit 1202 may also include a program / utility 1225 having a set (at least one) of program modules 1224, such program modules 1224 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0156] The computing device 1200 can also communicate with one or more external devices 1204 (e.g., keyboard, pointing device, etc.). This communication can be performed via the input / output (I / O) interface 1205. Furthermore, the computing device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter 1206. Figure 12As shown, network adapter 1206 communicates with other modules of computing device 1200 via bus 1203. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0157] It should be noted that although several units / modules or sub-units / modules of the task processing apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0158] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0159] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A task processing method applied to a first cluster, the task processing method comprising: Retrieve status data for the task being processed; Obtain the first number and the second number, where the first number is the number of state data items and the second number is the number of resource partitions in the first cluster; Determine the ratio of the first number to the second number; In response to the ratio being less than or equal to the operating baseline value of the resource load of the resource partition, the resource partition is controlled to send the status data to the second cluster, so that when the second cluster detects a cluster-level anomaly in the first cluster, it continues to execute the processing task based on the status data. The operating baseline value is used to adjust the resource load of the resource partition; The step of controlling the resource partition to send the status data to the second cluster includes: Based on the resource load of the resource partition, the status data is split to obtain multiple sets of data; Multiple sets of data are stored separately in storage media corresponding to different resource partitions; The system controls different resource partitions to read data from their corresponding storage media and sends the read data and the corresponding storage address to the second cluster, so that the storage addresses of each group of data in the first cluster and the second cluster are the same.

2. The task processing method according to claim 1, wherein acquiring the status data of the task being processed includes: A monitoring status directory is used to store the status data of the task being processed. Upon detecting newly added status data in the status directory, the newly added status data is obtained. Sending the status data to the second cluster includes: sending the newly added status data to the second cluster.

3. The task processing method according to claim 2, further comprising, before acquiring the newly added status data: Determine the path of the newly added status data in the status directory; Based on the path, determine the status file corresponding to the newly added status data in the status directory; The status file is serialized to obtain a serialized file; The process of obtaining the newly added status data includes: Obtain the serialized file; The serialized file is deserialized to obtain the newly added state data.

4. The task processing method according to claim 2, further comprising, before the monitoring status directory: Before the task is executed, the status directory address of the task is obtained. The monitoring status directory includes: monitoring the status directory based on the status directory address.

5. The task processing method according to claim 1, further comprising: In response to the ratio being greater than the operating baseline value, a target number of resource partitions is requested, the target number being determined by the first number, the second number, and the operating baseline value.

6. The task processing method according to claim 1, further comprising, after sending the status data to the second cluster: A consistency check is performed on the status data in the first cluster and the status data in the second cluster.

7. The task processing method according to claim 6, wherein the consistency check of the state data in the first cluster and the state data in the second cluster includes: Determine the first data size and first hash value of the state data in the first cluster, and the second data size and second hash value of the state data in the second cluster; In response to the first data size being equal to the second data size and the first hash value being equal to the second hash value, it is determined that the verification of the state data in the first cluster and the state data in the second cluster is successful.

8. A task processing method applied to a second cluster, the task processing method comprising: The first cluster is used to monitor whether a cluster-level anomaly occurs in the first cluster, and the first cluster is used to send the status data to the second cluster according to the method of any one of claims 1-7; Upon detecting a cluster-level anomaly in the first cluster, the processing task continues to be executed based on the status data.

9. A task processing method, comprising: The first cluster monitoring status directory is used to store status data of processing tasks; The first cluster responds by detecting newly added status data in the status directory and then acquires the newly added status data. The first cluster obtains a first number and a second number, wherein the first number is the number of status data and the second number is the number of resource partitions in the first cluster; Determine the ratio of the first number to the second number; in response that the ratio is less than or equal to the operating baseline value of the resource load of the resource partition, split the status data according to the resource load of the resource partition to obtain multiple sets of data; store the multiple sets of data in the storage media corresponding to different resource partitions; control different resource partitions to read data in the corresponding storage media, and send the read data and the storage address corresponding to the data to the second cluster respectively, so that the storage addresses of each set of data in the first cluster and the second cluster are the same; The operating baseline value is used to adjust the resource load of the resource partition; The second cluster monitors whether the first cluster experiences cluster-level anomalies. The second cluster responds to the detection of a cluster-level anomaly in the first cluster, and continues to execute the task being processed based on the status data of the task being processed in the first cluster, wherein the status data comes from the first cluster.

10. A medium storing computer-executable instructions, which, when executed by a processor, are used to implement the task processing method as described in any one of claims 1 to 9.

11. A task processing apparatus, applied to a first cluster, comprising: The acquisition module is used to acquire status data during task processing; The sending module is used to send the status data to the second cluster so that when the second cluster detects a cluster-level anomaly in the first cluster, it can continue to execute the processing task based on the status data. Obtain the first number and the second number, where the first number is the number of state data items and the second number is the number of resource partitions in the first cluster; Determine the ratio of the first number to the second number; In response to a ratio less than or equal to the operating baseline value of the resource load of the resource partition, the resource partition is controlled to send the status data to the second cluster, so that when the second cluster detects a cluster-level anomaly in the first cluster, it can continue to execute the processing task based on the status data; the operating baseline value is used to adjust the resource load of the resource partition. When the sending module controls the resource partition to send the status data to the second cluster, it is specifically used to: split the status data according to the resource load of the resource partition to obtain multiple sets of data; and store the multiple sets of data into the storage media corresponding to different resource partitions respectively. The system controls different resource partitions to read data from their corresponding storage media and sends the read data and the corresponding storage address to the second cluster, so that the storage addresses of each group of data in the first cluster and the second cluster are the same.

12. The task processing apparatus according to claim 11, wherein the acquisition module, when acquiring the status data of the task being processed, is specifically configured to: monitor a status directory, the status directory being used to store the status data of the task being processed; in response to the detection of newly added status data in the status directory, acquire the newly added status data; and send the status data to the second cluster, comprising: The newly added status data is sent to the second cluster.

13. The task processing apparatus according to claim 12, wherein the acquisition module is further configured to: determine the path of the newly added status data in the status directory before acquiring the newly added status data; determine the status file corresponding to the newly added status data in the status directory according to the path; perform serialization processing on the status file to obtain a serialized file; wherein when acquiring the newly added status data, the acquisition module is specifically configured to: acquire the serialized file; perform deserialization processing on the serialized file to obtain the newly added status data.

14. The task processing apparatus according to claim 13, wherein the acquisition module is further configured to: acquire the address of the status directory of the task being processed before the monitoring status directory is established, and before the task being processed is executed; the monitoring status directory includes: Monitor the status directory based on the status directory address.

15. The task processing apparatus according to claim 11, further comprising: The application module is used to apply for a target number of resource partitions in response to the ratio being greater than the operating baseline value, wherein the target number is determined by the first number, the second number, and the operating baseline value.

16. The task processing apparatus according to claim 11, wherein the sending module is further configured to: after sending the status data to the second cluster, perform a consistency check on the status data in the first cluster and the status data in the second cluster.

17. The task processing apparatus according to claim 16, wherein when the sending module performs consistency verification on the status data in the first cluster and the status data in the second cluster, it is specifically configured to: determine the first data size and the first hash value of the status data in the first cluster, and the second data size and the second hash value of the status data in the second cluster; and, in response to the first data size being equal to the second data size and the first hash value being equal to the second hash value, determine that the verification of the status data in the first cluster and the status data in the second cluster is successful.

18. A task processing apparatus, applied to a second cluster, comprising: The monitoring module is used to monitor whether the first cluster generates a cluster-level anomaly, and the first cluster is used to send the status data to the second cluster according to the method described in any one of claims 1-7; The execution module is used to respond to the detection of a cluster-level anomaly in the first cluster and, based on the status data, continue to execute the task being processed.

19. A task processing apparatus, comprising: The first control module is used to control the first cluster to perform the following steps: monitoring the status directory, which is used to store status data of processing tasks; Upon detecting newly added status data in the status directory, the system acquires a first number and a second number, where the first number represents the number of status data items and the second number represents the number of resource partitions in the first cluster. It then determines the ratio of the first number to the second number. If the ratio is less than or equal to the operating baseline value of the resource load of the resource partition, the system splits the status data according to the resource load of the resource partition, obtaining multiple sets of data. These multiple sets of data are stored in storage media corresponding to different resource partitions. The system controls different resource partitions to read data from their respective storage media and sends the read data and its corresponding storage address to the second cluster, ensuring that the storage addresses of each set of data in the first cluster and the second cluster are the same. The operating baseline value is used to adjust the resource load of the resource partitions. The second control module is used to control the second cluster to perform the following steps: monitor whether the first cluster generates a cluster-level anomaly; Upon detecting a cluster-level anomaly in the first cluster, the system continues to execute the task being processed based on the status data of the task being processed in the first cluster, where the status data originates from the first cluster.

20. A task processing system, comprising: The first cluster is configured to execute the task processing method according to any one of claims 1 to 7; The second cluster is used to execute the task processing method of claim 8.

21. A computing device, comprising: A processor, and a memory connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the task processing method as described in any one of claims 1 to 8.

Images