Classified recovery method, device and system for task faults
By employing a tiered recovery method in the distributed computing architecture, and selecting engine terminals for task recovery step by step between the same central data center and different central data centers, the problem of insufficient fault recovery rate during task execution in the distributed computing architecture is solved, achieving high availability and efficient fault recovery.
Patent Information
- Application Number
- CN202411108194.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-08
AI Technical Summary
In a distributed computing architecture, existing technologies struggle to effectively improve fault recovery rates during task execution, resulting in insufficient high availability for system request execution.
By implementing a tiered recovery method for task failures at the service platform level, the first step is to select engine terminals in different clusters within the same central data center to perform task recovery. If recovery still fails, the next step is to select engine terminals in different central data centers to continue recovery, thus improving the recovery rate through a tiered task recovery approach.
It improves the fault recovery rate during task execution in a distributed computing architecture, makes task execution highly available, reduces fault recovery time, and improves system reliability and efficiency.
Smart Images

Figure CN118916206A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present specification relate to the field of computer technology, and more particularly, to a hierarchical recovery method, device, and system for task failure. Background Art
[0002] Against the backdrop of the rapid development of content, especially video, service platforms are also undergoing transformation and upgrading towards content and video. In this context, the amount of computation required for task computations on various data in the service platform has also increased significantly. Therefore, service platforms usually use a distributed engine computing architecture to implement the computation of multiple tasks. In the process of task computation, service platforms need to improve the efficient and high-quality processing capabilities of tasks while protecting data privacy. In particular, for the processing of real-time tasks of streaming media data, in order to cope with possible failures and abnormal situations, the streaming media data computing engine needs to design a fault-tolerant mechanism.
[0003] At present, it is hoped that there will be an improved solution to improve the failure recovery rate during task execution under the distributed computing architecture, so that the system request execution has high availability. Summary of the invention
[0004] One or more embodiments of this specification describe a hierarchical recovery method, device and system for task failures to solve the problem of improving the failure recovery rate during task execution in a distributed computing architecture so that system request execution has high availability. The specific technical solution is as follows.
[0005] In a first aspect, an embodiment provides a hierarchical recovery method for a task failure, which is executed through a service platform and includes:
[0006] When it is determined that the first task executed in the first engine terminal meets the first retry condition, a second engine terminal capable of executing the first task is determined in the second cluster included in the first central computer room, and the first task is re-executed in the second engine terminal; the first engine terminal belongs to the first cluster included in the first central computer room, and the second cluster is different from the first cluster;
[0007] When it is determined that the first task executed in the second engine terminal meets the second retry condition, a third engine terminal capable of executing the first task is determined in a second central computer room, and the first task is re-executed in the third engine terminal; the second central computer room is different from the first central computer room.
[0008] In one embodiment, the step of determining the second engine terminal capable of executing the first task includes:
[0009] When the tenant who submits the first task is set with a resource pool, a second cluster is determined from the clusters of the first central computer room included in the resource pool; the resource pool includes a cluster having the service capability required by the tenant.
[0010] In one embodiment, the step of re-executing the first task in the second engine terminal includes: sending task information of the first task to the second engine terminal, so that the second engine terminal re-executes the first task based on the task information and an identifier carried by the task information for indicating re-execution of the task.
[0011] In one implementation, the first task is a stream data task, and the task information carries the storage address of the input stream data, the storage address of the output stream data, and task processing parameters.
[0012] In one implementation, when it is determined that the first task executed in the first engine terminal meets the first retry condition, the method further includes: stopping execution of the first task in the first engine terminal.
[0013] In one embodiment, before determining that the first task satisfies the first retry condition, the method further includes: determining a query strategy and a timeout strategy for the first task; querying the state of the first task in the first engine terminal based on the query strategy;
[0014] The step of determining whether the first task executed in the first engine terminal satisfies a first retry condition includes: when the state of the first task in the first engine terminal reaches an abnormal state and its abnormal state reaches the timeout policy, determining that the first task satisfies the first retry condition.
[0015] In one embodiment, the method further includes: determining a retry strategy for the first task, the retry strategy including a first number of times the first task is allowed to be retried in the same central computer room;
[0016] The step of determining that the first task satisfies the first retry condition includes: when the state of the first task in the first engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries does not exceed the first number, determining that the first task satisfies the first retry condition.
[0017] In one embodiment, the step of determining that the first task executed in the second engine terminal satisfies the second retry condition includes: when the state of the first task in the second engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries exceeds the first number, determining that the first task satisfies the second retry condition.
[0018] In one embodiment, the method also includes: when the status of the first task in the second engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries does not exceed the first number, determining a second engine terminal capable of executing the first task in other clusters included in the first central computer room, and returning to execute the step of re-executing the first task in the second engine terminal.
[0019] In a second aspect, an embodiment provides a hierarchical recovery method for a task failure, which is executed by an engine terminal, wherein the engine terminal includes a first engine terminal, a second engine terminal, and a third engine terminal, and a first central computer room includes a first cluster and a second cluster. The first engine terminal belongs to the first cluster, the second engine terminal belongs to the second cluster, and the third engine terminal belongs to the second central computer room; the method includes:
[0020] The second engine terminal receives the first task to be re-executed issued by the service platform, and re-executes the first task; wherein the first task is issued by the service platform when it is determined in the second cluster that the second engine terminal can execute the first task when the service platform determines that the first task executed in the first engine terminal meets the first retry condition;
[0021] The third engine terminal receives the first task to be re-executed issued by the service platform, and re-executes the first task; wherein, the first task is issued by the service platform when it is determined in the second central computer room that the third engine terminal is able to execute the first task when it is determined that the first task executed in the second engine terminal meets the second retry condition.
[0022] In one embodiment, the first engine terminal includes a management component and an execution component, wherein the execution component includes a process for running the first task, and the process includes a plurality of threads; the method further includes:
[0023] When there is a first thread in the plurality of threads with an abnormal state, the execution component restarts the first thread;
[0024] When the state of the first thread is still abnormal after the restart, the execution component reports the abnormal state of the first task to the management component;
[0025] The management component restarts the process upon receiving the abnormal state reported by the execution component;
[0026] When the state of the process is still abnormal after the restart, the management component reports the abnormal state of the first task to the service platform, so that the service platform determines whether the first task satisfies the first retry condition.
[0027] In one embodiment, the first engine terminal includes a plurality of physical machines; the step of the management component reporting the abnormal state of the first task to the service platform includes:
[0028] When the state of the process is still abnormal after the restart, the management component rebuilds and starts the process for running the first task in other physical machines included in the first engine terminal;
[0029] When the state of the first task in the reconstructed process is still abnormal, the management component reports the abnormal state of the first task to the service platform.
[0030] In one embodiment, when the second engine terminal re-executes the first task, the method also includes: when the execution component in the second engine terminal reports the abnormal status of the first task to the management component in the second engine terminal, the management component in the second engine terminal directly reports the abnormal status of the first task to the service platform, so that the service platform determines whether the first task meets the second retry condition.
[0031] In a third aspect, an embodiment provides a hierarchical recovery system for task failures, including a service platform, a first engine terminal in a first cluster, a second engine terminal in a second cluster, and a third engine terminal in a second central computer room; the first cluster and the second cluster belong to the first central computer room;
[0032] The service platform is configured to, when determining that the first task executed in the first engine terminal meets a first retry condition, determine a second engine terminal in the second cluster that can execute the first task, and send the first task to be re-executed to the second engine terminal;
[0033] a second engine terminal, configured to receive the first task sent by the service platform and re-execute the first task;
[0034] The service platform is configured to determine a third engine terminal capable of executing the first task in the second central computer room when it is determined that the first task executed in the second engine terminal meets a second retry condition, and send the first task to be re-executed to the third engine terminal;
[0035] The third engine terminal is used to receive the first task sent by the service platform and re-execute the first task.
[0036] In a fourth aspect, the embodiment provides a hierarchical recovery device for task failures, which is deployed in a service platform and includes:
[0037] A first recovery module is configured to, when it is determined that the first task executed in the first engine terminal meets the first retry condition, determine a second engine terminal in the second cluster included in the first central computer room that can execute the first task, and re-execute the first task in the second engine terminal; the first engine terminal belongs to the first cluster included in the first central computer room, and the second cluster is different from the first cluster;
[0038] The second recovery module is configured to determine a third engine terminal capable of executing the first task in a second central computer room when it is determined that the first task executed in the second engine terminal meets a second retry condition, and re-execute the first task in the third engine terminal; the second central computer room is different from the first central computer room.
[0039] In a fifth aspect, an embodiment provides a hierarchical recovery device for a task failure, which is deployed in an engine terminal, wherein the engine terminal includes a first engine terminal, a second engine terminal, and a third engine terminal, wherein a first central computer room includes a first cluster and a second cluster, wherein the first engine terminal belongs to the first cluster, the second engine terminal belongs to the second cluster, and the third engine terminal belongs to the second central computer room; and the device deployed in the second engine terminal includes:
[0040] a first receiving module configured to receive a first task to be re-executed issued by a service platform; a first executing module configured to re-execute the first task; wherein the first task is issued by the service platform when it is determined in the second cluster that the second engine terminal is capable of executing the first task when the service platform determines that the first task executed in the first engine terminal meets a first retry condition;
[0041] The device deployed in the third engine terminal includes:
[0042] a second receiving module configured to receive the first task to be re-executed issued by the service platform to the third engine terminal; a second executing module configured to re-execute the first task; wherein the first task is issued by the service platform when it is determined in the second central computer room that the third engine terminal is able to execute the first task when the service platform determines that the first task executed in the second engine terminal meets the second retry condition.
[0043] In a sixth aspect, an embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any one of the methods in the first aspect and the second aspect.
[0044] In a seventh aspect, an embodiment provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the first aspect and the second aspect is implemented.
[0045] In the method and device provided in the embodiments of this specification, when the status of a task is abnormal and needs to be retried, the service platform can gradually select engine terminals from different clusters in the same central computer room and from different central computer rooms to restore the task. This improves the fault recovery rate during task execution under the distributed computing architecture, making the execution of the task highly available. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0047] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;
[0048] Figure 2 A schematic diagram of the logical relationship between resource pools, clusters, and engine terminals;
[0049] Figure 3 A flowchart of a hierarchical recovery method for a task failure provided by an embodiment;
[0050] Figure 4 A schematic diagram of the internal structure of the engine terminal;
[0051] Figure 5 A schematic diagram of the internal structure of the execution component;
[0052] Figure 6 A schematic block diagram of a hierarchical recovery system for task failures provided in an embodiment;
[0053] Figure 7 A schematic block diagram of a hierarchical recovery device for a task failure provided by an embodiment;
[0054] Figure 8 A schematic block diagram of another hierarchical recovery device for task failure provided in an embodiment. DETAILED DESCRIPTION
[0055] The solution provided in this specification is described below in conjunction with the accompanying drawings.
[0056] Figure 1 This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. It includes a service platform and multiple engine terminals. The multiple engine terminals belong to the central computer room SH and the central computer room BJ respectively, among which only the engine terminal in the central computer room SH is shown, and the engine terminal in the central computer room BJ is similar to that in the central computer room SH and is omitted here. In the same central computer room, multiple engine terminals are also divided into several clusters. For example, engine terminals 1, 2 and 3 belong to cluster A, engine terminals 4, 5 and 6 belong to cluster B, and the engine terminal in cluster C is omitted and not drawn, and its engine terminal is similar to that in clusters A and B. Figure 1 This is just an example of a distributed computing platform. The actual distributed computing architecture can be Figure 1 different.
[0057] Among them, the service platform is used to provide services to tenants. A service is an online running entity that can provide an application programming interface (API) / software development kit (SDK) to support external access and use. A tenant represents a business group, generally composed of several students or groups. The services provided by the service platform to tenants include video, voice, image and other services that require large amounts of data calculation. Tenants can submit tasks to the service platform, and the data objects processed by the task may include streaming data (video data, voice data, and audio and video data), images, and user-related data.
[0058] A cluster is a physical unit to which tasks are submitted. An engine is a computing engine that actually runs in a cluster. The engine is responsible for running tasks. An engine is an abstraction of an engine terminal. When an engine has specific business functions, it is called an engine terminal. An engine terminal is an entry service for a cluster, through which tasks are submitted to the corresponding cluster. An engine terminal corresponds to a virtual space (namespace) in a cluster. Each cluster can contain one or more virtual spaces, and each namespace runs an engine terminal, that is, each cluster can contain several engine terminals. An engine terminal is implemented by one or more physical machines. A physical machine usually refers to an actual hardware device, that is, a real computer.
[0059] The central computer room contains a large number of physical machines, which are allocated to the virtual space of the cluster. Generally, the central computer room is set up by region, such as the Shanghai central computer room, the Beijing central computer room, etc.
[0060] Usually, tenants submit task requests to the service platform to use the various services provided by the service platform. Task requests are used to describe one or more tasks. Generally, one request corresponds to one task. Under the same "service type", one request can also correspond to multiple tasks. For the sake of convenience, we take one request corresponding to one task as an example.
[0061] When tenants submit tasks, the service platform will perform calculations and provide services within the resource pool set by the tenant. That is, the service platform supports multi-tenancy with independent logical resource pools and real resources. Different businesses enable different clusters through service activation and registration, and the corresponding cluster is a collection of instances that provide the same service.
[0062] The services provided by the service platform are oriented to the underlying engines with multiple technology stacks and different technical modes. Therefore, the service platform defines three core concepts for abstracting resources, namely resource pools, clusters, and engines. A resource pool is a logical grouping of clusters. The same resource pool has multiple homogeneous clusters. Homogeneous clusters refer to clusters with the same or similar service capabilities. If a resource pool contains N clusters, each cluster contains M engine terminals, and each engine terminal has a service capability, then this resource pool is essentially N clusters with M capabilities, where M and N are integers greater than 0. The following combines Figure 2 Provide explanation.
[0063] Figure 2 The figure is a logical relationship diagram of resource pools, clusters and engine terminals. Different resource pools, such as resource pools a, b and c, contain corresponding clusters respectively. Taking resource pool b as an example, resource pool b contains cluster A and cluster B. Cluster A contains engine terminals 1, 2 and 3, and cluster B contains engine terminals 4, 5 and 6. Taking video services as an example, the service platform can provide products in the form of content such as live broadcast and on-demand. The data object processed by the engine terminal is the video stream, which is also the data object of the task. The services provided by the service platform may include real-time stream slicing, real-time stream screenshot and live voice recognition. The stream or stream data includes video stream or audio stream or audio and video stream. The engine terminal used to provide real-time stream slicing service has the real-time stream slicing service capability, the engine terminal used to provide real-time stream screenshot service has the real-time stream screenshot service capability, and the engine terminal used to provide live voice recognition service has the live voice recognition service capability. The engine terminals 1, 2 and 3 in cluster A have the above three capabilities respectively, and the engine terminals 4, 5 and 6 in cluster B have the above three capabilities respectively. Cluster A and cluster B have the same or similar service capabilities and belong to homogeneous clusters.
[0064] When using the services provided by the service platform, tenants will choose to configure the service capabilities they need, thereby obtaining the corresponding resource pool. Initially, when a tenant submits a service request to the service platform, for example, when a host requests a live broadcast service from the service platform, the service platform will perform the following determination operations:
[0065] Configure resource pools based on tenants' choices → Filter clusters that meet resource requirements → Filter engines that meet the type → Further select engine terminals based on weights.
[0066] Among them, resources include service capabilities, computing capabilities, etc. Weights include region, load, etc. For example, you can choose based on the central computer room where the engine terminal is located. The N clusters contained in a resource pool can belong to different central computer rooms, that is, tenants are allowed to select clusters across regions, and the physical machines contained in a cluster usually belong to only the same central computer room.
[0067] When it is necessary to explain, the above service request can be understood as live broadcast service or on-demand service, etc. When a tenant submits a live broadcast service to the service platform, the service platform can receive the tenant's live video and transmit the live video to the video distribution network, and send it to the tenant who requested the live video through the video distribution network. When a tenant requests a live video on demand from the service platform, the service platform will send the live video to the tenant through the video distribution network.
[0068] The above organizational relationship between resource pools, clusters, and engines is only a specific division of a business application. In actual applications, there may be different divisions. For example, multiple engine terminals in a cluster may have the same service capabilities, or different clusters in the same resource pool may not be homogeneous clusters, but clusters with different service capabilities, etc. In short, the resource pool contains clusters with the service capabilities required by tenants. When a tenant submits a task request to the service platform, the service platform selects the corresponding engine terminal from the resources contained in the resource pool to perform task processing.
[0069] When a task executed in an engine terminal fails, in order to ensure high availability of services, the task is usually recovered. For example, the task can be restarted in the original engine terminal or by replacing the engine terminal. This fault recovery method can recover the task to a certain extent. However, in the context of large-scale data processing, the number of tenants is huge, the number of tasks is huge, and the number of faulty tasks will also increase. Sometimes the original fault recovery method cannot resolve the fault.
[0070] In order to improve the fault recovery rate during task execution under a distributed computing architecture and make the execution of tasks highly available, the embodiments of this specification provide a hierarchical recovery method for task faults, that is, when the engine terminal cannot recover the task, the task is recovered hierarchically through the service layer, that is, first select engine terminals in different clusters to recover the task, and when the task still cannot be recovered, select engine terminals in different central computer rooms to continue to recover the task. This hierarchical and step-by-step task recovery method can ensure the recovery of tasks as much as possible and improve the task recovery rate. Figure 3 This embodiment is described.
[0071] Figure 3A flowchart of a hierarchical recovery method for a task failure provided in an embodiment. The method is executed by a service platform and multiple engine terminals in the system. The engine terminal, cluster and central computer room used in the method are described in a logical sense below. The engine terminal includes several engine terminals such as a first engine terminal, a second engine terminal and a third engine terminal, each engine terminal includes several physical machines, and the central computer room includes several central computer rooms such as a first central computer room SH and a second central computer room BJ. The first central computer room SH includes several clusters such as a first cluster A and a second cluster B, the first engine terminal belongs to the first cluster A, the second engine terminal belongs to the second cluster B, and the third engine terminal belongs to the second central computer room BJ. The second cluster B is a different cluster from the first cluster A. The first central computer room SH and the second central computer room BJ are different central computer rooms. In order to more clearly explain the recovery process of the first task, the initial issuance process of the first task and the status query process when executed in the first engine terminal are first described below.
[0072] Initially, the service platform receives a task request sent by a tenant, which carries the first task Task1 to be executed. The first task Task1 can be any task initiated by the tenant. Then, the service platform determines the engine terminal used to execute the first task Task1, that is, the first engine terminal, based on the resource pool configured by the tenant in an existing manner (such as the aforementioned determination operation). The first engine terminal executes the first task Task1 and feeds back the task status of the first task Task1 to the service platform.
[0073] When receiving a task request, the service platform can parse the tenant and service type data from the task request, and query the service level value (Service Level Agreement, SLA) and policy id (identifier) corresponding to the task based on the tenant and service type data. Strategies include query strategies, timeout strategies, and retry strategies. Service type data includes the type of service, such as real-time stream segmentation service, real-time stream screenshot, live voice recognition, etc.
[0074] The engine layer is implemented through a general distributed architecture, which can be implemented based on the Actor model. It is mainly implemented by a management component Coordinator and multiple execution components Executor. The management component performs specific task distribution and process scheduling based on the multiple tasks received, while the execution component Executor performs multi-threaded execution by parsing a domain specific language (DSL) internally.
[0075] Figure 4This is a schematic diagram of the internal structure of the engine terminal. The first engine terminal includes a management component Coordinator1 and several execution components such as Executor1 and Executor2. One execution component can be used to execute a task, and multiple execution components can be used to execute multiple tasks respectively. For example, the execution component Executor1 is used to execute the first task Task1, that is, it includes a process running the first task Task1. The task process can also include several threads.
[0076] Figure 5 A schematic diagram of the internal structure of the execution component. For example, the execution component Executor1 of the first task Task1 includes the main thread, task scheduling, and threads such as input, output, encoding, and model reasoning. Input, output, encoding, and model reasoning are subtasks in the first task Task1. The execution component Executor2 of the second task Task2 includes the main thread, task scheduling, and threads such as download, upload, encoding, and image segmentation. Download, upload, encoding, and image segmentation are subtasks in the second task Task2.
[0077] Different tasks are used to process data (such as streaming data) differently. The main thread schedules the threads of each subtask through task scheduling, which includes obtaining the status of the subtask, restarting the subtask, etc. The task status or subtask status includes running, stopped, and abnormal. The main thread can sense the status of the first task Task1, and feed back the task status to the management component Coordinator1, and the management component Coordinator1 feeds back the task status to the service platform.
[0078] The service platform can query the status of the first task Task1 in the first engine terminal based on the query strategy parsed from the task request. In this way, the service platform and the first engine terminal establish a status query mechanism, and the service platform can perceive the status of the first task, including abnormal status.
[0079] The reason for the abnormal state of the first task Task1 may be accidental, or it may be a physical machine failure or a network failure. In order to reduce the abnormal recovery time and improve the timeliness of task recovery, the first engine terminal can start thread or process level recovery, that is, the recovery in steps S310 and S320. This abnormal recovery takes a short time. Firstly, thread or process level recovery is performed instead of directly performing task recovery through the service platform, which can significantly save the time of task recovery.
[0080] Step S310: when the first thread in the first task Task1 is abnormal, restart the first thread.
[0081] The first thread is any one or more threads among the threads included in the first task Task1. When a subtask is in an abnormal state, the corresponding thread will feed back the abnormal state to the main thread. The main thread can detect the state of each subtask, that is, each subthread. When the main thread in the execution component Executor1 senses that the first thread is in an abnormal state, the main thread can restart the first thread through the task scheduling thread. When the first thread is back in operation, the normal state of the task can be fed back to the main thread.
[0082] Restarting is not rebuilding the thread, and restarting the thread takes a shorter time. The execution component automatically uses the backoff method to retry the task thread, which can avoid task status abnormalities caused by accidental reasons. In addition, this task status abnormality will not be fed back to the management component Coordinator1, and thus will not be fed back to the service platform, but will only be digested inside the first engine terminal.
[0083] Step S320: When the state of the first thread is still abnormal after the restart, the execution component Executor1 reports the abnormal state of the first task to the management component Coordinator1. The management component Coordinator1 restarts the process of the first task upon receiving the abnormal state reported by the execution component Executor1.
[0084] After the restart, the state of the first thread is still abnormal, including that the state of the first thread has not been restored after the restart operation is performed on the first thread, and that the first thread cannot be restarted after the restart operation is performed on the first thread. In short, the first thread has not reached a normal operating state. In this case, the execution component Executor1 will request the upper level to recover the task, that is, to recover the task through the management component Coordinator1. Before performing the recovery in this step, the management component Coordinator1 will not report the abnormal state of the first task Task1 to the service layer, but will only digest it inside the first engine terminal.
[0085] When the management component Coordinator1 restarts the first task Task1, it will rebuild the threads of all subtasks in the process corresponding to the first task Task1, run the rebuilt threads, and destroy the threads of the original subtasks in the process. The management component Coordinator1 can restart the process through Task DAG / Pipeline. Restarting a process does not mean rebuilding the process. For example, the main thread does not need to be rebuilt. Restarting a process takes less time than rebuilding a process. When the task status returns to normal after the process is restarted, the execution component Executor1 will feedback the normal status to the management component Coordinator1.
[0086] Next, in step S330, when the state of the process is still abnormal after the restart, the management component Coordinator1 rebuilds the process for running the first task in other physical machines included in the first engine terminal, and starts the rebuilt process.
[0087] After the restart, the state of the process is still abnormal, including that the task state is not restored after the restart operation is performed on the process, and that the process cannot be restarted after the restart operation is performed on the process. In short, the task state has not reached the normal operating state. In this case, the management component Coordinator1 will perform the next level of task recovery. Before performing the recovery in this step, the management component Coordinator1 will not report the abnormal state of the first task Task1 to the service layer, but only digest it inside the first engine terminal.
[0088] The first engine terminal is a virtual space namespace in the first cluster, which includes several physical machines. The management component Coordinator1 is used to manage these physical machines. The management component Coordinator1 can select a new physical machine from these physical machines, rebuild and start the newly created process to execute the first task Task1. When selecting a physical machine, you can choose it based on the load situation of each physical machine and the principle of load balancing. The management component Coordinator1 can be rebuilt in the form of TaskDAG / Pipeline. Rebuilding the process of the first task Task1 in the new physical machine is essentially the same execution process as resubmitting the task.
[0089] The first engine terminal can be used to execute multiple tasks, that is, there are multiple execution components, each of which runs a process of a task, and multiple execution components can correspond to multiple physical machines respectively. Among them, one execution component usually runs in one physical machine, and one physical machine can run multiple execution components.
[0090] The above steps S310 to S330 correspond to the multi-level task recovery methods of restarting threads, restarting processes, and replacing physical machines (i.e. rebuilding processes), respectively. These task recoveries are all executed inside the first engine terminal, without the involvement of the service platform, and the fault recovery process is relatively fast. With each level of fault recovery, the time overhead and cost will increase. However, after the above three levels of task recovery, the task status is still abnormal. In this case, the task can be restored through the scheduling of the service platform so as to achieve the recovery of the first task as much as possible.
[0091] When the state of the first task Task1 is still abnormal after the process is rebuilt in other physical machines in step S330, the management component Coordinator1 may report the abnormal state of the first task Task1 to the service platform.
[0092] For the service platform, it faces a large number of tasks running in a large number of engine terminals. The service platform can store the task-related data of a large number of tasks in the database as persistent data. Task-related data may include the tenant corresponding to the task, service type data, SLA value, policy id, etc., as well as the task status. The task status in each engine terminal can be sent to the service platform through task status asynchronous notification. The service platform is also used to record task operation events, including task submission, re-execution, and timeout. The service platform can enable the TaskInquisitor thread, which is used to filter out the current list of tasks to be tracked from the persistent data. The service platform can query all tasks in non-terminal states and abnormal states based on the SLA value and query strategy, and add them to the current list of tasks to be tracked. The service platform performs recovery operations on tasks in the task list that meet the conditions and have abnormal states.
[0093] That is, in step S340, when the service platform determines that the first task Task1 executed in the first engine terminal meets the first retry condition, the service platform determines a second engine terminal in the second cluster B included in the first central computer room SH that can execute the first task Task1, and sends the first task Task1 to be re-executed to the second engine terminal. The second engine terminal receives the first task Task1 to be re-executed issued by the service platform, and re-executes the first task Task1.
[0094] The service platform can determine the query strategy and timeout strategy for the first task Task1, and query the status of the first task in the first engine terminal from the persistent data based on the query strategy. Tasks with different SLA levels have different query strategies and timeout strategies. The query strategy defines the frequency of queries, and the timeout strategy defines the retry countdown after the task is abnormal. When the first task Task1 times out, it means that the task or process has not completed its execution within the predetermined time.
[0095] As mentioned above, the status of a task includes executing, stopped, and abnormal. Task abnormality usually means that the task has not been completed and the engine terminal system automatically interrupts or terminates the task. Abnormalities can be detected through a timeout mechanism. SLA levels include high-security and low-security categories. For high-security categories, the query frequency will be higher and the retry countdown will be shorter, so as to achieve faster fault recovery for high-security services.
[0096] The service platform determines whether the first retry condition is met based on the abnormal state of the first task Task1, that is, when the state of the first task Task1 in the first engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, it is determined that the first task Task1 meets the first retry condition. The first retry condition is determined based on the abnormal state of the task, the timeout countdown duration, etc.
[0097] Retrying means recreating a task process to execute the task. In the service platform, retrying means changing different engine terminals to execute the task. In the engine layer, retrying can be restarting a process or thread. For the second engine terminal, the task processing process of receiving the first task Task1 reissued by the service platform is the same as that of the newly issued task, but the recovery process of the task failure can be slightly different.
[0098] The service platform determines an engine terminal capable of executing the first task Task1 in a different cluster in the same central computer room as the first engine terminal as the second engine terminal.
[0099] It should be noted that when it is determined that the first task Task1 executed in the first engine terminal meets the first retry condition, it is also necessary to stop executing the first task Task1 in the first engine terminal, release the resources in the first engine terminal, and avoid task conflicts caused by the task being executed in another engine terminal.
[0100] When the first task Task1 times out, the first engine terminal system may automatically interrupt or terminate the first task Task1, or the service platform may stop executing the first task Task1 in the first engine terminal.
[0101] When determining the second engine terminal, when the tenant who submits the first task Task1 is set with a resource pool, the resource pool contains a cluster with the service capabilities required by the tenant. The service platform can determine the second cluster B in the cluster of the first central computer room SH included in the resource pool, and determine the second engine terminal capable of executing the first task Task1 in the second cluster B.
[0102] When determining the second engine terminal from the second cluster B, selection can be made according to existing methods, for example, selection can be made in combination with the service capability and load of the terminal engine, etc. When a cluster is selected according to a resource pool set by a tenant, an available cluster can be selected more accurately.
[0103] When the service platform sends the re-executed first task Task1 to the second engine terminal, it can specifically send the task information of the first task Task1 to the second engine terminal. The second engine terminal can re-execute the first task based on the received task information and the identifier carried by it for indicating the re-execution of the task. The task information of the re-executed first task is different from that of the initially executed first task, and the difference includes the presence of a restart identifier indicating re-execution in the task information of the re-executed first task, and other information can be the same as the task information of the initially executed first task.
[0104] In a multimedia scenario, when the first task is a stream data task, the task information also carries a storage address of the input stream data, a storage address of the output stream data, and a task processing parameter.
[0105] The first task Task1 of this stage is issued by the service platform when it is determined that the first task Task1 executed in the first engine terminal meets the first retry condition and when it is determined in the second cluster B that the second engine terminal can execute the first task Task1.
[0106] When the second engine terminal receives the first task Task1 sent by the service platform, it can obtain the stream data to be processed based on the storage address of the input stream data in the task information, process the stream data based on the task processing parameters, and store the processed stream data in the storage address of the output stream data. The same operation is performed when other engine terminals re-execute the first task Task1.
[0107] The second engine terminal may also include a management component Coordinator2 and several execution components such as Executor3 and Executor4. For example, the execution component Executor3 is used to execute the first task Task1, that is, it includes a process for running the first task Task1. The task process may include several threads.
[0108] When the second engine terminal re-executes the first task, when the execution component Executor3 in the second engine terminal reports the abnormal status of the first task Task1 to the management component Coordinator2 in the second engine terminal, the management component Coordinator2 directly reports the abnormal status of the first task Task1 to the service platform, so that the service platform determines whether the first task Task1 meets the second retry condition based on the abnormal status of the first task Task1.
[0109] In this embodiment, when a task failure occurs, since the second engine terminal re-executes the first task Task1, the management component Coordinator2 does not restart the process or replace the physical machine, but directly enters the next level of recovery. This can avoid the situation of rescheduling to the first cluster A. This reflects the difference between the fault recovery process when re-executing a task and the fault recovery process of a new task.
[0110] If the first task Task1 is successfully executed in the second terminal engine, if a status abnormality occurs again later, the second engine terminal can start execution from the first layer recovery in step S310.
[0111] In another embodiment, the recovery process of step S340 can be performed multiple times. When determining the task-related data, the service platform can also determine a retry strategy for the first task Task1, and the retry strategy includes the first number of times the first task Task1 is allowed to be retried in the same central computer room. Re-executing a task in a cluster is recorded as a retry, and the first number defines how many clusters can be attempted to recover in the same central computer room. The retry strategy can be set by the tenant or determined by the service platform based on the SLA value.
[0112] In this embodiment, the service platform determines that the first task Task1 meets the first retry condition when the following conditions are met: the state of the first task Task1 in the first engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries does not exceed the first number. Whenever a task is retried in the first central computer room SH, the current number of retries increases by one.
[0113] When the first task Task1 is re-executed in the second engine terminal, the service platform can also obtain the task status of the first task Task1. When the status of the first task Task1 in the second engine terminal reaches an abnormality, and its abnormal status reaches the timeout policy, and the current number of retries does not exceed the first number, the service platform can determine the second engine terminal that can execute the first task Task1 in other clusters included in the first central computer room SH, and re-execute the first task Task1 in the second engine terminal, that is, repeat step S340. Other clusters are clusters other than the first cluster A and the second cluster B, or clusters other than the clusters that have executed the first task.
[0114] Here, in terms of the naming rules of concepts, in order to avoid conceptual complexity, the engine terminals that are determined in the first central machine room SH and can execute the first task Task1 are all called second engine terminals.
[0115] When the current number of retries exceeds the first number, the service platform determines that the first task Task1 meets the second retry condition and needs to replace the central computer room for fault recovery.
[0116] That is, in step S350, when the service platform determines that the first task Task1 executed in the second engine terminal meets the second retry condition, the service platform determines the third engine terminal that can execute the first task Task1 in the second central computer room BJ, and sends the first task Task1 to be re-executed to the third engine terminal. The third engine terminal receives the first task Task1 to be re-executed issued by the service platform, and re-executes the first task Task1. The second central computer room SH is different from the first central computer room BJ. At this time, the first task Task1 is issued by the service platform when it is determined that the first task Task1 executed in the second engine terminal meets the second retry condition and when it is determined in the second central computer room BJ that the third engine terminal can execute the first task Task1.
[0117] According to the naming convention, the engine terminal determined in other computer rooms (including the second central computer room BJ) except the first central computer room SH is called the third engine terminal.
[0118] The service platform may determine a query strategy and a timeout strategy for the first task Task1, and query the state of the first task in the second engine terminal from the persistent data based on the query strategy.
[0119] The service platform determines whether the second retry condition is met based on the abnormal state of the first task Task1, that is, when the state of the first task Task1 in the second engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, it is determined that the first task Task1 meets the second retry condition. The second retry condition is determined based on the abnormal state of the task, the timeout countdown duration, etc.
[0120] When it is determined that the first task Task1 executed in the second engine terminal meets the second retry condition, it is also necessary to stop executing the first task Task1 in the second engine terminal to release resources in the second engine terminal and avoid task conflicts caused by the task being executed in another engine terminal.
[0121] When the first task Task1 times out, the first engine terminal system may automatically interrupt or terminate the first task Task1, or the service platform may stop executing the first task Task1 in the first engine terminal.
[0122] When determining the third engine terminal, when the tenant who submits the first task Task1 is set with a resource pool, the resource pool contains a cluster with the service capabilities required by the tenant. The service platform can determine the third cluster D in the cluster of the second central computer room BJ included in the resource pool, and determine the third engine terminal capable of executing the first task Task1 in the third cluster D.
[0123] When determining the third engine terminal from the third cluster D, it can be selected according to existing methods, for example, it can be selected in combination with the service capability and load of the terminal engine, etc. When selecting a cluster according to the resource pool set by the tenant, an available cluster can be selected more accurately.
[0124] When the service platform sends the re-executed first task Task1 to the third engine terminal, the task information of the first task Task1 can be sent to the third engine terminal. The third engine terminal can re-execute the first task Task1 based on the received task information and the identifier carried by it for indicating the re-execution of the task. The task information of the re-executed first task Task1 is different from the task information of the initially executed first task Task1, and the difference includes the presence of a restart identifier indicating re-execution in the task information of the re-executed first task, and other information can be the same as the task information of the initially executed first task.
[0125] The third engine terminal may also include a management component and several execution components. When the execution component for executing the first task in the second engine terminal reports the abnormal state of the first task Task1 to the management component, the management component directly reports the abnormal state of the first task Task1 to the service platform, so that the service platform determines whether the first task Task1 meets the retry condition based on the abnormal state of the first task Task1.
[0126] In this embodiment, when the first task state is abnormal, the management component in the second engine terminal does not restart the process or replace the physical machine, but directly enters the next recovery, so as to avoid the situation of rescheduling to the same cluster.
[0127] In another embodiment, the recovery process of step S350 can be performed multiple times. The above-mentioned retry strategy can also include parameters such as the maximum total number of retries, the number of retries in the same computer room, and the number of central computer rooms in which retries are allowed.
[0128] When the first task Task1 is re-executed in the third engine terminal, the service platform can also obtain the task status of the first task Task1. When the status of the first task Task1 in the third engine terminal reaches an abnormality, and its abnormal status reaches the timeout policy, and the current number of retries does not exceed the first number, the service platform can determine a third engine terminal that can execute the first task Task1 in other clusters included in the second central computer room BJ, and re-execute the first task Task1 in the third engine terminal, that is, repeat step S350.
[0129] Looking back at step S340, when there are multiple homogeneous engine terminals in the first cluster A where the first engine terminal is located in the resource pool setting, other engine terminals in the same cluster may also be selected when the first task Task1 in the first engine terminal is in an abnormal state. When the first task in this engine terminal is still abnormal, step S340 may be executed to select an engine terminal from a different cluster.
[0130] In summary, steps S310 to S350 are a complete multi-level task fault recovery process. It is also possible not to execute steps S310 to S330, and only execute the fault recovery process of steps S340 and S350, that is, recover the task at the service layer. It is also possible not to execute step S330, and only execute the fault recovery process of steps S310 to S320 and steps S340 and S350. In other words, the hierarchical fault recovery method provided in this specification can select the appropriate level for fault recovery, or it can completely execute all recovery.
[0131] In any level of recovery, if the task reaches a normal state, there is no need to enter the next level of recovery. Only when the task in the previous level has not been recovered will the next level of fault recovery be entered.
[0132] The above embodiment adds distributed high-availability services on top of the running engine layer through a multi-layer scheduling approach, supports multi-tenant isolation, multi-computer room rescheduling and automatic anomaly detection, solves the problem of unrecoverable problems caused by service machines, networks, and disks, and constructs five different layers of recovery strategies, so that availability has corresponding fast recovery methods at the process, machine, and cluster levels, and can fall back to a higher-level automatic retry solution, achieving high availability of the overall system request execution.
[0133] In this specification, the word "first" in terms such as first task, first cluster, first central computer room, first engine terminal, etc., and the corresponding "second" (if any) in the text, etc., are merely for the convenience of distinction and description and do not have any limiting meaning.
[0134] In this specification, the engine terminal and service platform can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities.
[0135] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0136] Figure 6 A schematic block diagram of a hierarchical recovery system for task failures provided in an embodiment. Figure 3 The system includes a service platform 610, a first engine terminal 620 in a first cluster, a second engine terminal 630 in a second cluster, and a third engine terminal 640 in a second central computer room; the first cluster and the second cluster belong to the first central computer room;
[0137] The service platform 610 is configured to, when determining that the first task executed in the first engine terminal 620 satisfies a first retry condition, determine a second engine terminal 630 in the second cluster that can execute the first task, and send the first task to be re-executed to the second engine terminal 630;
[0138] The second engine terminal is used to receive the first task sent by the service platform 610 and re-execute the first task;
[0139] The service platform is used to determine a third engine terminal 640 capable of executing the first task in the second central computer room when it is determined that the first task executed in the second engine terminal 630 meets a second retry condition, and send the first task to be re-executed to the third engine terminal 640;
[0140] The third engine terminal is used to receive the first task sent by the service platform 610 and re-execute the first task.
[0141] The above system embodiment corresponds to the method embodiment. For specific description, please refer to the description of the method embodiment part, which will not be repeated here. The system embodiment has the same technical effect as the corresponding method embodiment. For specific description, please refer to Figure 3 Corresponding method embodiments.
[0142] Figure 7A schematic block diagram of a hierarchical recovery device for task failures provided in an embodiment. Figure 3 The device 700 is deployed in a service platform and includes:
[0143] The first recovery module 710 is configured to, when it is determined that the first task executed in the first engine terminal meets the first retry condition, determine a second engine terminal in the second cluster included in the first central computer room that can execute the first task, and re-execute the first task in the second engine terminal; the first engine terminal belongs to the first cluster included in the first central computer room, and the second cluster is different from the first cluster;
[0144] The second recovery module 720 is configured to determine a third engine terminal capable of executing the first task in a second central computer room when it is determined that the first task executed in the second engine terminal meets a second retry condition, and re-execute the first task in the third engine terminal; the second central computer room is different from the first central computer room.
[0145] In one embodiment, when determining the second engine terminal capable of executing the first task, the first recovery module 710 includes:
[0146] When the tenant who submits the first task is set with a resource pool, a second cluster is determined from the clusters of the first central computer room included in the resource pool; the resource pool includes a cluster having the service capability required by the tenant.
[0147] In one implementation, the first recovery module 710, when re-executing the first task in the second engine terminal, includes:
[0148] The task information of the first task is sent to the second engine terminal, so that the second engine terminal re-executes the first task based on the task information and an identifier carried by the task information and used to indicate re-execution of the task.
[0149] In one implementation, the first task is a stream data task, and the task information carries the storage address of the input stream data, the storage address of the output stream data, and task processing parameters.
[0150] In one embodiment, the apparatus 400 further includes:
[0151] The task stopping module (not shown in the figure) is configured to stop executing the first task in the first engine terminal when it is determined that the first task executed in the first engine terminal meets the first retry condition.
[0152] In one embodiment, the device 400 further includes: a policy determination module and a status query module (not shown in the figure). The policy determination module is configured to determine a query strategy and a timeout strategy for the first task before determining that the first task satisfies the first retry condition. The status query module is configured to query the status of the first task in the first engine terminal based on the query strategy. The first recovery module 710, when determining that the first task executed in the first engine terminal satisfies the first retry condition, includes: when the status of the first task in the first engine terminal reaches an abnormality, and its abnormal status reaches the timeout strategy, determining that the first task satisfies the first retry condition.
[0153] In one embodiment, the policy determination module is further configured to determine a retry policy for the first task, the retry policy including a first number of times the first task is allowed to be retried in the same central computer room. The first recovery module 710 determines that the first task satisfies the first retry condition, including: when the state of the first task in the first engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries does not exceed the first number, determining that the first task satisfies the first retry condition.
[0154] In one embodiment, the second recovery module 720, when determining that the first task executed in the second engine terminal satisfies the second retry condition, includes: when the state of the first task in the second engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries exceeds the first number, determining that the first task satisfies the second retry condition.
[0155] In one embodiment, the first recovery module 710 is further configured to determine a second engine terminal capable of executing the first task in other clusters included in the first central computer room, and re-execute the first task in the second engine terminal when the status of the first task in the second engine terminal reaches an abnormal state and its abnormal state reaches the timeout policy, and the current number of retries does not exceed the first number.
[0156] Figure 8 A schematic block diagram of another hierarchical recovery device for task failures provided in an embodiment. The device 800 is deployed in an engine terminal, the engine terminal includes a first engine terminal, a second engine terminal and a third engine terminal, the first central computer room includes a first cluster and a second cluster, the first engine terminal belongs to the first cluster, the second engine terminal belongs to the second cluster, and the third engine terminal belongs to the second central computer room. Figure 3 The method embodiments shown correspond.
[0157] The device 800_1 deployed in the second engine terminal includes:
[0158] A first receiving module 810 is configured to receive a first task to be re-executed issued by the service platform; a first executing module 820 is configured to re-execute the first task; wherein the first task is issued by the service platform when it is determined that the first task executed in the first engine terminal meets the first retry condition and the second engine terminal is determined in the second cluster to be able to execute the first task;
[0159] The device 800_2 deployed in the third engine terminal includes:
[0160] The second receiving module 830 is configured for the third engine terminal to receive the first task to be re-executed issued by the service platform; the second execution module 840 is configured to re-execute the first task; wherein, the first task is issued by the service platform when it is determined in the second central computer room that the third engine terminal is able to execute the first task when it is determined that the first task executed in the second engine terminal meets the second retry condition.
[0161] In one embodiment, the device deployed in the first engine terminal includes a management component and an execution component (not shown in the figure), wherein the execution component includes a process for running the first task, and the process includes a plurality of threads;
[0162] The execution component is configured to restart the first thread when there is a first thread with an abnormal state among the plurality of threads; and report the abnormal state of the first task to the management component when the state of the first thread is still abnormal after the restart;
[0163] The management component is configured to restart the process upon receiving the abnormal status reported by the execution component; when the status of the process is still abnormal after the restart, report the abnormal status of the first task to the service platform, so that the service platform determines whether the first task meets the first retry condition.
[0164] In one embodiment, the first engine terminal includes several physical machines; when the management component reports the abnormal state of the first task to the service platform, it includes: when the state of the process is still abnormal after restart, rebuilding and starting the process for running the first task in other physical machines included in the first engine terminal; when the state of the first task in the rebuilt process is still abnormal, reporting the abnormal state of the first task to the service platform.
[0165] In one embodiment, the first execution module includes a sub-management component and a sub-execution component (not shown in the figure);
[0166] When the sub-execution component reports the abnormal state of the first task to the sub-management component, the sub-management component directly reports the abnormal state of the first task to the service platform, so that the service platform determines whether the first task satisfies the second retry condition.
[0167] The above-mentioned device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.
[0168] The present specification also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute Figures 1 to 5 Any of the methods described above.
[0169] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, Figures 1 to 5 Any of the methods described above.
[0170] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0171] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0172] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.
Claims
1. A hierarchical recovery method for task failures, executed through a service platform, comprising: When it is determined that the first task executed in the first engine terminal meets the first retry condition, a second engine terminal capable of executing the first task is determined in the second cluster included in the first central computer room, and the first task is re-executed in the second engine terminal; The first engine terminal belongs to a first cluster included in the first central computer room, and the second cluster is different from the first cluster; When it is determined that the first task executed in the second engine terminal meets the second retry condition, a third engine terminal capable of executing the first task is determined in a second central computer room, and the first task is re-executed in the third engine terminal; the second central computer room is different from the first central computer room.
2. The method according to claim 1, wherein the step of determining a second engine terminal capable of executing the first task comprises: When the tenant who submitted the first task is set with a resource pool, determining a second cluster from the clusters of the first central computer room included in the resource pool; The resource pool includes a cluster having the service capabilities required by the tenants.
3. The method according to claim 1, wherein the step of re-executing the first task in the second engine terminal comprises: The task information of the first task is sent to the second engine terminal, so that the second engine terminal re-executes the first task based on the task information and an identifier carried by the task information and used to indicate re-execution of the task.
4. According to the method of claim 3, the first task is a stream data task, and the task information carries the storage address of the input stream data and the storage address of the output stream data and the task processing parameters.
5. The method according to claim 1, when it is determined that the first task executed in the first engine terminal meets the first retry condition, further comprising: Stop executing the first task in the first engine terminal.
6. The method according to claim 1, before determining that the first task satisfies a first retry condition, further comprising: Determining a query strategy and a timeout strategy for the first task; Querying the state of the first task in the first engine terminal based on the query strategy; The step of determining whether the first task executed in the first engine terminal satisfies a first retry condition includes: When the state of the first task in the first engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, it is determined that the first task meets a first retry condition.
7. The method according to claim 6, further comprising: Determine a retry strategy for the first task, the retry strategy including a first number of times the first task is allowed to be retried in the same central computer room; The step of determining that the first task satisfies the first retry condition includes: When the state of the first task in the first engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries does not exceed the first number, it is determined that the first task meets the first retry condition.
8. The method according to claim 7, wherein the step of determining whether the first task executed in the second engine terminal satisfies a second retry condition comprises: When the state of the first task in the second engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries exceeds the first number, it is determined that the first task meets the second retry condition.
9. The method according to claim 8, further comprising: When the status of the first task in the second engine terminal reaches an abnormal state, and its abnormal state reaches the timeout policy, and the current number of retries does not exceed the first number, a second engine terminal capable of executing the first task is determined in other clusters included in the first central computer room, and the step of re-executing the first task in the second engine terminal is returned.
10. A hierarchical recovery method for task failures, executed by engine terminals, the engine terminals comprising a first engine terminal, a second engine terminal and a third engine terminal, a first central computer room comprising a first cluster and a second cluster, the first engine terminal belonging to the first cluster, the second engine terminal belonging to the second cluster, and the third engine terminal belonging to the second central computer room; The method comprises: The second engine terminal receives the first task to be re-executed issued by the service platform, and re-executes the first task; wherein the first task is issued by the service platform when it is determined in the second cluster that the second engine terminal can execute the first task when the service platform determines that the first task executed in the first engine terminal meets the first retry condition; The third engine terminal receives the first task to be re-executed issued by the service platform, and re-executes the first task; wherein, the first task is issued by the service platform when it is determined in the second central computer room that the third engine terminal is able to execute the first task when it is determined that the first task executed in the second engine terminal meets the second retry condition.
11. The method according to claim 10, wherein the first engine terminal comprises a management component and an execution component, wherein the execution component comprises a process for running the first task, and the process comprises a plurality of threads; the method further comprises: When there is a first thread in the plurality of threads with an abnormal state, the execution component restarts the first thread; When the state of the first thread is still abnormal after the restart, the execution component reports the abnormal state of the first task to the management component; The management component restarts the process upon receiving the abnormal state reported by the execution component; When the state of the process is still abnormal after the restart, the management component reports the abnormal state of the first task to the service platform, so that the service platform determines whether the first task satisfies the first retry condition.
12. The method according to claim 11, wherein the first engine terminal comprises a plurality of physical machines; and the step of the management component reporting the abnormal state of the first task to the service platform comprises: When the state of the process is still abnormal after the restart, the management component rebuilds and starts the process for running the first task in other physical machines included in the first engine terminal; When the state of the first task in the reconstructed process is still abnormal, the management component reports the abnormal state of the first task to the service platform.
13. The method according to claim 11, when the second engine terminal re-executes the first task, the method further comprises: When the execution component in the second engine terminal reports the abnormal status of the first task to the management component in the second engine terminal, the management component in the second engine terminal directly reports the abnormal status of the first task to the service platform, so that the service platform determines whether the first task meets the second retry condition.
14. A hierarchical recovery system for task failures, comprising a service platform, a first engine terminal in a first cluster, a second engine terminal in a second cluster, and a third engine terminal in a second central computer room; the first cluster and the second cluster belong to the first central computer room; The service platform is configured to, when determining that the first task executed in the first engine terminal meets a first retry condition, determine a second engine terminal in the second cluster that can execute the first task, and send the first task to be re-executed to the second engine terminal; The second engine terminal is used to receive the first task sent by the service platform and re-execute the first task; The service platform is configured to determine a third engine terminal capable of executing the first task in the second central computer room when it is determined that the first task executed in the second engine terminal meets a second retry condition, and send the first task to be re-executed to the third engine terminal; The third engine terminal is used to receive the first task sent by the service platform and re-execute the first task.
15. A hierarchical recovery device for task failure, deployed in a service platform, comprising: A first recovery module is configured to, when it is determined that the first task executed in the first engine terminal meets the first retry condition, determine a second engine terminal in the second cluster included in the first central computer room that can execute the first task, and re-execute the first task in the second engine terminal; The first engine terminal belongs to a first cluster included in the first central computer room, and the second cluster is different from the first cluster; The second recovery module is configured to determine a third engine terminal capable of executing the first task in a second central computer room when it is determined that the first task executed in the second engine terminal meets a second retry condition, and re-execute the first task in the third engine terminal; the second central computer room is different from the first central computer room.
16. A hierarchical recovery device for task failure, deployed in an engine terminal, the engine terminal comprising a first engine terminal, a second engine terminal and a third engine terminal, the first central computer room comprising a first cluster and a second cluster, the first engine terminal belonging to the first cluster, the second engine terminal belonging to the second cluster, and the third engine terminal belonging to the second central computer room; The device deployed in the second engine terminal includes: a first receiving module configured to receive a first task to be re-executed issued by a service platform; a first executing module configured to re-execute the first task; wherein the first task is issued by the service platform when it is determined in the second cluster that the second engine terminal is capable of executing the first task when the service platform determines that the first task executed in the first engine terminal meets a first retry condition; The device deployed in the third engine terminal includes: The second receiving module is configured for the third engine terminal to receive the first task to be re-executed issued by the service platform; the second executing module is configured to re-execute the first task; wherein, the first task is issued by the service platform when it is determined in the second central computer room that the third engine terminal is able to execute the first task when the service platform determines that the first task executed in the second engine terminal meets the second retry condition.
17. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 13.
18. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 13 is implemented.
