Offline task scheduling system and method, electronic equipment and storage medium

Abstract

The invention provides an off-line task scheduling system and method, electronic equipment and a storage medium, and relates to the artificial intelligence field of distributed storage, automatic analysis, off-line data processing and the like. The system can comprise a scheduling layer and an execution layer, the scheduling layer comprises a group of scheduling nodes, and the execution layer comprises a group of execution nodes; each scheduling node has a peer-to-peer and complete scheduling capability and cooperatively works through a standardized application program interface; the scheduling node is used for determining the to-be-executed task as a target task in response to determining that any to-be-executed task meets the execution condition, determining a target node for executing the target task from the execution nodes, and sending a task execution instruction to the target node; and the execution node is used for completing task processing corresponding to the target task according to the obtained task execution instruction. By applying the scheme of the invention, the continuity, timeliness and the like of task processing can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and particularly to the fields of distributed storage, automated analysis and offline data processing, especially to offline task scheduling systems, methods, electronic devices and storage media. Background Technology

[0002] In traditional offline task scheduling systems, task scheduling relies primarily on a single central scheduling node. Therefore, if this central node fails, tasks may not be processed correctly. This presents a single point of failure risk, impacting the continuity of task processing. Summary of the Invention

[0003] This disclosure provides offline task scheduling systems, methods, electronic devices, and storage media.

[0004] An offline task scheduling system includes: a scheduling layer and an execution layer;

[0005] The scheduling layer includes a set of scheduling nodes, and the execution layer includes a set of execution nodes; each scheduling node has equal and complete scheduling capabilities and works collaboratively through a standardized application programming interface.

[0006] The scheduling node is configured to, in response to determining that any task to be executed meets the execution conditions, identify the task to be executed as a target task, determine the target node for executing the target task from the execution nodes, and send a task execution instruction to the target node;

[0007] The execution node is used to complete the task processing corresponding to the target task according to the obtained task execution instructions.

[0008] An offline task scheduling method, the method being applied in a scheduling node, includes:

[0009] In response to determining that any task to be executed meets the execution conditions, the task to be executed is identified as the target task, and a target node for executing the target task is identified from each execution node; wherein, both the scheduling node and the execution node are located in an offline task scheduling system, the offline task scheduling system includes a scheduling layer and an execution layer, the scheduling layer includes a set of scheduling nodes, the execution layer includes a set of execution nodes, each scheduling node has equal and complete scheduling capabilities, and they work together through a standardized application programming interface;

[0010] Send a task execution instruction to the target node to instruct the target node to complete the task processing corresponding to the target task.

[0011] An offline task scheduling method, the method being applied to an execution node, includes:

[0012] Obtain the task execution instructions issued by the scheduling node for the target task; wherein, both the scheduling node and the execution node are located in an offline task scheduling system, the offline task scheduling system includes a scheduling layer and an execution layer, the scheduling layer includes a group of scheduling nodes, the execution layer includes a group of execution nodes, each scheduling node has equal and complete scheduling capabilities, and they work together through a standardized application programming interface; the target task is any task to be executed that meets the execution conditions;

[0013] According to the task execution instructions, complete the task processing corresponding to the target task.

[0014] An electronic device, comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0018] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0019] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0022] Figure 1 This is a schematic diagram of the composition structure of the first embodiment 100 of the offline task scheduling system described in this disclosure;

[0023] Figure 2 This is a schematic diagram of the composition structure of the second embodiment 200 of the offline task scheduling system described in this disclosure;

[0024] Figure 3This is a schematic diagram of the composition structure of the third embodiment 300 of the offline task scheduling system described in this disclosure;

[0025] Figure 4 This is a flowchart of the first embodiment of the offline task scheduling method described in this disclosure;

[0026] Figure 5 This is a flowchart of the second embodiment of the offline task scheduling method described in this disclosure;

[0027] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0030] Figure 1 This is a schematic diagram of the structural composition of the first embodiment 100 of the offline task scheduling system described in this disclosure. Figure 1 As shown, it includes: scheduling layer 11 and execution layer 12.

[0031] The scheduling layer 11 includes a set of scheduling nodes (Master Server) 111, and the execution layer 12 includes a set of execution nodes (Worker Server) 121; each scheduling node 111 has equal and complete scheduling capabilities and works together through a standardized application programming interface (API). Figure 1 It is assumed that there are multiple scheduling nodes 111 and multiple execution nodes 121.

[0032] The scheduling node 111 is used to determine the target task as the target task in response to the determination that any task to be executed meets the execution conditions, and to determine the target node for executing the target task from each execution node 121 and send the task execution instruction to the target node.

[0033] Execution node 121 is used to complete the task processing corresponding to the target task according to the obtained task execution instructions.

[0034] As can be seen, by adopting the scheme described in the above system embodiment, a single central scheduling node is no longer set up. Instead, multiple scheduling nodes 111 can be deployed in the scheduling layer 11. Each scheduling node 111 has equal and complete scheduling capabilities and works together through standardized APIs. This eliminates the risk of single point of failure caused by the central scheduling node from the architectural level, thereby improving the continuity and timeliness of task processing. Moreover, both the scheduling layer 11 and the execution layer 12 are freely combinable distributed node clusters, which can independently and linearly expand the task scheduling and execution capabilities, making them very flexible and convenient, and thus meeting the usage needs of different scenarios.

[0035] Figure 2 This is a schematic diagram of the structural composition of the second embodiment 200 of the offline task scheduling system described in this disclosure. Figure 2 As shown, it includes: a front-end console 10, a scheduling layer 11, and an execution layer 12.

[0036] It can be seen that, compared to Figure 1 The embodiment shown, Figure 2 The illustrated embodiment further includes a front-end console 10. The front-end console 10 can be used to obtain task configuration requests submitted by users and can send the task configuration requests to the scheduling layer 11, such as through an API gateway.

[0037] For example, users (business parties) can fill in and submit task configuration requests through the relevant user interface (UI, UserInterface) provided by the front-end console 10. The task configuration request may include task type, execution logic, dependencies, triggering conditions, and resource requirements.

[0038] Task type can refer to the specific type of offline task, such as a distributed computing framework (Spark), a distributed stream processing and batch processing framework (Flink), a shell script, or a Hypertext Transfer Protocol (HTTP) task. Execution logic refers to the specific business operations and processing flow that the task needs to complete. Dependencies refer to the sequential execution relationships with other tasks, ensuring that different tasks are executed in the order of business logic. Triggering conditions refer to the specific rules that trigger task execution, such as time-triggered or event-triggered execution. Resource requirements refer to the hardware and software resource configurations required during task execution.

[0039] The front-end console 10 serves as the unified operation and maintenance entry point for the offline scheduling system. It can provide users with various interactive capabilities to meet their different usage needs. It also simplifies the interaction between users and the offline scheduling system and lowers the barrier to entry.

[0040] The scheduling layer 11 typically includes multiple scheduling nodes 111, each with equal and complete scheduling capabilities. They work together through standardized APIs without the need to elect a master node or share storage, thus adopting a decentralized scheduling mechanism to eliminate the risk of single point of failure caused by a central scheduling node. Furthermore, it supports the horizontal expansion of scheduling capabilities as the number of scheduling nodes increases.

[0041] For task configuration requests from the front-end console 10, a scheduling node 111 can be used to process them. There are no restrictions on how the scheduling node 111 is selected. For example, it can be selected randomly or according to the load priority principle (the lower the load, the higher the probability of being selected).

[0042] In some embodiments of this disclosure, the scheduling node 111 can verify the task configuration request. In response to determining that the verification fails, it can return an error message through the front-end console 10. In response to determining that the verification passes, it can further orchestrate the offline task corresponding to the task configuration request.

[0043] In addition, in some embodiments of this disclosure, the verification may include one or any combination of the following: parameter integrity verification, legality verification, resource compatibility verification, and dependency rationality verification.

[0044] Parameter integrity verification is a fundamental verification step after the task configuration request is submitted. It involves verifying whether all key parameters required for the task configuration are complete, such as checking for missing fields like task type and resource requirements, to prevent task failure due to incomplete parameters. Legality verification involves verifying whether the format and values ​​of the submitted parameters conform to system specifications. Resource compatibility verification involves verifying whether the resources required by the task are within the system's supported range. Dependency reasonableness verification involves verifying whether the task's dependencies are reasonable.

[0045] By validating task configuration requests, invalid or erroneous configurations can be intercepted in advance, preventing invalid tasks from entering the subsequent scheduling process, thereby improving the overall stability and resource utilization of the system.

[0046] If the verification fails, the scheduling node 111 can return an error message to the user through the front-end console 10, and can also indicate the reason for the failure so that the user can make corrections.

[0047] If the verification passes, the scheduling node 111 can further orchestrate the offline tasks corresponding to the task configuration request. In some embodiments of this disclosure, in response to determining that there is no dependency between the offline task and each task to be executed, the scheduling node 111 can generate a directed acyclic graph (DAG) corresponding to the offline task, and can determine that the task orchestration for the offline task is complete, and the task to be executed is an offline task with completed task orchestration. In response to determining that there is a dependency between the offline task and at least one task to be executed, the scheduling node 111 can update the corresponding DAG according to the dependency, and can determine that the task orchestration for the offline task is complete.

[0048] If the offline task has no dependency relationship with any of the other tasks to be executed in the system, i.e., it is an independent task, then a DAG corresponding to the offline task can be generated. This DAG only includes the offline task, but other tasks may be added later.

[0049] If there is a dependency between the offline task and at least one pending task in the system, such as the offline task depending on pending task B (which can only be executed after pending task B is completed), and pending task B depending on pending task A, then the DAG containing pending task B and pending task A can be updated to add the dependency between the offline task and pending task B.

[0050] By dynamically constructing and updating the DAG, each task to be executed can be executed in an orderly manner according to its dependencies, avoiding logical errors and resource conflicts. Moreover, the processing method described in this disclosure can achieve real-time updates of the DAG, making task configurations effective within seconds.

[0051] For each task to be executed in the system, scheduling can be performed in accordance with the manner described in this disclosure. For example, in response to determining that any task to be executed meets the execution conditions, scheduling node 111 can identify the task to be executed as the target task, and can identify the target node for executing the target task from among the execution nodes, and then send a task execution instruction to the target node.

[0052] For the same task, the scheduling node 111 that verifies and orchestrates the task and the scheduling node 111 that performs the scheduling process may be the same node or different nodes. In addition, if the task is a periodically executed task, the scheduling node 111 that performs the scheduling process for the task each time may be a different node.

[0053] In some embodiments of this disclosure, the dependency relationship may include: a pre-dependency relationship and a post-dependency relationship. Accordingly, for any task to be executed, meeting the execution conditions may include: determining, according to the DAG, that the task to be executed has no dependency relationship or only a post-dependency relationship, and determining the execution time of the task to be executed; determining, according to the DAG, that the task to be executed has a pre-dependency relationship, and determining that other tasks to be executed that have a pre-dependency relationship with the task to be executed have all been executed.

[0054] For example, assuming the task to be executed is task x, and task x has no dependencies, then task x can be determined to meet the execution condition each time its execution time arrives. For instance, the execution time could refer to 8 AM every day; that is, task x's triggering condition is time-triggered. As another example, assuming task x has dependencies, but only post-dependencies, such as task y depending on task x, then task x can also be determined to meet the execution condition each time its execution time arrives. Furthermore, assuming task x has pre-dependencies, such as task x depending on task z, then task x can be determined to meet the execution condition after task z has completed its execution; that is, task x's triggering condition is event-triggered.

[0055] By adopting the above processing method, the orderliness and logical correctness of task execution can be improved. By identifying the dependencies and time conditions between tasks, tasks are scheduled only when all prerequisites are met, thereby improving the reliability and accuracy of system scheduling.

[0056] For tasks that meet the execution conditions, the scheduling node 111 can identify them as target tasks and determine the target node for executing the target task from among the execution nodes 121. In some embodiments of this disclosure, the scheduling node 111 can periodically obtain node attribute information reported by each execution node 121, and determine the target node based on the latest obtained node attribute information, and send task execution instructions to the target node.

[0057] In addition, in some embodiments of this disclosure, the node attribute information may include: capability tags, resource status, and health status.

[0058] Capability tags can refer to the types of tasks supported. Resource status can refer to resource usage, such as memory usage and Central Processing Unit (CPU) usage. Health status can refer to whether the heartbeat is normal and whether the process is alive.

[0059] Accordingly, the scheduling node 111 can determine the target node by combining the newly acquired node attribute information of each execution node 121, thereby accurately routing the target task to the most suitable execution node 121, thereby improving the success rate of task execution and the resource utilization of the execution node 121.

[0060] In addition to the functions described above, in practical applications, scheduling node 111 can also support some other functions, such as backtracking, high availability guarantee mechanism, permission approval, etc., depending on actual needs.

[0061] The execution layer 12 typically includes multiple execution nodes 121. As the name suggests, execution nodes 121 are the nodes responsible for actually executing tasks. They can be deployed in different computing environments, such as physical machines, virtual machines, container platforms, cloud hosts, and serverless platforms, depending on business needs, and are not dependent on a host environment. Execution nodes 121 can dynamically join or leave the system, and like the scheduling layer 11, they support horizontal scaling, thus supporting rapid growth in business scale. In addition, execution nodes 121 can also support online upgrades, thereby further improving task execution efficiency and the accuracy of execution results.

[0062] Furthermore, the number of target nodes corresponding to a target task can be one or more, depending on actual needs, making it very flexible and convenient. Thanks to its cross-platform task execution capabilities, scheduling node 111 can ignore differences in the underlying environment and freely select and combine execution nodes 121 from the heterogeneous resource pool, thereby effectively realizing the free combination and elastic scaling of the execution layer 12 and maximizing resource utilization.

[0063] In some embodiments of this disclosure, the execution node 121 can periodically report its own node attribute information to the scheduling layer 11, and after obtaining the task execution instruction, it can obtain the task source code matching the target task and complete the task execution environment configuration corresponding to the target task, and then complete the task processing corresponding to the target task according to the task source code and the task execution environment.

[0064] For example, execution node 121 can automatically pull the task source code from the specified code repository using the built-in code puller based on the source code identifier carried in the task execution instruction. Furthermore, execution node 121 can pull or start the corresponding environment image based on the task type of the target task, or parse the environment dependency configuration list obtained through a predetermined method into specific dependency installation and configuration commands, thereby configuring the task execution environment.

[0065] By adopting the above processing method, source code retrieval and environment configuration operations can be automatically triggered by commands without manual intervention. This reduces environmental differences or missing dependencies caused by manual operations. Moreover, it greatly simplifies the task deployment and execution process and achieves a high degree of automation, environmental consistency and process reliability in task execution.

[0066] During the execution of the target task, execution node 121 can generate task logs and report task execution progress and other information to the scheduling layer 11. Correspondingly, the scheduling layer 11 can also display task execution progress and other information to the user through the front-end console 10, so that the user can understand the task processing status in a timely manner and realize task monitoring. In addition, after the target task is completed, execution node 121 can automatically archive and store the task logs and send the task execution results to the scheduling layer 11.

[0067] Figure 3 This is a schematic diagram of the structural composition of the third embodiment 300 of the offline task scheduling system described in this disclosure. Figure 3 As shown, it includes: a front-end console 10, a scheduling layer 11, an execution layer 12, and a Sentinel service module 13.

[0068] It can be seen that, compared to Figure 2 The embodiment shown, Figure 3 The illustrated embodiment further includes a sentinel service module 13. The sentinel service module 13 can be used to perform real-time status monitoring of each scheduling node 111 and each execution node 121, and in response to determining that any node has failed based on the monitoring results, it can execute the fault response plan corresponding to the fault.

[0069] For example, the sentinel service module 13 can monitor the heartbeat, health status and resource load of each scheduling node 111 and each execution node 121 in real time. When a node is found to be faulty based on the monitoring results, an alarm can be issued, and fault response solutions such as automatic node switching, task transfer or rescheduling can be triggered, thereby improving fault recovery efficiency and task execution success rate, and providing an automatic disaster recovery mechanism to improve the overall availability and stability of the system.

[0070] in addition, Figure 1 , Figure 2 and Figure 3 In the embodiments shown, the front-end console 10 and the scheduling layer 11, as well as the scheduling layer 11 and the execution layer 12, can communicate using a RESTful API.

[0071] Correspondingly, adopting RESTful API communication can achieve interface standardization and component decoupling, allowing each layer to be developed, deployed, and extended independently, thereby reducing system complexity and maintenance costs.

[0072] The above is an introduction to the system embodiments. The following method embodiments will further illustrate the solution described in this disclosure.

[0073] Figure 4 This is a flowchart of a first embodiment of the offline task scheduling method described in this disclosure. This method can be applied to scheduling nodes. Figure 4 As shown, the specific implementation methods are as follows.

[0074] In step 401, in response to determining that any task to be executed meets the execution conditions, the task to be executed is identified as the target task, and the target node for executing the target task is identified from the execution nodes; wherein, both the scheduling node and the execution node are located in the offline task scheduling system, which includes a scheduling layer and an execution layer. The scheduling layer includes a set of scheduling nodes, and the execution layer includes a set of execution nodes. Each scheduling node has equal and complete scheduling capabilities and works collaboratively through a standardized API.

[0075] In step 402, a task execution instruction is sent to the target node to instruct the target node to complete the task processing corresponding to the target task.

[0076] As can be seen, by adopting the scheme described in the above method embodiment, a single central scheduling node is no longer set up. Instead, multiple scheduling nodes can be deployed in the scheduling layer. Each scheduling node has equal and complete scheduling capabilities and works together through standardized APIs. This eliminates the risk of single point of failure caused by the central scheduling node from the architectural level, thereby improving the continuity and timeliness of task processing. Moreover, both the scheduling layer and the execution layer are freely combinable distributed node clusters, which can independently and linearly expand the task scheduling and execution capabilities, making them very flexible and convenient, and thus meeting the usage needs of different scenarios.

[0077] The offline task scheduling system may also include a front-end console. Accordingly, in some embodiments of this disclosure, the task configuration request submitted by the user through the front-end console can be obtained and the task configuration request can be verified. In response to the determination that the verification fails, an error message can be returned through the front-end console. In response to the determination that the verification passes, the offline task corresponding to the task configuration request can be further orchestrated.

[0078] In some embodiments of this disclosure, the verification may include one or any combination of the following: parameter integrity verification, legality verification, resource compatibility verification, and dependency rationality verification.

[0079] By validating task configuration requests, invalid or erroneous configurations can be intercepted in advance, preventing invalid tasks from entering the subsequent scheduling process, thereby improving the overall stability and resource utilization of the system.

[0080] In some embodiments of this disclosure, the method of task orchestration may include: in response to determining that there is no dependency between the offline task and each task to be executed, generating a DAG corresponding to the offline task, and determining that the task orchestration for the offline task is completed, wherein the task to be executed is an offline task whose task orchestration has been completed; in response to determining that there is a dependency between the offline task and at least one task to be executed, updating the corresponding DAG according to the dependency, and determining that the task orchestration for the offline task is completed.

[0081] By dynamically constructing and updating the DAG, each task to be executed can be executed in an orderly manner according to its dependencies, avoiding logical errors and resource conflicts. Moreover, the processing method described in this disclosure can achieve real-time updates of the DAG, making task configurations effective within seconds.

[0082] In some embodiments of this disclosure, the dependency relationship may include: a pre-dependency relationship and a post-dependency relationship. Accordingly, any task to be executed meeting the execution conditions may include: determining, according to the DAG, that the task to be executed does not have the aforementioned dependency relationship or only has a post-dependency relationship, and determining the execution time of the task to be executed; determining, according to the DAG, that the task to be executed has a pre-dependency relationship, and determining that other tasks to be executed that have a pre-dependency relationship with the task to be executed have all been executed.

[0083] By adopting the above processing method, the orderliness and logical correctness of task execution can be improved. By identifying the dependencies and time conditions between tasks, tasks are scheduled only when all prerequisites are met, thereby improving the reliability and accuracy of system scheduling.

[0084] In some embodiments of this disclosure, node attribute information reported by each execution node can be obtained periodically, and the target node can be determined based on the latest obtained node attribute information.

[0085] In some embodiments of this disclosure, the node attribute information may include: capability tags, resource status, and health status.

[0086] Accordingly, the target task can be accurately routed to the most suitable execution node, thereby improving the success rate of task execution and the resource utilization of the execution node.

[0087] Figure 5 This is a flowchart of a second embodiment of the offline task scheduling method described in this disclosure. This method can be applied to execution nodes. Figure 5 As shown, the specific implementation methods are as follows.

[0088] In step 501, the task execution instruction issued by the scheduling node for the target task is obtained; wherein, both the scheduling node and the execution node are located in the offline task scheduling system, which includes a scheduling layer and an execution layer. The scheduling layer includes a set of scheduling nodes, and the execution layer includes a set of execution nodes. Each scheduling node has equal and complete scheduling capabilities and works together through a standardized API. The target task is any task to be executed that meets the execution conditions.

[0089] In step 502, the task processing corresponding to the target task is completed according to the task execution instruction.

[0090] As can be seen, by adopting the scheme described in the above method embodiment, a single central scheduling node is no longer set up. Instead, multiple scheduling nodes can be deployed in the scheduling layer. Each scheduling node has equal and complete scheduling capabilities and works together through standardized APIs. This eliminates the risk of single point of failure caused by the central scheduling node from the architectural level, thereby improving the continuity and timeliness of task processing. Moreover, both the scheduling layer and the execution layer are freely combinable distributed node clusters, which can independently and linearly expand the task scheduling and execution capabilities, making them very flexible and convenient, and thus meeting the usage needs of different scenarios.

[0091] In some embodiments of this disclosure, node attribute information can be periodically reported to the scheduling layer, and after obtaining the task execution instruction, the task source code matching the target task can be obtained, and the task execution environment configuration corresponding to the target task can be completed, so that the task processing can be completed according to the task source code and the task execution environment.

[0092] It can automatically trigger source code retrieval and environment configuration operations through commands without manual intervention, thereby reducing environmental differences or missing dependencies caused by manual operation. Moreover, it greatly simplifies the task deployment and execution process and achieves a high degree of automation, environmental consistency and process reliability in task execution.

[0093] The specific implementation of the above method embodiments can be referred to the relevant descriptions in the foregoing system embodiments, and will not be repeated here.

[0094] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly in areas such as distributed storage, automated analysis, and offline data processing. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0095] Furthermore, the tasks and execution results described in the embodiments of this disclosure are not targeted at any specific user and are not intended to reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

[0096] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0097] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0098] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0099] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0100] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0101] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0106] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0107] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An offline task scheduling system, comprising: Scheduling layer and execution layer; The scheduling layer includes a set of scheduling nodes, and the execution layer includes a set of execution nodes; Each of the aforementioned scheduling nodes has equal and complete scheduling capabilities and works collaboratively through a standardized application programming interface; The scheduling node is configured to, in response to determining that any task to be executed meets the execution conditions, identify the task to be executed as a target task, determine the target node for executing the target task from the execution nodes, and send a task execution instruction to the target node; The execution node is used to complete the task processing corresponding to the target task according to the obtained task execution instructions.

2. The offline task scheduling system according to claim 1 further includes: The front-end console is used to obtain the task configuration request submitted by the user and send the task configuration request to the scheduling layer.

3. The offline task scheduling system according to claim 2, wherein, The scheduling node is further configured to: validate the task configuration request; return an error message through the front-end console in response to a failure to validate; and orchestrate the offline task corresponding to the task configuration request in response to a success to validate.

4. The offline task scheduling system according to claim 3, wherein, The verification includes one or any combination of the following: parameter integrity verification, legality verification, resource compatibility verification, and dependency relationship rationality verification.

5. The offline task scheduling system according to claim 3, wherein, In response to determining that there is no dependency between the offline task and each of the tasks to be executed, the scheduling node generates a directed acyclic graph corresponding to the offline task, and determines that the task orchestration for the offline task is complete. The tasks to be executed are offline tasks that have completed the task orchestration. In response to determining that there is a dependency between the offline task and at least one of the tasks to be executed, the scheduling node updates the corresponding directed acyclic graph according to the dependency, and determines that the task orchestration for the offline task is complete.

6. The offline task scheduling system according to claim 5, wherein, The dependencies include: pre-dependencies and post-dependencies; The execution conditions for any task to be executed include: Based on the directed acyclic graph, it is determined that the task to be executed does not have the dependency relationship or only has the subsequent dependency relationship, and the execution time of the task to be executed is determined. Based on the directed acyclic graph, it is determined that the task to be executed has the aforementioned prerequisite dependency, and all other tasks to be executed that have the aforementioned prerequisite dependency with the task to be executed have been completed.

7. The offline task scheduling system according to claim 1, wherein, The scheduling node is further configured to periodically acquire node attribute information reported by each execution node, and determine the target node based on the latest acquired node attribute information.

8. The offline task scheduling system according to claim 7, wherein, The node attribute information includes: capability tags, resource status, and health status.

9. The offline task scheduling system according to claim 7, wherein, The execution node is further configured to periodically report the node attribute information to the scheduling layer, and after receiving the task execution instruction, obtain the task source code matching the target task, complete the task execution environment configuration corresponding to the target task, and complete the task processing according to the task source code and the task execution environment.

10. The offline task scheduling system according to claim 1, further comprising: The sentinel service module is used to perform real-time status monitoring on each of the scheduling nodes and each of the execution nodes, and in response to determining that any node has failed based on the monitoring results, executes the corresponding fault response plan.

11. The offline task scheduling system according to claim 2, wherein, The front-end console and the scheduling layer, as well as the scheduling layer and the execution layer, communicate using a descriptive state transition application programming interface.

12. An offline task scheduling method, the method being applied in a scheduling node, comprising: In response to determining that any task to be executed meets the execution conditions, the task to be executed is identified as the target task, and a target node for executing the target task is identified from each execution node; wherein, both the scheduling node and the execution node are located in an offline task scheduling system, the offline task scheduling system includes a scheduling layer and an execution layer, the scheduling layer includes a set of scheduling nodes, the execution layer includes a set of execution nodes, each scheduling node has equal and complete scheduling capabilities, and they work together through a standardized application programming interface; Send a task execution instruction to the target node to instruct the target node to complete the task processing corresponding to the target task.

13. An offline task scheduling method, the method being applied to an execution node, comprising: Obtain the task execution instructions issued by the scheduling node for the target task; wherein, both the scheduling node and the execution node are located in an offline task scheduling system, the offline task scheduling system includes a scheduling layer and an execution layer, the scheduling layer includes a group of scheduling nodes, the execution layer includes a group of execution nodes, each scheduling node has equal and complete scheduling capabilities, and they work together through a standardized application programming interface; the target task is any task to be executed that meets the execution conditions; According to the task execution instructions, complete the task processing corresponding to the target task.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 12-13.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 12-13.

16. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 12-13.

Images