Task scheduling method, electronic equipment and computer readable storage medium

Abstract

The invention provides a task scheduling method, electronic equipment and a computer readable storage medium, the method is applied to the field of cloud computing and distributed computing, and the method comprises the following steps: receiving a task flow submitted by a user, and extracting a dependency relationship of each subtask in the task flow; constructing a directed acyclic graph of the task flow based on the dependency relationship of the subtasks, and generating an execution sequence of the task flow based on the directed acyclic graph; a current subtask label is generated according to the execution sequence and the dependency relationship of the current subtask, and the current subtask is any subtask in the task flow; and processing the current sub-task based on the current sub-task label, and obtaining a calculation result of the current sub-task. According to the method, the problems of waste of memory and computing power resources and low task execution efficiency caused by repeated execution of the same input intermediate computing steps by different tasks when a multi-task-flow and cross-node containerized task is executed can be avoided.

Description

Technical Field

[0001] This application relates to the fields of cloud computing and distributed computing, and in particular to a task scheduling method, electronic device, and computer-readable storage medium. Background Technology

[0002] In the era of cloud computing and big data, Kubernetes (K8s), as the mainstream platform in the field of container orchestration, has been widely used for the management and scheduling of large-scale containerized tasks and is a core technology platform supporting distributed computing.

[0003] As Kubernetes is increasingly used in multi-task workflows and cross-node containerized scenarios, different tasks often involve intermediate computation steps that share the same input. While existing technologies have implemented persistent storage and shared storage solutions to some extent, their core focus remains on data storage and hardware resource scheduling.

[0004] However, when executing containerized tasks across multiple tasks and nodes, intermediate computation steps with the same input are often repeatedly executed by different tasks. This results in wasted memory and computing resources and low task execution efficiency. Summary of the Invention

[0005] This application provides a task scheduling method, an electronic device, and a computer-readable storage medium. This method can avoid the problems of wasted memory and computing resources and low task execution efficiency caused by the repeated execution of intermediate calculation steps with the same input by different tasks when executing multi-task streams and cross-node containerized tasks.

[0006] Firstly, a task scheduling method is provided, applied to a distributed computing platform. This method includes: receiving a task flow submitted by a user and extracting the dependencies of each subtask in the task flow; constructing a directed acyclic graph (DAG) of the task flow based on the dependencies of the subtasks, and generating an execution sequence of the task flow based on the DAG; generating a label for the current subtask according to the execution sequence and the dependencies of the current subtask, where the current subtask is any subtask in the task flow; processing the current subtask based on the current subtask label, and obtaining the computation result of the current subtask.

[0007] Using the above method, when the distributed computing platform receives a user-submitted task flow, it decomposes the task flow into subtasks. A directed acyclic graph (DAG) is generated based on the dependencies of each subtask, constructing the execution sequence of all subtasks in the task flow to avoid scheduling errors caused by dependency confusion during cross-node execution. A unique subtask label is generated based on the execution sequence and dependencies of the current subtask. This unique subtask label is like giving each intermediate subtask a unique "fingerprint," and the execution sequence and dependencies contained in the label directly reflect the core characteristics of the subtask. If an intermediate subtask in different task flows has the same subtask label, it means that the subtask has the basis for reuse. Finally, the current subtask is processed based on its label, and the computation result is obtained. When a new task flow or other node needs to execute this intermediate subtask, it directly associates the label with the existing computation result, reducing memory resource consumption and wasted computing power.

[0008] In conjunction with the first aspect, in some possible implementations, the above-mentioned processing of the current subtask based on the current subtask label and obtaining the calculation result of the current subtask includes: retrieving the local cache based on the current subtask label; if the current subtask label exists in the local cache, retrieving the calculation result of the current subtask from the local cache; if the current subtask label does not exist in the local cache, retrieving the remote storage based on the current subtask label; if the current subtask label exists in the remote storage, retrieving the calculation result of the current subtask from the remote storage; and if the current subtask label does not exist in the remote storage, calculating the result of the current subtask.

[0009] Through the methods described above, the task scheduling method provided in this application, by employing hierarchical retrieval and result reuse, enables the distributed computing platform to not only leverage the low-latency characteristics of local caching to quickly and frequently respond to the result reuse needs of subtasks, reducing data transfer at key points, but also to reuse the computation results of historical subtasks stored for a long time through the persistent storage capabilities of remote storage, avoiding duplicate computations caused by local cache invalidation. Furthermore, computation is only performed when no sub-tag exists locally or remotely, ensuring the complete execution of the user-submitted task flow.

[0010] Combining the first aspect and the above implementation methods, in some possible implementation methods, the above-mentioned method of obtaining the calculation result of the current subtask from the local cache when the current subtask label exists includes: determining the storage path associated with the current subtask label based on the current subtask label; searching the local cache for the calculation result corresponding to the storage path based on the storage path; when the calculation result corresponding to the storage path exists in the local cache, using the calculation result as the calculation result of the current subtask; when the calculation result corresponding to the storage path does not exist in the local cache, obtaining the calculation result of the current subtask from the remote storage based on the storage path.

[0011] By binding subtask tags, computation results, and storage paths together, the task scheduling method provided in this application can reuse results when the same subtask tag is retrieved, and avoid the loss of computation results for subtask tags in the local cache by retrieving results from remote storage. This makes the reuse process of the task scheduling method more resilient and further improves the reliability of multi-task flow and cross-node reuse.

[0012] Combining the first aspect and the above implementation methods, in some possible implementation methods, the above-mentioned method of obtaining the calculation result of the current subtask from the remote storage when the current subtask label exists includes: determining the storage path associated with the current subtask label based on the current subtask label, and using the calculation result corresponding to the storage path in the remote storage as the calculation result of the current subtask.

[0013] By employing the methods described above, the task scheduling method provided in this application reduces redundant calculations caused by incomplete retrieval by constructing a retrieval process that prioritizes local operations over remote ones. This ensures continuous execution of the task flow while further improving the resource utilization efficiency and reliability of result reuse on the distributed computing platform.

[0014] In combination with the first aspect and the above implementation methods, in some possible implementation methods, when the current subtask label does not exist in the remote storage, the method further includes: saving the result of the current subtask to the local cache and the remote storage.

[0015] By using the above method, the task scheduling method provided in this application can achieve long-term persistence of the results by writing the calculation results of the current subtask to remote storage, avoiding the problem of result loss due to insufficient local cache space being cleared or the failure of nodes in the local cache, and ensuring the reliability of the calculation results and cross-node reuse capability under different conditions.

[0016] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the above-mentioned saving the result of the current subtask to the local cache and remote storage includes: storing the result of the current subtask to the target storage path, wherein the target storage path is associated with the label of the current subtask; and saving the result of the current subtask, the target storage path and the label of the current subtask to the local cache and remote storage.

[0017] By establishing a mapping relationship between "subtask label, storage path, and computation result" in both local cache and remote storage, the task scheduling method provided in this application achieves the following: First, associating storage in the local cache ensures that results can be quickly located via labels in high-frequency reuse scenarios, improving retrieval efficiency. Second, synchronously establishing a mapping relationship in remote storage guarantees the reusability of results in cross-node and long-term storage scenarios. This bidirectional association design strengthens the unique identification and accurate retrieval capabilities of task flow results in the distributed computing platform, while maintaining data consistency through the consistency of local and remote mappings, thus improving the reuse efficiency of computation results.

[0018] In combination with the first aspect and the above implementation methods, in some possible implementation methods, the method further includes: detecting the duration for which subtask tags stored in the local cache have not been retrieved; when the duration exceeds a preset lifespan, deleting the subtask tags stored in the local cache and the calculation results corresponding to the subtask tags.

[0019] By employing the methods described above, the task scheduling method provided in this application constructs a dynamic local cache resource management mechanism by performing timeliness detection and timeout cleanup on unretrieved subtask tags and their results in the local cache. This avoids invalid data occupying resources and thus limiting the storage of effective data, significantly improving the resource utilization rate of the local cache.

[0020] In conjunction with the first aspect and the above implementation methods, in some possible implementation methods, before generating the execution sequence of the task flow based on the directed acyclic graph, the method further includes: detecting whether there is a circular dependency in the directed acyclic graph, and whether the predecessor subtasks that the subtasks depend on are complete; when there is no circular dependency in the directed acyclic graph, and the predecessor subtasks that each subtask depends on are complete, generating the execution sequence based on the directed acyclic graph; when there is a circular dependency in the directed acyclic graph, and / or the predecessor subtasks that the subtasks depend on are incomplete, terminating the scheduling of the task flow, and returning a verification failure reminder to the user who submitted the task flow.

[0021] By employing the above method, the task scheduling method provided in this application can avoid subsequent scheduling deadlocks, execution interruptions, and ineffective resource consumption caused by the aforementioned problems by pre-checking the legality of the directed acyclic graph of the task flow in layers, thereby improving the reliability of task scheduling and the efficiency of task execution.

[0022] Secondly, a task scheduling apparatus is provided, comprising: an acquisition module, a generation module, a processing module, and a storage module. These modules cooperate to enable the task scheduling apparatus to implement the task scheduling function described in the method design of the first aspect. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more sub-modules corresponding to the aforementioned functions.

[0023] Thirdly, an electronic device is provided, comprising: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the task scheduling method in the first aspect or any possible implementation thereof.

[0024] Fourthly, a distributed computing system is provided, which includes a computing platform, a local cache, and remote storage. The computing platform is capable of implementing the task scheduling method in the first aspect or any possible implementation of the first aspect.

[0025] Fifthly, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, are used to implement the task scheduling method in the first aspect or any possible implementation thereof.

[0026] Sixthly, embodiments of this application provide a chip system including one or more processors and may also include a memory. The processors are used to retrieve and execute instructions stored in the memory, causing the task scheduling method in any of the first to fifth aspects or any possible implementation thereof to be executed. The chip system may consist of a single chip or may include multiple chips and other discrete devices, such as a memory controller and interface expansion circuits working together.

[0027] The chip system is equipped with output circuits or interfaces for sending information or data, such as sending scheduling instructions and result reading requests to computing nodes and storage devices, as well as input circuits or interfaces for receiving information or data, such as receiving task flow data and cached / stored result feedback. It interacts with external devices through input / output interfaces to support signal and data transmission throughout the entire task scheduling process.

[0028] In a seventh aspect, a computer program product is provided, comprising: a computer program (also referred to as code or instructions) that, when executed, causes a computer to perform a method in any possible implementation of any of the above aspects. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of an example implementation system provided in an embodiment of this application; Figure 2 This is a schematic diagram of the system architecture of an example implementation scheme provided in this application embodiment; Figure 3 This is a schematic flowchart of an example task scheduling method provided in an embodiment of this application; Figure 4 This is a schematic flowchart of a method for validating the directed acyclic graph in a task scheduling method provided in an embodiment of this application. Figure 5 This is an interaction diagram of an example task scheduling method provided in an embodiment of this application; Figure 6 This is a schematic flowchart of a local cache management method in a task scheduling method provided in an embodiment of this application; Figure 7 This is a schematic block diagram of a task scheduling apparatus provided in an embodiment of this application; Figure 8 This is a schematic block diagram of a task scheduling device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0031] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0033] With the deepening application of cloud computing and distributed computing technologies, complex task flows composed of multiple subtasks are increasingly common. These task flows involve requirements such as cross-node scheduling, ensuring environment consistency, and dynamic resource allocation, making containerized deployment the preferred method to meet these complex task flow needs. Kubernetes (K8s), as the current mainstream container orchestration platform, has become the core carrier for large-scale containerized task flows due to its powerful task scheduling, resource management, and fault self-healing capabilities. When a user submits a task flow to the K8s container orchestration platform, each subtask is executed through containerized deployment.

[0034] It should be understood that containerization refers to a lightweight virtualization technology in distributed computing platforms, which packages the application and all its dependencies, including code, runtime environment, system tools and configurations, into an independent and portable "container" to ensure that the application can run consistently and efficiently in any environment.

[0035] On the one hand, when a container executes a subtask to generate intermediate or final computation results, Kubernetes will mount the results to the local node's storage directory through a Persistent Volume Claim (PVC), or write them directly to a shared storage service, such as Network Attached Storage (NAS) or SimpleStorage Service (S3).

[0036] The core purpose of the existing stored procedures described above is to prevent data loss. Even if the subtask container is destroyed after execution, its computation results can still be saved through storage volumes or shared storage.

[0037] In subsequent cross-node tasks, if this result needs to be accessed, it needs to be manually configured by the user or the upper-layer application, or the data read operation needs to be manually invoked. The access address of the shared storage should be specified in the task configuration file, and the stored result should be obtained through path matching.

[0038] On the other hand, when a user submits a task flow, the user needs to declare parameters related to resource requests such as CPU and memory in the task configuration. The scheduler in the Kubernetes platform allocates subtask containers to suitable nodes based on rules such as the remaining amount of node resources and task priority.

[0039] The core of the existing scheduling process described above is to avoid resource constraints and conflicts. Tasks with high CPU requirements are assigned to nodes with high CPU idle rates, and closely related tasks are assigned to the same node to reduce network latency. However, this scheduling decision is based on the matching of resource supply and demand, and does not involve judging the task dependencies in the task computation logic or the specific task execution order.

[0040] However, in the task processing scenarios of the Kubernetes platform, although existing technologies have achieved the retention of computation results to prevent data loss through persistent storage volumes and shared storage, and have achieved the allocation of hardware resources to avoid resource conflicts through scheduling mechanisms based on the remaining amount of node resources and task priorities, there is still a problem of repeated execution of intermediate computation steps for the same input in multi-task flow and cross-node scenarios. This leads to the ineffective waste of memory and computing resources, as well as the problem of low overall task execution efficiency.

[0041] To address this, this application provides a task scheduling method to avoid the waste of memory and computing resources and low task execution efficiency caused by the repeated execution of intermediate calculation steps with the same input by different tasks when executing containerized tasks across multiple task flows and nodes.

[0042] Figure 1 This is a schematic diagram of a system provided in an embodiment of this application.

[0043] To address the aforementioned issues, this application provides a basic system architecture for the application of the task scheduling method, such as... Figure 1 As shown, the system 100 of this task scheduling method includes an electronic device with a distributed computing platform 101 deployed, a local cache 102, and a remote storage 103. A local hardware-level connection is established between the electronic device and the local cache 102. The local cache 102 is typically integrated inside the electronic device or directly connected to the electronic device via an interface such as a local bus, enabling low-latency read / write operations on cached data and providing hardware support for the distributed computing platform 101 running on the electronic device. The electronic device and the remote storage 103 are connected via network communication. The remote storage 103 establishes a remote data connection with the distributed computing platform 101 deployed on the electronic device. The distributed computing platform 101, relying on the network interface of the electronic device, initiates data interaction requests to the remote storage 103, realizing cross-node data interaction and persistent storage.

[0044] It should be understood that electronic devices provide hardware resources to support the deployment and operation of the distributed computing platform 101 on it. These hardware resources include, but are not limited to, computing power, network, and basic I / O. The electronic devices may include rack-mounted servers, edge computing nodes, virtual machines, etc. Local cache 102 provides low-latency data read and write for the distributed computing platform 101 on the electronic devices. Local cache 102 may include local high-speed solid-state drives (SSDs), processor-integrated high-speed caches, local cache servers, hardware cache accelerator cards, container local temporary cache volumes, etc. Remote storage 103 is used to achieve persistent storage of computation results and sharing across electronic devices and nodes. It may include the Hadoop Distributed File System (HDFS), the Ceph File System (CephFS), cloud disks, unified storage systems, etc.

[0045] Figure 2 This is a schematic diagram of the system architecture of an example implementation of the scheme provided in this application.

[0046] On top of the basic system, this application provides a system architecture for a distributed computing platform, such as... Figure 2 As shown, the execution carrier of the task scheduling method in this embodiment is a distributed computing platform 101, which also includes a local cache 102 and a remote storage 103. The distributed computing platform 101 may include multiple software modules that collaboratively implement task scheduling. The specific division, naming, and functional coupling of these modules can be adjusted according to the actual scenario, and this embodiment does not limit this.

[0047] For example, the distributed computing platform 101 may include core modules such as an interface server 201, a task flow controller 202, a directed acyclic graph dependency analysis module 203, computing nodes 204, a node cache manager 205, a persistent storage volume controller 206, and a lifecycle management module 207. However, this is only an exemplary module splitting and collaboration method. In practice, task scheduling can also be achieved by merging some module functions, adding other functional modules, and adjusting module interaction logic. The embodiments of this application are not limited to the above-described module division and collaboration forms.

[0048] It should be understood that the interface server 201 can serve as the interaction entry point between the system and the outside world to receive task flows submitted by users. However, its specific communication protocol, interface form, deployment quantity, etc., can be selected according to requirements, and this application embodiment does not limit this. The task flow controller 202 and the directed acyclic graph dependency analysis module 203 can be used to parse task dependencies, construct directed acyclic graphs, and generate execution sequences. They can also coordinate computing nodes and storage modules to complete the scheduling, execution, and result management of subtasks. However, the functional boundaries and specific algorithms of the two can be flexibly adjusted, and this application embodiment does not limit this. The computing node 204 is the software carrier that executes specific subtask calculations. It can correspond to computing instances on a distributed platform, such as containers, virtual machines, processes, etc. Its specific resource allocation method, CPU and memory scheduling strategy, runtime environment, operating system, and runtime framework can be determined according to actual computing needs, and this application embodiment does not limit this. The node cache manager 205 is responsible for the read and write control of the local cache 102, and the persistent storage volume controller 206 is responsible for the persistent management of the remote storage 103. The specific management strategies for storage, such as cache eviction policies, persistent synchronization policies, and storage media selection, can be configured as needed, and are not limited in this embodiment. The lifecycle management module 207 is responsible for monitoring the lifecycle status of tasks and each module. Its monitoring granularity and fault recovery strategies can also be flexibly configured according to the system scale and reliability requirements, and are not limited in this embodiment.

[0049] Based on the architecture of the aforementioned system and distributed computing platform, this application provides a task scheduling method that is suitable for scenarios where subtasks have clear dependencies, are prone to repetitive intermediate calculations, and require optimized utilization of computing / storage resources. Specifically, it can be applied to distributed computing platforms, such as the Kubernetes container orchestration platform and the Hadoop distributed computing framework. This application does not impose any restrictions on the application scenarios or the specific type of platform.

[0050] In this embodiment, the task scheduling method can be applied to a distributed computing platform including local cache and remote storage. The distributed computing platform acts as the main body for task scheduling, establishing data interaction with both the local cache and remote storage. In other words, the distributed computing platform is the carrier that coordinates the sub-task scheduling logic and the reuse of computation results between the local cache and remote storage.

[0051] It should be understood that the embodiments of this application can also be applied to more real-world scenarios, such as AI model training and inference scenarios involving the reuse of intermediate results during batch inference, cloud-native and cloud computing scenarios, enterprise complex business process automation scenarios, and scientific computing and engineering simulation scenarios. The embodiments of this application do not limit the application scenarios of the task scheduling method provided in this application.

[0052] Figure 3 This is a schematic flowchart of a task scheduling method provided in an embodiment of this application.

[0053] To address the issues of wasted memory and low execution efficiency in task flows composed of multiple subtasks within the distributed computing platform 101, this application provides a task scheduling method, exemplarily as follows: Figure 3 As shown, this method 300 can be applied to Figure 2 In the system 200 shown, the method 300 includes the following steps: S301 receives the task flow submitted by the user and extracts the dependencies of each subtask in the task flow.

[0054] The distributed computing platform 101 receives the task flow submitted by the user through the external interface provided by the interface server 201 or the network, and the task flow controller 202 in the distributed computing platform 101 parses the structure of the task flow.

[0055] For example, a user-submitted task flow contains multiple related subtasks. The distributed computing platform 101 identifies all the subtasks in the task flow and extracts the dependencies between them.

[0056] It should be understood that the dependency between subtasks can be a data dependency. For example, subtask B needs the output of subtask A as input. Alternatively, the dependency between subtasks can be an execution order dependency. For example, subtask C can only start after subtask B has finished executing. There may be no direct data transfer between subtask C and subtask B, but there are timing constraints between them. The dependency between subtasks can be a data dependency or an execution order dependency, and this application embodiment does not limit this.

[0057] The distributed computing platform 101 saves the dependencies between the extracted subtasks and temporarily stores them in the local cache 102 of the electronic device for subsequent steps.

[0058] S302, construct a directed acyclic graph of the task flow based on the dependencies between subtasks.

[0059] It should be understood that a Directed Acyclic Graph (DAG) is a special type of directed graph that does not contain any cycles. In a DAG, nodes represent tasks, directed edges represent dependencies between nodes, and arrows indicate directions. In a DAG, each edge has a direction, and there cannot be a path starting from a vertex, traversing several edges, and returning to that vertex.

[0060] The distributed computing platform 101 uses subtasks as nodes and the dependencies between subtasks as directed edges to construct a directed acyclic graph structure of task flow in the memory of the electronic device. For example, pointing from subtask A to subtask B means that subtask A is a predecessor subtask that subtask B depends on.

[0061] Table 1

[0062] Based on the task dependencies in Table 1, the dependencies between subtasks in this embodiment can exist in various ways. It is understood that subtask A is a starting subtask with no prerequisite dependencies, and is a fundamental subtask that can be executed first in the task flow. Subtask B depends only on one prerequisite subtask, subtask A, meaning that subtask B can only start after subtask A has been completed. Subtask D depends on both subtask A and subtask C, meaning that subtask D can only start execution after both subtask A and subtask C have been completed.

[0063] For example, the workflow controller deployed in the distributed computing platform 101 identifies the dependencies between subtasks by automatically parsing the implicit call relationships and data flow in the task code. Subsequently, the workflow controller passes these dependencies to the directed acyclic graph dependency analysis module 203, which constructs a structured directed acyclic graph in the local cache 102 as the basis for subsequent steps.

[0064] It should be understood that in the distributed computing platform 101, the method for constructing the task directed acyclic graph can be implemented in different ways. It can use the construction logic proprietary to the distributed computing framework, or use manual coding to customize the task directed acyclic graph. This application embodiment does not limit this.

[0065] After constructing the directed acyclic graph (DAG) of the task flow through the above steps, directly generating an execution sequence based on this DAG may lead to scheduling deadlocks, execution interruptions, or ineffective resource consumption due to hidden circular dependencies or missing preceding subtasks in the DAG. Therefore, this application requires the validity of the constructed DAG to be verified first.

[0066] Figure 4 This is a schematic flowchart illustrating a method for validating the directed acyclic graph in a task scheduling method provided in an embodiment of this application. Figure 4 As shown, this method 400 can be applied to Figure 2 In the system 200 shown, the logic of step S401 is consistent with that of step S302. After the construction of the directed acyclic graph is completed in step S302, the method 400 may further include the following steps: S402, check if there is a circular dependency in a directed acyclic graph.

[0067] After constructing the directed acyclic graph of the task flow, the task flow controller 202 of the distributed computing platform 101 triggers the circular dependency verification mechanism of the directed acyclic graph dependency analysis module 203. The sub-task nodes and dependency edges in the directed acyclic graph can be traversed through the graph traversal algorithm.

[0068] If, during the traversal, a subtask is found to be able to return to itself via a dependency path, then a circular dependency exists. Otherwise, a circular dependency does not exist.

[0069] For example, based on the task dependencies in Table 1, there exists a situation where subtask D depends on subtask C, subtask C depends on subtask E, and subtask E depends on subtask D. It can be understood that these three subtasks are mutually dependent, each requiring one to complete before the other can execute, forming a closed loop of subtask dependencies. Therefore, the directed acyclic graph dependency analysis module 203 determines that there is a risk of circular subtask dependency scheduling in this directed acyclic graph.

[0070] It should be understood that the verification algorithm for circular dependencies can be depth-first search, breadth-first search, or indirect verification based on topological sorting. The embodiments of this application do not limit the specific algorithm.

[0071] S403: If a circular dependency exists in the directed acyclic graph, terminate the scheduling of the task flow.

[0072] If a circular dependency is found in step S402, the task flow controller 202 will immediately terminate the scheduling process of the current task flow and notify the user that the directed acyclic graph verification failed because the task flow has a circular dependency and cannot be executed.

[0073] This can avoid task deadlock caused by circular dependencies and reduce invalid resource allocation and scheduling attempts.

[0074] S404. If there are no circular dependencies in the directed acyclic graph, verify whether the predecessor subtask of each subtask in the directed acyclic graph is complete.

[0075] After confirming that there are no circular dependencies, the task flow controller 202 controls the directed acyclic graph dependency analysis module 203 to further verify the integrity of the prerequisite dependencies of each subtask in the directed acyclic graph.

[0076] For example, for each subtask, check whether all its declared dependencies on the preceding subtasks are valid nodes in the directed acyclic graph of the task flow, referring to subtasks that have not been omitted or referenced.

[0077] For example, based on the task dependencies in Table 1, if there is a subtask F that declares a dependency on subtask Z, then the directed acyclic graph dependency analysis module 203 verifies that subtask Z does not exist in the directed acyclic graph of the current task flow. Therefore, it is determined that there is an incomplete pre-dependency of a subtask in the directed acyclic graph.

[0078] S405: If any subtask in the directed acyclic graph depends on an incomplete preceding subtask, terminate the scheduling of the task flow.

[0079] If step S404 checks and finds that a prerequisite subtask for a certain subtask is missing, meaning that subtask E, which subtask F depends on, does not exist in the directed acyclic graph of the task flow, the task flow controller 202 will terminate the scheduling process and return a prompt message to the user indicating that the prerequisite subtask is incomplete. Furthermore, the prompt message may include the identifier of the missing prerequisite subtask and the identifiers of the subtasks that depend on it.

[0080] This avoids insufficient input data during the execution of subsequent subtasks due to missing dependencies, ensuring that the task flow has the basic conditions for execution from the source.

[0081] S406. If all validity checks in the directed acyclic graph pass, generate the execution sequence of the task flow based on the directed acyclic graph.

[0082] After the verifications in steps S402 and S604 pass, it means that there are no circular dependencies in the directed acyclic graph (DAG) of the task flow, and that the predecessor subtasks dependent on each subtask in the DAG are complete. At this point, the task flow controller 202 determines that the DAG structure is valid through the DAG dependency analysis module 203, and then generates the execution sequence of the task flow based on the dependencies of the DAG. Specifically, the subtasks in the DAG are sorted using a topological sorting algorithm to obtain a linear execution order of all subtasks, providing a clear logical basis for the scheduling and execution of subsequent subtasks.

[0083] It should be understood that the generated execution sequence can be a unique linear sequence or a sequence containing parallel execution groups. It can be subtasks with no dependencies that can be executed in parallel. The specific form of the execution sequence is not limited in the embodiments of this application.

[0084] Thus, the method 400 provided in this application can improve the reliability of task scheduling by verifying in advance whether the directed acyclic graph of the task flow has circular dependencies and whether the dependent predecessor tasks are complete, thereby avoiding subsequent scheduling deadlocks, execution interruptions, and invalid resource consumption caused by the above problems.

[0085] S303, generates the execution sequence of task flow based on a directed acyclic graph.

[0086] For example, the workflow controller of the distributed computing platform 101 first calculates the in-degree for each subtask in the directed acyclic graph. It should be understood that the in-degree represents the number of directed edges pointing to a node. For instance, based on the subtask dependencies in Table 1, subtask A, which has no prerequisite dependencies, has an in-degree of 0; subtasks B and C, which depend on one prerequisite subtask, have an in-degree of 1; and subtask D, which depends on two prerequisite subtasks, has an in-degree of 2, meaning that subtask D can only be executed after subtasks A and C have both been completed. Simultaneously, the workflow controller initializes a queue of tasks to be executed, adding all subtasks with an in-degree of 0 (i.e., subtasks that can be executed directly) to the queue. Then, it decrements the in-degree of all subsequent subtasks of the subtasks in the queue by 1. If the in-degree of a subsequent subtask becomes 0, it is added to the queue of tasks to be executed as the next batch of executable tasks. This process is repeated until there are no more subsequent subtasks, meaning that all subtasks in the task flow have been included in the queue of tasks to be executed.

[0087] The linear execution sequence generated by the above process is stored in local cache 102, serving as the execution sequence for the distributed computing platform 101 to schedule subtasks. During subsequent execution scheduling, the distributed computing platform 101 can directly and quickly read the execution sequence from local cache 102 and trigger the processing flow of subtasks in sequence.

[0088] In this way, a directed acyclic graph is generated based on the dependency relationship of each subtask, and the execution sequence of all subtasks in the task flow is constructed, avoiding the problem of incorrect scheduling caused by dependency confusion when executing across nodes.

[0089] It should be understood that in the distributed computing platform 101, different algorithms can be used to generate the execution sequence. The classic topological sorting algorithm or the classic algorithm optimized by the distributed computing platform 101 can be used to generate the execution sequence of the task flow. This application embodiment does not limit this.

[0090] S304, Generate the current subtask label according to the execution sequence and the dependencies of the current subtask.

[0091] When the distributed computing platform 101 schedules and executes the current subtask in the execution sequence, it needs to generate a unique current subtask label based on the subtask's position in the execution sequence, the identity information of its dependent predecessor subtasks, and its own identity information. This label provides an identifiable basis for subsequent scheduling, result reuse, and other operations.

[0092] In one possible implementation, the current subtask tag includes at least the subtask's identity information, dependencies, and core parameters. The subtask's identity information includes its Identity Document (ID) and version number. The subtask's dependencies include at least the tags of all its predecessor subtasks. The subtask's core parameters include at least its input parameters. The tag generation process is handled by the task flow controller 202 of the distributed computing platform 101. After generation, the tag is stored in the local cache 102 and bound to the current subtask, serving as its unique tag within the workflow.

[0093] Thus, a unique subtask label is generated based on the execution sequence and dependencies of the current subtask. This unique subtask label is equivalent to assigning a unique "fingerprint" to each intermediate subtask; the execution sequence and dependencies contained in the label directly reflect the core characteristics of the subtask. If an intermediate subtask in different task flows has completely identical subtask labels, it means that the subtask has the basis for reuse.

[0094] It should be understood that the algorithm for generating tags can be a hash algorithm based on the combination of information from various dimensions, or it can be an algorithm with custom encoding rules, etc. The specific implementation method can be adjusted according to the performance requirements or compatibility requirements of the distributed computing platform 101, and this application embodiment does not limit it.

[0095] S305: Process the current subtask based on the current subtask label and obtain the calculation result of the current subtask.

[0096] The task flow controller 202 of the distributed computing platform 101 searches for the existence of a subtask label that is identical to the current subtask label. If an identical subtask label exists, the result is directly reused. If no identical subtask label exists, the computing node 204 is scheduled to execute the current subtask.

[0097] After the current subtask is completed, the computing node 204 will feed back the final calculation result to the task flow controller 202.

[0098] Based on the execution sequence generated in step S303, execute the next subtask and repeat the operations of steps S304 to S305 until all subtasks are executed.

[0099] In this way, the current subtask is processed and the calculation result is obtained based on the current subtask label. When a new task flow or other nodes need to execute this intermediate subtask, the existing calculation result is directly associated with the label, reducing the consumption of memory resources and the waste of computing power.

[0100] The above method has achieved ordered scheduling of task flow through the construction of directed acyclic graphs and the generation of execution sequences, which will enable the efficient reuse of subtask computation results.

[0101] Figure 5 This is an interaction diagram of an example task scheduling method provided in an embodiment of this application.

[0102] To achieve the processing of the current subtask based on the current subtask label and the acquisition of calculation results in method 300, this application provides method 500, which covers different retrieval and storage scenarios through the following steps, such as... Figure 5 As shown, step S305 specifically includes the following under different search conditions: S501 retrieves the current subtask tag from the local cache.

[0103] The task flow controller 202 in the distributed computing platform 101 uses the current subtask tag as an index and queries the local cache 102 through the node cache manager 205 to see if there is a matching cache record.

[0104] In one possible implementation, the task flow controller 202 sends a tag retrieval request to the node cache manager 205, which performs a retrieval based on the subtask tags stored in the local cache 102. If a subtask tag that exactly matches the current subtask tag is found, the node cache manager 205 returns a presence signal to the task flow controller 202. If no matching subtask tag is found, a non-existent signal is returned.

[0105] It should be understood that when the node cache manager 205 retrieves subtask tags based on the local cache 102, it can either use a full traversal matching method or achieve fast retrieval through a preset index structure. The specific algorithm logic for tag retrieval in this application embodiment is not limited.

[0106] S502, if the current subtask label exists in the local cache, determine the storage path associated with the current subtask label.

[0107] Once the task flow controller 202 of the distributed computing platform 101 confirms through the node cache manager 205 that the current subtask tag exists in the local cache 102, it needs to further determine the storage path associated with the tag.

[0108] Specifically, local cache 102 stores the mapping relationship between subtask tags and storage paths. Node cache manager 205 queries this mapping relationship based on the current subtask tag to obtain the corresponding local storage path. This path is used to locate the specific storage location of the calculation result in local cache 102, providing a basis for subsequent result retrieval.

[0109] It should be understood that the mapping relationship between subtask tags and storage paths can be generated by preset naming rules, or the tag hash value can be used as the path name, or it can be recorded by a separate metadata file. This application embodiment does not limit the maintenance method of the mapping.

[0110] S503: If the calculation result corresponding to the associated storage path exists in the local cache, return the calculation result under the storage path from the local cache.

[0111] Specifically, after the node cache manager 205 returns an existence signal to the task flow controller 202, the task flow controller 202 sends a result retrieval request to the node cache manager 205. Based on the current subtask label, the node cache manager 205 reads the corresponding calculation result from the local cache 102 and returns the calculation result to the task flow controller 202. The task flow controller 202 associates the retrieved result with the current subtask, marks the subtask status as completed (reused by the local cache 102), and provides data support for other tasks that depend on this subtask.

[0112] In one possible implementation, after obtaining the associated storage path, the node cache manager 205 checks whether a valid computation result exists under that path. If the result is confirmed to exist and be valid, the node cache manager 205 reads the computation result from that path and returns it to the task flow controller 202 through an interface.

[0113] The task flow controller 202 binds the result to the current subtask, marks its status as completed (reused by the local cache 102), and provides support for reusing computation results for subsequent tasks that depend on the subtask.

[0114] S504: If the calculation result corresponding to the associated storage path does not exist in the local cache, a request to retrieve the calculation result under the storage path is initiated from the remote storage.

[0115] If the node cache manager 205 finds that there are no valid computation results in the storage path associated with the local cache 102, for example, due to file corruption, deletion, or the computation results in that storage path never being saved, it sends a signal to the task flow controller 202 indicating that the local results are missing.

[0116] The task flow controller 202 then initiates a pull request to the remote storage 103 via the persistent storage volume controller 206. The pull request carries the current subtask tag and its associated storage path. This path can remain consistent between the local cache 102 and the remote storage 103, as both use the same path generation rules, facilitating cross-storage media location. It is used to instruct the remote storage 103 to return the computation results under the corresponding path.

[0117] In addition to the method described above that retrieves data based on the same storage path, the associated path can also be re-queried in remote storage 103 using tags. This can be a mapping relationship between tags and paths independent of remote storage 103, to accommodate scenarios where local and remote storage 103 have different path rules. This application will not elaborate on the implementation process for this method.

[0118] S505 returns the computation results from the remote storage path.

[0119] After receiving a pull request, the persistent storage volume controller 206 reads the calculation result from the remote storage 103 based on the storage path obtained from the local cache 102, or by relocating the corresponding path in the remote storage 103 through the subtask tag. This embodiment does not limit the specific location method of the remote path. After reading, the persistent storage volume controller 206 directly returns the calculation result to the task flow controller 202, which associates the result with the current subtask, marks the subtask status as completed (remote pull reuse), and provides calculation result support for subsequent tasks that depend on this subtask.

[0120] It should be understood that when returning results from remote storage 103, either the complete original result data can be returned, or the results can be compressed before transmission to reduce the data transmission overhead across storage media. This application embodiment does not limit the transmission format of the calculation results.

[0121] Thus, by obtaining the storage path of the computation result through local cache 102 and retrieving it from remote storage 103 when the computation result is missing, the scenario where local cache 102 has a label but no valid result is resolved. The association between subtask labels and storage paths ensures the accuracy of locating computation results in remote storage 103. Furthermore, the persistent storage feature of remote storage 103 provides support for cases where results from local cache 102 are missing, preventing task interruption or duplicate computations due to unavailable computation results in local cache 102.

[0122] For example, if the current subtask tag does not exist in the local cache 102, specifically, if the node cache manager 205 returns a non-existent signal, the task flow controller 202 in the distributed computing platform 101 will use the current subtask tag as an index to query whether a matching record exists in the remote storage 103 through the persistent storage volume controller 206.

[0123] S506: If the current subtask tag does not exist in the local cache, retrieve the current subtask tag from the remote storage.

[0124] In one possible implementation, the task flow controller 202 sends a tag retrieval request to the persistent storage volume controller 206, which performs a retrieval based on the subtask tags stored in the remote storage 103. If a subtask tag that exactly matches the current subtask tag is found, an existence signal is returned to the task flow controller 202. If no matching record is found, a non-existent signal is returned.

[0125] It should be understood that when the persistent storage volume controller 206 queries the remote storage 103, it can either retrieve a single subtask tag at a time or retrieve tags of multiple subtasks in the current execution sequence in batches to improve efficiency. This application embodiment does not limit the batch processing logic for tag retrieval.

[0126] In addition to the "direct retrieval based on subtask tags" described above, the search scope can also be narrowed by combining the tag's supplementary information, such as the task ID of the current subtask, creation time, and other information, so as to reduce the query overhead of the remote storage 103. This application will not elaborate on the implementation process of this.

[0127] For example, if the current subtask tag exists in remote storage 103, the calculation result of the current subtask is retrieved from remote storage 103. After persistent storage volume controller 206 returns an existence signal to task flow controller 202, task flow controller 202 sends a result retrieval request to it. Based on the current subtask tag, persistent storage volume controller 206 reads the corresponding calculation result from remote storage 103 and returns the result to task flow controller 202. Task flow controller 202 associates this result with the current subtask and marks its status as completed (reused by remote storage 103).

[0128] In one possible implementation, if the local cache 102 does not find the current subtask tag, the task flow controller 202 triggers a retrieval of the subtask tag in the remote storage 103. The task flow controller 202 initiates a tag retrieval request to the remote storage 103 through the persistent storage volume controller 206. This can be done by querying the database of subtask tags in the remote storage 103, or by traversing the storage directory of subtask tags in the remote storage 103. This embodiment of the application does not limit the specific implementation.

[0129] S507: If the current subtask label exists in the remote storage, return the calculation result from the storage path associated with the current subtask label in the remote storage.

[0130] If the current subtask tag is found in remote storage 103, it is confirmed that the current subtask has reusable calculation results.

[0131] Specifically, the current subtask label exists in remote storage 103, and the persistent storage volume controller 206 returns the path of the remote storage 103 corresponding to the label, providing a location basis for subsequent reading of calculation results.

[0132] The current subtask is associated with the remote storage path 103. The persistent storage volume controller 206 retrieves the result from this storage path in the remote storage 103 and returns it. The task flow controller 202 binds the result to the current subtask and marks its status as completed.

[0133] It should be understood that when returning results, the original data can be returned directly, or the results can be formatted according to the task requirements, such as converting binary data to structured data. This application embodiment does not limit the return format of the results.

[0134] Thus, the task scheduling method provided in this application, when the subtask tag is not found in the local cache 102, triggers a search in the remote storage 103. If the tag exists in the remote storage 103, the calculation result in the remote storage 103 is reused, optimizing the cross-storage retrieval logic for subtask tags. By adopting a search process that prioritizes local storage over remote storage, redundant calculations caused by incomplete searches are reduced, thereby ensuring the continuous execution of the task flow while further improving the resource utilization efficiency and result reuse reliability of the distributed computing platform.

[0135] S508: If the current subtask tag is not found in the remote storage, calculate the result of the current subtask.

[0136] When the task flow controller 202 of the distributed computing platform 101 searches the remote storage 103 through the persistent storage volume controller 206 and confirms that the current subtask tag does not exist in the remote storage 103, it means that neither the local cache 102 nor the remote storage 103 has any historical computation results for this subtask that can be reused. At this time, the persistent storage volume controller 206 returns a signal to the task flow controller 202 that the remote tag does not exist, triggering the actual computation process of the current subtask.

[0137] It should be understood that the retrieval of remote storage 103 can be either a full scan of the tag storage directory or a query of the preset subtask tag index library. This application embodiment does not limit the specific retrieval method.

[0138] Upon receiving a signal that the remote tag does not exist, the task flow controller 202 schedules the computing node 204 in the distributed computing platform 101 to execute the computing logic of the current subtask.

[0139] In one possible implementation, the task flow controller 202, based on the dependencies of the current subtask, obtains the computation results of all preceding subtasks from the local cache 102 or remote storage 103 through the node cache manager 205 or persistent storage volume controller 206, and uses them as input data for the current subtask.

[0140] S509 writes the calculation results to the local cache.

[0141] After obtaining the calculation results returned by the computing node 204, the task flow controller 202 writes the results to the local cache 102 through the node cache manager 205.

[0142] In one possible implementation, the node cache manager 205 generates a corresponding local storage path based on the current subtask label and stores the calculation result in the path. After the computing node 204 completes the calculation of the current subtask and feeds back the result to the task flow controller 202, the task flow controller 202 writes the calculation result into the local cache 102 through the node cache manager 205.

[0143] For example, the node cache manager 205 generates a target storage path according to preset rules, stores the calculation results in that path, completes the physical storage of the results locally, and provides a data foundation for rapid reuse in the short term.

[0144] S510 associates and stores the calculation results, target storage path, and current subtask label.

[0145] After writing the calculation results to the local cache 102, the node cache manager 205 establishes a mapping relationship between the current subtask label, the target storage path, and the calculation results in the index table of the local cache 102, and builds and maintains the local label index library.

[0146] Optionally, the calculation results may also include information such as the checksum, size, and generation time of the results, for subsequent verification of the validity of the results. This mapping relationship ensures that when searching through subtask tags, the corresponding storage path and calculation result information can be directly located, improving the efficiency and accuracy of local retrieval.

[0147] It should be understood that the carrier of associated storage can be either an independent data file or a hash table in memory. The embodiments of this application do not limit the specific form of associated storage.

[0148] S511 writes the calculation results to remote storage.

[0149] After completing the writing to the local cache 102, the task flow controller 202 synchronously writes the calculation results to the remote storage 103 through the persistent storage volume controller 206.

[0150] For example, the persistent storage volume controller 206 can reuse the storage path generated in step S509, or it can generate a storage path in the remote storage 103 based on the same rules and store the calculation result in the corresponding location of the remote storage 103.

[0151] In this way, writing to remote storage 103 can achieve long-term persistence of results, avoiding the problem of results being lost due to insufficient space in local cache 102 or failure of nodes in local cache 102, thus ensuring the reliability of calculation results and cross-node reuse capability under different conditions.

[0152] It should be understood that when writing to remote storage 103, either real-time synchronization or asynchronous batch writing can be used according to task priority, or the results of multiple subtasks can be aggregated and stored uniformly. This application embodiment does not limit the timing of writing.

[0153] S512 stores the calculation results, target storage path, and current subtask label together in remote storage.

[0154] After the calculation results are written to the remote storage 103, the persistent storage volume controller 206 builds and maintains a local tag index library in the remote storage 103.

[0155] For example, the mapping relationship between the current subtask label, the target storage path, and the calculation result identifier is stored in the remote storage 103 to ensure that when searching across nodes and across time, the storage location of the calculation result in the remote storage 103 can be directly located through the subtask label, thus ensuring the traceability and reuse reliability of the results in the remote storage 103.

[0156] In addition to the associated storage of "subtask tags, storage paths, and computation results" as described above, other subtask information such as dependencies and computation node information can be added to provide other extended information for task flow tracing. This application will not elaborate on the implementation process.

[0157] Thus, by establishing a mapping relationship between "subtask tag, storage path, and computation result" in the local cache 102 and the remote storage 103 respectively, and associating the storage in the local cache 102, it is ensured that results can be quickly located through tags in high-frequency reuse scenarios in the short term, improving retrieval efficiency. The mapping relationship is simultaneously established in the remote storage 103, ensuring the reusability of results in cross-node and long-term storage scenarios. Furthermore, by constructing indexes containing this mapping relationship in both the local cache and the remote storage, the retrieval capability of unique identifiers for task flow results in the distributed computing platform is strengthened, and data consistency is maintained through the consistency of the local and remote mappings, improving result reuse efficiency.

[0158] After the result processing, reuse, computation, and storage of the current subtask are completed, the distributed computing platform 101 schedules the next subtask to be executed based on the execution sequence generated in S303, repeats the processing steps from S501 to S512, until all subtasks in the task flow are executed, and finally completes the scheduling and execution of the entire task flow.

[0159] Figure 6 This is a schematic flowchart of a local cache management method in a task scheduling method provided in an embodiment of this application.

[0160] To avoid the problem of long-term unused subtask tags and corresponding calculation results occupying space in the local cache 102, thus limiting effective data storage, and to optimize resource utilization in the local cache 102 and ensure its response efficiency to frequently reused results, the task scheduling method provided in this application can also manage and optimize the space of the local cache 102. For example, ... Figure 6 As shown, this method 600 can be applied to Figure 2 In the system 200 shown, the method 600 further includes the following steps: S601, detect the duration for which subtask tags stored in the local cache have not been retrieved.

[0161] The node cache manager 205 of the distributed computing platform 101 will periodically, or may trigger a detection task when the storage space of the local cache 102 reaches a preset threshold. For all subtask tags stored in the local cache 102, the interval between the last time it was retrieved and the current detection time is calculated, which refers to the duration during which each subtask tag stored in the local cache 102 has not been retrieved.

[0162] For example, the node cache manager 205 pre-records the last retrieval time of each subtask tag in the index table of the local cache 102, and the last retrieval time is updated synchronously every time a subtask tag is retrieved.

[0163] When a detection task is triggered, the unsearched duration of each subtask tag is calculated by subtracting the last search time from the current system time, forming a list of subtask tag durations to be judged.

[0164] It should be understood that the detection triggering method can be either a fixed period, such as once per hour or after a preset time, or a dynamic triggering, such as triggering a detection task when the local cache 102 usage rate exceeds 80%. This application embodiment does not limit the detection frequency and triggering conditions.

[0165] S602, when the time exceeds the preset survival time, delete the subtask tags and the calculation results corresponding to the subtask tags stored in the local cache.

[0166] The node cache manager 205 compares the unretrieved duration calculated in step S601 with the preset subtask tag survival time.

[0167] For example, if a subtask tag is not retrieved for longer than its lifespan, the node cache manager 205 will perform a deletion operation. This involves deleting the calculation result file corresponding to that subtask tag from the local cache 102 and removing the mapping between the subtask tag and the storage path, thus completely releasing the occupied cache space.

[0168] If the time during which a subtask tag is not retrieved does not exceed its lifespan, then the subtask tag and its corresponding calculation result are retained and can be reused in subsequent possible scenarios.

[0169] After the deletion operation is completed, the node cache manager 205 can send a signal to the task flow controller 202 to indicate that the cache cleanup is complete, so that the task flow controller 202 can obtain the resource status of the local cache 102.

[0170] It should be understood that the preset survival time can be dynamically adjusted according to the subtask type, and the specific value of the survival time is not limited in the embodiments of this application.

[0171] Thus, the task scheduling method provided in this application constructs a dynamic local cache 102 resource management mechanism by performing timeliness detection and timeout cleanup on the unretrieved subtask tags and their results in the local cache 102, avoiding invalid data occupying resources and causing limited storage of effective data, and significantly improving the resource utilization of the local cache 102.

[0172] It should be understood that Figures 1 to 6 The scene diagrams or interaction diagrams shown are for illustrative purposes only and are not intended to limit the embodiments of this application to the examples illustrated. In fact, those skilled in the art can understand the implications based on these examples. Figures 1 to 6 The examples in the document can be transformed into equivalent ways to obtain more implementations.

[0173] The above text combined Figures 1 to 6 This document describes in detail the task scheduling method provided in the embodiments of this application. The following will combine... Figures 7 to 8 The apparatus embodiments of this application are described in detail below. It should be understood that the task scheduling apparatus of this application embodiments can execute the various task scheduling methods described in the foregoing embodiments of this application. That is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0174] In the embodiments described above, the distributed computing platform 101 can execute some or all of the steps in each embodiment. These steps or operations are merely examples, and other operations or variations thereof can also be performed in the embodiments of this application. Furthermore, the steps can be executed in different orders as presented in the embodiments, and it is not necessary to execute all the operations in the embodiments of this application. Moreover, the sequence number of each step does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0175] Figure 7 This is a schematic block diagram of a task scheduling apparatus provided in an embodiment of this application.

[0176] For example, such as Figure 7 As shown, the task scheduling device 700 includes an acquisition module 701, a generation module 702, a processing module 703, and a storage module 704.

[0177] The acquisition module 701 is used to receive the task flow submitted by the user through the interface and parse and extract the dependency relationship of each subtask in the task flow.

[0178] The generation module 702 is used to construct a directed acyclic graph of the task flow based on the subtask dependency relationship provided by the acquisition module 701; perform topological sorting on the directed acyclic graph to generate the execution sequence of the task flow; and generate a unique label for the current subtask by combining the execution sequence with the dependency relationship between the current subtask.

[0179] The processing module 703 is used to perform operations to retrieve local cache and remote storage based on the subtask tags output by the generation module 702; if there is a calculation result that matches the subtask tag, the calculation result is reused; if there is no matching calculation result, the computing node is scheduled to execute the current subtask and obtain the calculation result.

[0180] The storage module 704 is used to receive the calculation results output by the processing module 703, write the calculation results into the local cache and the remote storage respectively; and establish an association mapping between the subtask tag, the storage path of the calculation result and the calculation result itself in the local cache and the remote storage, so as to provide data support for subsequent retrieval and reuse.

[0181] Optionally, the functions of the above modules can be implemented by hardware or by executing corresponding software programs in hardware, and the hardware or software may contain sub-units corresponding to the above functions.

[0182] In one possible design, the task scheduling device 700 may correspond to the distributed computing platform 101 in the above method embodiments, or to a component (such as a circuit, chip, or chip system) configured in the distributed computing platform. The task scheduling device 700 can be used to execute the steps or processes performed by the distributed computing platform in any of the above method embodiments.

[0183] For example, the acquisition module 701 is used to implement the process of receiving the task flow submitted by the user and extracting the subtask dependencies in the above method, which can be S301 in method 300. The generation module 702 is used to implement the process of constructing a directed acyclic graph, generating execution sequences and subtask labels in the above method, which can be S302, S303, and S304 in method 300, and S607 in method 600 for generating execution sequences based on DAG. The processing module 703 is used to implement the process of retrieving cache, reusing results, or scheduling computing nodes to execute subtasks based on subtask labels in the above method, which can be S501 to S508 in method 500. The storage module 704 is used to implement the process of storing computing results and establishing a mapping between labels and results in the above method, which can be S509 to S512 in method 500. For simplicity, these will not be described in detail here.

[0184] The above are merely examples; for detailed steps or procedures, please refer to the descriptions in the foregoing embodiments.

[0185] Figure 8 This is a schematic block diagram of a task scheduling device provided in an embodiment of this application.

[0186] like Figure 8 As shown, the device 800 includes a processor 801, a memory 802, an executable program 803, and a communication interface 804.

[0187] It should be understood that the device 800 may be a chip, chip system, or processor, etc., in a terminal device or network device that implements the above methods. The device 800 can be used to implement the methods described in the above method embodiments, and for details, please refer to the description in the above method embodiments.

[0188] Among them, the processor 801 serves as the computing core, running software logic related to task scheduling. It can execute tasks such as task flow controller, directed acyclic graph analysis, and subtask label processing, and can also execute the entire process of instructions from receiving task flows to scheduling subtasks and reusing computation results.

[0189] The memory 802 can be used to temporarily store task flow data, directed acyclic graph structures, subtask execution sequences, locally cached computation results, and the mapping relationship between tags and results, providing the processor with low-latency data access capabilities and ensuring the real-time performance of the scheduling process. It can also be used to persistently store remote computation results, historical task tag and result association information, and the executable code of the task scheduler, which can be the executable program 803 in the diagram, to achieve long-term retention of task logic and results.

[0190] The communication interface 804 is used for external interaction. It can be used to receive user-submitted task flows, synchronize scheduling status between distributed nodes, and transmit computing data with remote storage. It supports cross-device transmission and distributed collaboration of task flows and serves as a bridge for interaction between the device and the external environment.

[0191] Device 800 also includes a bus, which serves as an internal data channel connecting the processor, memory, and communication interface to achieve efficient transmission of address, data, and control signals, ensuring collaborative operation between various hardware components.

[0192] With the cooperation of the aforementioned hardware components, the device 800 can complete the entire process of task flow reception, dependency analysis, scheduling execution, and result reuse, providing hardware support for efficient task scheduling in distributed scenarios.

[0193] According to the method provided in the embodiments of this application, this application also provides an electronic device for implementing the task scheduling method described in any of the foregoing embodiments. This electronic device can serve as a core control node, computing node, or management node in a distributed computing platform 101, achieving efficient scheduling of task flows through hardware and software collaboration.

[0194] Specifically, the electronic device includes a processor and a memory connected to the processor via a bus. The memory stores computer-executed instructions, which correspond to various steps of task scheduling in the aforementioned method embodiments. These instructions can be stored in the form of executable programs, function libraries, or configuration scripts, etc.

[0195] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, Digital Video Discs (DVDs), Compact Disc Read-Only Memory (CD-ROMs), microdrives, and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), dynamic random access memory (DRAMs), video random access memory (VRAMs), flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0196] The processor reads and executes the aforementioned computer execution instructions from memory, realizing the entire process of the task scheduling method by calling hardware resources. Through the collaborative work of the processor and memory, this electronic device can efficiently execute any of the aforementioned task scheduling methods, solving problems such as resource waste, low execution efficiency, and insufficient reliability of result reuse in distributed computing scenarios, thereby improving the stability and resource utilization of task flow processing.

[0197] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium for storing computer execution instructions that implement any of the task scheduling methods in the foregoing embodiments. When the instructions in the storage medium are executed by a processor, the electronic device where the processor is located can complete the entire task flow scheduling process.

[0198] The computer-readable storage medium can be any medium with data storage capabilities, including but not limited to volatile memory, such as random access memory, which can temporarily store instructions and intermediate data being executed by the processor, suitable for high-frequency access to temporary data processing during task scheduling. Alternatively, it can be non-volatile memory, such as read-only memory, flash memory, hard disk, solid-state drive, etc., which can store the core instructions of the task scheduling method for a long time, ensuring that instructions are not lost after power failure. It can also be a removable storage medium, such as an optical disc, USB flash drive, etc., facilitating the distribution and deployment of the task scheduler.

[0199] The computer execution instructions stored in the storage medium can be read and executed by the processor of the electronic device, and the steps implemented are completely consistent with the process described in the foregoing method embodiments.

[0200] Through this computer-readable storage medium, the aforementioned task scheduling method can be embedded into an executable program, which facilitates portability between different electronic devices or distributed computing nodes, ensures the standardized implementation of the method and cross-platform compatibility, and further expands the application scenarios of the task scheduling scheme.

[0201] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0202] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are illustrative. For example, the division of units is a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0204] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0205] In summary, the above are merely preferred embodiments of the technical solutions of this application and are not intended to limit the scope of protection of this application.

[0206] Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A task scheduling method applied to a distributed computing platform, characterized in that, The method includes: Receive the task flow submitted by the user and extract the dependency relationship of each subtask in the task flow; Based on the dependencies of the subtasks, a directed acyclic graph of the task flow is constructed, and an execution sequence of the task flow is generated based on the directed acyclic graph. According to the execution sequence and the dependency relationship of the current subtask, a label for the current subtask is generated, wherein the current subtask is any subtask in the task flow; The current subtask is processed based on the current subtask label, and the calculation result of the current subtask is obtained.

2. The method according to claim 1, characterized in that, The step of processing the current subtask based on the current subtask label and obtaining the calculation result of the current subtask includes: Based on the current subtask tag, the local cache is retrieved. If the current subtask tag exists in the local cache, the calculation result of the current subtask is obtained from the local cache. If the current subtask tag is not present in the local cache, retrieve the remote storage based on the current subtask tag; If the current subtask tag exists in the remote storage, retrieve the calculation result of the current subtask from the remote storage; If the current subtask tag is not present in the remote storage, calculate the result of the current subtask.

3. The method according to claim 2, characterized in that, The step of retrieving the calculation result of the current subtask from the local cache when the current subtask label exists in the local cache includes: Based on the current subtask tag, determine the storage path associated with the current subtask tag; Based on the storage path, check if the calculation result corresponding to the storage path exists in the local cache; When the calculation result corresponding to the storage path exists in the local cache, the calculation result is used as the calculation result of the current subtask; When the calculation result corresponding to the storage path is not found in the local cache, the calculation result of the current subtask is obtained from the remote storage based on the storage path.

4. The method according to claim 2, characterized in that, The step of retrieving the calculation result of the current subtask from the remote storage when the current subtask tag exists in the remote storage includes: Based on the current subtask label, determine the storage path associated with the current subtask label, and use the calculation result corresponding to the storage path in the remote storage as the calculation result of the current subtask.

5. The method according to any one of claims 2 to 4, characterized in that, When the current subtask tag is not present in the remote storage, the method further includes: The result of the current subtask is saved to the local cache and the remote storage.

6. The method according to claim 5, characterized in that, Saving the result of the current subtask to the local cache and the remote storage includes: Store the result of the current subtask to the target storage path, wherein the target storage path is associated with the tag of the current subtask; The result of the current subtask, the target storage path, and the label of the current subtask are saved to the local cache and the remote storage.

7. The method according to any one of claims 2 to 6, characterized in that, The method further includes: The duration for which subtask tags stored in the local cache have not been retrieved is detected; If the duration exceeds the preset survival time, delete the subtask tag and the calculation result corresponding to the subtask tag stored in the local cache.

8. The method according to any one of claims 1 to 7, characterized in that, Before generating the execution sequence of the task flow based on the directed acyclic graph, the method further includes: Detect whether there is a circular dependency in the directed acyclic graph, and whether the predecessor subtasks that the subtask depends on are complete; When the directed acyclic graph does not contain the cyclic dependency, and the preceding subtasks that each subtask depends on are complete, the execution sequence is generated based on the directed acyclic graph; If the directed acyclic graph contains the cyclic dependency, and / or the preceding subtask of the subtask is incomplete, the scheduling of the task flow is terminated, and a verification failure reminder is returned to the user who submitted the task flow.

9. An electronic device, characterized in that, Includes a processor and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the task scheduling method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the task scheduling method as described in any one of claims 1 to 8.

Images