Task execution method and system

Abstract

A task execution method and system, which belong to the technical field of computing. The task execution system schedules, to at least one sub-cluster in a computing cluster for execution, a plurality of sub-tasks comprised in a task to be executed, wherein at least two compute nodes in a first sub-cluster among the at least one sub-cluster are scheduled to execute at least two of the plurality of sub-tasks. The task execution system determines a first file that is required to be loaded when each of the at least two compute nodes executes a corresponding sub-task, and determines, on the basis of the first files that are required to be loaded by the at least two compute nodes, second files, the required loading quantity of which meets a threshold value. The task execution system loads the second files into a shared storage space of the first sub-cluster, and instructs compute nodes that are required to load the second files to acquire the second files from the shared storage space, so as to execute corresponding sub-tasks. The present application can improve task execution efficiency.

Description

Task execution method and system

[0001] The present application claims priority from Chinese Patent Application No. 202411280421.4, filed on September 12, 2024, entitled "Task execution method and system", the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of computing, and in particular, to a task execution method and system. BACKGROUND

[0003] A high performance computing (HPC) cluster, an artificial intelligence (AI) computing cluster, a data center computing cluster, etc. usually includes a scheduler and multiple compute nodes (NCs), and the scheduler is used to schedule the multiple compute nodes to execute tasks. Before a compute node executes a task, the compute node needs to load files required by the task from a storage system to a storage space of the compute node. The files required by the task are also referred to as files depended by the task or dependency files of the task.

[0004] A large-scale task usually includes multiple sub-tasks. For a large-scale task, the scheduler usually schedules multiple compute nodes to jointly execute the task, and each compute node executes at least one sub-task included in the task. Before each compute node of the multiple compute nodes executes a corresponding sub-task, the compute node needs to load files required by the sub-task from a storage system to a storage space of the compute node, and the multiple compute nodes concurrently load files required by corresponding sub-tasks from the storage system.

[0005] However, when multiple compute nodes concurrently load files from the storage system, congestion of the storage system is likely to occur, which affects the efficiency of task execution. SUMMARY

[0006] The present application provides a task execution method and system. The technical solution of the present application is as follows.

[0007] In a first aspect, a task execution method is provided. The method comprises: obtaining a task to be executed, the task comprising a plurality of sub-tasks; scheduling the plurality of sub-tasks to at least one sub-cluster in a computing cluster for execution, the at least one sub-cluster comprising a first sub-cluster, at least two computing nodes in the first sub-cluster being scheduled to execute at least two sub-tasks in the plurality of sub-tasks; determining a first file required to be loaded by each of the at least two computing nodes when executing a corresponding sub-task; determining a second file required to be loaded in an amount satisfying a threshold based on the first files required to be loaded by the at least two computing nodes; loading the second file to a shared storage space of the first sub-cluster, the shared storage space being accessible by the at least two computing nodes; and instructing a computing node in the at least two computing nodes that requires the second file to execute the corresponding sub-task to obtain the second file from the shared storage space.

[0008] In some embodiments, each sub-cluster comprises a plurality of computing nodes, and scheduling the plurality of sub-tasks to the at least one sub-cluster for execution specifically comprises scheduling the plurality of sub-tasks to the computing nodes in the at least one sub-cluster for execution. At least one computing node in each sub-cluster in the at least one sub-cluster is scheduled to execute at least one sub-task in the plurality of sub-tasks. One computing node can be scheduled to execute at least one sub-task, and each sub-task is scheduled to one computing node for execution. Optionally, the plurality of sub-tasks correspond to a plurality of processes one-to-one, and each sub-task in the plurality of sub-tasks is specifically scheduled to one process in one computing node for execution. The corresponding sub-task of each computing node is the sub-task scheduled to the each computing node for execution. The corresponding sub-task of each computing node is also referred to as the sub-task executed by the each computing node.

[0009] In some embodiments, the number of the first files required to be loaded by each computing node in the at least two computing nodes in the first sub-cluster when executing a corresponding sub-task is one or more. The second file is all or part of the first files, and the required loading amount of the second file is the number of the computing nodes in the first sub-cluster that require the second file to be loaded when executing a corresponding sub-task. The number of the second files is one or more. In the case where the number of the second files is multiple, the required loading amount of each second file satisfies the threshold.

[0010] In some embodiments, the second file is stored in a storage system (also referred to as a storage cluster), and loading the second file to the shared storage space of the first sub-cluster specifically comprises loading the second file from the storage system to the shared storage space. The shared storage space can be a memory space or a non-memory space. In the case where the shared storage space is a memory space, the shared storage space is also referred to as a shared memory or a shared memory space.

[0011] The task execution method provided in the present application is executed by a task execution system. In the technical solution provided in the present application, at least two computing nodes in a first sub-cluster are scheduled to execute at least two sub-tasks included in a to-be-executed task, and a shared storage space of the first sub-cluster can be accessed by the at least two computing nodes. The task execution system determines a second file needed to be loaded in an amount satisfying a threshold, according to a first file needed to be loaded by the at least two computing nodes when executing corresponding sub-tasks. The task execution system loads the second file from a storage system to the shared storage space, and instructs a computing node needing to load the second file among the at least two computing nodes to obtain the second file from the shared storage space to execute a corresponding sub-task. In this way, the computing node needing to load the second file does not need to load the second file from the storage system concurrently, which can reduce the probability of congestion of the storage system, improve the speed and efficiency of loading the second file, and thus improve the task execution efficiency. Moreover, the at least two computing nodes can be prevented from repeatedly loading the second file from the storage system, so as to avoid the waste of the storage space of the computing node and the fragmentation of the storage space of the computing node.

[0012] Optionally, the scheduling the plurality of sub-tasks to at least one sub-cluster in the computing cluster for execution includes: scheduling the plurality of sub-tasks to the at least one sub-cluster for execution according to a number of computing cores needed by each sub-task in the plurality of sub-tasks and a file needed by the each sub-task. The number of computing cores needed by the each sub-task is a number of computing cores needed for executing the each sub-task. The file needed by the each sub-task is a file needed for executing the each sub-task, and is also a file needed to be loaded when the computing node executes the each sub-task. The file needed by the each sub-task can also be referred to as a file depended by the each sub-task or a dependent file of the each sub-task.

[0013] Optionally, a number of same dependent files in the at least two sub-tasks scheduled to the first sub-cluster for execution (i.e., the files needed by the at least two sub-tasks) is greater than a first number.

[0014] The technical scheme provided in the application can schedule the subtasks with more same dependency files into the same subcluster, so that more second files can be determined in the dependency files of the at least two subtasks, and the second files are loaded from the storage system to the shared storage space of the first subcluster. Without the need for the computing nodes in the first subcluster to load the second files from the storage system, the pressure on the storage system and the probability of congestion of the storage system can be reduced, the speed and efficiency of loading the second files can be improved, and the efficiency of task execution can be improved.

[0015] Optionally, the scheduling the plurality of subtasks to the at least one subcluster for execution according to the number of computing cores required by each of the plurality of subtasks and the files required by the each of the plurality of subtasks comprises: obtaining affinity between the plurality of subtasks according to dependency files of the plurality of subtasks, that is, the files required by the plurality of subtasks, the affinity between any two of the plurality of subtasks is positively correlated with the number of same dependency files in the dependency files of the any two of the plurality of subtasks; and scheduling the plurality of subtasks to the at least one subcluster for execution according to the number of computing cores required by each of the plurality of subtasks and the affinity between the plurality of subtasks, wherein the affinity between the above at least two subtasks scheduled to the first subcluster for execution is greater than a first affinity.

[0016] Since the affinity between the above at least two subtasks scheduled to the first subcluster for execution is greater than the first affinity, the affinity between the at least two subtasks is greater. Since the affinity between any two of the plurality of subtasks is positively correlated with the number of same dependency files in the dependency files of the any two of the plurality of subtasks, the number of same dependency files in the dependency files of the at least two subtasks is greater. The first affinity is related to the first number, for example, the number of same dependency files in the dependency files of the two subtasks with the affinity greater than the first affinity is greater than the first number.

[0017] The technical scheme provided in the application can schedule the subtasks with more same dependency files into the same subcluster, so that more second files can be determined in the dependency files of the at least two subtasks, and the second files are loaded from the storage system to the shared storage space of the first subcluster. Without the need for the computing nodes in the first subcluster to load the second files from the storage system, the pressure on the storage system and the probability of congestion of the storage system can be reduced, the speed and efficiency of loading the second files can be improved, and the efficiency of task execution can be improved.

[0018] Optionally, the affinity between the plurality of sub-tasks is obtained according to the dependency files of the plurality of sub-tasks, comprising: obtaining the affinity between any two sub-tasks in the plurality of sub-tasks according to the number of dependency files of the any two sub-tasks and the number of same dependency files in the dependency files of the any two sub-tasks. In this implementation manner, the affinity between any two sub-tasks in the plurality of sub-tasks is positively correlated with the number of same dependency files in the dependency files of the any two sub-tasks.

[0019] Optionally, the affinity between the plurality of sub-tasks is obtained according to the dependency files of the plurality of sub-tasks, comprising: obtaining the affinity between any two sub-tasks in the plurality of sub-tasks according to the data amount of dependency files of the any two sub-tasks and the data amount of same dependency files in the dependency files of the any two sub-tasks. In this implementation manner, the affinity between any two sub-tasks in the plurality of sub-tasks is positively correlated with the data amount of same dependency files in the dependency files of the any two sub-tasks. For example, the affinity between any two sub-tasks in the plurality of sub-tasks is positively correlated with the total data amount of same dependency files in the dependency files of the any two sub-tasks, and in the case that the total data amount of same dependency files in the dependency files of the any two sub-tasks is large, it is also possible to indicate that the number of same dependency files in the dependency files of the any two sub-tasks is large.

[0020] Optionally, the first file needed to be loaded when each computing node in the at least two computing nodes executes a corresponding sub-task is determined, comprising: determining the sub-task executed by each computing node in the at least two computing nodes; determining the file needed by each sub-task executed by each computing node in the at least two computing nodes; and determining the union set of the file needed by the sub-task executed by each computing node in the at least two computing nodes as the first file needed to be loaded when the each computing node executes a corresponding sub-task.

[0021] In the present application, the file needed by each sub-task executed by each computing node is also referred to as the dependency file of the each sub-task, and thus the union set of the file needed by each sub-task executed by each computing node is the union set of the dependency files of at least one sub-task executed by the each computing node.

[0022] Optionally, determining the second files needed to be loaded in quantity satisfying the threshold according to the first files needed to be loaded by the at least two computing nodes comprises: sorting the first files needed to be loaded by the at least two computing nodes in descending order of quantity needed to be loaded; and determining the first files whose quantity needed to be loaded is greater than or equal to the threshold as the second files. That is, the first files whose quantity needed to be loaded is greater than or equal to the threshold are determined as the second files. Wherein, the quantity needed to be loaded of each first file is the number of computing nodes in the at least two computing nodes that need to load the each first file. The quantity needed to be loaded of the second file is the number of computing nodes in the at least two computing nodes that need to load the second file. The quantity needed to be loaded of the second file being greater than or equal to the threshold means that the number of computing nodes in the at least two computing nodes that need to load the second file is greater than or equal to the threshold. The number of the second files is one or more, and in the case that the number of the second files is more, the quantity needed to be loaded of each second file is greater than or equal to the threshold. That is, the number of computing nodes in the at least two computing nodes that need to load each second file is greater than or equal to the threshold.

[0023] The technical scheme provided in the present application can load the files (i.e., the second files) needed to be loaded by more computing nodes in the at least two computing nodes from the storage system to the shared storage space of the first sub-cluster without the need for these computing nodes to load the second files from the storage system concurrently, which can reduce the pressure on the storage system and the probability of congestion of the storage system, improve the performance of the computing nodes in loading files from the storage system, and thus improve the efficiency of task execution.

[0024] Optionally, instructing the computing nodes in the at least two computing nodes that need to load the second files to obtain the second files from the shared storage space to execute corresponding sub-tasks comprises: instructing each computing node in the at least two computing nodes that need to load the second files to mount the shared storage space to the each computing node, so that the each computing node obtains the second files from the shared storage space.

[0025] Optionally, the method further comprises: determining the size of the shared storage space according to the size of the second files; and allocating the shared storage space in the storage pool of the first sub-cluster according to the size of the shared storage space. In the case that the number of the second files is more, the size of the shared storage space is determined according to the size of the plurality of second files, and the size of the shared storage space is greater than the total size of the plurality of second files. Wherein, the first sub-cluster comprises a plurality of computing nodes, and the computing nodes in the first sub-cluster share the storage resources in the storage pool of the first sub-cluster.

[0026] The technical solution provided in the application determines the size of the shared storage space according to the size of the second file, which can facilitate allocation of the shared storage space capable of storing the second file in the storage pool of the first sub-cluster according to the size of the shared storage space.

[0027] Optionally, the task execution system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster.

[0028] The task execution system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster.

[0029] The task execution system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster.

[0030] The task execution system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster.

[0031] The task execution system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster.

[0032] The task execution system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster.

[0033] The task execution system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster.

[0034] Optionally, the method further comprises determining a first loading node in the first sub-cluster. The first loading node is any computing node in the first sub-cluster. Alternatively, the first loading node is a computing node in the first sub-cluster that is scheduled to execute at least one of the plurality of sub-tasks; that is, the first loading node is one of the at least two computing nodes in the first sub-cluster that is scheduled to execute at least one of the plurality of sub-tasks.

[0035] Optionally, the first loading node is a computing node in the first sub-cluster that is scheduled to execute at least one of the plurality of sub-tasks (that is, the first loading node is one of the at least two computing nodes in the first sub-cluster that is scheduled to execute at least one of the plurality of sub-tasks), and the number of files (that is, the second files) in the first files that need to be loaded by the first loading node to execute the corresponding sub-task meets the threshold is greater than the target number; or the total data amount of the files (that is, the second files) in the first files that need to be loaded by the first loading node to execute the corresponding sub-task meets the threshold is greater than the target data amount. For example, the first loading node is a computing node in the first sub-cluster that needs to load the most second files to execute the corresponding sub-task (that is, the first loading node is one of the at least two computing nodes in the first sub-cluster that needs to load the most second files to execute the corresponding sub-task); or the first loading node is a computing node in the first sub-cluster that needs to load the most total data amount of second files to execute the corresponding sub-task (that is, the first loading node is one of the at least two computing nodes in the first sub-cluster that needs to load the most total data amount of second files to execute the corresponding sub-task).

[0036] Optionally, determining the first loading node in the first sub-cluster comprises determining the first loading node among the at least two computing nodes in the first sub-cluster according to the second files that need to be loaded by the at least two computing nodes.

[0037] Optionally, the first manager is a management node located in the first sub-cluster (that is, the first manager is located in the first sub-cluster and is deployed independently of the computing nodes); or the first manager is located in the same device as any computing node in the first sub-cluster; or the first manager is located in the same device as the scheduler. In the case where the first manager is located in the same device as the computing nodes in the first sub-cluster, or in the case where the first manager is located in the same device as the scheduler, the first manager can be implemented based on software, hardware, or a combination of software and hardware.

[0038] Optionally, the sub-cluster is a supernode.

[0039] Optionally, the storage resources in the storage pool comprise at least one of volatile storage resources or non-volatile storage resources.

[0040] In a second aspect, a task execution system is provided, which comprises at least one function module for executing the task execution method provided in the first aspect or any possible implementation of the first aspect. The at least one function module can be implemented based on software, hardware or a combination of software and hardware, and the at least one function module can be combined or split based on specific implementation.

[0041] In a third aspect, a task execution system is provided, which comprises a memory and a processor; the memory is configured to store a computer program; and the processor is configured to execute the computer program stored in the memory to enable the task execution system to execute the task execution method provided in the first aspect or any possible implementation of the first aspect. The task execution system can be an internal system of a device.

[0042] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed to implement the task execution method provided in the first aspect or any possible implementation of the first aspect.

[0043] In a fifth aspect, a computer program product is provided, which comprises a program or code, and the program or code is executed to implement the task execution method provided in the first aspect or any possible implementation of the first aspect.

[0044] In a sixth aspect, a chip is provided, which comprises a programmable logic circuit and / or program instructions, and the chip is configured to implement the task execution method provided in the first aspect or any possible implementation of the first aspect when the chip is running.

[0045] The technical effects of the second aspect to the sixth aspect can refer to the technical effects of the first aspect and the possible implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0046] FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0047] FIG. 2 is a flowchart of a task execution method provided by an embodiment of the present application;

[0048] FIG. 3 is a schematic diagram of a manager instructing a computing node to mount a shared storage space provided by an embodiment of the present application;

[0049] FIG. 4 is a schematic diagram of a file loading node instructing a computing node to mount a shared storage space provided by an embodiment of the present application;

[0050] FIG. 5 is a schematic diagram of a task execution method in a conventional technology;

[0051] FIG. 6 is a schematic diagram of a task execution method according to an embodiment of the present application;

[0052] FIG. 7 is a schematic diagram of a task execution system according to an embodiment of the present application;

[0053] FIG. 8 is a schematic diagram of another task execution system according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] A high performance computing (HPC) cluster, an artificial intelligence (AI) computing cluster, a data center computing cluster, etc. usually includes a scheduler and multiple compute nodes (NCs), and the scheduler is configured to schedule the multiple compute nodes to execute a task. Files required by the task are usually stored in a storage system (also referred to as a storage cluster), and the compute nodes are connected to the storage system via a network. Before the compute nodes execute the task, the compute nodes need to load (i.e., read) the files required by the task from the storage system to the storage space of the compute nodes. The task is also referred to as a job.

[0056] A large-scale task usually includes multiple sub-tasks. Moreover, the larger the scale of the task is, the more sub-tasks the task usually includes. For a large-scale task, the scheduler usually schedules multiple compute nodes to jointly execute the task, and each compute node executes at least one sub-task included in the task. Before each compute node executes the corresponding sub-task, the compute node needs to load the files required by the sub-task from the storage system to the storage space of the compute node, and the multiple compute nodes concurrently load the files required by the corresponding sub-tasks from the storage system. The files required by the task are also referred to as the files depended on by the task or the dependent files of the task. The files required by the sub-task are also referred to as the files depended on by the sub-task or the dependent files of the sub-task.

[0057] However, multiple computing nodes concurrently load files from the storage system, which easily causes the storage system to be congested, and thus causes the input output (IO) efficiency of the storage system to be low, the performance (e.g., slow speed) of the computing nodes to load files from the storage system to be poor, and the efficiency of task execution to be affected. Specifically, the storage system includes a management node and multiple storage nodes, the multiple storage nodes are configured to store files, the management node is configured to manage the multiple storage nodes, and each computing node in the computing cluster is connected to the management node through a network. When the computing node loads files from the storage system, the computing node sends a load request (also referred to as a read request) for requesting to load files to the management node. The management node buffers each received load request in a cache queue of the management node, and processes the load requests in the cache queue according to the order of the load requests in the cache queue. When the management node processes any load request in the cache queue, the management node reads a file requested to be loaded by the load request from the multiple storage nodes according to the load request, and sends a load response (also referred to as a read response) including the file to the computing node corresponding to the load request (i.e., the computing node sending the load request). After the computing node receives the read load response, the computing node obtains the file from the load response, and stores the file to a storage space of the computing node, thereby realizing loading the file from the storage system to the storage space of the computing node. In a scenario where a scheduler schedules multiple computing nodes to jointly execute a task, the multiple computing nodes concurrently send load requests to the management node, and the management node buffers the load requests sent by the multiple computing nodes in the cache queue of the management node, which easily causes the management node to be congested, and thus causes the response time of the management node to the load requests to be too long, the IO efficiency of the storage system to be too low, the performance of the computing nodes to load files from the storage system to be poor, and the efficiency of task execution to be affected. Moreover, the larger the size of the task is, the more computing nodes the scheduler needs to schedule, and the more load requests these computing nodes concurrently send to the management node, which easily causes the management node to be congested, and thus causes the IO efficiency of the storage system to be lower. The IO efficiency of the storage system refers to the efficiency of writing data (or files) to the storage system and reading data (or files) from the storage system.

[0058] In addition, the files required by a task (i.e., the dependent files of the task) usually include program files, library files, configuration files, data files, and the like. The program files, library files, configuration files, and the like required by different sub-tasks of the same task are usually the same. There can also be the same files in the data files required by different sub-tasks of the same task. In the scenario where a scheduler schedules multiple computing nodes to execute multiple sub-tasks of the same task, the multiple computing nodes concurrently load the same files from the storage system, for example, the multiple computing nodes concurrently load the same program files, the same library files, and the same configuration files from the storage system. This easily causes a large instantaneous load of the storage system, affecting the file loading speed and efficiency. Moreover, this easily causes the waste of the storage space of the computing nodes and the fragmentation of the storage space of the computing nodes. It also causes the waste of bandwidth between the computing cluster and the storage system. The larger the size of a task is, the more computing nodes the scheduler needs to schedule, and the more same files the computing nodes repeatedly load from the storage system, and the more obvious these problems caused by repeatedly loading files are.

[0059] The embodiment of the present application provides a technical solution, which can reduce the probability that the computing nodes loading files from the storage system causes the storage system to be congested, improve the IO efficiency of the storage system and the performance of the computing nodes loading files from the storage system, and thus improve the task execution efficiency. Moreover, the same files loaded repeatedly from the storage system by the computing nodes can be avoided, and thus the waste and fragmentation of the storage space of the computing nodes can be avoided.

[0060] The technical solution of the embodiment of the present application is introduced below. First, the application scenario of the embodiment of the present application is introduced.

[0061] Please refer to FIG. 1, which shows a schematic diagram of an application scenario provided by the embodiment of the present application. The application scenario includes a computing cluster 10 and a storage system 20. The computing cluster 10 and the storage system 20 are connected through a network. The storage system 20 is used to store files required by a task. The computing cluster 10 is used to load the files required by the task from the storage system 20 and execute the task based on the files required by the task. The files required by the task are also called the dependent files of the task or the dependent files of the task. The computing cluster 10 can be an HPC cluster, an AI computing cluster, or a data center computing cluster, and the computing cluster 10 can be a distributed computing cluster. The network used to connect the computing cluster 10 and the storage system 20 includes a local area network, the Internet, a private network, and the like, and the embodiment of the present application does not limit the network used to connect the computing cluster 10 and the storage system 20.

[0062] In the embodiments of the present application, the computing cluster 10 includes a scheduler 100 and a plurality of sub-clusters. Each sub-cluster includes a manager, a storage pool and a plurality of computing nodes. The scheduler 100 is connected with the managers of the plurality of sub-clusters. The manager of each sub-cluster is connected with the computing nodes in the each sub-cluster and the storage pool in the each sub-cluster respectively, and the computing nodes in each sub-cluster are connected with the storage pool in the each sub-cluster. The manager of each sub-cluster is configured to manage the computing nodes in the each sub-cluster and the storage pool in the each sub-cluster. The computing nodes in each sub-cluster share the storage resources in the storage pool in the each sub-cluster. The manager of any sub-cluster can be a management node deployed independently in the any sub-cluster. Alternatively, the manager of any sub-cluster is located in the same device as any computing node in the any sub-cluster. Alternatively, the manager of any sub-cluster is located in the same device as the scheduler 100. In the case that the manager of a sub-cluster is located in the same device as the computing nodes in the sub-cluster, or in the case that the manager of the sub-cluster is located in the same device as the scheduler 100, the manager of the sub-cluster can be implemented based on software, hardware or a combination of software and hardware. The managers of the plurality of sub-clusters can be the same manager, or the managers of at least two sub-clusters in the plurality of sub-clusters are different, and the embodiments of the present application do not limit this.

[0063] The storage resources in the storage pool can include at least one of volatile storage resources or non-volatile storage resources. The volatile storage resources can be volatile memory resources. The non-volatile storage resources can include non-volatile memory resources, non-memory resources (such as disk resources) and the like. The storage resources in the storage pool in each sub-cluster can include memory resources of the computing nodes in the each sub-cluster or non-memory resources of the computing nodes in the each sub-cluster. For any sub-cluster, in the case that the sub-cluster further includes storage devices independent of the computing nodes, the storage resources in the storage pool in the sub-cluster can further include storage resources of the storage devices independent of the computing nodes. The storage pool in any sub-cluster in the plurality of sub-clusters can be a memory resource pool formed by converging memory resources in the any sub-cluster (including memory resources of the computing nodes in the any sub-cluster and / or memory resources of other nodes in the any sub-cluster), or a storage resource pool formed by converging non-memory resources in the any sub-cluster (including non-memory resources of the computing nodes in the any sub-cluster and / or non-memory resources of other nodes in the any sub-cluster). The memory resource pool is also referred to as a memory pool (MP), and the storage resource pool formed by converging non-memory resources can be a high-speed shared storage pool (HSSP).

[0064] The computing nodes in each sub-cluster are physically close computing nodes, and the computing nodes in each sub-cluster are connected through a high-speed network. Each computing node includes a processor, and each processor includes at least one computing core, which can be a processor core.

[0065] Figure 1 illustrates an example in which the computing cluster 10 includes k sub-clusters, each sub-cluster includes m computing nodes, and the manager of each sub-cluster is an independently deployed management node. k is a positive integer, and m is an integer greater than 1. For example, both k and m are integers greater than 1. As shown in Figure 1, the sub-cluster 1 includes the manager 1, the storage pool 1, and the computing nodes 11-1m, the manager 1 is connected with the scheduler 100, the computing nodes 11-1m, and the storage pool 1 respectively, and the computing nodes 11-1m are connected with the storage pool 1, the manager 1 is used to manage the computing nodes 11-1m and the storage pool 1, the storage pool 1 can be a storage resource pool formed by aggregating the memory resources or non-memory resources of the computing nodes 11-1m, and the computing nodes 11-1m share the storage resources in the storage pool 1. The sub-cluster 2 includes the manager 2, the storage pool 2, and the computing nodes 21-2m, the manager 2 is connected with the scheduler 100, the computing nodes 21-2m, and the storage pool 2 respectively, and the computing nodes 21-2m are connected with the storage pool 2, the manager 2 is used to manage the computing nodes 21-2m and the storage pool 2, the storage pool 2 can be a storage resource pool formed by aggregating the memory resources and / or non-memory resources of the computing nodes 21-2m, and the computing nodes 21-2m share the storage resources in the storage pool 2. Similarly, the sub-cluster k includes the manager k, the storage pool k, and the computing nodes k1-km, the manager k is connected with the scheduler 100, the computing nodes k1-km, and the storage pool k respectively, and the computing nodes k1-km are connected with the storage pool k, the manager k is used to manage the computing nodes k1-km and the storage pool k, the storage pool k can be a storage resource pool formed by aggregating the memory resources and / or non-memory resources of the computing nodes k1-km, and the computing nodes k1-km share the storage resources in the storage pool k. Each of the computing nodes 11-1m, the computing nodes 21-2m, and the computing nodes k1-km includes a processor (not shown in Figure 1), and each processor includes at least one computing core (not shown in Figure 1), which can be a processor core.

[0066] In the embodiments of the present application, the storage system 20 can be a parallel file system (PFS) or other distributed file system, and the storage system 20 stores files in a distributed storage manner. The storage system 20 is also referred to as a storage cluster or a distributed storage cluster. The storage system 20 can include a management node and a plurality of storage nodes (details of the storage system 20 are not shown in FIG. 1), the management node is connected with the plurality of storage nodes through a network, the management node is used to manage the plurality of storage nodes, and the plurality of storage nodes are used to store files. The computing cluster 10 is connected with the management node through a network (for example, each computing node in the computing cluster 10 is connected with the management node through a network). The computing nodes in the computing cluster 10 can load files required by tasks from the storage system 20. When each computing node in the computing cluster 10 loads a file from the storage system 20, the computing node sends a loading request for requesting to load the file to the management node. The management node caches each loading request in a cache queue of the management node as the management node receives the loading request, and the management node processes the loading request according to an order of the loading request in the cache queue. When the management node processes any loading request, the management node reads a file requested to be loaded by the loading request from the plurality of storage nodes according to the loading request, and sends a loading response carrying the file to a computing node corresponding to the loading request; after the computing node receives the loading response, the computing node obtains the file from the loading response, and the computing node stores the file to a storage pool in a sub-cluster where the computing node is located. In this way, the file is loaded from the storage system 20 to the computing cluster 10.

[0067] In the embodiments of the present application, the computing node can be a computing device or a node deployed in a computing device and having computing capability, and the computing device can be a computing server or other device having computing capability. The storage node can be a storage device, which can be a storage server, a redundant array of independent drives (RAID), a disk, a hard disk, an optical disk, a host logical unit number (LUN), a host volume, or other device having storage capability. The scheduler can be a physical device having scheduling capability or software deployed in a physical device and having scheduling capability. The manager can be a physical device having management capability or software deployed in a physical device and having management capability. In the case where the computing node is a node deployed in a computing device and having computing capability, different computing nodes can be deployed in the same computing device or different computing devices. In the case where the manager is software deployed in a physical device and having management capability, different managers can be deployed in the same physical device or different physical devices. All or part of the computing nodes, managers, and schedulers can be deployed in the same physical device or different physical devices, and the embodiments of the present application do not limit this.

[0068] In the embodiments of the present application, the scheduler 100 is configured to: obtain a to-be-executed task, the to-be-executed task including a plurality of subtasks; schedule the plurality of subtasks to at least one subcluster in the computing cluster for execution, the at least one subcluster including a first subcluster, at least two computing nodes in the first subcluster being scheduled to execute at least two subtasks in the plurality of subtasks; determine a first file needed to be loaded by each computing node in the at least two computing nodes when executing the corresponding subtask; and determine a second file needed to be loaded in an amount satisfying a threshold X_th according to the first files needed to be loaded by the at least two computing nodes. The first loading node is configured to: load the second file to a shared storage space of the first subcluster, and instruct the computing nodes in the at least two computing nodes that need to load the second file to obtain the second file from the shared storage space to execute the corresponding subtask. Alternatively, the first loading node is configured to: load the second file to a shared storage space of the first subcluster; and the first manager is configured to: instruct the computing nodes in the at least two computing nodes that need to load the second file to obtain the second file from the shared storage space to execute the corresponding subtask. The shared storage space can be accessed by the at least two computing nodes.

[0069] Alternatively, the scheduler 100 is configured to: obtain a task to be executed, the task comprising a plurality of sub-tasks; schedule the plurality of sub-tasks to at least one sub-cluster in the computing cluster for execution, the at least one sub-cluster comprising a first sub-cluster, at least two computing nodes in the first sub-cluster being scheduled to execute at least two sub-tasks in the plurality of sub-tasks. The first manager is configured to: determine a first file required to be loaded by each of the at least two computing nodes for executing the corresponding sub-task; determine a second file required to be loaded based on the first files required to be loaded by the at least two computing nodes, the second file required to be loaded satisfying a threshold X_th in quantity. The first loading node is configured to: load the second file to a shared storage space of the first sub-cluster, and instruct the at least two computing nodes requiring the second file to obtain the second file from the shared storage space for executing the corresponding sub-task. Alternatively, the first loading node is configured to: load the second file to a shared storage space of the first sub-cluster; and the first manager is configured to: instruct the at least two computing nodes requiring the second file to obtain the second file from the shared storage space for executing the corresponding sub-task. Wherein the shared storage space is accessible by the at least two computing nodes.

[0070] The scheduling the plurality of sub-tasks to the at least one sub-cluster for execution is specifically scheduling the plurality of sub-tasks to computing nodes in the at least one sub-cluster for execution. At least one computing node in each sub-cluster of the at least one sub-cluster is scheduled to execute at least one sub-task of the plurality of sub-tasks. One computing node can be scheduled to execute at least one sub-task, and each sub-task is scheduled to one computing node for execution. The sub-task corresponding to each computing node is the sub-task scheduled to the each computing node for execution. The sub-task corresponding to each computing node is also referred to as the sub-task executed by the each computing node. The second file is stored in the storage system 20, and loading the second file to the shared storage space of the first sub-cluster is specifically loading the second file from the storage system 20 to the shared storage space. The shared storage space is a shared storage space allocated in the storage pool of the first sub-cluster for storing the second file. The first manager is a manager of the first sub-cluster. The first manager can be a management node deployed independently in the first sub-cluster. Alternatively, the first manager is located in the same device as any computing node in the first sub-cluster. Alternatively, the first manager is located in the same device as the scheduler 100. In the case that the first manager is located in the same device as the computing node in the first sub-cluster, or in the case that the first manager is located in the same device as the scheduler 100, the first manager can be implemented based on software, hardware, or a combination of both. The first loading node is one computing node in the first sub-cluster, and the first loading node is a file loading node in the first sub-cluster. The file loading node is also referred to as a file loading computing node, a data loading node, a data loading computing node, or an input output computing node (IOCN), which is not limited in the embodiments of the present application.

[0071] In an optional embodiment, the first loading node is a computing node in the first sub-cluster that is scheduled to execute at least one sub-task of the plurality of sub-tasks, i.e., the first loading node is one of the at least two computing nodes in the first sub-cluster that is scheduled to execute the at least one sub-task, and the number of files in the first files that need to be loaded by the first loading node to execute the corresponding sub-task and have a loading quantity satisfying the threshold X_th (i.e., the second files) is greater than the target number Q_tar, or the total data quantity of the files in the first files that need to be loaded by the first loading node to execute the corresponding sub-task and have a loading quantity satisfying the threshold X_th (i.e., the second files) is greater than the target data quantity D_tar. For example, the first loading node is a computing node in the first sub-cluster that has the largest number of second files that need to be loaded to execute the corresponding sub-task (i.e., the first loading node is one of the at least two computing nodes in the first sub-cluster that has the largest number of second files that need to be loaded to execute the corresponding sub-task), or the first loading node is a computing node in the first sub-cluster that has the largest total data quantity of second files that need to be loaded to execute the corresponding sub-task (i.e., the first loading node is one of the at least two computing nodes in the first sub-cluster that has the largest total data quantity of second files that need to be loaded to execute the corresponding sub-task).

[0072] In the embodiments of the present application, the at least two computing nodes in the first sub-cluster are scheduled to execute at least two sub-tasks included in the to-be-executed task, and the shared storage space of the first sub-cluster can be accessed by the at least two computing nodes. The files in the first files that need to be loaded by one of the computing nodes in the first sub-cluster (i.e., the first loading node) to execute the corresponding sub-task and satisfy the threshold X_th (i.e., the second files) are loaded from the storage system 20 to the shared storage space of the first sub-cluster for the computing nodes in the at least two computing nodes that need to load the second files to obtain. In this way, the computing nodes that need to load the second files do not need to concurrently load the second files from the storage system 20, the amount of data read by the computing nodes from the storage system 20 and the number of loading requests (i.e., read requests) sent to the storage system 20 by the computing nodes to load the files from the storage system 20 can be reduced, the pressure on the storage system 20 and the probability of congestion of the storage system 20 are reduced, the IO efficiency of the storage system 20 and the performance of the computing nodes to load the files from the storage system 20 are improved, and thus the execution speed and efficiency of the task are improved. Moreover, repeated loading of the same files from the storage system 20 by multiple computing nodes in the computing cluster 10 can be avoided, and thus the waste of storage space of the computing nodes and the fragmentation of the storage space of the computing nodes are avoided. In addition, the bandwidth waste between the computing cluster 10 and the storage system 20 is avoided.

[0073] Referring to FIG. 1, the first sub-cluster can be any one of the sub-clusters 1-k, and the first loading node can be any computing node in the first sub-cluster. Which sub-cluster of the sub-clusters 1-k is the first sub-cluster depends on the number of computing cores required by each of the plurality of sub-tasks, the files required by each of the plurality of sub-tasks, and the actual situation of the sub-clusters 1-k (for example, the number of idle computing cores in the computing nodes in the sub-clusters 1-k). Which computing node in the first sub-cluster is the first loading node depends on the number of second files or the total data volume required to be loaded when executing the corresponding sub-task among the computing nodes in the first sub-cluster that are scheduled to execute the at least two sub-tasks described above. For example, the first sub-cluster is the sub-cluster 1, and the first loading node is the computing node 11, which is not limited by the embodiments of the present application.

[0074] It should be noted that FIG. 1 illustrates an example in which each of the sub-clusters 1-k includes m computing nodes. Those skilled in the art can understand that this is merely an example, and in actual applications, the number of computing nodes in each sub-cluster can be determined according to actual conditions, and the number of computing nodes in different sub-clusters can be equal or not equal. In optional embodiments, the sub-cluster is a super node, the storage pool in the sub-cluster is a super node MP, and the manager of the sub-cluster is a super node resource manager. The super node is a cluster formed by computing nodes with close physical distance through high-speed network connection. The communication bandwidth between the computing nodes in the super node is larger and the communication latency is smaller than that between the computing nodes in a conventional cluster. The super node MP is a memory resource pool formed by aggregating the memory resources of the computing nodes in the super node based on super node MP technology. The super node MP is managed by the super node resource manager, and the memory resources in the super node MP are shared by the computing nodes in the super node. The super node resource manager can dynamically allocate memory resources in the super node MP of the super node to the computing nodes in the super node according to the usage of the memory resources in the super node MP. The memory resources allocated by the super node resource manager to any computing node in the super node in the super node MP can be local memory resources of the computing node or memory resources of other computing nodes in the super node. Thus, each computing node in the super node can access not only the local memory resources of the computing node but also the memory resources of other computing nodes in the super node. Since the computing nodes in the super node are connected through high-speed network, each computing node in the super node can access the memory resources of other computing nodes in the super node at a high speed and with small latency. The super node MP technology makes the use of the memory resources of the computing nodes more flexible and helps to improve the memory utilization of the computing nodes. In optional embodiments, the super node is a computing rack, and the super node resource manager is a rack manager (RM). That is, the sub-cluster in the embodiments of the present application can be a computing rack. The computing rack usually includes computing nodes, RMs, switches and other devices, the computing nodes in the computing rack are computing nodes with relatively close contact in the computing cluster, and the RM is a management device responsible for managing and controlling the computing nodes in the computing rack.

[0075] It should be understood that the above introduction of the application scenario is only for example and does not limit the technical solutions of the embodiments of the present application. In actual application, other devices can also be included in the application scenario. For example, the application scenario can also include a resource planner (not shown in FIG. 1), which is also called a task planner. The resource planner is configured to receive an execution request sent by a client for requesting execution of the to-be-executed task, and split the to-be-executed task into the plurality of sub-tasks according to the execution request. The scheduler 100 is configured to obtain the to-be-executed task including the plurality of sub-tasks from the resource planner. Those skilled in the art can know that the application scenario can be adjusted as appropriate with changes in business requirements, and the embodiments of the present application do not enumerate here.

[0076] The above is the introduction of the application scenario of the embodiments of the present application. The method embodiments of the present application are introduced below.

[0077] Please refer to FIG. 2, which shows a flowchart of a task execution method provided by an embodiment of the present application. The task execution method is executed by a task execution system. The task execution system includes a scheduler, a first manager and a first loading node in a computing cluster. The computing cluster includes a plurality of sub-clusters. Each sub-cluster includes a manager, a storage pool and a plurality of computing nodes sharing storage resources in the storage pool. Each computing node includes a processor, and each processor includes at least one computing core. The plurality of sub-clusters includes a first sub-cluster, the first manager is the manager of the first sub-cluster, and the first loading node is a file loading node in the first sub-cluster. For example, the computing cluster is the computing cluster 10 in FIG. 1, the scheduler is the scheduler 100, and the first sub-cluster is any one of the sub-clusters 1-k.

[0078] Referring to FIG. 2, the task execution method includes the following steps S201 to S206.

[0079] S201. Obtain a to-be-executed task A, which includes a plurality of sub-tasks.

[0080] The task execution system can obtain the to-be-executed task A from a resource planner. In a specific embodiment, the task execution system includes a scheduler, and the scheduler obtains the to-be-executed task A from the resource planner. The resource planner is configured to obtain an execution request for requesting execution of the to-be-executed task A, and split the to-be-executed task A into the plurality of sub-tasks according to the execution request. For example, a user submits the to-be-executed task A on a client (such as a user device) to trigger the execution request for requesting execution of the to-be-executed task A, the client sends the execution request to the resource planner, the resource planner receives the execution request, and the resource planner splits the to-be-executed task A into the plurality of sub-tasks according to the execution request.

[0081] In a specific embodiment, the execution request carries a number of processes required by the to-be-executed task A, and the processes required by the to-be-executed task A are the processes required for executing the to-be-executed task A. The resource planner splits the to-be-executed task A into a plurality of sub-tasks according to the number of processes required by the to-be-executed task A, and the plurality of sub-tasks correspond to the plurality of processes one by one. In an optional embodiment, the execution request further carries a number of computing cores required by each process in the plurality of processes, and the resource planner splits the to-be-executed task A into the plurality of sub-tasks according to the number of processes required by the to-be-executed task A and the number of computing cores required by each process in the plurality of processes.

[0082] In an optional embodiment, the execution request further carries information (for example, the identification of a file) of all or part of the files required by each process in the plurality of processes. The files required by each process include program files, library files, configuration files, data files, and the like. The files required by different processes can be the same or different. For example, the program files, the library files, and the configuration files required by the plurality of processes are the same, the data files required by some processes in the plurality of processes are the same, and the data files required by other processes are different. For each process in the plurality of processes: in the case that the execution request carries the information of all the files required by the each process, the resource planner determines all the files required by the each process according to the information of all the files required by the each process; in the case that the execution request carries the information of part of the files required by the each process, the resource planner determines the part of the files required by the each process according to the information of the part of the files required by the each process, and determines another part of the files required by the each process according to the part of the files required by the each process, so that the resource planner can determine all the files required by the each process. In an embodiment, the execution request carries the identification of the program files and the identification of the data files required by each process in the plurality of processes; the resource planner determines the program files required by the each process according to the identification of the program files required by the each process carried by the execution request; the resource planner determines the data files required by the each process according to the identification of the data files required by the each process carried by the execution request; and the resource planner determines the library files and the configuration files required by the each process according to the program files required by the each process and the data files required by the each process. For example, the resource planner determines the library files and the configuration files required for executing the program files based on the data files according to the program files and the data files required by the each process; the resource planner determines the library files required for executing the program files based on the data files as the library files required by the each process; and the resource planner determines the configuration files required for executing the program files based on the data files as the configuration files required by the each process.

[0083] After the resource planner splits the to-be-executed task A into the plurality of sub-tasks, the resource planner determines the number of computing cores required by the process corresponding to each of the plurality of sub-tasks as the number of computing cores required by the each of the plurality of sub-tasks, and determines the files required by the process corresponding to each of the plurality of sub-tasks as the files required by the each of the plurality of sub-tasks. The resource planner generates information of the to-be-executed task A according to the number of computing cores required by each of the plurality of sub-tasks and the information of the files required by each of the plurality of sub-tasks. In an optional embodiment, the resource planner generates a to-be-loaded file list corresponding to each of the plurality of sub-tasks according to the information of the files required by the each of the plurality of sub-tasks, each to-be-loaded file list corresponding to each of the plurality of sub-tasks includes the information of the files required by the each of the plurality of sub-tasks, and the resource planner generates the information of the to-be-executed task A according to the number of computing cores required by the process corresponding to each of the plurality of sub-tasks and the plurality of to-be-loaded file lists corresponding to the plurality of sub-tasks. Since the files required by different sub-tasks of the plurality of sub-tasks can be the same, there can be the same to-be-loaded file list in the plurality of to-be-loaded file lists. It can be understood that the files required by each of the plurality of sub-tasks include program files, library files, configuration files, data files, etc. The files required by different sub-tasks can be the same or different. For example, the program files, the library files, and the configuration files required by the plurality of sub-tasks are the same, the data files required by some of the plurality of sub-tasks are the same, and the data files required by other sub-tasks are different. It should be noted that the description herein is only by way of example, and in actual applications, the program files, the library files, and the configuration files required by different sub-tasks of the plurality of sub-tasks can also be different, and the number of each type of files in the files required by each of the plurality of sub-tasks can be one or more. The files required by each of the plurality of sub-tasks can depend on the files required by the process corresponding to the each of the plurality of sub-tasks. The files required by a sub-task are also referred to as the files depended on by the sub-task or the dependent files of the sub-task.

[0084] In an optional embodiment, the execution request further carries an identification of the task A to be executed. For example, the identification of the task A to be executed is ID-A, and the number of processes required by the task A to be executed is 10. Each of the 5 processes among the 10 processes requires 20 computing cores, and each of the 5 processes requires the program file 1, the library file 1, the configuration file 1, and the data file 1. Each of the other 5 processes among the 10 processes requires 30 computing cores, and each of the other 5 processes requires the program file 1, the library file 1, the configuration file 1, and the data file 2. Assume that the identification of the program file 1 is ID-C1, the identification of the library file 1 is ID-W1, the identification of the configuration file 1 is ID-P1, the identification of the data file 1 is ID-D1, and the identification of the data file 2 is ID-D2.

[0085] As an example, the content of the execution request includes: "the to-be-executed task "ID-A" requires 10 processes, each of the 5 processes requires 20 computing cores, and the files required by each of the 5 processes are program file "ID-C1", library file "ID-W1", configuration file "ID-P1" and data file "ID-D1", each of the other 5 processes requires 30 computing cores, and the files required by each of the other 5 processes are program file "ID-C1", library file "ID-W1", configuration file "ID-P1" and data file "ID-D2". The resource planner splits the to-be-executed task A into 10 sub-tasks according to the execution request, and the 10 sub-tasks correspond to the 10 processes one by one. And the resource planner determines that the files required by each of the 5 processes in the 10 processes are program file 1, library file 1, configuration file 1 and data file 1, and the files required by each of the other 5 processes are program file 1, library file 1, configuration file 1 and data file 2 according to the execution request. The resource planner determines the number of computing cores required by the process corresponding to each of the 10 sub-tasks as the number of computing cores required by the each sub-task, and determines the files required by the process corresponding to each of the 10 sub-tasks as the files required by the each sub-task. The resource planner generates the information of the to-be-executed task A according to the identification of the to-be-executed task A, the number of computing cores required by each of the 10 sub-tasks and the information of the files required by each of the 10 sub-tasks. The information of the to-be-executed task A can include: "the to-be-executed task "ID-A" includes 10 sub-tasks, each of the 5 sub-tasks requires 20 computing cores, and the files required by each of the 5 sub-tasks are program file "ID-C1", library file "ID-W1", configuration file "ID-P1" and data file "ID-D1", each of the other 5 sub-tasks requires 30 computing cores, and the files required by each of the other 5 sub-tasks are program file "ID-C1", library file "ID-W1", configuration file "ID-P1" and data file "ID-D2".

[0086] As another example, the content of the execution request includes: "the to-be-executed task "ID-A" requires 10 processes, each of the 5 processes requires 20 computing cores, and the files required by each of the 5 processes include the program file "ID-C1" and the data file "ID-D1", each of the other 5 processes requires 30 computing cores, and the files required by each of the other 5 processes include the program file "ID-C1" and the data file "ID-D2". The resource planner splits the to-be-executed task A into 10 sub-tasks according to the execution request, and the 10 sub-tasks correspond to the 10 processes one by one. And the resource planner determines that the files required by each of the 5 processes of the 10 processes include the program file 1 and the data file 1, and the files required by each of the other 5 processes include the program file 1 and the data file 2 according to the execution request. The resource planner determines that the library file 1 and the configuration file 1 are required for executing the program file 1 based on the data file 1, and the library file 1 and the configuration file 1 are required for executing the program file 1 based on the data file 2, so the resource planner determines that the files required by each of the 5 processes include the program file 1, the library file 1, the configuration file 1 and the data file 1, and the files required by each of the other 5 processes include the program file 1, the library file 1, the configuration file 1 and the data file 2. The resource planner determines the number of computing cores required by the process corresponding to each of the 10 sub-tasks as the number of computing cores required by each of the 10 sub-tasks, and determines the files required by the process corresponding to each of the 10 sub-tasks as the files required by each of the 10 sub-tasks. The resource planner generates the information of the to-be-executed task A according to the identification of the to-be-executed task A, the number of computing cores required by each of the 10 sub-tasks and the information of the files required by each of the 10 sub-tasks.

[0087] It should be noted that, in the case where the above execution request carries the information of a part of the files required by each of the above plurality of processes, the embodiments of the present application are illustrated by taking an example in which the resource planner determines a part of the files required by each process according to the information of the part of the files required by each process, determines another part of the files required by each process according to the part of the files required by each process, and determines all the files required by each process as all the files required by the sub-task corresponding to each process. In other embodiments, the resource planner determines a part of the files required by each process according to the information of the part of the files required by each process, and determines the part of the files required by each process as a part of the files required by the sub-task corresponding to each process, and the scheduler determines another part of the files required by each sub-task according to the part of the files required by each sub-task, and further determines all the files required by each sub-task, and the embodiments of the present application are not limited thereto.

[0088] S202. The plurality of sub-tasks are scheduled to be executed by at least one sub-cluster in the computing cluster, the at least one sub-cluster comprising the first sub-cluster, at least two computing nodes in the first sub-cluster are scheduled to execute at least two sub-tasks in the plurality of sub-tasks.

[0089] In an optional embodiment, the task execution system determines the number of computing cores required by each sub-task in the plurality of sub-tasks and the files required by each sub-task in the plurality of sub-tasks, and schedules the plurality of sub-tasks to be executed by the at least one sub-cluster according to the number of computing cores required by each sub-task in the plurality of sub-tasks and the files required by each sub-task. The number of computing cores required by each sub-task is the number of computing cores required to execute the sub-task, and the files required by each sub-task are the files required to execute the sub-task. The files required by a sub-task are also referred to as the files on which the sub-task depends or the dependent files of the sub-task. The task execution system schedules the plurality of sub-tasks to be executed by the at least one sub-cluster specifically by scheduling the plurality of sub-tasks to be executed by the computing nodes in the at least one sub-cluster. For any computing node, the number of idle computing cores (i.e. the number of idle computing cores in the processors in the computing node) in the computing node is greater than the number of computing cores required by any sub-task before the any sub-task is scheduled to be executed by the computing node. An idle computing core is a computing core in an idle state, i.e. an unoccupied computing core.

[0090] In the embodiments of the present application, at least two computing nodes in the first sub-cluster are scheduled to execute at least two sub-tasks in the plurality of sub-tasks, each of the at least two computing nodes is scheduled to execute one or more sub-tasks in the at least two sub-tasks, and each of the at least two sub-tasks is scheduled to be executed by one of the at least two computing nodes. For the convenience of description, the file required by a sub-task is referred to as a dependent file of the sub-task. In the embodiments of the present application, the number of the same dependent files in the dependent files of the at least two sub-tasks scheduled to be executed by the first sub-cluster (i.e., the at least two sub-tasks scheduled to be executed by the at least two computing nodes in the first sub-cluster) is greater than a first number. The first number can be set according to actual conditions, and in different scenarios, the first number can be equal or not equal. Since the number of the same dependent files in the dependent files of the at least two sub-tasks scheduled to be executed by the first sub-cluster is greater than the first number, the number of the same dependent files in the dependent files of the at least two sub-tasks is relatively large. Thus, the embodiments of the present application can realize scheduling sub-tasks with a relatively large number of the same dependent files to the same sub-cluster. For example, the embodiments of the present application can realize scheduling sub-tasks with more same dependent files to the same sub-cluster. The same dependent files in the dependent files of the at least two sub-tasks can be the dependent files of all the sub-tasks in the at least two sub-tasks, or can be the dependent files of part of the sub-tasks in the at least two sub-tasks. For example, the dependent files of all the sub-tasks in the at least two sub-tasks include file 1, and file 1 is the same dependent file in the dependent files of the at least two sub-tasks. For another example, the dependent files of part of the sub-tasks (the part of the sub-tasks is a plurality of sub-tasks) in the at least two sub-tasks include file 1, the dependent files of another part of the sub-tasks do not include file 1, and file 1 is also the same dependent file in the dependent files of the at least two sub-tasks. In one example, the at least two sub-tasks are five sub-tasks, and the dependent files of the five sub-tasks include program file 1, and program file 1 is the same dependent file in the dependent files of the five sub-tasks. In another example, the at least two sub-tasks are five sub-tasks, the dependent files of three sub-tasks in the five sub-tasks include program file 1, and the dependent files of the other two sub-tasks do not include program file 1, and program file 1 is also the same dependent file in the dependent files of the five sub-tasks.

[0091] In specific embodiments, the task execution system obtains information of a to-be-executed task A, the information of the to-be-executed task A including the number of computing cores required by each of the plurality of subtasks and information (e.g., the identifier of the file) of the file required by each of the plurality of subtasks; the task execution system determines the number of computing cores required by each of the plurality of subtasks and the file required by each of the plurality of subtasks according to the information of the to-be-executed task A. For ease of description, the file required by a subtask is referred to as the dependent file of the subtask. The task execution system obtains the affinity between the plurality of subtasks according to the dependent files of the plurality of subtasks; the task execution system schedules the plurality of subtasks to the at least one sub-cluster for execution according to the number of computing cores required by each of the plurality of subtasks and the affinity between the plurality of subtasks. The affinity between any two subtasks is positively correlated with the number of the same dependent files in the dependent files of the any two subtasks. The affinity between the at least two subtasks scheduled to the first sub-cluster for execution is greater than the first affinity. The first affinity can be set according to actual conditions, and in different scenarios, the first affinities can be equal or unequal. Since the affinity between the at least two subtasks scheduled to the first sub-cluster for execution is greater than the first affinity, the affinity between the at least two subtasks is greater, and since the affinity between any two subtasks is positively correlated with the number of the same dependent files in the dependent files of the any two subtasks, the number of the same dependent files in the dependent files of the at least two subtasks is greater. The first affinity is related to the first number. For example, the number of the same dependent files in the dependent files of any two subtasks with an affinity greater than the first affinity is greater than the first number.

[0092] In specific embodiments, the task execution system obtains the affinity between any two subtasks in the plurality of subtasks according to the dependent files of the any two subtasks, and the task execution system can obtain the affinity between each of the plurality of subtasks and each of the other subtasks in the plurality of subtasks. That is, the task execution system can obtain the affinity between the plurality of subtasks. The task execution system schedules the subtasks with an affinity greater than the first affinity in the plurality of subtasks to the same sub-cluster, and the same sub-cluster is the first sub-cluster.

[0093] In one embodiment, the task execution system obtains the affinity between any two of the plurality of sub-tasks according to the number of dependent files of the two sub-tasks and the number of identical dependent files of the two sub-tasks. In a specific embodiment, the task execution system calculates the affinity between any two of the plurality of sub-tasks according to the number of dependent files of the two sub-tasks and the number of identical dependent files of the two sub-tasks using a first affinity formula. The first affinity formula is C = 2xn / (x+y). C represents the affinity between the two sub-tasks, n represents the number of identical dependent files of the two sub-tasks, x represents the number of dependent files of one of the two sub-tasks, y represents the number of dependent files of the other of the two sub-tasks, n is an integer greater than or equal to 0, x and y are positive integers, and the symbol " / " represents a division sign. When n = 0, the affinity between the two sub-tasks is 0. In one example, the dependent files of one of the two sub-tasks are program file 1, library file 1, configuration file 1, and data file 1, and the dependent files of the other of the two sub-tasks are program file 1, library file 1, configuration file 1, and data file 2. Then, the identical dependent files of the two sub-tasks are program file 1, library file 1, and configuration file 1, the number of identical dependent files of the two sub-tasks is 3, the number of dependent files of the two sub-tasks is 4, that is, n = 3, x = 4, and y = 4, and the affinity between the two sub-tasks is C = 2xn / (x+y) = 2x3 / (4+4) = 3 / 4.

[0094] The above description takes the task execution system obtaining the affinity between any two of the plurality of sub-tasks according to the number of dependent files of the two sub-tasks as an example. The task execution system can also obtain the affinity between any two of the plurality of sub-tasks according to the data amount of dependent files of the two sub-tasks. The data amount of dependent files is also the size of dependent files. For example, the task execution system obtains the affinity between any two of the plurality of sub-tasks according to the data amount of dependent files of the two sub-tasks and the data amount of identical dependent files of the two sub-tasks. In this implementation, the affinity between any two of the plurality of sub-tasks is positively correlated with the data amount of identical dependent files of the two sub-tasks. In a specific embodiment, the task execution system calculates the affinity between any two of the plurality of sub-tasks according to the data amount of dependent files of the two sub-tasks and the data amount of identical dependent files of the two sub-tasks using a second affinity formula.

[0095] wherein the second affinity formula is

[0096] C represents the affinity between the two sub-tasks, n represents the number of same dependency files in the dependency files of the two sub-tasks, x represents the number of dependency files of one of the two sub-tasks, y represents the number of dependency files of the other of the two sub-tasks, D i represents the data amount of the i-th same dependency file in the dependency files of the two sub-tasks, D j represents the data amount of the j-th dependency file of the one sub-task, D p represents the data amount of the p-th dependency file of the other sub-task, n is an integer greater than or equal to 0, and x and y are positive integers. When n = 0, the affinity between the two sub-tasks is 0. In one example, in two of the plurality of sub-tasks, the dependency files of one sub-task are program file 1, library file 1, configuration file 1 and data file 1, and the dependency files of the other sub-task are program file 1, library file 1, configuration file 1 and data file 2, the data amount of program file 1 is D1, the data amount of library file 1 is D2, the data amount of configuration file 1 is D3, the data amount of data file 1 is D4, and the data amount of data file 2 is D5. The same dependency files in the dependency files of the two sub-tasks are program file 1, library file 1 and configuration file 1, the number of same dependency files in the dependency files of the two sub-tasks is 3, the number of dependency files of the two sub-tasks is 4, that is, n = 3, x = 4, y = 4, and the affinity between the two sub-tasks is C = 2 × (D1 + D2 + D3) / [(D1 + D2 + D3 + D4) + (D1 + D2 + D3 + D5)].

[0097] In an optional embodiment, the task execution system comprises a scheduler, and S202 is performed by the scheduler.

[0098] S203. Determine the first file needed to be loaded when each of the at least two computing nodes executes the corresponding sub-task.

[0099] The at least two computing nodes in S203 are the computing nodes in the first sub-cluster that are scheduled to execute the at least two sub-tasks. The number of first files needed to be loaded when each of the at least two computing nodes executes the corresponding sub-task is one or more.

[0100] In an optional embodiment, the task execution system determines a subtask performed by each of the at least two computing nodes. The task execution system determines files required by each subtask performed by each of the at least two computing nodes. The task execution system determines a union of the files required by each subtask performed by each of the at least two computing nodes. The task execution system determines the union of the files required by each subtask performed by each of the at least two computing nodes as first files required to be loaded by the each computing node when performing a corresponding subtask. Each subtask performed by each computing node is a subtask scheduled to be performed by the each computing node, also referred to as a corresponding subtask of the each computing node. Each of the at least two computing nodes corresponds to at least one subtask. Each file required by each subtask performed by each computing node is a file required to be loaded by the each computing node when performing the each subtask. The union of the files required by each subtask performed by each computing node is a union of files required by the at least one subtask performed by the each computing node. Each file required by each subtask is also referred to as a dependency file of the each subtask, thus the union of the files required by each subtask performed by each computing node is a union of dependency files of the at least one subtask performed by the each computing node.

[0101] Referring to FIG. 1, as an example, the first subcluster is subcluster 1, the above-mentioned multiple subtasks of task A to be executed include subtasks 1-10, the above-mentioned at least two subtasks of the multiple subtasks scheduled to be performed by the first subcluster are subtasks 1-5, the above-mentioned at least two computing nodes of the first subcluster scheduled to perform the at least two subtasks include computing node 11 and computing node 12, and subtasks 1 and 2 are scheduled to be performed by computing node 11 (subtasks 1 and 2 are subtasks performed by computing node 11), subtasks 3, 4 and 5 are scheduled to be performed by computing node 12 (subtasks 3, 4 and 5 are subtasks performed by computing node 12).

[0102] For ease of description, a file required by a subtask is referred to as a dependency file of the subtask.

[0103] The task execution system determines that the sub-tasks executed by the computing node 11 are sub-task 1 and sub-task 2, determines the dependent files of the sub-task 1 (i.e., the files required by the sub-task 1) and the dependent files of the sub-task 2 (i.e., the files required by the sub-task 2), determines the union of the dependent files of the sub-task 1 and the dependent files of the sub-task 2, and determines the union of the dependent files of the sub-task 1 and the dependent files of the sub-task 2 as the first files required to be loaded when the computing node 11 executes the corresponding sub-tasks (i.e., the sub-task 1 and the sub-task 2). Assuming that the dependent files of the sub-task 1 are program file 1, library file 1, configuration file 1 and data file 1, the dependent files of the sub-task 2 are program file 1, library file 1, configuration file 1 and data file 2, the task execution system determines that the union of the dependent files of the sub-task 1 and the dependent files of the sub-task 2 is {program file 1, library file 1, configuration file 1, data file 1, data file 2}, and thus determines the union of the dependent files of the sub-task 1 and the dependent files of the sub-task 2 {program file 1, library file 1, configuration file 1, data file 1, data file 2} as the first files required to be loaded when the computing node 11 executes the corresponding sub-tasks.

[0104] The task execution system determines that the sub-tasks executed by the computing node 12 are sub-task 3, sub-task 4 and sub-task 5, determines the dependent files of the sub-task 3 (i.e., the files required by the sub-task 3), the dependent files of the sub-task 4 (i.e., the files required by the sub-task 4) and the dependent files of the sub-task 5 (i.e., the files required by the sub-task 5), determines the union of the dependent files of the sub-task 3, the dependent files of the sub-task 4 and the dependent files of the sub-task 5, and determines the union of the dependent files of the sub-task 3, the dependent files of the sub-task 4 and the dependent files of the sub-task 5 as the first files required to be loaded when the computing node 12 executes the corresponding sub-tasks (i.e., the sub-task 3, the sub-task 4 and the sub-task 5). Assuming that the dependent files of the sub-task 3, the sub-task 4 and the sub-task 5 are all program file 1, library file 1, configuration file 1 and data file 1, the task execution system determines that the union of the dependent files of the sub-task 3, the dependent files of the sub-task 4 and the dependent files of the sub-task 5 is {program file 1, library file 1, configuration file 1, data file 1}, and thus determines the union of the dependent files of the sub-task 3, the dependent files of the sub-task 4 and the dependent files of the sub-task 5 {program file 1, library file 1, configuration file 1, data file 1} as the first files required to be loaded when the computing node 12 executes the corresponding sub-tasks.

[0105] In an optional embodiment, the task execution system comprises a scheduler and a first manager, the first manager is a manager of the first sub-cluster, and S203 can be performed by the scheduler or the first manager. In the case that S203 is performed by the first manager, after performing S202, the scheduler notifies the first manager of the files required by each of the at least two sub-tasks scheduled to be executed by the first sub-cluster and the at least two computing nodes in the first sub-cluster scheduled to execute the at least two sub-tasks, so as to facilitate the first manager to perform S203. In an optional embodiment, the scheduler generates a correspondence between the at least two computing nodes, the at least two sub-tasks and the files required by the at least two sub-tasks, in which each computing node corresponds to a sub-task executed by the each computing node, and each sub-task corresponds to a file required by the each sub-task. The scheduler sends the correspondence to the first manager, so as to notify the first manager of the files required by each of the at least two sub-tasks and the at least two computing nodes scheduled to execute the at least two sub-tasks.

[0106] S204. Determine, according to the first files required to be loaded by the at least two computing nodes, second files whose required loading quantities satisfy a threshold X_th.

[0107] The at least two computing nodes in S204 are computing nodes in the first sub-cluster scheduled to execute the at least two sub-tasks. The quantity of the first files required to be loaded by each of the at least two computing nodes (i.e., the first files required to be loaded by each of the at least two computing nodes when executing a corresponding sub-task) is one or more. The second files are preloading files of the first sub-cluster. The quantity of the second files is one or more. In the case that the quantity of the second files is multiple, the required loading quantity of each of the multiple second files satisfies the threshold X_th. Wherein, the required loading quantity of any first file is the quantity of the computing nodes in the at least two computing nodes that require to load the any first file. The required loading quantity of any second file is the quantity of the computing nodes in the at least two computing nodes that require to load the any second file. The required loading quantity of any second file satisfying the threshold X_th includes that the quantity of the computing nodes in the at least two computing nodes that require to load the any second file is greater than or equal to the threshold X_th. Since the required loading quantity of the second file satisfies the threshold X_th, the quantity of the computing nodes in the at least two computing nodes that require to load the second file is relatively large, and the embodiment of the present application can determine the files (i.e., the second files) required to be loaded by the relatively large number of computing nodes in the at least two computing nodes as the preloading files.

[0108] For each first file needed to be loaded by the at least two computing nodes, the task execution system determines the number of computing nodes in the at least two computing nodes that need to load the each first file; the task execution system determines the number of computing nodes in the at least two computing nodes that need to load the each first file as the needed loading number of the each first file. The task execution system sorts the first files needed to be loaded by the at least two computing nodes in descending order of the needed loading number; the task execution system determines the first files whose sorting is greater than or equal to a threshold value X th as second files. The first files whose sorting is greater than or equal to the threshold value X th are the first files whose needed loading number is greater than or equal to the threshold value X th. The number of computing nodes in the at least two computing nodes that need to load the second files is greater than or equal to the threshold value X th.

[0109] Continuing to illustrate with the example in S203. The at least two computing nodes are the computing node 11 and the computing node 12, the first files needed to be loaded by the computing node 11 are the program file 1, the library file 1, the configuration file 1, the data file 1 and the data file 2, and the first files needed to be loaded by the computing node 12 are the program file 1, the library file 1, the configuration file 1 and the data file 1. Therefore, the first files needed to be loaded by the at least two computing nodes are the program file 1, the library file 1, the configuration file 1, the data file 1 and the data file 2. The task execution system determines that the number of computing nodes in the at least two computing nodes that need to load the program file 1 is 2, the number of computing nodes in the at least two computing nodes that need to load the library file 1 is 2, the number of computing nodes in the at least two computing nodes that need to load the configuration file 1 is 2, the number of computing nodes in the at least two computing nodes that need to load the data file 1 is 2, and the number of computing nodes in the at least two computing nodes that need to load the data file 2 is 1. Therefore, the task execution system determines that the needed loading number of the program file 1, the needed loading number of the library file 1, the needed loading number of the configuration file 1 and the needed loading number of the data file 1 are all 2, and the needed loading number of the data file 2 is 1. The task execution system sorts the program file 1, the library file 1, the configuration file 1, the data file 1 and the data file 2 in descending order of the needed loading number, and the sorting result can be: the program file 1, the library file 1, the configuration file 1, the data file 1, the data file 2. Assuming that the threshold value X th is 2, therefore, the first files whose sorting is greater than or equal to the threshold value X th are the program file 1, the library file 1, the configuration file 1 and the data file 1, and the task execution system determines the program file 1, the library file 1, the configuration file 1 and the data file 1 as second files.

[0110] In an optional embodiment, the task execution system comprises a scheduler and the first manager, and S204 can be performed by the scheduler or the first manager. In the case that S203 is performed by the scheduler and S204 is performed by the first manager, after performing S203, the scheduler notifies the first manager of the first files required to be loaded by the at least two computing nodes in the first sub-cluster and the required loading quantity of each first file, so as to facilitate the first manager to perform S204. Alternatively, the scheduler notifies the first manager of the first files required to be loaded by the at least two computing nodes and the computing nodes in the at least two computing nodes required to load each first file, so as to facilitate the first manager to perform S204.

[0111] S205. Loading the second files to a shared storage space S of the first sub-cluster, the shared storage space S being accessible by the at least two computing nodes.

[0112] The at least two computing nodes in S205 are the computing nodes in the first sub-cluster scheduled to perform the at least two sub-tasks. The shared storage space S is a storage space in a storage pool of the first sub-cluster, and the computing nodes in the first sub-cluster share the storage resources in the storage pool. The shared storage space S is used to store the second files (i.e., the pre-loaded files of the first sub-cluster).

[0113] After determining the second files, the task execution system loads the second files from the storage system to the shared storage space S. In an optional embodiment, the task execution system comprises a first loading node, and the first loading node is a file loading node in the first sub-cluster, and the first loading node loads the second files from the storage system to the shared storage space S. In a specific embodiment, the first loading node sends a loading request to the storage system, and the loading request comprises the identifier of the second file. The storage system receives the loading request. The storage system obtains the second file from the storage system according to the loading request. The storage system sends a loading response to the first loading node, and the loading response comprises the second file. The first loading node receives the loading response. The first loading node obtains the second file from the loading response. The first loading node stores the second file to the shared storage space S. In the case that the quantity of the second files is multiple, the first loading node can load the multiple second files from the storage system through one loading request, or can load the multiple second files from the storage system through multiple loading requests, and the embodiments of the present application do not limit this.

[0114] As mentioned above, S204 can be performed by the scheduler or the first manager. In one embodiment, S204 is performed by the scheduler, after the scheduler performs S204, the scheduler generates a preloading task, and sends the preloading task to the first manager, the preloading task comprising the identification of the second file. The first manager receives the preloading task, and the first manager controls the first loading node to load the second file from the storage system to the shared storage space S according to the preloading task. In the case that the number of the second files is multiple, the preloading task comprises the identification of each of the multiple second files, and the first manager controls the first loading node to load the multiple second files from the storage system to the shared storage space S according to the preloading task. In another embodiment, S204 is performed by the first manager, after the first manager performs S204, the first manager controls the first loading node to load the second file from the storage system to the shared storage space S. As a specific implementation of the first manager controlling the first loading node to load the second file from the storage system to the shared storage space S, the first manager generates a preloading request according to the identification of the second file, the preloading request comprising the identification of the second file. The first manager sends the preloading request to the first loading node. The first loading node receives the preloading request, and the first loading node loads the second file from the storage system to the shared storage space S according to the identification of the second file comprised in the preloading request. In the case that the number of the second files is multiple, the preloading request comprises the identification of the multiple second files, and the first loading node loads the multiple second files from the storage system to the shared storage space S according to the identification of the multiple second files comprised in the preloading request.

[0115] In an optional embodiment, before loading the second file into the shared storage space S, the task execution system determines the size of the shared storage space S according to the size of the second file, and allocates the shared storage space S in the storage pool of the first sub-cluster according to the size of the shared storage space S. After allocating the shared storage space S, the task execution system loads the second file from the storage system into the shared storage space S. In the case where the second files are multiple, the task execution system determines the size of the shared storage space S according to the sizes of the multiple second files, and the size of the shared storage space S is greater than the total size of the multiple second files. In a specific embodiment, the task execution system comprises a scheduler and a first manager. The scheduler determines the size of the shared storage space S according to the size of the second file, and the first manager allocates the shared storage space S in the storage pool of the first sub-cluster according to the size of the shared storage space S. Alternatively, the first manager determines the size of the shared storage space S according to the size of the second file, and the first manager allocates the shared storage space S in the storage pool of the first sub-cluster according to the size of the shared storage space S. In the case where the scheduler determines the size of the shared storage space S according to the size of the second file, after the scheduler determines the size of the shared storage space S, the scheduler notifies the first manager of the size of the shared storage space S, so that the first manager allocates the shared storage space S in the storage pool of the first sub-cluster according to the size of the shared storage space S. In one embodiment, S204 is executed by the scheduler. After the scheduler executes S204, the scheduler determines the size of the shared storage space S according to the size of the second file, and the above-mentioned preloading task sent by the scheduler to the first manager further comprises the size of the shared storage space S (i.e., the preloading task comprises the identification of the second file and the size of the shared storage space S) to instruct the first manager to allocate the shared storage space S in the storage pool of the first sub-cluster. After the first manager receives the preloading task, the first manager allocates the shared storage space S in the storage pool of the first sub-cluster according to the size of the shared storage space S included in the preloading task, and the first manager controls the first loading node to load the second file from the storage system into the shared storage space S. In another embodiment, S204 is executed by the first manager. After the first manager executes S204, the first manager determines the size of the shared storage space S according to the size of the second file, the first manager allocates the shared storage space S in the storage pool of the first sub-cluster according to the size of the shared storage space S, and the first manager controls the first loading node to load the second file from the storage system into the shared storage space S. In a specific embodiment, after the first manager allocates the shared storage space S, the first manager controls the first loading node to mount the shared storage space S to the first loading node, and then controls the first loading node to load the second file from the storage system into the shared storage space S.For example, the preloading request sent by the first manager to the first loading node further comprises an address of the shared storage space S (e.g. comprising a first address of the shared storage space S and a size of the shared storage space S), and the first loading node determines the shared storage space S according to the address of the shared storage space S and mounts the shared storage space S to the first loading node.

[0116] In an optional embodiment, before the first manager controls the first loading node to load the second file from the storage system to the shared storage space S, the scheduler determines the first loading node in the first sub-cluster, or the first manager determines the first loading node in the first sub-cluster. In one embodiment, the scheduler determines the first loading node in the first sub-cluster, and after the scheduler determines the first loading node, the scheduler notifies the first manager of the identity of the first loading node so as to facilitate the first manager to determine the first loading node and control the first loading node to load the second file from the storage system to the shared storage space S. For example, S204 is performed by the scheduler, and after the scheduler performs S204, the scheduler determines the first loading node in the first sub-cluster, and the preloading task sent by the scheduler to the first manager further comprises the identity of the first loading node (i.e. the preloading task comprises the identity of the second file and the identity of the first loading node, and can further comprise the size of the shared storage space S) to indicate that the first loading node loads the second file to the shared storage space S. After the first manager receives the preloading task, the first manager determines the first loading node according to the identity of the first loading node comprised in the preloading task, and controls the first loading node to load the second file from the storage system to the shared storage space S. In another embodiment, the first manager determines the first loading node in the first sub-cluster, and after the first manager determines the first loading node, the first manager controls the first loading node to load the second file from the storage system to the shared storage space S.

[0117] In the embodiments of the present application, the first loading node is any computing node in the first sub-cluster. Alternatively, the first loading node is one computing node in the first sub-cluster that is scheduled to execute at least one subtask in the plurality of subtasks (i.e., the first loading node is one computing node in the at least two computing nodes in the first sub-cluster). In the case where the first loading node is one computing node in the first sub-cluster that is scheduled to execute at least one subtask in the plurality of subtasks, the number of files (i.e., the second files) in the first files that need to be loaded by the first loading node when executing the corresponding subtask and the number of which satisfy the threshold X th is greater than the target number Q tar; or the total data amount of the files (i.e., the second files) in the first files that need to be loaded by the first loading node when executing the corresponding subtask and the total data amount of which satisfy the threshold X th is greater than the target data amount D tar. For example, the first loading node is one computing node in the first sub-cluster that needs to load the most second files when executing the corresponding subtask (i.e., the first loading node is one computing node in the at least two computing nodes in the first sub-cluster that needs to load the most second files when executing the corresponding subtask); or the first loading node is one computing node in the first sub-cluster that needs to load the most total data amount of the second files when executing the corresponding subtask (i.e., the first loading node is one computing node in the at least two computing nodes in the first sub-cluster that needs to load the most total data amount of the second files when executing the corresponding subtask).

[0118] The following is an example of the first loading node being one computing node in the first sub-cluster that is scheduled to execute at least one subtask in the plurality of subtasks, and the scheduler determining the first loading node in the first sub-cluster. The implementation of the first manager determining the first loading node in the first sub-cluster can refer to the implementation of the scheduler determining the first loading node in the first sub-cluster.

[0119] In one embodiment, the first loading node is one of the at least two computing nodes in the first sub-cluster that is scheduled to perform at least one of the plurality of sub-tasks, i.e., the first loading node is one of the at least two computing nodes in the first sub-cluster, and the number of files in the first file that need to be loaded by the first loading node when performing the corresponding sub-task is greater than the target number Q tar. The target number Q tar can be set according to actual conditions, and in different scenarios, the target number Q tar can be equal or not equal. The scheduler determines the number of second files that need to be loaded by each of the at least two computing nodes when performing the corresponding sub-task, and determines the first loading node from the at least two computing nodes according to the number of second files that need to be loaded by each of the at least two computing nodes when performing the corresponding sub-task, and the number of second files that need to be loaded by the first loading node when performing the corresponding sub-task is greater than the target number Q tar. Since the number of second files that need to be loaded by the first loading node when performing the corresponding sub-task is greater than the target number Q tar, the number of second files that need to be loaded by the first loading node when performing the corresponding sub-task is relatively large. Thus, the embodiment of the present application determines the computing node that needs to load the second file in the at least two computing nodes to load the second file from the storage system. In an optional embodiment, the first loading node is the computing node that needs to load the most second files when performing the corresponding sub-task from the at least two computing nodes. That is, the scheduler determines the computing node that needs to load the most second files from the at least two computing nodes as the first loading node to load the second file from the storage system.

[0120] In another embodiment, the first loading node is one of the at least two computing nodes in the first sub-cluster that is scheduled to perform at least one of the plurality of sub-tasks, and the total data amount of the second file that needs to be loaded by the first loading node when performing the corresponding sub-task satisfies the threshold X th. The target data amount D tar is set according to actual conditions, and in different scenarios, the target data amount D tar can be equal or not equal. The scheduler determines the total data amount of the second file that needs to be loaded by each of the at least two computing nodes when performing the corresponding sub-task, and determines the first loading node from the at least two computing nodes according to the total data amount of the second file that needs to be loaded by each of the at least two computing nodes when performing the corresponding sub-task, and the total data amount of the second file that needs to be loaded by the first loading node when performing the corresponding sub-task is greater than the target data amount D tar. Wherein, the total data amount of the second file that needs to be loaded by each of the at least two computing nodes when performing the corresponding sub-task is the total data amount of at least one second file that needs to be loaded by the each computing node when performing the corresponding sub-task. For any computing node in the at least two computing nodes: in the case that the number of second files that needs to be loaded by the computing node when performing the corresponding sub-task is one, the total data amount of the second file that needs to be loaded by the computing node when performing the corresponding sub-task is the data amount of the one second file; in the case that the number of second files that needs to be loaded by the computing node when performing the corresponding sub-task is multiple, the total data amount of the second file that needs to be loaded by the computing node when performing the corresponding sub-task is the sum of the data amounts of the multiple second files. Since the total data amount of the second file that needs to be loaded by the first loading node when performing the corresponding sub-task is greater than the target data amount D tar, the total data amount of the second file that needs to be loaded by the first loading node when performing the corresponding sub-task is relatively large. Thus, the embodiment of the present application determines the computing node with the relatively large total data amount of the second file that needs to be loaded from the at least two computing nodes as the first loading node to load the second file from the storage system. In an optional embodiment, the first loading node is the computing node with the largest total data amount of the second file that needs to be loaded from the at least two computing nodes when performing the corresponding sub-task. That is, the scheduler determines the computing node with the largest total data amount of the second file that needs to be loaded from the at least two computing nodes as the first loading node to load the second file from the storage system.

[0121] In an optional embodiment, S204 is performed by the scheduler, and the preloading task sent by the scheduler to the first manager further comprises an identification of the computing nodes in the first subset of clusters that need to load the second files, to instruct the computing nodes in the at least two computing nodes that need to load the second files. In the case that the number of the second files is more than one, the preloading task comprises an identification of the computing nodes that need to load each of the second files, and the number of the computing nodes that need to load each of the second files is greater than or equal to the threshold X th. In a specific embodiment, the preloading task comprises a preloading file list, and the preloading file list comprises an identification of the second files and an identification of the computing nodes in the at least two computing nodes that need to load the second files. For example, the preloading file list comprises a correspondence between at least one file identification and at least two node identifications, in which the number of the corresponding node identifications for each file identification is greater than or equal to the threshold X th, each file identification is used to identify a second file, and each file identification corresponds to the node identification used to identify the computing nodes in the at least two computing nodes that need to load the second file identified by the file identification.

[0122] As a specific implementation, S204 is performed by the scheduler, and the scheduler determines the size of the shared storage space S according to the size of the second files, and determines the first loading node in the first subset of clusters. The scheduler generates a preloading file list according to the identification of the at least one second file and the identification of the computing nodes in the at least two computing nodes that need to load the second files, and the preloading file list comprises a correspondence between at least one file identification and at least two node identifications, in which each file identification is used to identify a second file in the at least one second file, and each file identification corresponds to the node identification used to identify the computing nodes in the at least two computing nodes that need to load the second file identified by the file identification. The scheduler generates the preloading task according to the preloading file list, the size of the shared storage space S, and the identification of the first loading node, and the preloading task comprises the preloading file list, the size of the shared storage space S, and the identification of the first loading node. The scheduler sends the preloading task to the first manager. The first manager determines the computing nodes that need to load the at least one second file according to the preloading file list comprised in the preloading task; the first manager allocates the shared storage space S in the storage pool of the first subset of clusters according to the size of the shared storage space S comprised in the preloading task; and the first manager determines the first loading node according to the identification of the first loading node comprised in the preloading task. The first manager controls the first loading node to mount the shared storage space S to the first loading node, and controls the first loading node to load the at least one second file from the storage system to the shared storage space S.

[0123] In optional embodiments, S201 to S204 are performed by a scheduler, and part of S205 (e.g., generating the preloading task) is performed by the scheduler. For example, the scheduler includes an analysis unit and a construction unit, S201 to S204 are specifically performed by the analysis unit, and the part of S205 is specifically performed by the construction unit. The analysis unit is also referred to as a data analysis unit, a file analysis unit, a data loading analysis unit, a file loading analysis unit, etc.; the construction unit is also referred to as a preloading unit, a preloading task construction unit, etc., which are not limited in the embodiments of the present application.

[0124] S206. The computing node that needs to load the second file from the shared storage space S to execute the corresponding subtask.

[0125] The at least two computing nodes in S205 are the computing nodes in the first sub-cluster that are scheduled to execute the at least two subtasks.

[0126] In optional embodiments, the task execution system instructs each computing node that needs to load the second file from the at least two computing nodes to mount the shared storage space S to the each computing node, so that the each computing node obtains the second file from the shared storage space S to execute the corresponding subtask. The task execution system includes a first manager and a first loading node, and S206 is performed by the first manager or the first loading node.

[0127] In one embodiment, S206 is performed by the first manager. In the case that S204 is performed by the scheduler, the preloading task sent by the scheduler to the first manager includes an identification of the computing node of the at least two computing nodes that needs to load the second file, the first manager determines the computing node of the at least two computing nodes that needs to load the second file according to the identification of the computing node of the at least two computing nodes that needs to load the second file, after the first manager controls the first loading node to load the second file to the shared storage space S, the first manager instructs the computing node of the at least two computing nodes that needs to load the second file to obtain the second file from the shared storage space S to execute the corresponding subtask. In the case that S204 is performed by the first manager, the first manager can determine the computing node of the at least two computing nodes that needs to load the second file by performing S204, after the first manager controls the first loading node to load the second file to the shared storage space S, the first manager instructs the computing node of the at least two computing nodes that needs to load the second file to obtain the second file from the shared storage space S to execute the corresponding subtask. For example, the first manager instructs each of the computing node of the at least two computing nodes that needs to load the second file to mount the shared storage space S to the each of the computing node, so that the each of the computing node obtains the second file from the shared storage space S. In a specific embodiment, the first manager generates a space mounting request, the space mounting request includes an address of the shared storage space S (for example, includes a first address of the shared storage space S and a size of the shared storage space S). The first manager sends the space mounting request to each of the computing node of the at least two computing nodes that needs to load the second file. After each of the computing node of the at least two computing nodes that needs to load the second file receives the space mounting request, the each of the computing node determines the shared storage space S according to the address of the shared storage space S included in the space mounting request, and mounts the shared storage space S in the each of the computing node. In this way, the first manager instructs each of the computing node of the at least two computing nodes that needs to load the second file to mount the shared storage space S to the each of the computing node, so that each of the computing node of the at least two computing nodes that needs to load the second file can directly access the shared storage space S to obtain the second file from the shared storage space S. In an optional embodiment, after each of the computing node of the at least two computing nodes that needs to load the second file mounts the shared storage space S in the each of the computing node, the each of the computing node can generate a space mounting response and send the space mounting response to the first manager. The space mounting response sent by each of the computing node to the first manager includes a mounting result of the shared storage space S by the each of the computing node, and the mounting result can be mounting success or failure. The first manager determines the mounting result of the shared storage space S by each of the computing node of the at least two computing nodes that needs to load the second file according to the space mounting response sent by the each of the computing node.

[0128] As an example, as shown in FIG. 1, the first sub-cluster is sub-cluster 1, the first loading node is computing node 11, the first manager is manager 1, and the computing node in the at least two computing nodes in the first sub-cluster that needs to load the second file includes computing node 12. FIG. 3 shows a schematic diagram in which manager 1 instructs computing node 12 to mount shared storage space S to computing node 12. After computing node 11 loads the second file to shared storage space S, as shown in FIG. 3, manager 1 generates a space mounting request, which includes the address of shared storage space S. Manager 1 sends the space mounting request to computing node 12. Computing node 12 receives the space mounting request. Computing node 12 determines shared storage space S according to the address of shared storage space S included in the space mounting request. Computing node 12 mounts shared storage space S to computing node 12. Computing node 12 generates a space mounting response, which includes the mounting result of shared storage space S by computing node 12. Computing node 12 sends the space mounting response to manager 1. Manager 1 receives the space mounting response. Manager 1 determines the mounting result of shared storage space S by computing node 12 according to the space mounting response. In a specific embodiment, as shown in FIG. 3, manager 1 includes message processing unit 11 and storage sharing unit 12, and computing node 12 includes shared storage management unit 120 and manager client 121. Manager 1 and computing node 12 interact through message processing unit 11 and manager client 121. Storage sharing unit 12 in manager 1 generates the space mounting request. Message processing unit 11 in manager 1 sends the space mounting request to computing node 12. Manager client 121 in computing node 12 receives the space mounting request. Shared storage management unit 120 in computing node 12 determines shared storage space S according to the address of shared storage space S included in the space mounting request, and mounts shared storage space S to computing node 12. Shared storage management unit 120 in computing node 12 generates the space mounting response. Manager client 121 in computing node 12 sends the space mounting response to manager 1. Message processing unit 11 in manager 1 receives the space mounting response. Storage sharing unit 12 in manager 1 determines the mounting result of shared storage space S by computing node 12 according to the space mounting response.

[0129] In another embodiment, S206 is performed by the first loading node. In the case that S204 is performed by the scheduler, the preloading task sent by the scheduler to the first manager includes the identification of the computing nodes in the at least two computing nodes that need to load the second file, the preloading request sent by the first manager to the first loading node includes the identification of the computing nodes in the at least two computing nodes that need to load the second file, the first loading node determines the computing nodes in the at least two computing nodes that need to load the second file according to the identification of the computing nodes in the at least two computing nodes that need to load the second file included in the preloading request, after the first loading node loads the second file to the shared storage space S, the first loading node instructs the computing nodes in the at least two computing nodes that need to load the second file to obtain the second file from the shared storage space S to perform the corresponding subtasks. In the case that S204 is performed by the first manager, the first manager can determine the computing nodes in the at least two computing nodes that need to load the second file by performing S204, the preloading request sent by the first manager to the first loading node includes the identification of the computing nodes in the at least two computing nodes that need to load the second file, the first loading node determines the computing nodes in the at least two computing nodes that need to load the second file according to the identification of the computing nodes in the at least two computing nodes that need to load the second file included in the preloading request, after the first loading node loads the second file to the shared storage space S, the first loading node instructs the computing nodes in the at least two computing nodes that need to load the second file to obtain the second file from the shared storage space S to perform the corresponding subtasks. For example, the first loading node instructs each of the computing nodes in the at least two computing nodes that need to load the second file to mount the shared storage space S to the each of the computing nodes (also referred to as the first loading node shares the shared storage space S with each of the computing nodes in the at least two computing nodes that need to load the second file), so that the each of the computing nodes obtains the second file from the shared storage space S. In an optional embodiment, the computing nodes in the first sub-cluster communicate through the first manager, and the first loading node shares the shared storage space S with each of the computing nodes in the at least two computing nodes that need to load the second file through the first manager. In a specific embodiment, the first loading node sends a space sharing request to the first manager, the space sharing request including the address of the shared storage space S (for example, including the first address of the shared storage space S and the size of the shared storage space S). The first manager receives the space sharing request. The first manager generates a space mounting request according to the space sharing request, the space mounting request including the address of the shared storage space S. The first manager sends the space mounting request to each of the computing nodes in the at least two computing nodes that need to load the second file.After each of the at least two computing nodes that need to load the second file receives the space mounting request, the each of the at least two computing nodes determines the shared storage space S according to the address of the shared storage space S included in the space mounting request, and mounts the shared storage space S at the each of the at least two computing nodes. Thus, the first loading node shares the shared storage space S with each of the at least two computing nodes that need to load the second file, so that each of the at least two computing nodes that need to load the second file can directly access the shared storage space S to obtain the second file from the shared storage space S. In an optional embodiment, after each of the at least two computing nodes that need to load the second file mounts the shared storage space S at the each of the at least two computing nodes, the each of the at least two computing nodes generates a space mounting response, and sends the space mounting response to the first manager. The space mounting response sent by each of the at least two computing nodes to the first manager includes a mounting result of the shared storage space S at the each of the at least two computing nodes, and the mounting result can be a mounting success or a mounting failure. The first manager receives the space mounting response sent by each of the at least two computing nodes that need to load the second file. The first manager generates a space sharing response according to the space mounting response sent by each of the at least two computing nodes that need to load the second file, and sends the space sharing response to the first loading node, where the space sharing response includes the mounting result of the shared storage space S at each of the at least two computing nodes that need to load the second file. The first loading node receives the space sharing response. The first loading node determines the mounting result of the shared storage space S at each of the at least two computing nodes that need to load the second file according to the space sharing response.

[0130] As an example, as shown in FIG. 1, the first sub-cluster is sub-cluster 1, the first loading node is computing node 11, the first manager is manager 1, and the computing node in the first sub-cluster that needs to load the second file from the at least two computing nodes in the first sub-cluster includes computing node 12. FIG. 4 shows a schematic diagram in which computing node 11 instructs computing node 12 to mount shared storage space S to computing node 12. After computing node 11 loads the second file to shared storage space S, as shown in FIG. 4, computing node 11 sends a space sharing request to manager 1, and the space sharing request includes the address of shared storage space S. Manager 1 receives the space sharing request. Manager 1 generates a space mounting request according to the space sharing request, and the space mounting request includes the address of shared storage space S. Manager 1 sends the space mounting request to computing node 12. Computing node 12 receives the space mounting request. Computing node 12 determines shared storage space S according to the address of shared storage space S included in the space mounting request. Computing node 12 mounts shared storage space S to computing node 12. Computing node 12 generates a space mounting response, and the space mounting response includes the mounting result of shared storage space S by computing node 12. Computing node 12 sends the space mounting response to manager 1. Manager 1 receives the space mounting response. Manager 1 generates a space sharing response according to the space mounting response, and the space sharing response includes the mounting result of shared storage space S by computing node 12. Manager 1 sends the space sharing response to computing node 11. Computing node 11 receives the space sharing response. Computing node 11 determines the mounting result of shared storage space S by computing node 12 according to the space sharing response. As shown in FIG. 4, computing node 11 includes shared storage management unit 110 and manager client 111, computing node 12 includes shared storage management unit 120 and manager client 121, and manager 1 includes message processing unit 11 and storage sharing unit 12. Computing node 11 interacts with manager 1 through manager client 111 and message processing unit 11, and computing node 12 interacts with manager 1 through manager client 121 and message processing unit 11. Shared storage management unit 110 in computing node 11 generates the space sharing request. Manager client 111 in computing node 11 sends the space sharing request to manager 1. Message processing unit 11 in manager 1 receives the space sharing request. Storage sharing unit 12 in manager 1 generates the space mounting request according to the space sharing request. Message processing unit 11 in manager 1 sends the space mounting request to computing node 12. Manager client 121 in computing node 12 receives the space mounting request. Shared storage management unit 120 in computing node 12 determines shared storage space S according to the address of shared storage space S included in the space mounting request, and mounts shared storage space S to computing node 12. Shared storage management unit 120 in computing node 12 generates the space mounting response.The manager client 121 in the computing node 12 sends the space mounting response to the manager 1. The message processing unit 11 in the manager 1 receives the space mounting response. The storage sharing unit 12 in the manager 1 generates a space sharing response according to the space mounting response. The message processing unit 11 in the manager 1 sends the space sharing response to the computing node 11. The manager client 111 in the computing node 11 receives the space sharing response. The shared storage management unit 110 in the computing node 11 determines the mounting result of the computing node 12 to the shared storage space S according to the space sharing response.

[0131] It should be noted that the second file described in S204 to S206 is a preloaded file of the first sub-cluster. The first file required to be loaded by the at least two computing nodes in the first sub-cluster when performing the corresponding sub-tasks can further include a third file. For example, the file other than the second file in the first file required to be loaded by the at least two computing nodes when performing the corresponding sub-tasks is the third file, and the third file is a non-preloaded file of the first sub-cluster. Each computing node of the at least two computing nodes that needs to load the third file loads the third file from the storage system to the storage space of the each computing node to perform the corresponding sub-task. After each computing node of the at least two computing nodes performs the corresponding sub-task based on the second file, the first manager can reclaim the shared storage space S. For example, the first manager instructs the computing nodes of the at least two computing nodes that have obtained the second file from the shared storage space S to unload the shared storage space S.

[0132] The above S202 to S206 are described by taking the first sub-cluster as an example. In an optional embodiment, the at least one sub-cluster further includes a second sub-cluster, at least two computing nodes in the second sub-cluster are scheduled to perform at least two sub-tasks of the plurality of sub-tasks, and the sub-tasks performed by the at least two computing nodes in the second sub-cluster are different from the sub-tasks performed by the at least two computing nodes in the first sub-cluster. The task execution system further determines the first file required to be loaded by each computing node of the at least two computing nodes in the second sub-cluster when performing the corresponding sub-task, determines the second file required to be loaded in the first file required to be loaded by the at least two computing nodes in the second sub-cluster according to the threshold (the threshold is the same as or different from the threshold X_th), loads the second file required to be loaded in the first file required to be loaded by the at least two computing nodes in the second sub-cluster to the shared storage space of the second sub-cluster, and instructs the computing nodes of the at least two computing nodes in the second sub-cluster that need to load the second file to obtain the second file from the shared storage space of the second sub-cluster to perform the corresponding sub-task. For specific implementation process, refer to S203 to S206, which will not be described here.

[0133] To sum up, the technical solution provided by the embodiments of the present application, at least two computing nodes in the first sub-cluster are scheduled to execute at least two sub-tasks included in the to-be-executed task, and the shared storage space of the first sub-cluster can be accessed by the at least two computing nodes. The task execution system determines a second file needed to be loaded according to the first file needed to be loaded by the at least two computing nodes when executing the corresponding sub-tasks, and the number of the second file needed to be loaded meets a threshold value. The task execution system loads the second file from the storage system to the shared storage space, and instructs the computing nodes that need to load the second file among the at least two computing nodes to obtain the second file from the shared storage space to execute the corresponding sub-tasks. Thus, without the computing nodes that need to load the second file among the at least two computing nodes loading the second file from the storage system concurrently, the probability of congestion of the storage system can be reduced, the speed and efficiency of loading the second file can be improved, and thus the task execution efficiency can be improved. Moreover, the at least two computing nodes can avoid loading the same dependent file from the storage system repeatedly, and thus the waste of the storage space of the computing nodes and the fragmentation of the storage space of the computing nodes can be avoided.

[0134] The task execution method in the prior art and the task execution method provided by the embodiments of the present application will be compared and described below in combination with the accompanying drawings.

[0135] Please refer to FIG. 5, which shows a schematic diagram of the task execution method in the prior art. As shown in FIG. 5, the computing cluster 10 includes a scheduler 100, computing nodes 11-13 and computing nodes 21-23, each computing node includes a processor, and each processor includes at least one computing core.

[0136] The scheduler 100 acquires a to-be-executed task A, which includes a plurality of sub-tasks. The scheduler 100 schedules the plurality of sub-tasks to the computing nodes 11-13 and the computing nodes 21-23 for execution. The scheduler 100 determines files required by each of the computing nodes 11-13 and the computing nodes 21-23 for execution of a sub-task. The scheduler 100 determines files required by each of the computing nodes 11-13 and the computing nodes 21-23 for execution of a corresponding sub-task according to the files required by the each of the computing nodes 11-13 and the computing nodes 21-23 for execution of the sub-task. The scheduler 100 sends information of the files required by each of the computing nodes 11-13 and the computing nodes 21-23 for execution of a corresponding sub-task to the each of the computing nodes 11-13 and the computing nodes 21-23. Each of the computing nodes 11-13 and the computing nodes 21-23 loads the files required by the each of the computing nodes 11-13 and the computing nodes 21-23 for execution of a corresponding sub-task from the storage system 20 to a storage space of the each of the computing nodes 11-13 and the computing nodes 21-23 according to the information of the files required by the each of the computing nodes 11-13 and the computing nodes 21-23 for execution of a corresponding sub-task sent by the scheduler 100. For example, the files required by each of the computing nodes 11-13 for execution of a corresponding sub-task are program file 1, library file 1, configuration file 1 and data file 1, and the files required by each of the computing nodes 21-23 for execution of a corresponding sub-task are program file 1, library file 1, configuration file 1 and data file 2. Each of the computing nodes 11-13 sends a read request to the storage system 20 to load program file 1, library file 1, configuration file 1 and data file 1 from the storage system 20 to a storage space of the each of the computing nodes 11-13. Each of the computing nodes 21-23 sends a read request to the storage system 20 to load program file 1, library file 1, configuration file 1 and data file 2 from the storage system 20 to a storage space of the each of the computing nodes 21-23. Moreover, the computing nodes 11-13 and the computing nodes 21-23 concurrently send read requests to the storage system 20 to load the files required by the computing nodes 11-13 and the computing nodes 21-23 for execution of a corresponding sub-task from the storage system 20. Assuming that one read request sent by each of the computing nodes 11-13 and the computing nodes 21-23 to the storage system 20 can load all the files required by the each of the computing nodes 11-13 and the computing nodes 21-23 for execution of a corresponding sub-task, and the pressure caused by one read request to the storage system 20 is F, when the computing nodes 11-13 and the computing nodes 21-23 concurrently send read requests to the storage system 20, the concurrency of the read requests received by the storage system 20 is 6 (the concurrency of the read requests is the number of the computing nodes), and the pressure caused by the read requests to the storage system 20 is 6F. The pressure of the storage system 20 is large, and the probability of congestion of the storage system 20 is large, which easily affects the efficiency of task execution.Further, each of the computing nodes 11-13 needs to load the program file 1, the library file 1, the configuration file 1 and the data file 1 from the storage system 20 to the storage space of the each computing node, each of the computing nodes 21-23 needs to load the program file 1, the library file 1, the configuration file 1 and the data file 2 from the storage system 20 to the storage space of the each computing node. That is, the four files of the program file 1, the library file 1, the configuration file 1 and the data file 1 are loaded repeatedly by the computing nodes 11-13, and the four files of the program file 1, the library file 1, the configuration file 1 and the data file 2 are loaded repeatedly by the computing nodes 21-23. These repeated loadings easily lead to the waste of the storage space of the computing nodes 11-13 and the computing nodes 21-23, and lead to the fragmentation of the storage space of the computing nodes.

[0137] Please refer to FIG. 6, which shows a schematic diagram of a task execution method provided by an embodiment of the present application. As shown in FIG. 6, the computing cluster 10 includes a scheduler 100 and sub-clusters 1-k. The sub-cluster 1 includes computing nodes 11-13, a manager 1 and a storage pool 1, the manager 1 is used to manage the computing nodes 11-13 and the storage pool 1, and the computing nodes 11-13 share the storage resources in the storage pool 1. The sub-cluster 2 includes computing nodes 21-23, a manager 2 and a storage pool 2, the manager 2 is used to manage the computing nodes 21-23 and the storage pool 2, and the computing nodes 21-23 share the storage resources in the storage pool 2. The sub-cluster k includes computing nodes k1-k3, a manager k and a storage pool k, the manager k is used to manage the computing nodes k1-k3 and the storage pool k, and the computing nodes k1-k3 share the storage resources in the storage pool k. Each computing node includes a processor, and each processor includes at least one computing core.

[0138] The scheduler 100 obtains a to-be-executed task A, the to-be-executed task A including a plurality of subtasks. The scheduler 100 schedules the plurality of subtasks to the subcluster 1 and the subcluster 2 for execution. The computing nodes 11-13 in the subcluster 1 are scheduled to execute at least two subtasks in the plurality of subtasks, the computing nodes 21-23 in the subcluster 2 are scheduled to execute at least two subtasks in the plurality of subtasks, the at least two subtasks executed by the computing nodes 11-13 being different from the at least two subtasks executed by the computing nodes 21-23. The scheduler 100 determines a first file needed to be loaded by each of the computing nodes 11-13 and the computing nodes 21-23 when executing the corresponding subtask. The scheduler 100 determines a second file (preloaded file of the subcluster 1) whose number of needed loading satisfies a threshold according to the first files needed to be loaded by the computing nodes 11-13. The scheduler 100 determines a second file (preloaded file of the subcluster 2) whose number of needed loading satisfies a threshold according to the first files needed to be loaded by the computing nodes 21-23. The scheduler 100 generates a preloaded file list of the subcluster 1 according to the second files whose number of needed loading satisfies the threshold in the first files needed to be loaded by the computing nodes 11-13, the preloaded file list of the subcluster 1 including the identifiers of the second files whose number of needed loading satisfies the threshold in the first files needed to be loaded by the computing nodes 11-13; the scheduler 100 sends the preloaded file list of the subcluster 1 to the manager 1. The scheduler 100 generates a preloaded file list of the subcluster 2 according to the second files whose number of needed loading satisfies the threshold in the first files needed to be loaded by the computing nodes 21-23, the preloaded file list of the subcluster 2 including the identifiers of the second files whose number of needed loading satisfies the threshold in the first files needed to be loaded by the computing nodes 21-23; the scheduler 100 sends the preloaded file list of the subcluster 2 to the manager 2. For example, the second files whose number of needed loading satisfies the threshold in the first files needed to be loaded by the computing nodes 11-13 are program file 1, library file 1, configuration file 1 and data file 1, and the second files whose number of needed loading satisfies the threshold in the first files needed to be loaded by the computing nodes 21-23 are program file 1, library file 1, configuration file 1 and data file 2, then the preloaded file list of the subcluster 1 includes the identifier of the program file 1, the identifier of the library file 1, the identifier of the configuration file 1 and the identifier of the data file 1, and the preloaded file list of the subcluster 2 includes the identifier of the program file 1, the identifier of the library file 1, the identifier of the configuration file 1 and the identifier of the data file 2.The manager 1 controls the file loading nodes (e.g. the computing node 11) in the sub-cluster 1 to send read requests to the storage system 20 according to the preloaded file list of the sub-cluster 1, so as to load the program file 1, the library file 1, the configuration file 1 and the data file 1 from the storage system 20 to the shared storage space 1 in the storage pool 1, and the manager 1 instructs the computing nodes 12-13 to obtain the program file 1, the library file 1, the configuration file 1 and the data file 1 from the shared storage space 1. The manager 2 controls the file loading nodes (e.g. the computing node 21) in the sub-cluster 2 to send read requests to the storage system 20 according to the preloaded file list of the sub-cluster 2, so as to load the program file 1, the library file 1, the configuration file 1 and the data file 2 from the storage system 20 to the shared storage space 2 in the storage pool 2, and the manager 2 instructs the computing nodes 22-23 to obtain the program file 1, the library file 1, the configuration file 1 and the data file 2 from the shared storage space 2. The computing node 11 and the computing node 21 can concurrently send read requests to the storage system 20 to load the preloaded files of the respective sub-cluster from the storage system 20. Assuming that sending one read request to the storage system 20 by the computing node 11 can load all the preloaded files of the sub-cluster 1, sending one read request to the storage system 20 by the computing node 21 can load all the preloaded files of the sub-cluster 2, and one read request causes a pressure F on the storage system 20, when the computing node 11 and the computing node 21 concurrently send read requests to the storage system 20, the concurrency of the read requests received by the storage system 20 is 2 (the concurrency of the read requests is the number of sub-clusters), and the pressure caused on the storage system 20 by the read requests is 2F. Compared with the prior art, the concurrency of the read requests is reduced by 2 / 3, and the pressure on the storage system 20 is reduced by 2 / 3. In addition, for the sub-cluster 1, only the computing node 11 needs to load the program file 1, the library file 1, the configuration file 1 and the data file 1 from the storage system 20, and the computing nodes 12-13 do not need to load the program file 1, the library file 1, the configuration file 1 and the data file 1 from the storage system 20. For the sub-cluster 2, only the computing node 21 needs to load the program file 1, the library file 1, the configuration file 1 and the data file 2 from the storage system 20, and the computing nodes 22-23 do not need to load the program file 1, the library file 1, the configuration file 1 and the data file 2 from the storage system 20. The same dependent files are avoided from being loaded repeatedly by multiple computing nodes in the same sub-cluster, and the amount of data to be loaded is reduced.

[0139] The above is the introduction of the method embodiment of the present application. The device embodiment of the present application is introduced below, and the device of the present application is used to execute the method of the present application. For details not disclosed in the device embodiment, please refer to the method embodiment.

[0140] The embodiment of the present application provides a task execution system, which comprises at least one function module used for executing all or part of steps of the task execution method provided by the embodiment shown in Fig. 2. The at least one function module can be realized based on software, hardware or the combination of software and hardware, and the at least one function module can be combined or divided in any mode based on specific implementation.

[0141] Please refer to Fig. 7, which shows a schematic diagram of a task execution system 700 provided by the embodiment of the present application. The task execution system 700 comprises an acquisition module 710, a scheduling module 720, a first determination module 730, a second determination module 740, a loading module 750 and an indication module 760.

[0142] The acquisition module 710 is used for acquiring a task to be executed, wherein the task to be executed comprises a plurality of subtasks.

[0143] The scheduling module 720 is used for scheduling the plurality of subtasks to at least one subcluster in a computing cluster for execution, wherein the at least one subcluster comprises a first subcluster, and at least two computing nodes in the first subcluster are scheduled to execute at least two subtasks in the plurality of subtasks.

[0144] The first determination module 730 is used for determining a first file needed to be loaded by each computing node in the at least two computing nodes when executing a corresponding subtask.

[0145] The second determination module 740 is used for determining a second file needed to be loaded in a quantity satisfying a threshold value according to the first file needed to be loaded by the at least two computing nodes.

[0146] The loading module 750 is used for loading the second file to a shared storage space of the first subcluster, wherein the shared storage space can be accessed by the at least two computing nodes.

[0147] The indication module 760 is used for instructing a computing node needing to load the second file in the at least two computing nodes to acquire the second file from the shared storage space to execute a corresponding subtask.

[0148] The function implementation of the acquisition module 710 can refer to the related description in S201; the function implementation of the scheduling module 720 can refer to the related description in S202; the function implementation of the first determination module 730 can refer to the related description in S203; the function implementation of the second determination module 740 can refer to the related description in S204; the function implementation of the loading module 750 can refer to the related description in S205; and the function implementation of the indication module 760 can refer to the related description in S206.

[0149] In an optional embodiment, the scheduling module 720 is configured to: schedule the plurality of sub-tasks to the at least one sub-cluster according to the number of computing cores required by each of the plurality of sub-tasks and the files required by the each of the plurality of sub-tasks.

[0150] In an optional embodiment, the file required by any of the plurality of sub-tasks is a file required to be loaded by a computing node to execute the any of the plurality of sub-tasks. The file required by the any of the plurality of sub-tasks can be referred to as a dependent file of the any of the plurality of sub-tasks. The number of identical dependent files in the dependent files of the at least two sub-tasks scheduled to the first sub-cluster (i.e., the files required by the at least two sub-tasks) is greater than the first number.

[0151] In an optional embodiment, the scheduling module 720 is configured to: obtain affinity between any two of the plurality of sub-tasks according to the dependent files of the any two of the plurality of sub-tasks (i.e., the files required by the any two of the plurality of sub-tasks), the affinity between the any two of the plurality of sub-tasks being positively correlated with the number of identical dependent files in the dependent files of the any two of the plurality of sub-tasks; and schedule the plurality of sub-tasks to the at least one sub-cluster according to the number of computing cores required by each of the plurality of sub-tasks and the affinity between the plurality of sub-tasks, wherein the affinity between the at least two sub-tasks scheduled to the first sub-cluster is greater than a first affinity.

[0152] In an optional embodiment, the scheduling module 720 is configured to: obtain affinity between any two of the plurality of sub-tasks according to the number of dependent files of the any two of the plurality of sub-tasks and the number of identical dependent files in the dependent files of the any two of the plurality of sub-tasks.

[0153] In an optional embodiment, the scheduling module 720 is configured to: obtain affinity between any two of the plurality of sub-tasks according to the data amount of dependent files of the any two of the plurality of sub-tasks and the data amount of identical dependent files in the dependent files of the any two of the plurality of sub-tasks.

[0154] In an optional embodiment, the first determining module 730 is configured to: determine a sub-task executed by each of the at least two computing nodes; determine a file required to be loaded by each of the sub-tasks executed by each of the at least two computing nodes; and determine a union of the files required by the sub-tasks executed by each of the at least two computing nodes as the first file required to be loaded by the each of the at least two computing nodes to execute the corresponding sub-task.

[0155] In an optional embodiment, the second determining module 740 is configured to: sort the first files required to be loaded by the at least two computing nodes in descending order of required loading quantity; and determine a first file whose sorting is greater than or equal to the threshold value as a second file.

[0156] In an optional embodiment, the indication module 760 is configured to instruct each of the at least two computing nodes that needs to load the second file to mount the shared storage space to the each of the at least two computing nodes, so that the each of the at least two computing nodes obtains the second file from the shared storage space.

[0157] In an optional embodiment, the task execution system 700 further comprises a space allocation module 770 configured to determine the size of the shared storage space according to the size of the second file, and allocate the shared storage space in the storage pool of the first sub-cluster according to the size of the shared storage space.

[0158] In an optional embodiment, the task execution system 700 comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first sub-cluster, and the first loading node is a file loading node of the first sub-cluster; the obtaining module 710 and the scheduling module 720 are located in the scheduler; the first determination module 730 is located in the scheduler or the first manager; the second determination module 740 is located in the scheduler or the first manager; the space allocation module 770 is located in the scheduler or the first manager; the loading module 750 is located in the first loading node; and the indication module 760 is located in the first manager or the first loading node.

[0159] In an optional embodiment, the first manager is a management node in the first sub-cluster; or the first manager is located in the same device as any computing node in the first sub-cluster; or the first manager is located in the same device as the scheduler.

[0160] In an optional embodiment, the task execution system 700 further comprises a third determination module 780 configured to determine the first loading node in the first sub-cluster. The first loading node is any computing node in the first sub-cluster, or the first loading node is a computing node in the first sub-cluster that is scheduled to execute any of the plurality of sub-tasks. The third determination module 780 is located in the scheduler or the first manager.

[0161] In an optional embodiment, the first loading node is a computing node scheduled to execute at least one subtask in the plurality of subtasks (i.e., the first loading node is one of the at least two computing nodes in the first subcluster), and the number of the second files in the first files required to be loaded by the first loading node to execute the corresponding subtask and satisfy the threshold is greater than the target number; or the total data amount of the second files in the first files required to be loaded by the first loading node to execute the corresponding subtask and satisfy the threshold is greater than the target data amount. For example, the first loading node is a computing node in the first subcluster that requires the largest number of the second files to execute the corresponding subtask (i.e., the first loading node is one of the at least two computing nodes in the first subcluster that requires the largest number of the second files to execute the corresponding subtask); or the first loading node is a computing node in the first subcluster that requires the largest total data amount of the second files to execute the corresponding subtask (i.e., the first loading node is one of the at least two computing nodes in the first subcluster that requires the largest total data amount of the second files to execute the corresponding subtask).

[0162] In an optional embodiment, the subcluster is a supernode.

[0163] In an optional embodiment, the storage resource in the storage pool includes at least one of a volatile storage resource or a non-volatile storage resource.

[0164] In summary, the technical scheme provided by the embodiments of the present application is that at least two computing nodes in the first subcluster are scheduled to execute at least two subtasks in the to-be-executed task, and the shared storage space of the first subcluster can be accessed by the at least two computing nodes. The task execution system loads the second files in the first files required to be loaded by the at least two computing nodes to execute the corresponding subtasks and satisfy the threshold from the storage system to the shared storage space, and instructs the computing nodes requiring to load the second files among the at least two computing nodes to obtain the second files from the shared storage space to execute the corresponding subtasks. Thus, the computing nodes requiring to load the second files among the at least two computing nodes do not need to load the second files from the storage system concurrently, which can reduce the probability of congestion of the storage system, improve the speed and efficiency of loading the second files, and thus improve the task execution efficiency. Moreover, the computing nodes requiring to load the second files among the at least two computing nodes can avoid repeatedly loading the second files from the storage system, thereby avoiding the waste of the storage space of the computing nodes and the fragmentation of the storage space of the computing nodes.

[0165] The task execution system provided by the embodiments of the present application can also be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), which can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The task execution method provided by the embodiments of the present application can also be implemented by software. When the task execution method provided by the embodiments of the present application is implemented by software, each module in the task execution system can be a software module.

[0166] The embodiments of the present application provide a task execution system, which comprises a memory and a processor. The memory is configured to store a computer program. The processor is configured to execute the computer program stored in the memory, so that the task execution system executes all or part of the steps of the method provided by the embodiments shown in FIG. 2. Optionally, the task execution system is a system inside a device.

[0167] Please refer to FIG. 8, which shows a schematic diagram of another task execution system 800 provided by the embodiments of the present application. The task execution system 800 comprises a processor 801, a memory 802, a bus 803 and a communication interface 804. The processor 801, the memory 802 and the communication interface 804 are connected through the bus 803. FIG. 8 illustrates that the processor 801 and the memory 802 are independent of each other, and the processor 801 and the memory 802 can also be integrated together, which is not limited by the embodiments of the present application.

[0168] The memory 802 is used to store a computer program, which includes an operating system and program codes. The memory 802 can be various types of storage media. For example, the memory 802 is a random access memory (RAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), a flash memory, a register, an optical disc storage, a magnetic disc storage, or other magnetic storage devices.

[0169] The processor 801 can be a general processor or a special-purpose processor. The general processor is a processor that performs specific steps and / or operations by reading and executing a computer program stored in a memory, and the general processor can use the computer program stored in the memory in the process of executing the above steps and / or operations. The computer program is executed, for example, to implement the functions of the above-mentioned acquisition module 710, scheduling module 720, first determination module 730, second determination module 740, loading module 750, indication module 760, spatial allocation module 770, and third determination module 780, etc. The general processor is, for example but not limited to, a central processing unit (CPU). The special-purpose processor is a processor specially designed to perform specific steps and / or operations, and the special-purpose processor is, for example but not limited to, an application-specific integrated circuit (ASIC), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 801 can be a single-CPU processor or a multi-CPU processor. The processor 801 includes at least one circuit to perform all or part of the steps of the task execution method provided by the above embodiments.

[0170] The communication interface 804 includes a physical interface, a logical interface, and the like for realizing the interconnection of devices inside the task execution system 800, and an interface for realizing the interconnection of the task execution system 800 with other devices, networks, and the like. The physical interface can be an Ethernet interface, a fast Ethernet (FE) interface, a gigabit Ethernet (GE) interface, a Terabit Ethernet (TbE) interface, a 400GE interface, an asynchronous transfer mode (ATM) interface, and the like, which can be used to realize the interconnection of the task execution system 800 with other devices. The logical interface can be an interface inside the task execution system 800, which can be used to realize the interconnection of devices inside the task execution system 800. It is easy to understand that the communication interface 804 can be used for the task execution system 800 to communicate with other devices. For example, the communication interface 804 is used for the transmission and reception of information between the task execution system 800 and other devices, networks. The communication interface 804 can also be any kind of device such as a transmitter, a receiver, etc.

[0171] The bus 803 can be any type of communication bus for interconnecting the internal components of the task execution system 800. For example, the bus 803 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 803 can be divided into an address bus, a data bus, a control bus, etc. The embodiments of the present application take the above-mentioned components in the task execution system 800 as an example to illustrate that the above-mentioned components in the task execution system 800 can also be connected to each other in other ways. For example, the above-mentioned components in the task execution system 800 are interconnected through a logical interface in the task execution system 800, and the embodiments of the present application do not limit this.

[0172] The above-mentioned components can be respectively arranged on independent chips, or at least partially or entirely arranged on the same chip. Whether to arrange each component on a different chip or integrate the components on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned components.

[0173] Those skilled in the art can understand that the task execution system 800 shown in FIG. 8 is only exemplary, and the task execution system 800 can also include other components, which are not listed one by one herein. The task execution system 800 shown in FIG. 8 can load the files required by the task from the storage system to execute the task by executing all or part of the steps of the task execution method provided by the above-mentioned embodiments.

[0174] Based on the same inventive concept, the embodiments of the present application provide a computer readable storage medium, which stores a computer program. When the computer program is executed (for example, executed by the task execution system, the scheduler, the first manager, the first loading node, one or more processors, etc.), all or part of the steps of the task execution method provided by the method embodiment shown in FIG. 2 are implemented.

[0175] Based on the same inventive concept, the embodiments of the present application provide a computer program product, which includes a program or code. When the program or code is executed (for example, executed by the task execution system, the scheduler, the first manager, the first loading node, one or more processors, etc.), all or part of the steps of the task execution method provided by the method embodiment shown in FIG. 2 are implemented.

[0176] Based on the same inventive concept, the chip according to an embodiment of the present application comprises a programmable logic circuit and / or program instructions, and the chip is used to implement all or part of the steps of the task execution method provided by the method embodiment shown in FIG. 2 when running.

[0177] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium of the computer, or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media, or semiconductor media (such as solid state disk) and the like.

[0178] It should be understood that the term "at least one" in the present application refers to one or more, and the term "multiple" refers to two or more. In the present application, unless otherwise specified, the symbol " / " generally represents the meaning of or, for example, A / B can represent A or B. The term "and / or" in the present application is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, in order to clearly describe, the present application uses "first", "second", "third" and the like to distinguish the same items or similar items with basically the same function and role. Those skilled in the art can understand that "first", "second", "third" and the like do not limit the quantity and execution order.

[0179] The method embodiments and device embodiments provided by the embodiments of the present application can be mutually referenced, the order of the method embodiments can be adjusted appropriately, and the operations can be increased or decreased according to the situation. Any person skilled in the art can easily think of changes within the technical range disclosed in the present application, which should be covered within the protection scope of the present application, therefore, it is not necessary to repeat it.

[0180] In the corresponding embodiments provided by the present application, it should be understood that the disclosed apparatus, etc. can be implemented by other construction manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the modules is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, apparatuses or modules, and can be electrical or other forms. The modules described as separate components can be or can not be physically separated, and the components described as modules can be or can not be physical modules, and can be located in one place or distributed on a plurality of network nodes. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs.

[0181] The above describes only exemplary embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A task execution method characterized by, The method comprises: obtaining a task to be executed, the task to be executed comprising a plurality of subtasks; scheduling the plurality of subtasks to at least one subcluster in a computing cluster for execution, the at least one subcluster comprising a first subcluster, at least two computing nodes in the first subcluster being scheduled to execute at least two subtasks in the plurality of subtasks; determining first files required to be loaded by each of the at least two computing nodes when executing a corresponding subtask; determining, according to the first files required to be loaded by the at least two computing nodes, second files required to be loaded in a quantity satisfying a threshold; loading the second files to a shared storage space of the first subcluster, the shared storage space being accessible by the at least two computing nodes; instructing the computing nodes of the at least two computing nodes that require loading of the second files to obtain the second files from the shared storage space to execute corresponding subtasks.

2. The method of claim 1, wherein, The scheduling of the plurality of subtasks to at least one subcluster in a computing cluster for execution comprises: scheduling the plurality of subtasks to the at least one subcluster for execution according to a quantity of computing cores required by each of the plurality of subtasks and files required by the each of the plurality of subtasks.

3. The method according to claim 1 or 2, characterized in that, The determining of the first files required to be loaded by each of the at least two computing nodes when executing a corresponding subtask comprises: determining a subtask executed by each of the at least two computing nodes; determining files required by each of the subtasks executed by each of the at least two computing nodes; determining a union of the files required by the subtask executed by each of the at least two computing nodes as the first files required to be loaded by the each of the at least two computing nodes when executing a corresponding subtask.

4. The method according to any one of claims 1 to 3, characterized in that, The determining of the second files required to be loaded in a quantity satisfying a threshold according to the first files required to be loaded by the at least two computing nodes comprises: sorting the first files required to be loaded by the at least two computing nodes in descending order of the quantity required to be loaded; determining, as the second files, the first files whose quantity required to be loaded is greater than or equal to the threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The instructing of the computing nodes of the at least two computing nodes that require loading of the second files to obtain the second files from the shared storage space to execute corresponding subtasks comprises: instructing each of the computing nodes of the at least two computing nodes that require loading of the second files to mount the shared storage space to the each of the computing nodes, so that the each of the computing nodes obtains the second files from the shared storage space.

6. A task execution system characterized by comprising: The system comprises: an obtaining module, configured to obtain a task to be executed, the task to be executed comprising a plurality of subtasks; a scheduling module, configured to schedule the plurality of subtasks to at least one subcluster in a computing cluster for execution, the at least one subcluster comprising a first subcluster, at least two computing nodes in the first subcluster being scheduled to execute at least two subtasks in the plurality of subtasks; The first determining module is configured to determine first files required to be loaded by each of the at least two computing nodes when performing corresponding subtasks. The second determining module is configured to determine second files required to be loaded in a quantity satisfying a threshold according to the first files required to be loaded by the at least two computing nodes. The loading module is configured to load the second files to a shared storage space of the first subcluster, the shared storage space being accessible by the at least two computing nodes. The indicating module is configured to instruct the computing nodes of the at least two computing nodes requiring the second files to obtain the second files from the shared storage space to perform corresponding subtasks.

7. The system of claim 6, wherein, The scheduling module is configured to: schedule the plurality of subtasks to the at least one subcluster for execution according to the number of computing cores required by each of the plurality of subtasks and the files required by the each of the plurality of subtasks.

8. The system of claim 6 or 7, wherein, The first determining module is configured to: determine subtasks performed by each of the at least two computing nodes; determine files required to be loaded by each of the subtasks performed by each of the at least two computing nodes; determine a union of the files required by the subtasks performed by each of the at least two computing nodes as the first files required to be loaded by the each of the at least two computing nodes when performing corresponding subtasks.

9. The system of any one of claims 6 to 8, wherein, The second determining module is configured to: sort the first files required to be loaded by the at least two computing nodes in descending order of a quantity required to be loaded; determine the first files greater than or equal to the threshold as the second files.

10. The system of any one of claims 6 to 9, wherein, The indicating module is configured to: instruct each of the computing nodes of the at least two computing nodes requiring the second files to mount the shared storage space to the each of the computing nodes, so that the each of the computing nodes obtains the second files from the shared storage space.

11. The system of any one of claims 6 to 10, wherein, The system comprises a scheduler, a first manager and a first loading node, the first manager is configured to manage the first subcluster, and the first loading node is a file loading node of the first subcluster; The obtaining module and the scheduling module are in the scheduler; The first determining module is in the scheduler or the first manager; The second determining module is in the scheduler or the first manager; The loading module is in the first loading node; The indicating module is in the first manager or the first loading node.

12. The system of claim 11, wherein: the first manager is a management node in the first subcluster; or the first manager and any computing node in the first subcluster are in a same device; or the first manager and the scheduler are in a same device. comprise a memory and a processor; 13. A task execution system characterized by comprising: the memory is configured to store a computer program; the processor is configured to execute the computer program stored in the memory to enable the task execution system to perform the task execution method of any one of claims 1 to 5. ​ 14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed to implement the task execution method in any one of claims 1 to 5.

15. A computer program product, characterised in that, The computer program product comprises a program or code, and the program or code is executed to implement the task execution method in any one of claims 1 to 5.

Images