Task scheduling method, device, electronic device and storage medium

Abstract

Embodiments of the present invention provide a task scheduling method, apparatus, electronic device, and storage medium. The method comprises: receiving a task request submitted by a user; determining a target resource for a task corresponding to a task identifier from candidate resources based on the resources required for the task; downloading a task data packet corresponding to the task based on the download address information of the task data packet; determining whether the resource type of the target resource is a container resource type; if so, constructing a container based on a preset container image, and sending the task data packet to a queue of the constructed container. The present invention can improve resource reusability and the success rate of task execution.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a task scheduling method, device, electronic equipment and storage medium. Background Art

[0002] Big data services, through a scalable underlying big data platform and various upper-layer big data applications, support organizations and individuals in collecting, transmitting, storing, processing, exchanging, and destroying massive, heterogeneous, and rapidly changing data, covering a wide range of activities related to the data lifecycle. Existing big data tasks are primarily handled by big data platform components based on the characteristics of distributed systems.

[0003] YARN (Yet Another Resource Negotiator) is a resource manager that provides unified resource management and scheduling for upper-layer applications. Existing big data task scheduling methods use a YARN cluster-based task scheduling approach. A YARN cluster consists of virtual machines or physical machines. After receiving a task submitted by a user, the YARN cluster determines the resource status of each node (virtual machine or physical machine) in the cluster. Based on the required resources, it identifies the node that can execute the task. The task is then dispatched to the identified node, which then processes the dispatched task and returns the execution result.

[0004] However, existing task scheduling based on YARN clusters, which are composed of virtual machines or physical machines, occupies cluster resources after tasks are assigned to nodes, making it impossible to elastically increase or decrease task resources, and thus, to fully reuse cluster resources. Furthermore, because YARN clusters are developed based on the Java Virtual Machine, the cluster's inherent resource management, garbage collection mechanisms, and memory settings can affect the success rate of task execution. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a task scheduling method, device, electronic device, and storage medium to improve resource reusability and increase the success rate of task execution. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a task scheduling method, the method comprising:

[0007] Receive a task request submitted by a user, wherein the task request includes a task identifier, resources required for the task, and download address information of a task data packet;

[0008] Determining a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task;

[0009] Downloading the task data packet corresponding to the task based on the download address information of the task data packet;

[0010] Determine whether the resource type of the target resource is a container resource type;

[0011] If the resource type of the target resource is a container resource type, a container is constructed according to a preset container image, and the task data packet is sent to a queue of the constructed container.

[0012] Optionally, the method further includes:

[0013] If the resource type of the target resource is not a container resource type, the task data packets are assembled into a target request, and the target request is sent to the YARN resource queue of the target resource.

[0014] Optionally, the step of determining a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task includes:

[0015] determining a state of a first resource, where the state of the first resource includes a remaining available state of the first resource, and the first resource is a resource among the candidate resources;

[0016] determining, based on the state of the first resource, whether the first resource satisfies the resource requirements of the task;

[0017] If the first resource meets the resource requirements of the task, the first resource is determined as the target resource.

[0018] Optionally, when the task request includes a task-specified resource type, the first resource is a resource corresponding to the task-specified resource type.

[0019] Optionally, when the resource type priority of the first resource is not the lowest, after determining whether the first resource meets the resource requirements of the task, the method further includes:

[0020] If the first resource does not meet the resource requirements of the task, determining whether the task can be downgraded;

[0021] If the task can be downgraded, determining whether the second resource meets the resource requirements of the task, the resource type priority of the first resource is higher than the resource type priority of the second resource;

[0022] If the second resource meets the resource requirements of the task, the second resource is determined as the target resource.

[0023] Optionally, when the task request does not include a task-specified resource type, the first resource is a resource corresponding to a resource type with the highest priority among the candidate resources.

[0024] Optionally, after determining whether the first resource meets the resources required for the task, the method further includes:

[0025] If the first resource does not meet the resource requirements of the task, determining whether the task can be downgraded;

[0026] If the task can be downgraded, determining whether a third resource meets the resource requirements of the task, wherein the resource type priority of the first resource is higher than the resource type priority of the third resource;

[0027] If the third resource meets the resource requirements of the task, determining the third resource as the target resource;

[0028] If the third resource does not meet the resource requirements of the task, determining whether a fourth resource meets the resource requirements of the task, the resource type priority of the third resource being higher than the resource type priority of the fourth resource;

[0029] If the fourth resource meets the resource requirements of the task, the fourth resource is determined as the target resource.

[0030] Optionally, the method further includes:

[0031] Receive the task execution result information returned by the target resource.

[0032] Optionally, the candidate resources include resources of different resource type priorities, which are used to execute the task corresponding to the task identifier; resources with higher resource type priorities are used to process tasks with higher computing resource requirements; resources with lower resource type priorities are used to process tasks with lower computing efficiency requirements.

[0033] In a second aspect, an embodiment of the present invention provides a task scheduling device, the device comprising:

[0034] A first receiving module is used to receive a task request submitted by a user, wherein the task request includes a task identifier, resources required for the task, and download address information of a task data packet;

[0035] a determination module, configured to determine a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task;

[0036] A download module, configured to download the task data packet corresponding to the task based on the download address information of the task data packet;

[0037] A judgment module, configured to judge whether the resource type of the target resource is a container resource type;

[0038] The first sending module is used to build a container according to a preset container image when the judging module determines that the resource type of the target resource is a container resource type, and send the task data packet to a queue of the built container.

[0039] Optionally, the device further comprises:

[0040] The second sending module is used to assemble the task data packets into a target request and send the target request to the YARN resource queue of the target resource when the judgment module determines that the resource type of the target resource is not a container resource type.

[0041] Optionally, the determining module includes:

[0042] A first determining submodule is configured to determine a state of a first resource, where the state of the first resource includes a remaining available state of the first resource, and the first resource is a resource among the candidate resources;

[0043] A first judgment submodule, configured to judge, based on the state of the first resource, whether the first resource satisfies the resource requirements of the task;

[0044] The second determining submodule is configured to determine the first resource as a target resource when the first determining submodule determines that the first resource satisfies the resource requirements of the task.

[0045] Optionally, when the task request includes a task-specified resource type, the first resource is a resource corresponding to the task-specified resource type.

[0046] Optionally, when the resource type priority of the first resource is not the lowest, the apparatus further includes:

[0047] a second judgment submodule, configured to judge whether the task can be downgraded when the first judgment submodule judges that the first resource does not meet the resource requirements of the task;

[0048] a third judgment submodule, configured to, when the second judgment submodule determines that the task can be downgraded, determine whether the second resource meets the resource requirements of the task, wherein the resource type priority of the first resource is higher than the resource type priority of the second resource;

[0049] The third determining submodule is configured to determine the second resource as a target resource when the third judging submodule determines that the second resource satisfies the resource requirements of the task.

[0050] Optionally, when the task request does not include a task-specified resource type, the first resource is a resource corresponding to a resource type with the highest priority among the candidate resources.

[0051] Optionally, the device further comprises:

[0052] a fourth judgment submodule, configured to judge whether the task can be downgraded when the first judgment submodule judges that the first resource does not meet the resource requirements of the task;

[0053] a fifth judgment submodule, configured to, when the fourth judgment submodule determines that the task can be downgraded, determine whether a third resource satisfies the resource requirements of the task, wherein the resource type priority of the first resource is higher than the resource type priority of the third resource;

[0054] a fourth determining submodule, configured to determine the third resource as a target resource when the fifth determining submodule determines that the third resource satisfies the resource required for the task;

[0055] a sixth judgment submodule, configured to judge whether a fourth resource satisfies the resource requirements of the task when the fifth judgment submodule determines that the third resource does not meet the resource requirements of the task, wherein the resource type priority of the third resource is higher than the resource type priority of the fourth resource;

[0056] The fifth determining submodule is configured to determine the fourth resource as the target resource when the sixth determining submodule determines that the fourth resource satisfies the resource required by the task.

[0057] Optionally, the device further comprises:

[0058] The second receiving module is used to receive the task execution result information returned by the target resource.

[0059] Optionally, the candidate resources include resources of different resource type priorities, which are used to execute the task corresponding to the task identifier; resources with higher resource type priorities are used to process tasks with higher computing resource requirements; resources with lower resource type priorities are used to process tasks with lower computing efficiency requirements.

[0060] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0061] Memory for storing computer programs;

[0062] The processor is configured to implement the steps of the task scheduling method described in the first aspect above when executing the program stored in the memory.

[0063] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a task scheduling method described in the first aspect are implemented.

[0064] Beneficial effects of the embodiments of the present invention:

[0065] The embodiment of the present invention provides a task scheduling method, device, electronic device, and storage medium. The method can determine a target resource for a task corresponding to a task identifier among candidate resources based on the resources required for the task included in a received task request. Furthermore, the method can select a matching target resource for the task based on the actual situation of the resources required for the task, thereby improving the success rate of task execution. Furthermore, when the target resource is a container resource type, a container is constructed based on a preset container image, and the task data packet is sent to the queue of the constructed container. Because container resources are reusable, the reusability of the resources can be improved.

[0066] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 A schematic diagram of the structure of a big data scheduling system provided by an embodiment of the present invention;

[0069] Figure 2 A flowchart of a task scheduling method provided by an embodiment of the present invention;

[0070] Figure 3 A schematic diagram of an implementation method of a task scheduling method provided by an embodiment of the present invention;

[0071] Figure 4 A schematic diagram of the structure of a task scheduling device provided by an embodiment of the present invention;

[0072] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0074] To address the existing problem of task scheduling based on YARN clusters, which consists of virtual machines or physical machines. Once tasks are assigned to nodes, they occupy cluster resources, making it impossible to flexibly increase or decrease the resources occupied by tasks, and thus, to fully reuse cluster resources. Furthermore, because YARN clusters are developed based on the Java virtual machine, the cluster's inherent resource management, garbage collection mechanism, memory settings, and other factors can affect the success rate of task execution. Embodiments of the present invention provide a task scheduling method, apparatus, electronic device, and storage medium.

[0075] An embodiment of the present invention provides a task scheduling method, which may include:

[0076] Receive a task request submitted by a user, wherein the task request includes a task identifier, resources required for the task, and download address information of a task data packet;

[0077] Determining a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task;

[0078] Downloading the task data packet corresponding to the task based on the download address information of the task data packet;

[0079] Determine whether the resource type of the target resource is a container resource type;

[0080] If the resource type of the target resource is a container resource type, a container is constructed according to a preset container image, and the task data packet is sent to a queue of the constructed container.

[0081] A task scheduling method provided by an embodiment of the present invention can determine a target resource for a task corresponding to a task identifier among candidate resources based on the resources required for the task included in a received task request, and can then select a matching target resource for the task based on the actual situation of the resources required for the task, thereby improving the success rate of task execution. Furthermore, when the resource type of the target resource is a container resource type, a container is constructed based on a preset container image, and the task data packet is sent to the queue of the constructed container. Since container resources are reusable, the reusability of resources can be improved.

[0082] The following is a detailed description of a task scheduling method provided by an embodiment of the present invention:

[0083] The task scheduling method provided by the embodiment of the present invention can be applied to Figure 1 In the big data scheduling system shown, the scheduling system is based on the integration of YARN resources and container resources, and can provide users with a variety of operating resources. The scheduling system may include a scheduler, a shared resource pool, and a dedicated resource pool. The shared resource pool may include a container cluster and a shared YARN cluster. The container cluster may be composed of multiple container groups, each of which may include at least one container. The shared YARN cluster may be composed of multiple virtual machines or physical machines ( Figure 1 A dedicated resource pool may include a dedicated YARN cluster, and the dedicated YARN cluster may be composed of multiple virtual machines or physical machines.

[0084] In embodiments of the present invention, resource queues corresponding to users can be established in a shared YARN cluster, i.e., shared YARN resource queues, to provide operational resources to users. These shared YARN resource queues can be established on virtual machines or physical machines within the shared YARN cluster to provide services to multiple users. Specifically, a resource queue corresponding to each user can be established within the shared YARN cluster, and each user's corresponding resource queues occupy a consistent proportion within the shared YARN cluster, thereby achieving isolation between different users.

[0085] Container clusters can use K8s (Kubernetes) clusters, and namespaces can be created for each user within the container cluster to isolate different users. Dedicated YARN clusters can provide runtime resources for users on a specific whitelist. Within a dedicated YARN cluster, resource queues can be created for these whitelisted users, known as dedicated YARN resource queues. These dedicated YARN resource queues can be created on virtual machines or physical machines within the dedicated YARN cluster.

[0086] The embodiment of the present invention provides a task scheduling method, the execution subject can be Figure 1 The scheduler in the big data scheduling system shown can be any node in the scheduling system. For example, the scheduler can be any container or virtual machine in a shared resource pool, or any virtual machine in a dedicated resource pool.

[0087] like Figure 2 As shown, an embodiment of the present invention provides a task scheduling method, which may include the following steps:

[0088] S101: Receive a task request submitted by a user.

[0089] In an embodiment of the present invention, when performing task scheduling, the scheduler receives a task request submitted by a user. The task request may include a task identifier, resources required for the task, download address information for the task data packet, etc. The task identifier may be the task name, etc., and the resources required for the task may be resources such as the number of CPUs required for the task and the amount of memory required for the task.

[0090] The task type can be a Hadoop (Hadoop Distributed File System) task, a hive (big data offline data warehouse component) task, a flink (big data real-time stream computing component) task, or a spark (big data real-time batch computing component) task, etc.

[0091] Optionally, the content of the task corresponding to the task request submitted by the user may be a task such as processing a data stream or writing data.

[0092] For example, the structure format of the task request submitted by the user may be as follows:

[0093]

[0094] Among them, task_name defines the name of the task in the task request submitted by the user. This task name is test-spark-test. resource_type defines the resource type specified in the task request submitted by the user. access_token defines the authentication of the task request submitted by the user. force_run defines whether the task submitted by the user can be downgraded. task_jar defines the download address of the task data package submitted by the user. The maximum number of parallel tasks can be used to determine the number of subtasks that can be divided into during the task operation.

[0095] Of course, the embodiment of the present invention only introduces the task request submitted by the user in the above format, which does not constitute a specific limitation of the embodiment of the present invention. The parameters included in the task request submitted by the user in the embodiment of the present invention may be part, all or more of the above.

[0096] See also Figure 2 , S102, according to the resources required by the task, determine the target resource for the task corresponding to the task identifier in the candidate resources.

[0097] The scheduler receives a task request submitted by a user and, based on the resources required for the task contained in the task request, determines a target resource from the candidate resources for the task corresponding to the task identifier. The target resource is used to execute the task corresponding to the task request submitted by the user. The candidate resources may include resources of different resource type priorities for executing the task corresponding to the task identifier. Preferably, resources with higher resource type priorities are used to process tasks with higher computing resource requirements, while resources with lower resource type priorities are used to process tasks with lower computing efficiency requirements.

[0098] See also Figure 1 In a big data scheduling system, the candidate resources of an embodiment of the present invention may include container resources, shared YARN resources, and exclusive YARN resources, namely, corresponding container clusters, shared YARN clusters, and exclusive YARN cluster resources.

[0099] Optionally, the priority order of each resource type in the candidate resource can be: exclusive YARN resource types take precedence over shared YARN resource types, which in turn take precedence over container resource types. When a task request submitted by a user requires high computing resources, exclusive YARN resources or shared YARN resources can be used to process the task. When a task request submitted by a user requires low or moderate computing efficiency, or the number of tasks is small, container resources can be used to process the task. This ensures reasonable resource allocation and improves the success rate of task execution.

[0100] S103: Download the task data packet corresponding to the task based on the download address information of the task data packet.

[0101] After determining the target resource for running the task corresponding to the task identifier in the task request, the task data package corresponding to the task can be downloaded according to the download address information of the task data package in the task request submitted by the user to facilitate execution of the task.

[0102] S104: Determine whether the resource type of the target resource is a container resource type.

[0103] S105: If the resource type of the target resource is a container resource type, a container is constructed according to a preset container image, and the task data packet is sent to a queue of the constructed container.

[0104] After the scheduler determines the target resource for the task corresponding to the task identifier in the task request and downloads the task data packet corresponding to the task, it can further determine whether the resource type of the determined target resource is a container resource type. If it is determined that the resource type of the target resource is a container resource type, the scheduler builds a container for executing the task according to the pre-set container image, and then sends the task data packet to the resource queue of the constructed container so that the constructed container can run the task.

[0105] As an optional implementation method of an embodiment of the present invention, after the scheduler determines the target resource for the task corresponding to the task identifier in the task request and downloads the task data packet corresponding to the task, it can further determine whether the resource type of the determined target resource is a container resource type. If it is determined that the resource type of the target resource is not a container resource type, the task data packet can be assembled into a target request and the target request can be sent to the YARN queue of the target resource.

[0106] When it is determined that the resource type of the target resource is not a container resource type, it indicates that the resource type of the target resource is an exclusive YARN resource type or a shared YARN resource type. At this time, the task data packet can be assembled into a target request according to the assembly specification under the YARN resource platform, and the target request can be further sent to the YARN queue of the exclusive YARN resource or the YARN queue of the shared YARN resource.

[0107] Exemplarily, the target request can be a request in the form of a YARN REST API (Representational State Transfer Application Programming Interface). Since the YARN resource opens the REST API, the target request can be sent to a YARN queue of exclusive YARN resources or a YARN queue of shared YARN resources to execute tasks using exclusive YARN resources or shared YARN resources.

[0108] As an optional implementation of the embodiment of the present invention, after the scheduler sends the task data packet to the resource queue corresponding to the determined target resource, it may also receive task execution result information returned by the target resource.

[0109] A task scheduling method provided by an embodiment of the present invention can determine a target resource for the task corresponding to the task identifier among candidate resources based on the resources required for the task included in the received task request, and the candidate resources include resources with different resource type priorities for executing the task corresponding to the task identifier. The higher the resource type priority, the corresponding resource is used to process tasks with higher computing resource requirements, and the lower the resource type priority, the corresponding resource is used to process tasks with lower computing efficiency requirements. Then, according to the actual situation of the resources required for the task, a matching target resource can be selected for the task to improve the success rate of the task execution. Moreover, in the case where the resource type of the target resource is a container resource type, a container is constructed according to a preset container image, and the task data packet is sent to the queue of the constructed container. Since the container resources are reusable, the reusability of the resources can be improved.

[0110] As an optional implementation of the embodiment of the present invention, in the embodiment of the present invention, a method for determining a target resource for a task corresponding to a task identifier from candidate resources based on resources required by the task may include:

[0111] Determine a state of a first resource, wherein the state of the first resource may include a remaining available state of the first resource, and the first resource is a resource among candidate resources.

[0112] In an embodiment of the present invention, after receiving a task request submitted by a user, the scheduler may determine the status of any resource in the big data scheduling system to obtain the status of a first resource. The status of the first resource may include the remaining available state of the first resource and the current utilization state of the first resource. The first resource may be an exclusive YARN resource, a shared YARN resource, or a container resource.

[0113] Exemplarily, determining the remaining available state of the first resource may be determining the state of resources such as the available CPU and memory quantity in the first resource.

[0114] According to the state of the first resource, it is determined whether the first resource meets the resource requirements of the task.

[0115] After determining the status of the first resource, it is possible to further determine whether the first resource meets the resource requirements of the task. Specifically, the candidate resources include resources with different resource type priorities to be used to execute the task corresponding to the task identifier. Accordingly, based on the resource type priority, it is possible to first determine whether the resources with higher resource type priorities meet the resource requirements of the task, so that the resources with higher resource type priorities can be used to execute the task first, thereby improving the efficiency of task execution.

[0116] Determining whether the first resource meets the resource requirements of the task may include determining whether the remaining available CPU and memory quantity in the first resource meet the CPU and memory quantity required by the task.

[0117] If the first resource satisfies the resources required by the task, the first resource is determined as the target resource.

[0118] When it is determined that the first resource meets the resource requirements of the task, the first resource may be determined as the target resource.

[0119] Optionally, based on the priority of the resource type, it can be determined first whether the resource with a high priority type meets the resource requirements of the task, and if so, the resource with the high priority type is determined as the target resource, and no further determination is made whether the resources with other priority types meet the resource requirements of the task. When the resource with a high priority type does not meet the resource requirements of the task, it can be further determined whether the resource with the second highest priority type meets the resource requirements of the task, and if so, the resource is determined as the target resource, and no further determination is made whether the resources with other priority types meet the resource requirements of the task. When the resource with the second highest priority type does not meet the resource requirements of the task, it can be further determined whether the resource with a low priority type meets the resource requirements of the task, and if so, the resource is determined as the target resource.

[0120] When it is determined that any of the candidate resources does not meet the resource requirements of the task, a message indicating that the task submission has failed may be returned to the user.

[0121] In an embodiment of the present invention, the status of each resource in the candidate resources can be determined first, and then it can be judged whether the status of each resource meets the resource requirements of the task, so as to give priority to using resources with high resource type priority to execute the task and improve the efficiency of task execution.

[0122] As an optional implementation of an embodiment of the present invention, when the task request submitted by the user includes a task-specified resource type, the first resource in the above-mentioned determination of the state of the first resource may be a resource corresponding to the task-specified resource type. Accordingly, the above-mentioned implementation method of determining the target resource for the task corresponding to the task identifier from the candidate resources based on the resources required for the task may include:

[0123] Determine the status of the resource corresponding to the task specified resource type, and then judge whether the resource corresponding to the task specified resource type meets the resource requirements of the task. If so, determine the resource corresponding to the task specified resource type as the target resource.

[0124] As an optional implementation of an embodiment of the present invention, when a task request submitted by a user includes a task-specified resource type, the first resource is a resource corresponding to the task-specified resource type, and the resource type priority of the first resource is not the lowest, the above-mentioned process of determining a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task, after determining whether the first resource meets the resource requirements of the task, may further include:

[0125] If the first resource does not meet the resource requirements of the task, it is determined whether the task can be downgraded.

[0126] In an embodiment of the present invention, the task request submitted by the user may include an identifier of whether the task can be downgraded. When it is determined that the first resource corresponding to the resource type specified by the task does not meet the resources required by the task, it can be further determined whether the task can be downgraded based on the identifier of whether the task can be downgraded contained in the task request.

[0127] When it is determined that the first resource corresponding to the task-specified resource type does not meet the resource requirements of the task and the task cannot be downgraded, a task submission failure message may be returned to the user.

[0128] If the task can be downgraded, it is determined whether the second resource meets the resource requirements of the task, wherein the resource type priority of the first resource is higher than the resource type priority of the second resource.

[0129] If the second resource satisfies the resources required by the task, the second resource is determined as the target resource.

[0130] If it is determined that the task can be downgraded, it can be further determined whether a second resource having a lower resource type priority than the first resource satisfies the resource requirements of the task. If it is determined that the second resource having a lower resource type priority than the first resource satisfies the resource requirements of the task, the second resource is determined as the target resource.

[0131] Exemplarily, the first resource corresponding to the task-specified resource type is a shared YARN resource. When the shared YARN resource does not meet the resource requirements of the task and the task can be downgraded, it can be further determined whether the container resource whose resource type priority is lower than the resource type priority of the shared YARN resource meets the resource requirements of the task. If so, the container resource is determined as the target resource.

[0132] Exemplarily, the first resource corresponding to the task-specified resource type is an exclusive YARN resource. When the exclusive YARN resource does not meet the resource requirements of the task and the task can be downgraded, it can be further determined whether the shared YARN resource whose resource type priority is lower than the resource type priority of the exclusive YARN resource meets the resource requirements of the task. If so, the shared YARN resource is determined as the target resource. When the shared YARN resource does not meet the resource requirements of the task, it can be further determined whether the container resource whose resource type priority is lower than the resource type priority of the shared YARN resource meets the resource requirements of the task. If so, the container resource is determined as the target resource.

[0133] If it is determined that the first resource corresponding to the task specified resource type does not meet the resource requirements of the task, the task can be downgraded, and the second resource does not meet the resource requirements of the task, a task submission failure message can be returned to the user.

[0134] In an embodiment of the present invention, the task request may include an identification of whether the task can be downgraded, and then, if the resources corresponding to the task specified resource type do not meet the resources required by the task, the task is downgraded, and then the task is processed using resources whose resource type priority is lower than the task specified resource type priority, so as to improve the success rate of task execution.

[0135] As an optional implementation of an embodiment of the present invention, when the task request submitted by the user does not include a task-specified resource type, that is, the task does not specify a resource type, the first resource in the above-mentioned determination of the state of the first resource may be a resource corresponding to the highest resource type priority among the candidate resources. Accordingly, in the above-mentioned process of determining the target resource for the task corresponding to the task identifier among the candidate resources based on the resources required for the task, after determining whether the first resource meets the resource requirements of the task, the following may also be included:

[0136] If the first resource does not meet the resource requirements of the task, it is determined whether the task can be downgraded.

[0137] In an embodiment of the present invention, the task request submitted by the user may include an identification of whether the task can be downgraded. When it is determined that the first resource corresponding to the highest priority resource type does not meet the resources required for the task, or the user submitting the task request is not in the whitelist of the first resource corresponding to the highest priority resource type, it can be further determined whether the task can be downgraded based on the identification of whether the task can be downgraded contained in the task request.

[0138] When it is determined that the first resource corresponding to the resource type with the highest priority does not meet the resource requirements of the task and the task cannot be downgraded, a task submission failure message may be returned to the user.

[0139] If the task can be downgraded, it is determined whether the third resource meets the resource requirements of the task, wherein the resource type priority of the first resource is higher than the resource type priority of the third resource.

[0140] If the third resource satisfies the resources required by the task, the third resource is determined as the target resource.

[0141] If it is determined that the task can be downgraded, a further determination can be made as to whether a third resource having a lower resource type priority than the first resource satisfies the resource requirements of the task. If it is determined that the third resource having a lower resource type priority than the first resource satisfies the resource requirements of the task, the third resource is determined as the target resource.

[0142] Exemplarily, the first resource corresponding to the highest resource type priority is the exclusive YARN resource. When the exclusive YARN resource does not meet the resource requirements of the task, or the user who submits the task request is not in the whitelist of the exclusive YARN resource, and the task can be downgraded, it can be further determined whether the shared YARN resource (the third resource) with a lower resource type priority than the exclusive YARN resource meets the resource requirements of the task. If so, the shared YARN resource is determined as the target resource.

[0143] If the third resource does not meet the resource requirements of the task, it is determined whether the fourth resource meets the resource requirements of the task, wherein the resource type priority of the third resource is higher than the resource type priority of the fourth resource.

[0144] If the fourth resource satisfies the resources required by the task, the fourth resource is determined as the target resource.

[0145] If it is determined that a third resource having a lower resource type priority than the first resource does not meet the resource requirements of the task, the task may be downgraded (this determination has been made above and does not need to be made again here). Further, it may be determined whether a fourth resource having a lower resource type priority than the third resource meets the resource requirements of the task. If it is determined that the fourth resource having a lower resource type priority than the third resource meets the resource requirements of the task, the fourth resource is determined as the target resource.

[0146] Exemplarily, when the third resource shared YARN resource does not meet the resources required by the task and the task can be downgraded, it can be further determined whether the container resource (fourth resource) whose resource type priority is lower than the resource type priority of the shared YARN resource meets the resources required by the task, and if so, the container resource source is determined as the target resource.

[0147] If it is determined that the first resource corresponding to the highest priority resource type does not meet the resource requirements of the task, the task can be downgraded, and both the third resource and the fourth resource do not meet the resource requirements of the task, a task submission failure message can be returned to the user.

[0148] In an embodiment of the present invention, the task request may include an identification of whether the task can be downgraded, and then, if the resource corresponding to the highest priority resource type does not meet the resource requirements of the task, the task is downgraded, and then the task is processed using a resource with a lower priority resource type than the highest priority resource type, thereby realizing downgraded and graded processing of the task to improve the success rate of task execution.

[0149] As an optional implementation of the embodiment of the present invention, Figure 3 As shown, Figure 3 A schematic diagram of an implementation of a task scheduling method provided in an embodiment of the present invention may include:

[0150] Step 1: Receive a task request submitted by a user. The task request includes the task identifier, the resources required for the task, and the download address information of the task data package.

[0151] Step 2: determining the state of the first resource, where the state of the first resource includes the remaining available state of the first resource;

[0152] Step 3: judging whether the first resource meets the resource requirements of the task according to the status of the first resource;

[0153] Step 4: If the first resource meets the resource requirements of the task, the first resource is determined as the target resource;

[0154] Step 5: If the first resource does not meet the resource requirements of the task, and the task request includes a task-specified resource type, and the first resource is a resource corresponding to the task-specified resource type, and the resource type priority of the first resource is not the lowest, then determine whether the task can be downgraded;

[0155] Step 6: If the task can be downgraded, determine whether the second resource meets the resource requirements of the task, and the resource type priority of the first resource is higher than the resource type priority of the second resource;

[0156] Step 7: If the task cannot be downgraded, the task submission failure message is returned;

[0157] Step 8: If the second resource meets the resource requirements of the task, the second resource is determined as the target resource.

[0158] Step 9: If the second resource does not meet the resource requirements of the task, a task submission failure message is returned;

[0159] Step 10: If the first resource does not meet the resource requirements of the task, the task request does not include the task-specified resource type, and the first resource is the resource corresponding to the highest resource type priority among the candidate resources, then determine whether the task can be downgraded;

[0160] Step 11: If the task can be downgraded, determine whether the third resource meets the resource requirements of the task, and the resource type priority of the first resource is higher than the resource type priority of the third resource;

[0161] Step 12: If the task cannot be downgraded, a message indicating that the task submission failed is returned;

[0162] Step 13: If the third resource meets the resource requirements of the task, the third resource is determined as the target resource;

[0163] Step 14: If the third resource does not meet the resource requirements of the task, determine whether the fourth resource meets the resource requirements of the task, and the resource type priority of the third resource is higher than the resource type priority of the fourth resource;

[0164] Step 15: If the fourth resource meets the resource requirements of the task, the fourth resource is determined as the target resource.

[0165] Step 16: If the fourth resource does not meet the resource requirements of the task, a message indicating that the task submission failed is returned;

[0166] Step 17: Download the task data packet corresponding to the task based on the download address information of the task data packet;

[0167] Step 18: determine whether the resource type of the target resource is a container resource type;

[0168] Step 19: If the resource type of the target resource is a container resource type, a container is built according to a preset container image, and the task data packet is sent to the queue of the built container;

[0169] Step 20: If the resource type of the target resource is not a container resource type, assemble the task data packet into a target request and send the target request to the YARN resource queue of the target resource;

[0170] Step 21: Receive the task execution result information returned by the target resource.

[0171] Corresponding to the above method embodiments, the embodiments of the present invention also provide corresponding device embodiments.

[0172] like Figure 4 As shown, an embodiment of the present invention provides a task scheduling device, which may include:

[0173] The first receiving module 201 is configured to receive a task request submitted by a user. The task request includes a task identifier, resources required for the task, and download address information of a task data packet.

[0174] The determination module 202 is configured to determine a target resource for the task corresponding to the task identifier from candidate resources according to the resources required by the task.

[0175] The download module 203 is configured to download the task data packet corresponding to the task based on the download address information of the task data packet.

[0176] The judgment module 204 is used to judge whether the resource type of the target resource is a container resource type.

[0177] The first sending module 205 is configured to construct a container according to a preset container image and send the task data packet to a queue of the constructed container when the judging module 204 determines that the resource type of the target resource is a container resource type.

[0178] A task scheduling device provided by an embodiment of the present invention can determine a target resource for a task corresponding to a task identifier among candidate resources based on the resources required for the task included in a received task request, and can then select a matching target resource for the task based on the actual situation of the resources required for the task, thereby improving the success rate of task execution. Furthermore, when the resource type of the target resource is a container resource type, a container is constructed based on a preset container image, and the task data packet is sent to the queue of the constructed container. Since container resources are reusable, the reusability of resources can be improved.

[0179] Optionally, the above device may further include:

[0180] The second sending module is configured to assemble the task data packets into a target request and send the target request to the YARN resource queue of the target resource when the judging module 204 judges that the resource type of the target resource is not a container resource type.

[0181] Optionally, the determining module 202 includes:

[0182] The first determining submodule is configured to determine a state of a first resource, where the state of the first resource includes a remaining available state of the first resource, and the first resource is a resource among candidate resources.

[0183] The first judgment submodule is used to judge whether the first resource meets the resource requirements of the task according to the status of the first resource.

[0184] The second determining submodule is configured to determine the first resource as a target resource when the first determining submodule determines that the first resource satisfies the resource requirements of the task.

[0185] Optionally, when the task request includes a task-specified resource type, the first resource is a resource corresponding to the task-specified resource type.

[0186] Optionally, when the resource type priority of the first resource is not the lowest, the apparatus may further include:

[0187] The second judgment submodule is configured to judge whether the task can be downgraded when the first judgment submodule judges that the first resource does not meet the resource requirements of the task.

[0188] The third judgment submodule is used to judge whether the second resource meets the resource requirements of the task when the second judgment submodule judges that the task can be downgraded, and the resource type priority of the first resource is higher than the resource type priority of the second resource.

[0189] The third determining submodule is configured to determine the second resource as the target resource when the third judging submodule determines that the second resource satisfies the resource requirements of the task.

[0190] Optionally, when the task request does not include a task-specified resource type, the first resource is a resource corresponding to a resource type with the highest priority among the candidate resources.

[0191] Optionally, the above device further includes:

[0192] The fourth judgment submodule is configured to judge whether the task can be downgraded when the first judgment submodule judges that the first resource does not meet the resource requirements of the task.

[0193] The fifth judgment submodule is used to judge whether the third resource meets the resources required by the task when the fourth judgment submodule judges that the task can be downgraded, and the resource type priority of the first resource is higher than the resource type priority of the third resource.

[0194] The fourth determining submodule is configured to determine the third resource as the target resource when the fifth judging submodule determines that the third resource satisfies the resource requirements of the task.

[0195] The sixth judgment submodule is used to judge whether the fourth resource meets the resource requirements of the task when the fifth judgment submodule determines that the third resource does not meet the resource requirements of the task, and the resource type priority of the third resource is higher than the resource type priority of the fourth resource.

[0196] The fifth determining submodule is configured to determine the fourth resource as the target resource when the sixth judging submodule determines that the fourth resource satisfies the resource requirements of the task.

[0197] Optionally, the above device further includes:

[0198] The second receiving module is used to receive the task execution result information returned by the target resource.

[0199] Optionally, the candidate resources include resources of different resource type priorities, which are used to execute tasks corresponding to the task identifier; resources with higher resource type priorities are used to process tasks with higher computing resource requirements; resources with lower resource type priorities are used to process tasks with lower computing efficiency requirements.

[0200] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0201] Memory 303, for storing computer programs;

[0202] The processor 301 is configured to implement the steps of the above-mentioned task scheduling method when executing the program stored in the memory 303 to achieve the same technical effect.

[0203] The communication bus mentioned in the electronic device mentioned above may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the figure shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0204] The communication interface is used for communication between the above electronic device and other devices.

[0205] The memory may include RAM (Random Access Memory) or NVM (Non-Volatile Memory), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0206] The above-mentioned processor can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0207] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned task scheduling methods to achieve the same technical effect.

[0208] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When the computer program product is run on a computer, the computer executes the steps of any one of the task scheduling methods in the above embodiments to achieve the same technical effect.

[0209] In the above embodiments, the embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, DSL (Digital Subscriber Line)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD (Digital Versatile Disc)), or a semiconductor medium (eg, an SSD (Solid State Disk)).

[0210] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0211] Each embodiment in this specification is described in a related manner. Similar portions between the embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device / apparatus embodiments are generally similar to the method embodiments, so their description is relatively simple. For related portions, reference can be made to the description of the method embodiments.

[0212] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A task scheduling method, characterized in that: The method comprises: Receive a task request submitted by a user, wherein the task request includes a task identifier, resources required for the task, and download address information of a task data packet; Determining a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task; Downloading the task data packet corresponding to the task based on the download address information of the task data packet; Determine whether the resource type of the target resource is a container resource type; If the resource type of the target resource is a container resource type, a container is constructed according to a preset container image, and the task data packet is sent to a queue of the constructed container; The step of determining a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task includes: determining a state of a first resource, where the state of the first resource includes a remaining available state of the first resource, and the first resource is a resource among the candidate resources; determining, based on the state of the first resource, whether the first resource satisfies the resource requirements of the task; If the first resource meets the resource requirements of the task, determining the first resource as the target resource; In a case where the resource type priority of the first resource is not the lowest, if the first resource does not meet the resource requirements of the task, determining whether the task can be downgraded; If the task can be downgraded, determining whether the second resource meets the resource requirements of the task, the resource type priority of the first resource is higher than the resource type priority of the second resource; If the second resource meets the resource requirements of the task, the second resource is determined as the target resource.

2. The method according to claim 1, characterized in that The method further comprises: If the resource type of the target resource is not a container resource type, the task data packets are assembled into a target request, and the target request is sent to the YARN resource queue of the target resource.

3. The method according to claim 1, characterized in that In a case where the task request includes a task-specified resource type, the first resource is a resource corresponding to the task-specified resource type.

4. The method according to claim 1, wherein In a case where the task request does not include a task-specified resource type, the first resource is a resource corresponding to a resource type with the highest priority among the candidate resources.

5. The method according to claim 4, characterized in that After determining whether the first resource meets the resource requirements of the task, the method further includes: If the first resource does not meet the resource requirements of the task, determining whether the task can be downgraded; If the task can be downgraded, determining whether a third resource meets the resource requirements of the task, wherein the resource type priority of the first resource is higher than the resource type priority of the third resource; If the third resource meets the resource requirements of the task, determining the third resource as the target resource; If the third resource does not meet the resource requirements of the task, determining whether a fourth resource meets the resource requirements of the task, the resource type priority of the third resource being higher than the resource type priority of the fourth resource; If the fourth resource meets the resource requirements of the task, the fourth resource is determined as the target resource.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Receive the task execution result information returned by the target resource.

7. The method according to any one of claims 1 to 5, characterized in that: The candidate resources include resources of different resource type priorities, which are used to execute the tasks corresponding to the task identifier; resources with higher resource type priorities are used to process tasks with higher computing resource requirements; resources with lower resource type priorities are used to process tasks with lower computing efficiency requirements.

8. A task scheduling device, characterized in that: The device comprises: A first receiving module is used to receive a task request submitted by a user, wherein the task request includes a task identifier, resources required for the task, and download address information of a task data packet; a determination module, configured to determine a target resource for the task corresponding to the task identifier from candidate resources based on the resources required for the task; A download module, configured to download the task data packet corresponding to the task based on the download address information of the task data packet; A judgment module, configured to judge whether the resource type of the target resource is a container resource type; A first sending module is configured to, when the judging module determines that the resource type of the target resource is a container resource type, construct a container according to a preset container image and send the task data packet to a queue of the constructed container; Wherein, the determination module includes: A first determining submodule is configured to determine a state of a first resource, where the state of the first resource includes a remaining available state of the first resource, and the first resource is a resource among the candidate resources; A first judgment submodule, configured to judge, based on the state of the first resource, whether the first resource satisfies the resource requirements of the task; a second determining submodule, configured to determine the first resource as a target resource when the first determining submodule determines that the first resource satisfies the resource requirements of the task; a second judgment submodule, configured to, when the resource type priority of the first resource is not the lowest and the first judgment submodule determines that the first resource does not meet the resource requirements of the task, determine whether the task can be downgraded; a third judgment submodule, configured to, when the second judgment submodule determines that the task can be downgraded, determine whether the second resource meets the resource requirements of the task, wherein the resource type priority of the first resource is higher than the resource type priority of the second resource; The third determining submodule is configured to determine the second resource as a target resource when the third judging submodule determines that the second resource satisfies the resource requirements of the task.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 7 are implemented.

Images