Job instance scheduling method and device, equipment, medium and product

Abstract

The invention provides a job instance scheduling method and device, equipment, a medium and a product. Relates to the technical field of big data processing. The method comprises the following steps: acquiring at least one to-be-scheduled first job instance and related information of the first job instance; in response to a second job instance which is determined according to each piece of related information and meets a ready state, writing the second job instance into a ready job queue under the corresponding tenant; for a current second job instance in the ready job queue under each tenant, determining the type of the current second job instance, and obtaining a target available resource detection strategy matched with the type of the current second job instance; detecting available resources corresponding to the current second job instance based on the target available resource detection strategy; in response to the fact that the available resources meet the execution requirement of the current second job instance, submitting the current second job instance to a distributed queue; and scheduling the second job instance in the distributed queue to the corresponding physical resource group. According to the method provided by the invention, a job instance scheduling process is provided, the dependence of job scheduling on a bottom layer resource management framework is reduced, and the job instance scheduling efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a method, device, equipment, medium and product for scheduling a job instance. Background Art

[0002] Big data job scheduling is the process of planning, managing, and executing various data processing tasks in a big data processing environment. This process involves multiple types of jobs, each of which has its own specific computing framework. These frameworks have built-in resource management and control mechanisms, which can be combined with the upper-level task scheduling system to schedule job instances to optimize resource utilization and improve job execution efficiency.

[0003] Existing job scheduling is usually completed by a two-layer scheduling framework. The upper-layer task scheduling software can only perform coarse-grained job submission, while the actual job instance scheduling depends on the capabilities of the underlying resource management framework. Since the underlying frameworks corresponding to different types of job instances are quite different, it is difficult to achieve unified resource scheduling. In addition, the two-layer scheduling architecture leads to a long resource control link, which reduces scheduling efficiency. Summary of the invention

[0004] The present application provides a method, device, equipment, medium and product for scheduling a job instance, aiming to solve the technical problems of difficulty in unified resource scheduling and low scheduling efficiency in a two-layer scheduling framework due to strong dependence on the underlying resource management framework.

[0005] In a first aspect, the present application provides a method for scheduling a job instance, the method comprising:

[0006] Obtain at least one first job instance to be scheduled and related information of the first job instance;

[0007] In response to determining a second job instance that meets the ready state according to each of the relevant information, writing the second job instance into a ready job queue under the corresponding tenant;

[0008] For the current second job instance in the ready job queue under each tenant, determine the type of the current second job instance, and obtain a target available resource detection strategy that matches the type of the current second job instance;

[0009] Detecting available resources corresponding to the current second job instance based on the target available resource detection strategy;

[0010] In response to the available resources satisfying the execution requirement of the current second job instance, submitting the current second job instance to the dispatched queue;

[0011] The second job instance in the dispatched queue is scheduled to the corresponding physical resource group.

[0012] In a possible design, the relevant information of the first job instance includes a pre-dependency condition;

[0013] Determining a second job instance that meets the ready state according to the relevant information includes:

[0014] Determining whether the current condition satisfies the corresponding pre-dependency condition according to the pre-dependency condition of each first job instance;

[0015] In response to the current condition satisfying the pre-dependent condition, the first job instance whose current condition satisfies the pre-dependent condition is determined as the second job instance that satisfies the ready state.

[0016] In a possible design, the relevant information of the first job instance includes a tenant identifier:

[0017] Writing the second job instance to the ready job queue of the corresponding tenant includes:

[0018] According to the tenant identifier of the first job instance, writing the second job instance into the ready job queue of the tenant corresponding to the tenant identifier;

[0019] The method further comprises:

[0020] The relevant information of the second job instance is written into an ordered set of tenants corresponding to the tenant identifier, where the ordered set is located in a preset database.

[0021] In a possible design, the detecting the available resources corresponding to the current second job instance based on the target available resource detection strategy includes:

[0022] If the job instance type is Mpp type, the available Mpp cluster indicators and available resources are tested in sequence;

[0023] If the job instance type is a container type, the available resources and the load of the available cluster nodes are detected;

[0024] If the job instance type is Spark, the available resources are checked.

[0025] In a possible design, the indicators of the available Mpp cluster and the available resource amounts are detected in sequence, including:

[0026] Obtaining the indicators of the applied Mpp cluster and the applied resource amount in the relevant information of the second job instance;

[0027] In response to the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster, and the amount of resources applied being less than or equal to the amount of available resources, determining that the available resources meet the current second job instance execution requirement;

[0028] In response to the indicator of the applied Mpp cluster being greater than the indicator of the available Mpp cluster, or the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster and the applied resource amount being greater than the available resource amount, it is determined that the available resources do not meet the current second job instance execution requirements.

[0029] In a possible design, the detecting of the available resource amount and the load of the available cluster nodes includes:

[0030] Obtaining the applied cluster node load and the applied resource amount from the relevant information of the second job instance;

[0031] In response to the load of the applied cluster node being less than or equal to the load of the available cluster node, and the amount of the applied resources being less than or equal to the amount of the available resources, determining that the available resources meet the current second job instance execution requirement;

[0032] In response to the load of the requested cluster node being greater than the load of the available cluster node and / or the requested amount of resources being greater than the available amount of resources, it is determined that the available resources do not meet the current second job instance execution requirement.

[0033] In a possible design, the detecting of the available resource amount includes:

[0034] Obtaining the requested resource amount from the relevant information of the second job instance;

[0035] In response to the requested amount of resources being less than or equal to the available amount of resources, determining that the available resources meet the current second job instance execution requirement;

[0036] In response to the requested resource amount being greater than the available resource amount, it is determined that the available resources do not meet the current second job instance execution requirement.

[0037] In one possible design, the method further includes:

[0038] Periodically collect at least one of the following available resource data from the underlying computing platform:

[0039] The amount of resources available in the physical resource group, the metrics of the available Mpp clusters, and the load of the available cluster nodes;

[0040] The available Mpp cluster indicators include: the number of available user connections and the available queue length of the Mpp cluster.

[0041] In a possible design, after scheduling the second job instance in the dispatched queue to the corresponding physical resource group, the method further includes:

[0042] In response to a scheduling failure, a scheduling failure prompt message is issued;

[0043] The second job instance that fails in scheduling is rewritten into the ready job queue of the corresponding tenant according to the scheduling failure prompt information.

[0044] In a second aspect, the present application provides a job instance scheduling device, the device comprising:

[0045] An acquisition module, used to acquire at least one first job instance to be scheduled and related information of the first job instance;

[0046] a writing module, configured to, in response to determining a second job instance that satisfies a ready state according to each of the related information, write the second job instance into a ready job queue under a corresponding tenant;

[0047] A determination module, configured to determine the type of the current second job instance in the ready job queue of each tenant;

[0048] A detection module, configured to detect available resources corresponding to the current second job instance based on a target available resource detection strategy;

[0049] A submission module, configured to submit the current second job instance to a dispatched queue in response to available resources satisfying the execution requirement of the current second job instance;

[0050] A scheduling module, used for scheduling the second job instance in the dispatched queue to the corresponding physical resource group;

[0051] The acquisition module is further used to acquire a target available resource detection strategy that matches the type of the current second job instance.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0056] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0057] The job instance scheduling method, device, equipment, medium and product provided by the present application obtain at least one first job instance to be scheduled and the relevant information of the first job instance; in response to determining the second job instance that meets the ready state according to each of the relevant information, the second job instance is written into the ready job queue under the corresponding tenant; for the current second job instance in the ready job queue under each tenant, the type of the current second job instance is determined, and the target available resource detection strategy matching the type of the current second job instance is obtained; the available resources corresponding to the current second job instance are detected based on the target available resource detection strategy; in response to the available resources meeting the execution requirements of the current second job instance, the current second job instance is submitted to the dispatched queue; the second job instance in the dispatched queue is scheduled to the corresponding physical resource group. By obtaining the relevant information of the job instance and determining its ready state, the job instance that meets the scheduling conditions is screened out and placed in the ready job queue for subsequent job instance scheduling, which can avoid invalid scheduling and is conducive to improving scheduling efficiency. At the same time, for the job instance in the ready job queue, the target available resource detection strategy matching the job instance type is obtained, which is conducive to accurate detection of resources and unified scheduling of different types of job instances. In addition, using the concept of physical resource groups to characterize resources can also shield the differences in computing platforms to a certain extent, thereby reducing the difficulty of unified scheduling and improving scheduling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 An application scenario diagram of the job instance scheduling method provided in this application;

[0060] Figure 2 A flowchart of a method for scheduling a job instance provided in one embodiment of the present application;

[0061] Figure 3 A flowchart of a method for scheduling a job instance provided by another embodiment of the present application;

[0062] Figure 4 A schematic diagram of the structure of a job instance scheduling device provided in one embodiment of the present application;

[0063] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0064] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0066] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved in the technical solution of this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0067] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0068] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail.

[0069] Currently, the method used to schedule job instances usually relies on a two-layer scheduling framework. This two-layer architecture will result in a long resource management link, which reduces the efficiency of scheduling. In addition, the job instance scheduling process usually involves multiple types of jobs. The upper-level task scheduling software can only perform coarse-grained job submission, while the actual fine-grained scheduling of jobs depends on a specific underlying resource management framework. Due to the large differences between the underlying resource management frameworks, it is difficult to achieve unified scheduling of multiple types of job instances.

[0070] Therefore, in the face of technical problems in the prior art, in order to reduce the dependence on the underlying resource management framework and achieve unified scheduling of multiple types of jobs, the type of job instance can be determined first, and then a matching target available resource detection strategy can be used for different types. This method can improve the flexibility of the scheduling system. In order to improve the scheduling efficiency of job instances, job instances that meet the ready state can be first screened out, and based on the target available resource detection strategy, corresponding available resource detection can be performed on various types of job instances, thereby reducing the scheduling failure caused by job instances not meeting the ready state or insufficient available resources, thereby improving scheduling efficiency.

[0071] Figure 1 This is an application scenario diagram that can implement the job instance scheduling method provided by this application, such as Figure 1 As shown, the scenario diagram corresponding to the job instance scheduling method provided in the present application includes: an external device 101, a server 102 and a server cluster 103. The server cluster 103 includes multiple physical resource groups, and a physical resource group represents a computing resource in a computing platform. Among them, the external device 101 is used to send the first job instance to be scheduled and its related information. For example, it can be a device corresponding to a big data scheduling task, which can be a server, a server cluster, a cloud device, etc. The job instance scheduling device can be integrated in the server 102, wherein the server is a form of a job instance scheduling device, and the job instance scheduling device can also be in other forms, which is not limited in this embodiment. The job instance scheduling device is a type of electronic device.

[0072] Specifically, the external device 101 sends the first job instance to be scheduled and its related information to the server 102, and the server 102 receives and obtains at least one first job instance to be scheduled and its related information. Then, the server 102 determines the second job instance that meets the ready state based on these related information, and writes the second job instance into the ready job queue under the corresponding tenant. Next, the server 102 determines the type of the current second job instance in the ready job queue under each tenant, and obtains the target available resource detection strategy that matches the type of the current second job instance. The server 102 detects the available resources corresponding to the current second job instance based on the target available resource detection strategy. In response to the available resources meeting the execution requirements of the current second job instance, the server 102 submits the current second job instance to the dispatched queue, and finally dispatches the second job instance in the dispatched queue to the corresponding physical resource group in the server cluster 103.

[0073] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0074] Figure 2 A flowchart of a method for scheduling a job instance provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the execution subject of this embodiment is a job instance scheduling device, which can be implemented by a computer program. It can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or a job instance scheduling device. The job instance scheduling device can be a server, a server cluster or other electronic devices. The job instance scheduling method provided in this embodiment includes the following steps:

[0075] Step 201: Obtain at least one first job instance to be scheduled and related information of the first job instance.

[0076] The first job instance is a job instance to be scheduled, which is a specific and independent task or job execution instance.

[0077] The relevant information of the first job instance refers to all necessary information and configuration information of the first job instance.

[0078] Specifically, at least one first job instance can be obtained from the job instance scheduling system. Each first job instance will carry relevant information. Exemplarily, the relevant information may include the job type, input data, and configuration parameters of the first job instance.

[0079] Step 202: In response to determining that a second job instance meets the ready state according to the relevant information, the second job instance is written into a ready job queue under the corresponding tenant.

[0080] The ready state refers to the state corresponding to the first job instance that meets the job execution conditions. If a job instance meets the execution conditions, the job instance can be scheduled or executed immediately.

[0081] The second job instance refers to a job instance in a ready state. In this embodiment, if the first job instance is determined to be in a ready state based on the relevant information of the first job instance, the first job instance in this state is the second job instance.

[0082] The tenant refers to the user or user group to which the current second job instance belongs.

[0083] The ready job queue refers to a queue that contains multiple ready job instances maintained for each tenant.

[0084] Specifically, for each previously acquired relevant information of the first job instance, determine whether the first job instance meets the ready state. In this embodiment, the ready state may include that the configuration parameters and input data of the first job instance have been prepared, the current time is already within the execution time window of the first job instance, and the first job instance has no other blocking conditions. After judgment, if the first job instance meets the ready state, the first job instance can be determined as the second job instance, and the second job instance is written into the ready job queue of the tenant to which the job instance belongs.

[0085] Step 203: for the current second job instance in the ready job queue of each tenant, determine the type of the current second job instance, and obtain a target available resource detection strategy that matches the type of the current second job instance.

[0086] The target available resource detection strategy refers to a strategy used to determine whether the available resources of the computing platform meet the execution requirements of the second job instance.

[0087] In this embodiment, each type of job instance may correspond to different resource requirements and detection strategies. Exemplarily, if the job type to which the current second job instance belongs is a machine learning training type, the target available resource detection strategy that matches the job type may be to check the number of CPU cores and available GPUs of the current computing platform, check the available memory in the current computing platform, check the disk I / O and network bandwidth in the current computing platform, etc. Specifically, the second job instance to be scheduled is obtained from the ready job queue of the tenant, and the job type to which it belongs can be determined based on the relevant information of the second job instance, and then the target available resource detection strategy that matches it is obtained based on the determined type of the second job instance.

[0088] The type of the second job instance may be a machine learning job, a graph computing job, a data analysis job, etc. It is understandable that the type of the second job instance and the target available resource detection strategy matching it are not limited in this embodiment.

[0089] Step 204: Detect available resources corresponding to the current second job instance based on the target available resource detection strategy.

[0090] The available resources corresponding to the second job instance refer to the resources in the computing platform that can currently be provided to the second job instance for use.

[0091] Exemplarily, the current second job instance can be a data analysis type job. According to the target available resource detection strategy obtained by the data analysis type, the number of available CPU cores, available memory, and available disk I / O bandwidth in the computing platform are detected. The detected result can be the available resources corresponding to the current second job instance, or whether the available resources corresponding to the current second job instance meet the execution requirements of the current second job instance.

[0092] Step 205: In response to the available resources satisfying the execution requirement of the current second job instance, submitting the current second job instance to the dispatched queue.

[0093] The dispatched queue refers to a queue that stores job instances that have passed resource detection and are ready for immediate scheduling. The dispatched queue may be in the form of a data structure in memory, such as a queue, a list, etc. The form of the dispatched queue is not limited in this embodiment.

[0094] Specifically, the available resources of the computing platform are detected according to the target available resource detection strategy. If the detected available resources are sufficient to support the execution of the current second job instance, the current second job instance can be submitted to the dispatched queue to prepare for subsequent scheduling of the current second job instance.

[0095] Step 206: Schedule the second job instance in the dispatched queue to the corresponding physical resource group.

[0096] Among them, the physical resource group is a logical concept used to uniformly represent and manage the resources in the computing platform. In this embodiment, the computing platform is the infrastructure and software environment used to execute and manage computing tasks. Each physical resource group can be associated with a specific computing platform through a physical channel. In different types of computing platforms, the physical channel can represent different meanings. For example, in the Hadoop computing platform, the physical channel represents the Yarn queue; in the Kubernetes computing platform, the physical channel represents the Namespace. The role of the physical channel is to associate and manage the physical resource group and the specific resources in the computing platform.

[0097] Specifically, the second job instance may include identification information of the physical resource group to which it corresponds, and the second job instance may be scheduled to the corresponding physical resource group according to the identification information.

[0098] Among them, Yarn is the resource manager in the Hadoop ecosystem, and the Yarn queue is used to manage and schedule computing tasks.

[0099] Namespace is a logical partition in Kubernetes, which is used to group and isolate resources and objects.

[0100] The identification information of the physical resource group may be a unique identification ID of the physical resource group, which is not limited in this embodiment.

[0101] The job instance scheduling method provided in this embodiment obtains at least one first job instance to be scheduled and related information of the first job instance; in response to determining a second job instance that meets the ready state according to the related information, the second job instance is written into the ready job queue under the corresponding tenant; for the current second job instance in the ready job queue under each tenant, the type of the current second job instance is determined, and the target available resource detection strategy matching the type of the current second job instance is obtained; the available resources corresponding to the current second job instance are detected based on the target available resource detection strategy; in response to the available resources meeting the execution requirements of the current second job instance, the current second job instance is submitted to the dispatched queue; the second job instance in the dispatched queue is scheduled to the corresponding physical resource group. The job instances that meet the scheduling conditions are screened out and put into the ready job queue for subsequent job instance scheduling, so as to avoid invalid scheduling and improve scheduling efficiency. At the same time, by obtaining the target available resource detection strategy matching the job instance type in the ready job queue, resources can be accurately detected to achieve unified scheduling of job instances of different types. In addition, using the concept of physical resource groups to characterize resources can shield the differences in computing platforms to a certain extent, reduce the difficulty of unified scheduling, and further improve scheduling efficiency.

[0102] As an optional implementation, based on the above embodiment, the relevant information of the first job instance includes a pre-dependency condition;

[0103] Determining a second job instance that meets the ready state according to the relevant information includes:

[0104] Determining whether the current condition satisfies the corresponding pre-dependency condition according to the pre-dependency condition of each first job instance;

[0105] In response to the current condition satisfying the pre-dependent condition, the first job instance whose current condition satisfies the pre-dependent condition is determined as the second job instance that satisfies the ready state.

[0106] The pre-dependency condition refers to the condition that needs to be met before scheduling or executing the first job instance. The pre-dependency condition may include task dependency, data dependency, resource dependency, time dependency, event dependency, etc. For example, the pre-dependency condition may be that the first job instance needs to be performed after a certain job instance is completed, or that the first job instance needs to be executed within a certain time window.

[0107] Specifically, for the first job instance, the pre-dependency conditions of the first job instance will be checked first, and it will be determined whether the current conditions meet the pre-dependency conditions. Exemplarily, if the first job instance depends on the completion of another job instance, it will be checked whether the status of the dependent job instance is completed. If the current conditions meet the pre-dependency conditions, the first job instance can be determined as a second job instance that meets the ready state. In this embodiment, whether the first job instance meets the pre-dependency conditions can be checked in the job management service component, and the first job instance that meets the pre-dependency conditions can be determined as a second job instance that meets the ready state.

[0108] The job instance scheduling method provided in this embodiment, the relevant information of the first job instance includes a pre-dependency condition; determining the second job instance that meets the ready state according to each relevant information includes: determining whether the current condition meets the corresponding pre-dependency condition according to the pre-dependency condition of each first job instance; in response to the current condition meeting the pre-dependency condition, determining the first job instance whose current condition meets the pre-dependency condition as the second job instance that meets the ready state. By introducing the check of the pre-dependency condition, it can be ensured that only when all the pre-dependency conditions are met, the first job instance will be determined as the second job instance in the ready state and participate in the subsequent scheduling. This not only improves the accuracy and reliability of job instance scheduling, but also avoids scheduling failures and waste of computing resources caused by failure to meet the pre-conditions, thereby helping to improve scheduling efficiency.

[0109] As an optional implementation, based on the above embodiment, the relevant information of the first job instance includes a tenant identifier:

[0110] Write the second job instance to the ready job queue of the corresponding tenant, including:

[0111] According to the tenant identifier of the first job instance, the second job instance is written into the ready job queue of the tenant corresponding to the tenant identifier;

[0112] The method also includes:

[0113] The relevant information of the second job instance is written into an ordered set of tenants corresponding to the tenant identifier, and the ordered set is located in a preset database.

[0114] The ordered set refers to a data structure used to store and manage relevant information of a job instance.

[0115] Specifically, the obtained relevant information of the first job instance includes the tenant identification information to which the job instance belongs. In this embodiment, after determining that the job instance meets the ready state according to the relevant information of the first job instance, the job instance can be determined as the second job instance in the job management service component, and the relevant information of the current second job instance can be written into the ready queue corresponding to the tenant identification according to the obtained tenant identification information of the first job instance.

[0116] In this embodiment, the ready queue of the tenant is stored in a preset database in the form of an ordered set. The preset database may be a Redis database, and the ordered set may be an ordered set Zset in Redis. Exemplarily, for a first job instance of a tenant identified as A, if the instance satisfies the ready state and is determined to be a second job instance, the second job instance of tenant A may be stored in an ordered set Zset in the Redis database. The key of Zset is the ID of the tenant, and the information of each second job instance of the tenant will be a member of the Zset of the tenant. Each member can use the priority and submission timestamp of the job instance to calculate a score, and sort the members in the set according to the score. For example, for a job instance with a priority of 3, its score calculation method may be, score = priority * 1000000000 + submission timestamp. The members may be sorted according to their scores, that is, the ready job queue of the tenant is obtained. It is understandable that the sorting method of tenant member information is not limited in this embodiment.

[0117] The job instance scheduling method provided in this embodiment, the relevant information of the first job instance includes the tenant identifier: writing the second job instance into the ready job queue under the corresponding tenant includes: according to the tenant identifier of the first job instance, writing the second job instance into the ready job queue of the tenant corresponding to the tenant identifier; the method also includes: writing the relevant information of the second job instance into the ordered set of the tenant corresponding to the tenant identifier, and the ordered set is located in a preset database. By using a preset database to store the ordered set, the persistence and high availability of data can be ensured, which is conducive to improving the stability of job scheduling; in addition, by using the ordered set, it is conducive to efficient sorting of job instances and improving the efficiency of overall job instance scheduling.

[0118] As an optional implementation manner, based on the above embodiment, the available resources corresponding to the current second job instance are detected based on the target available resource detection strategy, including:

[0119] If the job instance type is Mpp type, the available Mpp cluster indicators and available resources are tested in sequence;

[0120] If the job instance type is a container type, the available resources and the load of the available cluster nodes are detected;

[0121] If the job instance type is Spark, the available resources are checked.

[0122] Among them, Mpp type refers to the type of job for large-scale parallel processing. Mpp type jobs usually involve a large amount of data processing and calculation, which can be executed simultaneously on multiple computing nodes through parallelization technology. Correspondingly, Mpp cluster is a computing architecture consisting of multiple independent computing nodes, designed to efficiently process large-scale data and complex computing tasks through parallel processing technology. It can be understood that the indicators and available resources of the available Mpp cluster are important parameters for measuring and managing computing resources in the cluster.

[0123] Exemplarily, the indicators of the available Mpp clusters are first detected, and then the available resources in the Mpp clusters are detected. It is understandable that for Mpp type jobs, the indicators of the available Mpp clusters need to be detected first, and after the detection is passed, the available resources are detected, so as to avoid invalid job scheduling caused by sufficient resources but busy clusters.

[0124] Among them, container type refers to jobs or tasks running in a containerized environment. Container jobs are usually managed and scheduled using container orchestration tools. Correspondingly, for container type jobs, the available resources refer to the unallocated part of the resource quota under a specific namespace in the Kubernetes cluster; the load of the available cluster node refers to the load that can carry new container type jobs without causing node resource overload.

[0125] The Spark type refers to a data processing task in Hadoop. In the Hadoop ecosystem, Yarn is a resource management and scheduling system responsible for managing computing resources in the cluster. Spark jobs can be run on Yarn, which uses Yarn to manage and schedule the resources required for Spark jobs. Correspondingly, for Spark job instances, the available resources can refer to the remaining load information of the Yarn queue.

[0126] Specifically, in this embodiment, the job dispatch service component can be used to select a target available resource detection strategy that matches the job instance type for different types of job instances, and detect the corresponding available resources. After the detection is completed, the information of the detected available resources can be saved in the Redis database. It can be understood that the method of saving the information of the available resources is not limited in this embodiment.

[0127] The job instance scheduling method provided in this embodiment detects the available resources corresponding to the current second job instance based on the target available resource detection strategy, including: if the job instance type is the Mpp type, the indicators of the available Mpp cluster and the available resource amount are detected in sequence; if the job instance type is the container type, the available resource amount and the load of the available cluster nodes are detected; if the job instance type is the Spark type, the available resource amount is detected. By adopting different resource detection strategies for job instances of Mpp type, container type and Spark type, the system can respond to various job requirements more flexibly. This not only helps to achieve unified scheduling of different types of jobs, but also optimizes resource utilization, thereby improving the efficiency of job instance scheduling.

[0128] As an optional implementation, based on the above embodiment, the indicators of the available Mpp cluster and the available resource amount are detected in sequence, including:

[0129] Obtain the index of the applied Mpp cluster and the applied resource amount in the relevant information of the second job instance;

[0130] In response to the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster, and the amount of resources applied being less than or equal to the amount of available resources, determining that the available resources meet the current second job instance execution requirement;

[0131] In response to the indicator of the applied Mpp cluster being greater than the indicator of the available Mpp cluster, or the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster and the applied resource amount being greater than the available resource amount, it is determined that the available resources do not meet the current second job instance execution requirements.

[0132] Among them, the index of the applied Mpp cluster refers to the index amount of the Mpp cluster required by the second job instance. Correspondingly, the amount of resources applied refers to the amount of resources required by the second job instance. Exemplarily, the index of the applied Mpp cluster may refer to the number of connections required by the current second job instance, and the amount of resources applied may refer to the amount of memory required by the current second job instance. This embodiment is not limited to this.

[0133] Exemplarily, if the type of the current second job instance is the Mpp type, then the indicators of the available Mpp cluster can be detected first, and then the available resources can be detected. Specifically, first, the indicator amount of the Mpp cluster required to apply for the current second job instance and the amount of resources required to apply for the current second job instance can be obtained from the relevant information of the current second job instance; then, the indicators of the available Mpp cluster and the available resources can be obtained from the Redis database. After obtaining the above information, it is necessary to first compare the indicators of the applied Mpp cluster and the indicators of the available Mpp cluster. If the indicator of the applied Mpp cluster is less than or equal to the indicator of the available Mpp cluster, and the amount of resources applied is less than or equal to the amount of available resources, it means that the current available resources are sufficient to support the execution of the current second job instance; correspondingly, if the indicator of the applied Mpp cluster is greater than the indicator of the available Mpp cluster, or the indicator of the applied Mpp cluster is less than or equal to the indicator of the available Mpp cluster and the amount of resources applied is greater than the amount of available resources, it means that the current available resources are insufficient to support the execution of the current second job instance. According to the above information comparison method, it can be determined whether the currently available resources can meet the execution requirements of the current second job instance.

[0134] The job instance scheduling method provided in this embodiment detects the indicators of the available Mpp cluster and the available resource amount in sequence, including: obtaining the indicators of the applied Mpp cluster and the applied resource amount in the relevant information of the second job instance; in response to the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster, and the applied resource amount being less than or equal to the available resource amount, determining that the available resources meet the current second job instance execution requirements; in response to the indicator of the applied Mpp cluster being greater than the indicator of the available Mpp cluster, or the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster and the applied resource amount being greater than the available resource amount, determining that the available resources do not meet the current second job instance execution requirements. By comparing the applied indicators and resource amount with the available indicators and resource amount, it is determined whether the currently available resources can support the current job instance, which can ensure that resources are efficiently utilized, avoid the situation where the job instance fails to be scheduled due to insufficient resources, and improve the efficiency of job instance scheduling; in addition, for the Mpp job type, the indicator detection is first performed, and then the resource amount detection is performed, which can avoid invalid scheduling of job instances and virtual occupation of resources to a certain extent.

[0135] As an optional implementation, based on the above embodiment, the available resource amount and the load of the available cluster nodes are detected, including:

[0136] Obtaining the applied cluster node load and the applied resource amount from the relevant information of the second job instance;

[0137] In response to the load of the applied cluster node being less than or equal to the load of the available cluster node, and the amount of the applied resources being less than or equal to the amount of the available resources, determining that the available resources meet the current second job instance execution requirement;

[0138] In response to the load of the requested cluster node being greater than the load of the available cluster node and / or the requested amount of resources being greater than the available amount of resources, it is determined that the available resources do not meet the current second job instance execution requirement.

[0139] The load of the cluster node applied for refers to the load of the cluster node required by the second job instance. Correspondingly, the amount of resources applied for refers to the amount of resources required by the second job instance. Exemplarily, the amount of resources applied for in this embodiment may refer to the amount of resource quota that the current second job instance needs to allocate to the Namespace. This embodiment does not limit this.

[0140] Exemplarily, if the type of the current second job instance is a container type, then the available resource amount and the load of the available cluster node can be detected respectively. Specifically, first, the resource amount required for the current second job instance and the load of the cluster node required for the current second job instance can be obtained from the relevant information of the current second job instance; then, the available resource amount and the load of the available cluster node can be obtained from the Redis database. After obtaining the above information, the applied resource amount and the available resource amount, the applied cluster node load and the available cluster node load can be compared respectively. In this embodiment, if the applied resource amount and the applied cluster node load are both less than or equal to the available resource amount and the available cluster node load, it means that the current available resources are sufficient to support the execution of the current second job instance; correspondingly, if the applied resource amount is greater than the available resource amount, and / or the applied cluster node load is greater than the available cluster node load, it means that the current available resources are insufficient to support the execution of the current second job instance. According to the above information comparison method, it can be determined whether the current available resources can meet the execution requirements of the current second job instance.

[0141] The job instance scheduling method provided in this embodiment detects the available resource amount and the load of the available cluster nodes, including: obtaining the load of the applied cluster node and the applied resource amount in the relevant information of the second job instance; in response to the load of the applied cluster node being less than or equal to the load of the available cluster node, and the applied resource amount being less than or equal to the available resource amount, determining that the available resources meet the current second job instance execution requirements; in response to the load of the applied cluster node being greater than the load of the available cluster node, and / or the applied resource amount being greater than the available resource amount, determining that the available resources do not meet the current second job instance execution requirements. By comparing the applied resource amount with the available resource amount, and the applied cluster node load with the available cluster node load, resource utilization can be optimized, which helps to achieve load balancing and avoid node overload or idleness. In addition, it is beneficial for the container job instance to be executed under the conditions of sufficient resources and load balancing, improve the reliability of scheduling, reduce job failures or delays caused by insufficient resources or node overload, and thus improve scheduling efficiency.

[0142] As an optional implementation manner, based on the above embodiment, the available resource amount is detected, including:

[0143] Obtaining the requested resource amount from the relevant information of the second job instance;

[0144] In response to the requested amount of resources being less than or equal to the available amount of resources, determining that the available resources meet the current second job instance execution requirement;

[0145] In response to the requested resource amount being greater than the available resource amount, it is determined that the available resources do not meet the current second job instance execution requirement.

[0146] For example, if the type of the current second job instance is Spark type, the amount of resources requested by the Spark job may be the amount of load that the Yarn queue needs to allocate, which is not limited in this embodiment.

[0147] Specifically, first, the amount of resources required for the current second job instance can be obtained from the relevant information of the current second job instance; then, the available amount of resources can be obtained from the Redis database. After obtaining the above information, the amount of resources applied for and the amount of available resources can be compared respectively. In this embodiment, if the amount of resources applied for is less than or equal to the amount of available resources, it means that the current available resources are sufficient to support the execution of the current second job instance; correspondingly, if the amount of resources applied for is greater than the amount of available resources, it means that the current available resources are insufficient to support the execution of the current second job instance. According to the above information comparison method, it can be determined whether the currently available resources can meet the execution requirements of the current second job instance.

[0148] The method for scheduling a job instance provided in this embodiment detects the available resource amount, including: obtaining the requested resource amount in the relevant information of the second job instance; in response to the requested resource amount being less than or equal to the available resource amount, determining that the available resources meet the current second job instance execution requirements; in response to the requested resource amount being greater than the available resource amount, determining that the available resources do not meet the current second job instance execution requirements. By comparing the requested resource amount with the available resource amount, it is possible to ensure that the job instance is scheduled when the resources are sufficient, to avoid the situation where the job instance fails to be scheduled due to insufficient resources to a certain extent, and to improve the efficiency of job scheduling.

[0149] As an optional implementation manner, based on the above embodiment, the method further includes:

[0150] Periodically collect at least one of the following available resource data from the underlying computing platform:

[0151] The amount of resources available in the physical resource group, the metrics of the available Mpp clusters, and the load of the available cluster nodes;

[0152] The indicators of the available Mpp cluster include: the number of available user connections and the available queue length of the Mpp cluster.

[0153] The underlying computing platform refers to various types of computing resource platforms, which may be Hadoop, Kubernetes, Mpp database cluster, etc. This embodiment does not limit this.

[0154] The number of available user connections refers to the number of unoccupied connections in the current Mpp cluster that can be allocated to new user connections. Specifically, the number of available user connections is calculated by subtracting the number of established user connections in the current cluster from the total number of user connections configured in the cluster system.

[0155] The available queue length refers to the queue length that is not occupied in the current Mpp cluster and can be allocated to new job instances. Specifically, the available queue length is calculated by subtracting the number of job instances currently in the queue from the total queue length configured by the cluster system.

[0156] Specifically, a scheduled task can be set to periodically collect available resource data in the underlying computing platform, and the collected available resource data can be saved in a database such as Redis. It can be understood that the available resources of the physical resource group, for Hadoop and Kubernetes computing platforms, can refer to the remaining load information of the Yarn queue and the unallocated margin information under the Namespace; the load of the available cluster nodes can refer to the available node load of container-type jobs in the Kubernetes platform.

[0157] The method for scheduling job instances provided in this embodiment also includes: periodically collecting at least one of the following available resource data from the underlying computing platform: the amount of resources available in the physical resource group, the indicators of the available Mpp cluster, and the load of the available cluster nodes; wherein the indicators of the available Mpp cluster include: the number of available user connections and the length of the available queue of the Mpp cluster. Periodically collecting available resource data helps to obtain more accurate available resource data information; in addition, the available resource data information collected contains many types, and can provide more comprehensive available resource data information for different types of job instances, which is conducive to achieving unified scheduling and improving scheduling accuracy and scheduling efficiency.

[0158] As an optional implementation manner, based on the above embodiment, after scheduling the second job instance in the dispatched queue to the corresponding physical resource group, the method further includes:

[0159] In response to a scheduling failure, a scheduling failure prompt message is issued;

[0160] The second job instance that failed in scheduling is rewritten into the ready job queue under the corresponding tenant according to the scheduling failure prompt information.

[0161] Among them, scheduling failure refers to the failure of job submission due to errors in resource detection. It is understandable that in the process of detection based on the target available resource detection strategy that matches the second job instance, there may be detection errors. In this embodiment, when obtaining available resource information from Redis, due to the cache delay of the Redis database, although the detection results show that the available resources meet the execution requirements of the current second job instance, in the subsequent job instance submission process, the actual available resources can no longer meet the execution requirements of the current second job instance, which leads to the scheduling failure of the current second job instance.

[0162] Specifically, in this embodiment, the job execution service component can be used to subscribe to the message of the dispatched queue, and the second job instance that needs to be scheduled can be scheduled to the corresponding physical resource group. If the current second job instance fails to be scheduled, the job execution service component can send a specific return code message to the job management service component to prompt the job scheduling failure. In addition, the second job instance that failed to be scheduled can also be rewritten into the tenant ready job queue corresponding to the job instance, waiting for the next scheduling.

[0163] The job instance scheduling method provided in this embodiment, after scheduling the second job instance in the dispatched queue to the corresponding physical resource group, further includes: in response to a scheduling failure, issuing a scheduling failure prompt message; and rewriting the second job instance that failed to be scheduled into the ready job queue under the corresponding tenant according to the scheduling failure prompt message. By issuing a prompt message when scheduling fails and rewriting the job instance into the ready job queue, it can be ensured that the job will not be lost due to a scheduling failure, and the long waiting time caused by insufficient resources or other reasons can be reduced, thereby improving the scheduling efficiency of the job instance.

[0164] Figure 3 A flowchart of a method for scheduling a job instance provided in another embodiment of the present application is shown in FIG. Figure 3 As shown, the job instance scheduling method provided in this embodiment includes the following steps:

[0165] Step 301 , configure multiple physical resource groups, each of which uniformly represents and manages computing resources on a corresponding computing platform.

[0166] Each physical resource group is linked to a physical channel, and the physical resource group is associated with a specific type of computing platform through the channel. For different computing platforms, the physical channel represents different meanings. For example, in the Hadoop computing platform, the physical channel represents the Yarn queue, and in the Kubernetes computing platform, the physical channel represents the Namespace.

[0167] Step 302: regularly collect available resource data of the physical resource group from the underlying computing platform.

[0168] The collected available resource data includes at least one of the available resource amount of the physical resource group, the index of the available Mpp cluster, and the load of the available cluster node.

[0169] Step 303: Obtain at least one first job instance to be scheduled and related information of the first job instance.

[0170] Step 304: In response to determining that a second job instance meets the ready state according to the relevant information, the second job instance is written into a ready job queue under the corresponding tenant.

[0171] Step 305: for the current second job instance in the ready job queue of each tenant, determine the type of the current second job instance, and obtain a target available resource detection strategy that matches the type of the current second job instance.

[0172] Step 306: Detect available resources corresponding to the current second job instance based on the target available resource detection strategy.

[0173] Step 307: In response to the available resources satisfying the execution requirement of the current second job instance, submitting the current second job instance to the dispatched queue.

[0174] Step 308: Schedule the second job instance in the dispatched queue to the corresponding physical resource group.

[0175] Step 309: For the job instance that fails to be scheduled, a specific scheduling failure prompt message is issued, and the instance is written into the ready job queue under the corresponding tenant, waiting for the next scheduling.

[0176] In this embodiment, the implementation method and technical effects of steps 301 to 309 are similar to the implementation methods of the corresponding solutions in the above embodiments, and will not be repeated here.

[0177] Figure 4 The schematic diagram of the structure of the job instance scheduling device provided in one embodiment of the present application is that the job instance scheduling device provided in this embodiment is located in the job instance scheduling device. The job instance scheduling device can be an electronic device. The job instance scheduling device 40 provided in this embodiment includes: an acquisition module 41, a writing module 42, a determination module 43, a detection module 44, a submission module 45 and a scheduling module 46.

[0178] Among them, the acquisition module 41 is used to obtain at least one first job instance to be scheduled and related information of the first job instance; the writing module 42 is used to write the second job instance into the ready job queue under the corresponding tenant in response to determining that the second job instance meets the ready state according to the relevant information; the determination module 43 is used to determine the type of the current second job instance in the ready job queue under each tenant; the detection module 44 is used to detect the available resources corresponding to the current second job instance based on the target available resource detection strategy; the submission module 45 is used to submit the current second job instance to the dispatched queue in response to the available resources meeting the execution requirements of the current second job instance; the scheduling module 46 is used to schedule the second job instance in the dispatched queue to the corresponding physical resources; the acquisition module 41 is also used to obtain the target available resource detection strategy that matches the type of the current second job instance.

[0179] Accordingly, the relevant information of the first job instance includes the pre-dependency condition.

[0180] Optionally, when determining the second job instance that meets the ready state based on relevant information, the determination module 43 is specifically used to: determine whether the current condition meets the corresponding pre-dependency condition based on the pre-dependency condition of each first job instance; in response to the current condition meeting the pre-dependency condition, determine the first job instance whose current condition meets the pre-dependency condition as the second job instance that meets the ready state.

[0181] Accordingly, the relevant information of the first job instance includes the tenant identifier.

[0182] Optionally, when writing the second job instance into the ready job queue under the corresponding tenant, the writing module 42 is specifically used to: write the second job instance into the ready job queue of the tenant corresponding to the tenant identifier according to the tenant identifier of the first job instance; the writing module 42 is also used to write the relevant information of the second job instance into the ordered set of the tenant corresponding to the tenant identifier. The ordered set is located in a preset database.

[0183] Optionally, when detecting the available resources corresponding to the current second job instance based on the target available resource detection strategy, the detection module 44 is specifically used to: if the job instance type is the Mpp type, then the indicators of the available Mpp cluster and the available resource volume are detected in sequence; if the job instance type is the container type, then the available resource volume and the load of the available cluster nodes are detected; if the job instance type is the Spark type, then the available resource volume is detected.

[0184] Optionally, the detection module 44, when successively detecting the indicators of the available Mpp cluster and the available resource amount, is specifically used to: obtain the indicators of the applied Mpp cluster and the applied resource amount in the relevant information of the second job instance; in response to the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster, and the applied resource amount being less than or equal to the available resource amount, determine that the available resources meet the current execution requirements of the second job instance; in response to the indicator of the applied Mpp cluster being greater than the indicator of the available Mpp cluster, or the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster and the applied resource amount being greater than the available resource amount, determine that the available resources do not meet the current execution requirements of the second job instance.

[0185] Optionally, when detecting the available resources and the load of the available cluster nodes, the detection module 44 is specifically used to: obtain the applied cluster node load and the applied resource amount in the relevant information of the second job instance; in response to the applied cluster node load being less than or equal to the available cluster node load, and the applied resource amount being less than or equal to the available resource amount, determine that the available resources meet the current second job instance execution requirements; in response to the applied cluster node load being greater than the available cluster node load, and / or the applied resource amount being greater than the available resource amount, determine that the available resources do not meet the current second job instance execution requirements.

[0186] Optionally, the detection module 44, when detecting the available amount of resources, is specifically used to obtain the requested amount of resources in the relevant information of the second job instance; in response to the requested amount of resources being less than or equal to the available amount of resources, determining that the available resources meet the current execution requirements of the second job instance; in response to the requested amount of resources being greater than the available amount of resources, determining that the available resources do not meet the current execution requirements of the second job instance.

[0187] Optionally, the job instance scheduling device provided in this embodiment also includes a collection module.

[0188] The collection module is used to periodically collect at least one of the following available resource data from the underlying computing platform: the amount of resources available in the physical resource group, the indicators of the available Mpp cluster, and the load of the available cluster nodes.

[0189] The available Mpp cluster indicators include: the number of available user connections and the available queue length of the Mpp cluster.

[0190] Optionally, the job instance scheduling device provided in this embodiment further includes an issuing module.

[0191] The issuing module is used for issuing a scheduling failure prompt message in response to a scheduling failure after scheduling the second job instance in the dispatched queue to the corresponding physical resource group.

[0192] Optionally, the writing module 42 is further configured to rewrite the second job instance that failed in scheduling into the ready job queue of the corresponding tenant according to the scheduling failure prompt information.

[0193] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: a processor 51 and a memory 52.

[0194] The memory 52 is used to store computer-executable instructions; the processor 51 is used to execute the computer-executable instructions stored in the memory 52 to implement the job instance scheduling method provided by any of the above embodiments. For details, please refer to the relevant description in the above method embodiments, and no further details will be given here.

[0195] The program may include program codes, and the program codes include computer-executable instructions. The memory 52 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0196] In this embodiment, the memory 52 is connected to the processor 51 via a bus. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0197] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the job instance scheduling method provided in any of the above embodiments. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0198] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the job instance scheduling method provided by any one of the above embodiments.

[0199] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0200] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0201] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0202] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0203] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0204] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0205] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification. Those skilled in the art will easily think of other implementation schemes of the present application after considering the specification and practicing the invention disclosed here. This application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the technical field that are not disclosed in this application. The description and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are indicated by the following claims.

[0206] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for scheduling a job instance, characterized in that: The method comprises: Obtain at least one first job instance to be scheduled and related information of the first job instance; In response to determining a second job instance that meets the ready state according to each of the relevant information, writing the second job instance into a ready job queue under the corresponding tenant; For the current second job instance in the ready job queue under each tenant, determine the type of the current second job instance, and obtain a target available resource detection strategy that matches the type of the current second job instance; Detecting available resources corresponding to the current second job instance based on the target available resource detection strategy; In response to the available resources satisfying the execution requirement of the current second job instance, submitting the current second job instance to the dispatched queue; The second job instance in the dispatched queue is scheduled to the corresponding physical resource group.

2. The method according to claim 1, characterized in that The relevant information of the first job instance includes a pre-dependency condition; Determining a second job instance that meets the ready state according to the relevant information includes: Determining whether the current condition satisfies the corresponding pre-dependency condition according to the pre-dependency condition of each first job instance; In response to the current condition satisfying the pre-dependent condition, the first job instance whose current condition satisfies the pre-dependent condition is determined as the second job instance that satisfies the ready state.

3. The method according to claim 1, characterized in that The relevant information of the first job instance includes a tenant identifier; Writing the second job instance into the ready job queue of the corresponding tenant includes: According to the tenant identifier of the first job instance, writing the second job instance into the ready job queue of the tenant corresponding to the tenant identifier; The method further comprises: The relevant information of the second job instance is written into an ordered set of tenants corresponding to the tenant identifier, where the ordered set is located in a preset database.

4. The method according to claim 1, characterized in that: The detecting the available resources corresponding to the current second job instance based on the target available resource detection strategy includes: If the job instance type is Mpp type, the available Mpp cluster indicators and available resources are tested in sequence; If the job instance type is a container type, the available resources and the load of the available cluster nodes are detected; If the job instance type is Spark, the available resources are checked.

5. The method according to claim 4, characterized in that The indicators of the available Mpp cluster and the available resources are detected in sequence, including: Obtaining the indicators of the applied Mpp cluster and the applied resource amount in the relevant information of the second job instance; In response to the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster, and the amount of resources applied being less than or equal to the amount of available resources, determining that the available resources meet the current second job instance execution requirement; In response to the indicator of the applied Mpp cluster being greater than the indicator of the available Mpp cluster, or the indicator of the applied Mpp cluster being less than or equal to the indicator of the available Mpp cluster and the applied resource amount being greater than the available resource amount, it is determined that the available resources do not meet the current second job instance execution requirements.

6. The method according to claim 4, characterized in that The detecting of the available resource amount and the load of the available cluster nodes includes: Obtaining the applied cluster node load and the applied resource amount from the relevant information of the second job instance; In response to the load of the applied cluster node being less than or equal to the load of the available cluster node, and the amount of the applied resources being less than or equal to the amount of the available resources, determining that the available resources meet the current second job instance execution requirement; In response to the load of the requested cluster node being greater than the load of the available cluster node and / or the requested amount of resources being greater than the available amount of resources, it is determined that the available resources do not meet the current second job instance execution requirement.

7. The method according to claim 4, characterized in that The detecting of the available resource amount includes: Obtaining the requested resource amount from the relevant information of the second job instance; In response to the requested amount of resources being less than or equal to the available amount of resources, determining that the available resources meet the current second job instance execution requirement; In response to the requested resource amount being greater than the available resource amount, it is determined that the available resources do not meet the current second job instance execution requirement.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Periodically collect at least one of the following available resource data from the underlying computing platform: The amount of resources available in the physical resource group, the metrics of the available Mpp clusters, and the load of the available cluster nodes; The available Mpp cluster indicators include: the number of available user connections and the available queue length of the Mpp cluster.

9. The method according to any one of claims 1 to 7, characterized in that: After scheduling the second job instance in the dispatched queue to the corresponding physical resource group, the method further includes: In response to a scheduling failure, a scheduling failure prompt message is issued; The second job instance that fails in scheduling is rewritten into the ready job queue of the corresponding tenant according to the scheduling failure prompt information.

10. A job instance scheduling device, characterized in that: The device comprises: An acquisition module, used to acquire at least one first job instance to be scheduled and related information of the first job instance; a writing module, configured to, in response to determining a second job instance that satisfies a ready state according to each of the related information, write the second job instance into a ready job queue under a corresponding tenant; A determination module, configured to determine the type of the current second job instance in the ready job queue of each tenant; A detection module, configured to detect available resources corresponding to the current second job instance based on a target available resource detection strategy; A submission module, configured to submit the current second job instance to a dispatched queue in response to available resources satisfying the execution requirement of the current second job instance; A scheduling module, used for scheduling the second job instance in the dispatched queue to the corresponding physical resource group; The acquisition module is further used to acquire a target available resource detection strategy that matches the type of the current second job instance.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images