A resource scheduling method and device, computer equipment and storage medium

Abstract

The present disclosure provides a resource scheduling method and device, computer equipment and storage medium, wherein the method comprises: receiving a job execution request, sending an application management (AM) module configuration request to a resource scheduling module in K8s based on a remote calling protocol between the K8s system; starting the AM module in response to first configuration success feedback information returned by the resource scheduling module; forwarding a container resource configuration request to the resource scheduling module in response to the container resource configuration request sent by the AM module; forwarding second configuration success feedback information to the AM module in response to the second configuration success feedback information returned by the resource scheduling module; and forwarding a container resource start request to each agent module in response to the container resource start request sent by the AM module. The present disclosure can schedule container resources in the K8s system to target jobs in the YARN system, thereby enabling more computing resources in the K8s to be used to execute big data jobs in the YARN.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to a resource scheduling method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of Internet technology, more and more work needs to use big data for processing and analysis. Cloud native resource management systems, such as container cluster management systems (Kubernetes, K8s), which can provide application deployment, maintenance and expansion functions, have received widespread attention.

[0003] Usually many big data jobs run in a distributed resource management system (Yet Another Resource Negotiator, YARN), which cannot take advantage of more computing resources on K8s. How to use more computing resources on K8s to execute big data jobs running in YARN is a problem to be solved. SUMMARY

[0004] The embodiments of the present disclosure at least provide a resource scheduling method and device, computer equipment and storage medium.

[0005] In a first aspect, the embodiments of the present disclosure provide a resource scheduling method, comprising:

[0006] receiving a job execution request; the job execution request is used to request execution of a target job;

[0007] based on a remote call protocol between a cloud native resource management K8s system, sending an application management AM module configuration request to a resource scheduling module in the K8s system; the AM module configuration request is used to configure an AM module for the target job;

[0008] in response to the first configuration success reply information returned by the resource scheduling module, starting the AM module of the target job;

[0009] in response to the container resource configuration request sent by the AM module, forwarding the container resource configuration request to the resource scheduling module;

[0010] in response to the second configuration success reply information returned by the resource scheduling module for the container resource configuration request, forwarding the second configuration success reply information to the AM module;

[0011] in response to the container resource start request sent by the AM module, forwarding the container resource start request to each proxy module in the K8s system; the proxy module is used to start the container resource corresponding to the target job.

[0012] In a possible implementation, before sending an application management (AM) module configuration request to a resource scheduling module in the K8s system, the method further includes:

[0013] Based on the job execution request, a queue containing multiple to-be-executed jobs is obtained;

[0014] Based on the used container resource quantity of each queue and the total container resource quantity required by each queue, a target queue whose first resource usage ratio meets a first preset condition is filtered out from each queue;

[0015] For each target queue, based on the used container resource quantity of each to-be-executed job contained in the target queue and the total container resource quantity required by each to-be-executed job, a to-be-executed target job whose second resource usage ratio meets a second preset condition is filtered out from each to-be-executed job.

[0016] In a possible implementation, the forwarding, in response to the container resource configuration request sent by the AM module, of the container resource configuration request to the resource scheduling module includes:

[0017] Converting, in response to the container resource configuration request sent by the AM module, the container resource configuration request in the first data structure form into a container resource configuration request in a second data structure form;

[0018] Forwarding the container resource configuration request in the second data structure form to the resource scheduling module, so that the resource scheduling module identifies the container resource configuration request in the second data structure form.

[0019] In a possible implementation, the target job includes multiple target jobs;

[0020] The forwarding, in response to the second configuration success receipt information returned by the resource scheduling module for the container resource configuration request, of the second configuration success receipt information to the AM module includes:

[0021] Converting, in response to the second configuration success receipt information returned by the resource scheduling module for the container resource configuration request, the second configuration success receipt information in the second data structure form into second configuration success receipt information in the first data structure form;

[0022] Forwarding, by multiple threads corresponding to the target jobs respectively, the second configuration success receipt information in the first data structure form corresponding to each target job to the AM module.

[0023] In a possible implementation, the method further includes, in response to the second configuration success feedback information returned by the resource scheduling module for the container resource configuration request:

[0024] In response to the second configuration success feedback information returned by the resource scheduling module for the container resource configuration request, the second configuration success feedback information is forwarded to the AM module according to a preset communication period.

[0025] In a possible implementation, the method further includes, before forwarding the container resource configuration request to the resource scheduling module in response to the container resource configuration request sent by the AM module:

[0026] The container resource start request sent by the AM module for the container resource through a target interface is received based on a pre-constructed request service agreement with the AM module; the target interface is an interface in the AM module used to send a container resource start request.

[0027] In a possible implementation, the method further includes, after the container resource start request sent by the AM module:

[0028] The state of the container resource corresponding to the agent module is changed based on the start meta information, so that the container resource corresponding to each agent module is started when the agent module detects that the state of the corresponding container resource is changed; the start meta information is used to start the container resource.

[0029] In a possible implementation, the method further includes:

[0030] Target information obtained in the resource scheduling process is stored in a target storage module in the K8s system, so that each target information in the target process is recovered from the cluster data persistence module when a target process is restarted; the target information includes at least one of the following information: cluster metadata information, attribute information of a queue in which the target job is located, attribute information of an application corresponding to the target job, attribute information of the container resource, and attribute information of a node in which the agent module is located.

[0031] In a second aspect, the embodiments of the present disclosure further provide a resource scheduling apparatus, including:

[0032] A first receiving module is configured to receive a job execution request; the job execution request is used to request execution of a target job.

[0033] The first sending module is configured to send an application management (AM) module configuration request to a resource scheduling module in the K8s system based on a remote calling protocol between the cloud-native resource management system; the AM module configuration request is used to configure the AM module for the target job;

[0034] The starting module is configured to start the AM module of the target job in response to first configuration success acknowledgement information returned by the resource scheduling module;

[0035] The second sending module is configured to forward a container resource configuration request sent by the AM module to the resource scheduling module;

[0036] The third sending module is configured to forward second configuration success acknowledgement information returned by the resource scheduling module for the container resource configuration request to the AM module;

[0037] The fourth sending module is configured to forward a container resource starting request sent by the AM module to each agent module in the K8s system; the agent module is configured to start the container resource corresponding to the target job.

[0038] In a third aspect, the embodiments of the present disclosure further provide a computer device, including a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the first aspect or any possible implementation manner of the first aspect, or perform the steps of the second aspect or any possible implementation manner of the second aspect.

[0039] In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the first aspect or any possible implementation manner of the first aspect, or perform the steps of the second aspect or any possible implementation manner of the second aspect.

[0040] The resource scheduling method provided by the embodiment of the present disclosure, the resource management module in the distributed resource management YARN system can configure the AM module through the resource scheduling module in the cloud native resource management K8s system, and can forward the container resource configuration request of the AM module to the resource scheduling module, so that the resource scheduling module can schedule the container resource in the K8s to the target job in the YARN, so that the container resource in the K8s executes the target job in the YARN, so that the big data job in the YARN can be executed by using more computing resources in the K8s; and the resource management module can forward the container resource start request sent by the AM module to each agent module in the K8s, so that each agent module starts the corresponding container resource, and the container resource corresponding to each agent module can provide a safe and isolated running environment for the target job, so that the big data job in the YARN can be executed by using the single machine isolation feature of the container resource in the K8s.

[0041] In order to make the above objectives, characteristics and advantages of the present disclosure more apparent, clear and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings required to be used in the embodiments. The drawings herein are incorporated into the description and form a part of the description, which show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0043] Figure 1 A flowchart of a resource scheduling method provided by an embodiment of the present disclosure is shown;

[0044] Figure 2 A flowchart of another resource scheduling method provided by an embodiment of the present disclosure is shown;

[0045] Figure 3 A flowchart of another resource scheduling method provided by an embodiment of the present disclosure is shown;

[0046] Figure 4 An architectural schematic diagram of a resource scheduling device provided by an embodiment of the present disclosure is shown;

[0047] Figure 5 A schematic diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0048] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0049] Generally, many big data jobs run in a distributed resource management YARN system, and in general cases, these big data jobs can only use the idle resources on K8s to execute, and thus cannot use more computing resources on K8s. In some solutions in which the computing engine in YARN is modified to migrate the big data jobs to K8s to use more computing resources on K8s, high modification costs are required, and more computing resources on K8s cannot be used. Therefore, how to use more computing resources on K8s to execute the big data jobs running in YARN is a problem to be solved.

[0050] Based on this, the present disclosure provides a resource scheduling method. A resource management module in a distributed resource management YARN system can configure an AM module through a resource scheduling module in a cloud-native resource management K8s system, and can forward a container resource configuration request of the AM module to the resource scheduling module, so that the resource scheduling module can schedule a container resource in K8s to a target job in YARN, so that the container resource in K8s executes the target job in YARN, thereby enabling more computing resources in K8s to be used to execute big data jobs in YARN. In addition, the resource management module can forward a container resource start request sent by the AM module to each agent module in K8s, so that each agent module starts a corresponding container resource. The container resource corresponding to each agent module can provide a secure and isolated running environment for the target job, so that the single machine isolation feature of the container resource in K8s can be used to execute the big data jobs in YARN.

[0051] The defects of the above solutions and the proposed solutions are the result of the inventors' practice and careful research, so the discovery process of the above problems and the solutions proposed by the present disclosure to solve the above problems should be the contribution of the inventors to the present disclosure in the process of the present disclosure.

[0052] It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0053] To facilitate the understanding of the present embodiment, first, a resource scheduling method disclosed by the present embodiment is introduced in detail, and the execution subject of the resource scheduling method provided by the present embodiment is generally a computer device with certain computing capability.

[0054] The resource scheduling method provided by the present embodiment can be applied to the scenario of big data job migration, and specifically can be applied to the scenario of migrating the big data job running on the distributed resource management system to the cloud native resource management system and executing the big data job by using the computing resource on the cloud native resource management system.

[0055] Figure 1 The flowchart of the resource scheduling method provided by the present embodiment is described below by taking the resource management (Resource Manager, RM) module in the distributed resource management (Hadoop Yet Another Resource Negotiator, Hadoop YARN, hereinafter referred to as YARN) system as an example. The method comprises S101-S106, wherein:

[0056] S101: receiving a job execution request; the job execution request is used to request to execute a target job.

[0057] S102: based on a remote calling protocol between the cloud native resource management K8s system, an application management (AM) module configuration request is sent to the resource scheduling module in the K8s system.

[0058] In the present embodiment, the cloud native resource management K8s system, i.e., the Kubernetes container cluster resource management system, is an online resource management system in the field of microservices.

[0059] The K8s system can include a cluster responsible for automatically managing multiple containerized programs, and is a container orchestration tool. The K8s system can realize binding and running the application program in a container resource, form a secure and isolated running environment, and can realize operating multiple container resources on one host, and has good scalability.

[0060] The resource scheduling module in the cloud-native resource management system, that is, the resource scheduling K8s Scheduler module in the K8s system, can control and manage the use of container resources by the application program, and realize scheduling of the container resources.

[0061] As shown in a flowchart of another resource scheduling method, Figure 2 As shown in a flowchart of another resource scheduling method,

[0062] The target job can be specifically created after the user or the user group is successfully authenticated. In specific implementation, the application program management (AM) module in the RM module can be used to create the job corresponding to the application program identification information. The created job can be added to the user-specified queue. In specific implementation, the queue permission control Admission Manager module in the RM module can be used to authenticate the identity information of the user or the user group, and to authenticate the user-specified queue.

[0063] In specific implementation, multiple users can simultaneously submit job execution applications to the YARN, and multiple queues containing jobs to be executed can be obtained. Each queue can contain multiple jobs to be executed. In order to realize the fair and fine management in the process of allocating container resources for the jobs to be executed, before step S102 is performed, the target job whose demand degree for the container resources meets a preset condition can be selected from the multiple jobs to be executed.

[0064] In an implementation, the target queue whose demand degree for the container resources meets a first preset condition can be selected from the multiple queues, and then the target job whose demand degree for the container resources meets a second preset condition can be selected from the target queue. Here, the demand degree for the container resources can be represented by a resource usage ratio. The resource usage ratio can be a ratio of the used container resources to the total container resources required. The smaller the resource usage ratio, the greater the demand degree for the container resources. For example, the first preset condition can be greater than a first set threshold, and the second preset condition can be greater than a second set threshold. The first set threshold and the second set threshold can be different in quantity.

[0065] An implementation of screening target jobs is provided as follows. Before sending an application management (AM) module configuration request to a resource scheduling K8s Scheduler module in a K8s system, the following can be included:

[0066] Step 1: Based on the job execution request, a queue containing multiple jobs to be executed is obtained.

[0067] Step 2: Based on the number of used container resources of each queue and the total number of required container resources of each queue, target queues whose first resource usage ratio meets a first preset condition are screened from each queue.

[0068] Step 3: For each target queue, based on the number of used container resources of each job to be executed in the target queue and the total number of required container resources of each job to be executed, target jobs to be executed whose second resource usage ratio meets a second preset condition are screened from each job to be executed.

[0069] Each container can contain a preset number of cores, for example, a container can contain 4 cores.

[0070] The total number of required container resources Quota1 of each queue can be represented by the total number of required containers of each queue and the number of cores contained in each container. The total number of required containers of each queue can be the sum of the number of containers applied by each job to be executed in each queue when the user submits a job application. In a specific implementation, the number of containers applied by each job to be executed when the user submits a job application can be fixed or not fixed, for example, a number range <min, max> can be applied, where min can represent the minimum value of the number of containers, and max can represent the maximum value of the number of containers.

[0071] The number of used container resources Usage1 of each queue can be represented by the number of used containers and the number of cores contained in each container.

[0072] The first resource usage ratio can be represented by the ratio of the number of used container resources to the total number of required container resources, i.e. Usage1 / Quota1.

[0073] According to the first resource usage ratio Usage1 / Quota1 of each queue, target queues whose first resource usage ratio meets a first preset condition can be screened. For example, target queues whose first resource usage ratio is greater than a first set threshold value can be screened.

[0074] The total container resource quantity Quota2 required by each to-be-executed job in each target queue can be represented by the total number of containers required by each to-be-executed job and the number of cores contained in each container.

[0075] The used container resource quantity Usage2 of each to-be-executed job can be represented by the number of used containers and the number of cores contained in each container.

[0076] The second resource usage ratio can be represented by the ratio of the used container resource quantity to the total required container resource quantity, i.e., Usage2 / Quota2.

[0077] According to the second resource usage ratio Usage2 / Quota2 of each queue, to-be-executed target jobs that meet the second preset condition in the second resource usage ratio can be screened out. For example, target jobs with a second resource usage ratio greater than a second set threshold can be screened out.

[0078] In specific implementations, the above process of screening target jobs can be performed each time a to-be-executed target job is created. In the process of executing jobs each time a to-be-executed target job is created, the used container resource of each target queue and each job in each target queue can be dynamically changed. The used container resource quantity corresponding to each target queue and each job in each target queue can be accumulated each time a container is applied for; and the used container resource quantity corresponding to each target queue and each job in each target queue can be reduced each time a container ends executing a job.

[0079] In specific implementations, the above process of screening target jobs can be performed by using the Quota Manager module in the RM module.

[0080] The AM module configuration request can be used to request the K8s Scheduler module to configure the AM module for the to-be-executed target job. The AM module configuration request can carry information such as a job created according to application identity information, user permission information, job type information, the type of container resource to be applied for, and the number of container resources to be applied for.

[0081] The data in YARN and the data in the K8s system are represented in different data structure forms, that is, the data in YARN is represented in a first data structure form, and the data in the K8s system is represented in a second data structure form, that is, a Plain Old Data (POD) data structure form. The application management module configuration request sent by the RM module to the K8s Scheduler module is represented in the first data structure form, and therefore the K8s system cannot recognize the application management module configuration request in the first data structure form. Therefore, in an embodiment, the Remote K8s Scheduler module in the RM module can be used to convert the AM module configuration request in the first data structure form into an AM module configuration request in the second data structure form. Then, the AM module configuration request in the second data structure form is sent to the K8s Scheduler module, so that the K8s Scheduler module can recognize the AM module configuration request in the second data structure form.

[0082] Here, for each target job to be executed, an AM module configuration request is sent to the K8s Scheduler module to request the K8s Scheduler module to configure an AM module for each target job to be executed. The target jobs to be executed can include different job types, such as a batch computing job type Spark and a streaming computing job type Flink. The AM modules configured for different job types can be different. The processing logic corresponding to different AM modules can be different.

[0083] The AM module can be used to interact with the RM module after being started to apply for container resources for executing a target job and manage all the container resources it has applied for.

[0084] S103: In response to receiving the first configuration success reply information returned by the resource scheduling module, starting the AM module.

[0085] The first configuration reply information is used to indicate that the K8s Scheduler module has configured an AM module matched with the target job. The first configuration success reply information can be configuration success reply information returned by the K8s Scheduler module to the RM module after receiving the AM module configuration request sent by the RM module and configuring an AM module matched with the target job.

[0086] Here, the first configuration success feedback information returned by the K8s Scheduler module is in the form of the second data structure. After receiving the first configuration success feedback information in the form of the second data structure, the RM module can convert the first configuration success feedback information in the form of the second data structure into first configuration success feedback information in the form of the first data structure by using the Remote K8s Scheduler module. After identifying the first configuration success feedback information in the form of the first data structure, the RM module can start the AM module.

[0087] In specific implementations, the AMs Manager module can be used to start and stop the AM module. In addition, after starting the AM module, the AMs Manager module can be used to maintain the communication between the AM module and the RM module at the same communication period, monitor the life state of the AM module, and the like.

[0088] S104: In response to the container resource configuration request sent by the AM module, the container resource configuration request is forwarded to the resource scheduling module.

[0089] The container resource configuration request carries the number of containers applied by the user, and the container resource configuration request can be used to request to configure container resources according to the number of containers. The container resources are used to execute target jobs after being started.

[0090] The container resource configuration request can be sent by the AM module to the RM module after the AM module is started. In specific implementations, the AM module and the RM module can communicate at the same communication period, so the container resource configuration request sent by the AM module can be received at a preset communication period.

[0091] In order to implement the execution of target jobs by using container resources in K8s, after receiving the container resource configuration request sent by the AM module, the RM module can forward the container resource configuration request to the K8s Scheduler module by using the Remote K8s Scheduler module. The Remote K8s Scheduler module can interact with the K8s Scheduler module in various ways.

[0092] Here, the container resource configuration request sent by the AM module to the RM module is in a first data structure form, and the K8s cannot recognize the container resource configuration request in the first data structure form. Therefore, in an embodiment, in response to the container resource configuration request sent by the AM module, the container resource configuration request in the first data structure form can be converted into a container resource configuration request in a second data structure form; and then the container resource configuration request in the second data structure form is forwarded to the resource scheduling module, so that the resource scheduling module recognizes the container resource configuration request in the second data structure form.

[0093] Here, the container resource configuration request can be converted into a container resource configuration request in a POD data structure form, so that the K8s Scheduler module can recognize the container resource configuration request in the POD data structure form, and configure the container resources matched with the target job according to the number of containers carried in the container resource configuration request.

[0094] S105: In response to the second configuration success feedback information returned by the resource scheduling module for the container resource configuration request, the second configuration success feedback information is forwarded to the AM module.

[0095] The second configuration success feedback information is used to indicate that the K8s Scheduler module has configured the container resources matched with the target job. The second configuration success feedback information can be the configuration success feedback information of the container resources matched with the target job returned by the K8s Scheduler module to the RM module after the K8s Scheduler module receives the container resource configuration request sent by the RM module and configures the container resources matched with the target job.

[0096] In a specific implementation, in response to receiving the second configuration success feedback information returned by the K8s Scheduler module for the container resource configuration request, the second configuration success feedback information can be forwarded to the AM module according to a preset communication period.

[0097] Here, the second configuration success feedback information returned by the K8s Scheduler module is in a second data structure form. Here, after the RM module receives the second configuration success feedback information in the second data structure form, the Remote K8s Scheduler module can be used to convert the second configuration success feedback information in the second data structure form into the second configuration success feedback information in a first data structure form. After the RM module recognizes the second configuration success feedback information in the first data structure form, the Remote K8s Scheduler module is used to forward the second configuration success feedback information to the AM module.

[0098] The Remote K8s Scheduler module can include multiple individual threads. In one embodiment, when the target job to be executed includes multiple target jobs, the second configuration success feedback information in the form of the first data structure corresponding to each target job can be respectively forwarded to the AM module by using the corresponding thread of the target job. By using multiple threads to forward the second configuration success feedback information in parallel, the resource scheduling efficiency can be improved.

[0099] S106: In response to the container resource start request sent by the AM module, the container resource start request is forwarded to each agent module in the K8s system respectively; the agent module is used to start the container resource corresponding to the target job.

[0100] Here, the container resource start request can be used to start the container resource configured for the target job. After receiving the container resource start request sent by the AM module, the Remote K8s Scheduler module can be used to convert the container resource start request in the form of the first data structure into a container resource start request in the form of the second data structure, and then the Remote Kubelet Service module in the RM module is used to forward the container resource start request in the form of the second data structure to each Kubelet module in the K8s.

[0101] Each single machine corresponding to the Kubelet module is allocated with a container resource configured for the target job, and the Kubelet module can be used to start the container resource allocated on each single machine.

[0102] In the embodiments of the present disclosure, a request service agreement is pre-built between the RM module and the AM module. Through the request service agreement, the AM module and the RM module can maintain the same communication period for communication, and the container resource start request of the AM module can be sent to the RM module. Therefore, in one embodiment, in the process of receiving the container resource start request sent by the AM module for the container resource, the container resource start request sent by the AM module for the container resource through the target interface can be received based on the pre-built request service agreement between the AM module.

[0103] The target interface can include a container resource start interface and a container resource stop interface. The container resource start interface can be an interface for sending a container resource start request, such as a Start Container interface; and the container resource stop interface can be an interface for sending a container resource stop request, such as a Stop Container interface.

[0104] In the embodiments of the present disclosure, in response to receiving the container resource startup request sent by the AM module for the container resource, the state of the container resource corresponding to the Kubelet module can also be changed based on the startup meta information, so that the Kubelet module is started to monitor the state change of the corresponding container resource.

[0105] Here, the container resource startup request supplemented with the startup meta information can be started by using the Remote K8s Scheduler module, and the state of the container resource corresponding to the Kubelet module can be changed according to the startup meta information. The startup meta information can include image Image, distributed file (Hadoop Distributed File System File, HDFS File) and the like.

[0106] In order to realize that in the resource scheduling process, when the target process is restarted, each target information in the target process can be recovered, in an implementation manner, the target information obtained in the resource scheduling process can be stored in a target storage module in the K8s system.

[0107] Here, the cluster data persistence ETCD State Store module in the RM module can be used to periodically store each target information in the resource scheduling process into the target storage module ETCD module through an application programming interface server (Application Program Interface Server, API Server).

[0108] The cluster meta data information can include initialization configuration information of the cluster runtime, global configuration information, tag configuration information and the like; the attribute information of the queue where the target job is located can include queue name, queue resource description information, queue access control list information and the like; the attribute information of the application corresponding to the target job can include name, user, queue where the application is located, submission text information, running information and the like; the attribute information of the node where the Kubelet module is located can include total container resource quantity information and state information of the node.

[0109] Referring to Figure 3 Fig. 4 is a flowchart of another resource scheduling method provided by the embodiments of the present disclosure, Figure 3 The resource scheduling method provided by the embodiments of the present disclosure will be described taking the K8s Scheduler module as an example. The method comprises S301-S303, wherein:

[0110] S301: In response to receiving a request for application management module configuration sent by a resource management module in a target distributed resource management system for a target job, configuring an application management module matched with the target job, and returning first configuration acknowledgement information to the resource management module.

[0111] In the embodiments of the present disclosure, the target distributed resource management system can refer to an offline resource management system in the field of big data, and can include a YARN resource management system. In the following, the resource scheduling method provided by the embodiments of the present disclosure will be introduced by taking the target distributed resource management system as YARN as an example.

[0112] The resource management module in the target distributed resource management system, that is, the RM module in YARN. The RM module can realize the interaction between the AM module and K8s.

[0113] Here, the K8s Scheduler module in K8s can be used to interact with the RM module and receive the application management module configuration request. The application management module configuration request carries the target job created according to the application identity information and the job type information of the target job. The application identity information is carried in the job application submitted by the user to YARN. The application management module configuration request can be used to configure the AM module for the target job according to the job type information of the target job.

[0114] After the AM module matched with the target job is configured, the first configuration acknowledgement information can be returned to the RM module. The first configuration acknowledgement information can be used to indicate that the application management module matched with the target job has been configured.

[0115] Here, the generated first configuration acknowledgement information is in the form of second data structure, for example, in the form of POD data structure. After the first configuration acknowledgement information in the form of second data structure is generated, the first configuration acknowledgement information in the form of second data structure can be converted into the first configuration acknowledgement information in the form of first data structure, and then the first configuration acknowledgement information in the form of first data structure is returned to the RM module, so that the RM module can recognize the first configuration acknowledgement information in the form of first data structure and start the configured AM module.

[0116] S302: In response to receiving a request for container resource configuration sent by the resource management module from the configured application management module, configuring a container resource matched with the target job, and returning second configuration acknowledgement information to the resource management module.

[0117] Here, the container resource configuration request is sent by the AM module to the RM module after the RM module starts the AM module, and is forwarded by the RM module to the K8s Scheduler module. The container resource configuration request can be sent by the AM module to the RM module according to a preset communication period.

[0118] The container resource configuration request carries the number of containers applied by the user, and the container resource configuration request can be used to request to configure container resources according to the number of containers. The container resources are used to start and execute target jobs.

[0119] After receiving the container resource configuration request, the number of containers applied by the user carried in the container resource configuration request can be used to configure container resources matched with the target job.

[0120] After the container resources matched with the target job are configured, a second configuration receipt information can be returned to the RM module. The second configuration receipt information is used to indicate that the container resources matched with the target job have been configured.

[0121] The generated second configuration receipt information can also be information in the form of a second data structure. Here, the second configuration receipt information in the form of the second data structure can be converted into the second configuration receipt information in the form of the first data structure, and then the second configuration receipt information in the form of the first data structure is returned to the RM module, so that the RM module can identify the second configuration receipt information in the form of the first data structure, and forward the second configuration receipt information to the AM module, and the AM module sends a container resource start request.

[0122] S303: In response to receiving the container resource start request sent by the resource management module for the container resource, starting the container resource corresponding to each agent module, so that the container resource executes the target job.

[0123] The container resource start request can be sent by the AM module to the RM module, and then forwarded by the RM module to each Kubelet module in K8s.

[0124] The container resource start request can be used to start the container resources configured for the target job. Each single machine corresponding to the Kubelet module is allocated with the container resources configured for the target job, and the Kubelet module receives the container resource start request, which can be used to start the container resources allocated on the corresponding single machine.

[0125] Those skilled in the art can understand that in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined by its function and possible inherent logic.

[0126] Based on the same inventive concept, the embodiments of the present disclosure also provide a resource scheduling device corresponding to the resource scheduling method. Since the principle of the device solving problems in the embodiments of the present disclosure is similar to the above-mentioned resource scheduling method of the present disclosure, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0127] Referring to Figure 4 As shown in the figure, an architecture schematic diagram of a resource scheduling device provided by the embodiments of the present disclosure, the device comprises:

[0128] The first receiving module 401 receives a job execution request; the job execution request is used to request to execute a target job;

[0129] The first sending module 402 is configured to send an application management AM module configuration request to a resource scheduling module in a cloud-native resource management K8s system based on a remote calling protocol between the K8s system; the AM module configuration request is used to configure the AM module for the target job;

[0130] The starting module 403 is configured to start the AM module of the target job in response to the first configuration success return information returned by the resource scheduling module;

[0131] The second sending module 404 is configured to forward the container resource configuration request to the resource scheduling module in response to the container resource configuration request sent by the AM module;

[0132] The third sending module 405 is configured to forward the second configuration success return information to the AM module in response to the second configuration success return information returned by the resource scheduling module for the container resource configuration request;

[0133] The fourth sending module 406 is configured to forward the container resource starting request to each proxy module in the K8s system in response to the container resource starting request sent by the AM module; the proxy module is configured to start the container resource corresponding to the target job.

[0134] In an optional implementation, before the first sending module 402 sends the application management AM module configuration request to the resource scheduling K8s Scheduler module in the K8s system, the device further comprises:

[0135] The acquisition module is configured to acquire a queue containing a plurality of to-be-executed jobs based on the job execution request;

[0136] The first screening module is configured to screen, from each of the queues, a target queue whose first resource usage ratio meets a first preset condition, based on a used container resource quantity of each of the queues and a total container resource quantity required by each of the queues.

[0137] The second screening module is configured to screen, from each of the target jobs in each of the target queues, a target job whose second resource usage ratio meets a second preset condition, based on a used container resource quantity of each of the target jobs in the target queue and a total container resource quantity required by each of the target jobs.

[0138] In an optional implementation, the second sending module 404 is specifically configured to:

[0139] In response to the container resource configuration request in the first data structure, the second sending module 404 is configured to convert the container resource configuration request in the first data structure into a container resource configuration request in a second data structure.

[0140] The second sending module 404 is configured to forward the container resource configuration request in the second data structure to the resource scheduling module, so that the resource scheduling module identifies the container resource configuration request in the second data structure.

[0141] In an optional implementation, the target jobs include a plurality of target jobs, and the third sending module 405 is specifically configured to:

[0142] In response to the second configuration success feedback information returned by the resource scheduling module for the container resource configuration request, the third sending module 405 is configured to convert the second configuration success feedback information in the second data structure into second configuration success feedback information in the first data structure.

[0143] The third sending module 405 is configured to forward the second configuration success feedback information in the first data structure to the AM module by using a thread corresponding to each of the target jobs.

[0144] In an optional implementation, the third sending module 405 is specifically configured to:

[0145] In response to the second configuration success feedback information returned by the resource scheduling module for the container resource configuration request, the third sending module 405 is configured to forward the second configuration success feedback information to the AM module according to a preset communication period.

[0146] In an optional implementation, the apparatus further includes a second receiving module configured to:

[0147] receive a container resource startup request sent by the AM module through a target interface for the container resource based on a pre-constructed request service agreement with the AM module; the target interface is an interface in the AM module used for sending a container resource startup request.

[0148] In an optional implementation, in response to receiving the container resource startup request sent by the application management module for the container resource, the apparatus further includes:

[0149] a changing module, configured to change a state of the container resource corresponding to the agent module based on startup meta-information, so that when the agent module monitors that the state of the corresponding container resource changes, the container resource corresponding to each agent module is started; the startup meta-information is used to start the container resource.

[0150] In an optional implementation, the apparatus further includes:

[0151] a storage module, configured to store target information acquired in a resource scheduling process in a target storage module in the K8s system, so as to recover each target information in the target process from the cluster data persistence module when the target process is restarted; wherein the target information includes at least one of the following information: cluster metadata information, attribute information of a queue in which the target job is located, attribute information of an application corresponding to the target job, attribute information of the container resource, and attribute information of a node in which the agent module is located.

[0152] The description of the processing procedure of each module in the apparatus and the interaction procedure between the modules can refer to the related description in the method embodiments, and will not be described in detail here.

[0153] Based on the same technical concept, the embodiments of the present disclosure further provide a computer device. Referring to FIG. 5, Figure 5 As shown in FIG. 5, a structure schematic diagram of the computer device 500 provided by the embodiments of the present disclosure includes a processor 501, a memory 502, and a bus 503. The memory 502 is used to store execution instructions, including an internal memory 5021 and an external memory 5022; the internal memory 5021 is also called an internal storage, used to temporarily store operation data in the processor 501 and exchange data with the external memory 5022 such as a hard disk, the processor 501 exchanges data with the external memory 5022 through the internal memory 5021, and when the computer device 500 is running, the processor 501 and the memory 502 communicate through the bus 503, so that the processor 501 executes the following instructions:

[0154] receive a job execution request; the job execution request is used to request to execute a target job;

[0155] based on a remote call protocol with a cloud-native resource management K8s system, sending an application management AM module configuration request to a resource scheduling module in the K8s system; the AM module configuration request is used to configure an AM module for the target job;

[0156] in response to first configuration success acknowledgement information returned by the resource scheduling module, starting the AM module of the target job;

[0157] in response to a container resource configuration request sent by the AM module, forwarding the container resource configuration request to the resource scheduling module;

[0158] in response to second configuration success acknowledgement information returned by the resource scheduling module for the container resource configuration request, forwarding the second configuration success acknowledgement information to the AM module;

[0159] in response to a container resource start request sent by the AM module, forwarding the container resource start request to each agent module in the K8s system; the agent module is used to start a container resource corresponding to the target job.

[0160] The embodiment of the present disclosure further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is run by a processor, the steps of the resource scheduling method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0161] The embodiment of the present disclosure further provides a computer program product, and the computer program product carries a program code. The instructions included in the program code can be used to execute the steps of the resource scheduling method described in the above method embodiment. For details, refer to the above method embodiment, which will not be described here.

[0162] The above computer program product can be specifically implemented by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0164] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0165] In addition, the functional units in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0166] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0167] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present disclosure, used to illustrate the technical solutions of the present disclosure, rather than limit the same, and the protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features within the technical scope disclosed by the present disclosure; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A resource scheduling method, characterized in that, Applications to YARN systems for distributed resource management include: Receive a job execution request; the job execution request is used to request the execution of a target job in the YARN distributed resource management system. Based on the remote call protocol between the YARN system and the cloud-native resource management Kubernetes (K8s) system, an application management AM module configuration request is sent to the resource scheduling module in the K8s system in a data structure format recognizable by the K8s system. The data structure format of the YARN system is different from that of the K8s system. The AM module configuration request is used to configure the AM module for the target job. In response to the first configuration success receipt returned by the resource scheduling module, the AM module of the target job is started; In response to the container resource configuration request sent by the AM module in the form of a data structure recognizable by the K8s system, the container resource configuration request is forwarded to the resource scheduling module; In response to the second configuration success receipt information returned by the resource scheduling module for the container resource configuration request, the second configuration success receipt information is forwarded to the AM module, wherein the second configuration success receipt information is used to indicate that the resource scheduling module has configured the container resources in the K8s system that match the target job; In response to the container resource startup request sent by the AM module in a data structure form recognizable by the K8s system, the container resource startup request is forwarded to each agent module in the K8s system; the agent module is used to start the container resource in the K8s system corresponding to the target job to execute the target job; the container resource provides a secure and isolated running environment for the target job to be executed after startup.

2. The method according to claim 1, characterized in that, Before sending the application management AM module configuration request to the resource scheduling module in the Kubernetes system, the method further includes: Based on the job execution request, obtain a queue containing multiple jobs to be executed; Based on the number of container resources used in each queue and the total number of container resources required by each queue, target queues whose first resource usage ratio meets the first preset condition are selected from each queue. For each target queue, based on the number of container resources used by each job to be executed contained in the target queue and the total number of container resources required by each job to be executed, target jobs to be executed that meet the second preset condition in terms of the second resource usage ratio are selected from each job to be executed.

3. The method according to claim 1, characterized in that, The step of forwarding the container resource configuration request to the resource scheduling module in response to the container resource configuration request sent by the AM module includes: In response to the container resource configuration request sent by the AM module, the container resource configuration request in the first data structure form is converted into a container resource configuration request in the second data structure form; The container resource configuration request in the second data structure form is forwarded to the resource scheduling module so that the resource scheduling module can recognize the container resource configuration request in the second data structure form.

4. The method according to claim 3, characterized in that, The target task includes multiple tasks; The step of forwarding the second configuration success receipt information to the AM module in response to the resource scheduling module's return of the container resource configuration request includes: In response to the second configuration success receipt information returned by the resource scheduling module for the container resource configuration request, the second configuration success receipt information in the second data structure form is converted into the second configuration success receipt information in the first data structure form. The threads corresponding to the multiple target jobs respectively forward the second configuration success receipt information in the form of the first data structure to the AM module.

5. The method according to claim 1, characterized in that, The step of forwarding the second configuration success receipt information to the AM module in response to the resource scheduling module's return of the container resource configuration request includes: In response to the second configuration success receipt information returned by the resource scheduling module for the container resource configuration request, the second configuration success receipt information is forwarded to the AM module according to a preset communication cycle.

6. The method according to claim 1, characterized in that, In response to the container resource configuration request sent by the AM module, before forwarding the container resource configuration request to the resource scheduling module, the method further includes: Based on a pre-built request service protocol with the AM module, the Kubelet module in the Kubernetes system receives a container resource startup request sent by the AM module through a target interface for the container resource; the target interface is the interface in the AM module used to send the container resource startup request.

7. The method according to claim 1, characterized in that, Following the container resource start request sent by the AM module, the method further includes: The state of the container resources corresponding to the proxy modules is changed based on the startup metadata, so that when the proxy modules detect that the state of the corresponding container resources has changed, they start the container resources corresponding to each proxy module; the startup metadata is used to start the container resources.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: The target information obtained during resource scheduling is stored in the target storage module of the K8s system so that when the target process is restarted, the target information in the target process can be restored from the cluster data persistence module. The cluster data persistence module is used to periodically persist the target information during resource scheduling to the target storage module. The target information includes at least one of the following: cluster metadata information, attribute information of the queue where the target job is located, attribute information of the application corresponding to the target job, attribute information of the container resource, and attribute information of the node where the agent module is located.

9. A resource scheduling device, characterized in that, Applications to YARN systems for distributed resource management include: The first receiving module is used to receive a job execution request; the job execution request is used to request the execution of a target job in the distributed resource management YARN system. The first sending module is used to send an application management AM module configuration request in a data structure form recognizable by the K8s system to the resource scheduling module in the K8s system based on the remote call protocol between the K8s system and the cloud-native resource management K8s system. The data structure form of the YARN system is different from that of the K8s system. The AM module configuration request is used to configure the AM module for the target job. The startup module is used to start the AM module of the target job in response to the first configuration success receipt information returned by the resource scheduling module; The second sending module is used to forward the container resource configuration request to the resource scheduling module in response to the container resource configuration request in the form of a data structure recognizable by the K8s system sent by the AM module. The third sending module is used to forward the second configuration success receipt information to the AM module in response to the second configuration success receipt information returned by the resource scheduling module for the container resource configuration request. The second configuration success receipt information is used to indicate that the resource scheduling module has configured the container resources in the K8s system that match the target job. The fourth sending module is used to respond to the container resource startup request in the form of a data structure recognizable by the K8s system sent by the AM module, and forward the container resource startup request to each proxy module in the K8s system respectively; the proxy module is used to start the container resource in the K8s system corresponding to the target job to execute the target job; the container resource provides a secure and isolated running environment for the target job to execute the target job after startup.

10. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the resource scheduling method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the resource scheduling method as described in any one of claims 1 to 8.

Images