Resource scheduling method, device, equipment and medium
By judging the job type and converting the resource configuration, the problem of insufficient resource isolation in the existing technology is solved, efficient resource allocation for container and non-container jobs is realized, and resource utilization is improved.
Patent Information
- Application Number
- CN202111501983.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-12-09
AI Technical Summary
The existing resource scheduling methods cannot achieve physical resource isolation, resulting in low resource utilization and the inability to effectively manage resource allocation for container type and non-container type jobs.
By obtaining the type of job to be executed, if it is the container run type, the configured resource group is converted into the physical resource amount of the container; if it is a non-container run type, the resource group is converted into each worker node according to the number of worker nodes and the operation strategy. to ensure that the job runs in a limited resource environment.
Reasonable resource allocation for container and non-container jobs is realized, resource utilization is improved, container jobs can be run in a container environment, and non-container jobs can be run on various nodes.
Smart Images

Figure CN114168292B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and in particular to a resource scheduling method, device, equipment and medium. Background Art
[0002] The main function of job scheduling is to review whether the system can meet the resource needs of user jobs based on the information in the job control block, and to select certain jobs from the backup queue in external memory and transfer them into the main memory according to a certain algorithm, and to create processes for them and allocate necessary resources.
[0003] However, existing resource scheduling can only provide concurrency control at the logical level, but cannot achieve resource isolation at the physical level, which can easily cause disordered resource competition and lead to low resource utilization. Summary of the invention
[0004] The present application provides a resource scheduling method, apparatus, device and medium to solve the problem of low resource utilization.
[0005] In a first aspect, the present application provides a resource scheduling method, the method being used in a job manager, comprising:
[0006] Obtain a job to be executed, and determine whether the type of the job is a container operation type;
[0007] If so, converting the configured resource group into the physical resource amount of the container so that the job runs in the container with limited resources;
[0008] If not, the configured resource group is converted into the physical resource amount of each working node according to the number of working nodes and the operation strategy, so that the job is run on the node with limited resources.
[0009] Optionally, the configured resource group includes a maximum CPU resource amount;
[0010] The step of converting the configured resource group into the physical resource amount of the container specifically includes:
[0011] The maximum CPU resource amount is converted into a container job application resource amount.
[0012] Optionally, converting the maximum CPU resource amount into the container job application resource amount specifically includes:
[0013] The maximum CPU resource amount is compressed according to a preset ratio and converted into the resource amount requested for the container job.
[0014] Optionally, the configured resource group includes a maximum memory resource amount, and the operation strategy includes: CPU priority or memory priority;
[0015] If the operation strategy includes CPU priority, the resource group to be configured is converted into the amount of physical resources of each working node according to the number of working nodes and the operation strategy, specifically including:
[0016] Calculate the number of tasks to be run by a single working node based on the maximum CPU resource, the number of working nodes, and the CPU resource of the master node;
[0017] The CPU resource amount of each task on each working node is calculated based on the number of tasks running on each working node, the number of working nodes, the CPU resource amount of a single task, and the CPU resource amount of the master node. The memory of the single task is fixed at 4G.
[0018] Optionally, if the operation strategy includes memory priority, converting the configured resource group into the amount of physical resources of each working node according to the number of working nodes and the operation strategy specifically includes:
[0019] The memory resource amount of each working node is calculated according to the memory resource amount of a single task, the number of working nodes and the memory resource amount of the master node, and the CPU of the single task is fixed to 1 core.
[0020] Optionally, determining whether the type of the job is a container operation type specifically includes:
[0021] Determine whether the type of the job is a command line type, where the command line type includes: python, perl, shell or java.
[0022] In a second aspect, the present application provides a resource scheduling method, which is used in a resource manager and includes:
[0023] Allocate resources to the container corresponding to the job to be executed according to the amount of physical resources of the container, so that the job runs in the container with limited resources; or allocate resources to the node corresponding to the job to be executed according to the amount of physical resources of each working node, so that the job runs on the node with limited resources;
[0024] The physical resource amount of the container is obtained by converting the resource group configured for the job when the job manager determines that the type of the job is a container running type;
[0025] The amount of physical resources of each working node is obtained by converting the resource group configured for the job according to the number of working nodes and the running strategy when the job manager determines that the type of the job is a non-container running type.
[0026] Optionally, the Kubernetes is used to allocate execution resources to the container corresponding to the job according to the physical resource amount of the container;
[0027] The yarn is used to allocate execution resources to the node corresponding to the job according to the physical resource amount of each working node.
[0028] In a third aspect, the present application provides a resource scheduling device, including:
[0029] A first acquisition module, used to acquire a job to be executed;
[0030] A determination module, used to determine whether the type of the job is a container operation type;
[0031] The second acquisition module is used to convert the configured resource group into the physical resource amount of the container when the type of the job is a container running type, so that the job can run in the container with limited resources; and if the configured resource group is converted into the physical resource amount of each working node according to the number of working nodes and the running strategy, so that the job can run on the node with limited resources.
[0032] In a fourth aspect, the present application provides an electronic device, including: a memory and a processor;
[0033] The memory is used to store instructions; the processor is used to call the instructions in the memory to execute the resource scheduling method in the first aspect and any possible design of the first aspect or the resource scheduling method in the second aspect and any possible design of the second aspect.
[0034] In a fifth aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When at least one processor of an electronic device executes the computer instructions, the electronic device executes the resource scheduling method in the first aspect and any possible design of the first aspect, or the resource scheduling method in the second aspect and any possible design of the second aspect.
[0035] In a sixth aspect, the present application provides a computer program product, which includes computer instructions. When at least one processor of an electronic device executes the computer instructions, the electronic device executes the resource scheduling method in the first aspect and any possible design of the first aspect, or the resource scheduling method in the second aspect and any possible design of the second aspect.
[0036] The resource scheduling method provided in the present application obtains the job to be executed, determines whether the type of the job is a container operation type, and if so, converts the configured resource group into the physical resource amount of the container so that the job runs in the container with limited resources. If not, converts the configured resource group into the physical resource amount of each working node according to the number of working nodes and the operation strategy so that the job runs on the node with limited resources. In this way, jobs of container operation type and jobs of non-container operation type are reasonably encapsulated, and different resource allocation methods are provided according to the job type, so as to reasonably set container resources and non-container resources and improve resource utilization. And it makes it possible for jobs of container operation type to run conveniently in a container environment, and jobs of non-container operation type to run conveniently on each node. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0038] Figure 1 A schematic diagram of a scenario of a resource scheduling method provided in an embodiment of the present application;
[0039] Figure 2 A flowchart of a resource scheduling method provided in one embodiment of the present application;
[0040] Figure 3 A flowchart of a resource scheduling method provided in one embodiment of the present application;
[0041] Figure 4 A schematic diagram of the structure of a resource scheduling device provided in one embodiment of the present application;
[0042] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0044] Traditional resource scheduling methods include scheduling systems based on Control-M, scheduling systems based on Dolphin Scheduler, or scheduling systems based on Airflow for resource scheduling.
[0045] The scheduling system based on Control-M realizes the separation of management, scheduling and job execution through a three-layer architecture. The three-layer architecture is Control-M / EM, Control-M / Server and Control-M / Agent. Control-M / EM is the management module, responsible for job uploading, operation monitoring, etc.; Control-M / Server is the core of the entire scheduling system, responsible for the instantiation of job flows, job scheduling, resource configuration, job operation management, etc.; Control-M / Agent is the job operation node, responsible for running jobs issued by Control-M / Server, saving job operation status, and synchronizing job operation status to Control-M / Server. Control-M provides resource management at the logical level and controls according to job concurrency. It does not have resource scheduling methods at the physical resource level and cannot be directly connected to Kubernetes and Hadoop platforms.
[0046] The scheduling system based on Dolphin Scheduler provides a special spark plug-in for spark engine type jobs, and submits tasks through spark-sumbimt at the bottom layer. In terms of specific resource management and control, the resource parameters of spark are completely open to users, including the number of driver cores, the number of driver memory, the number of executors, the number of memories used by a single executor, and the number of CPU cores. However, all resource parameters of spark need to be filled in manually by users, and no container type plug-in is provided, nor does it have a resource control strategy for containers.
[0047] The Airflow-based scheduling system provides SparkSubmit Operator and Kubernetes Pod Operator plug-ins for container-type and spark-type jobs respectively. For Spark Submit Operator, users are required to set resource parameters, and for Kubernetes Pod Operator, some parameters of Kubernetes YAML files are open to users for setting, but parameter control of CPU and memory resources is not provided.
[0048] Therefore, traditional resource scheduling methods cannot achieve resource isolation at the physical level, which can easily cause disorderly resource competition and lead to low resource utilization.
[0049] In response to the above problems, the present application proposes a resource scheduling method, which obtains the job to be executed, determines whether the type of the job is a container operation type, and if so, converts the configured resource group into the physical resource amount of the container so that the job runs in the container with limited resources. If not, convert the configured resource group into the physical resource amount of each working node according to the number of working nodes and the operation strategy, so that the job runs on the node with limited resources. In this way, jobs of container operation type and jobs of non-container operation type are reasonably encapsulated, and different resource allocation methods are provided according to the job type, so as to reasonably set container resources and non-container resources and improve resource utilization. And it makes it possible for jobs of container operation type to run conveniently in a container environment, and jobs of non-container operation type to run conveniently on each node.
[0050] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0051] Figure 1 A scenario diagram of a resource scheduling method provided by an embodiment of the present application is shown. The job manager 101 obtains the job to be executed uploaded by the client 102, and determines whether the type of the job is a container operation type. When the type of the job to be executed is a container operation type, the resource group configured by the user through the client 102 is obtained, and the configured resource group is converted into the physical resource amount of the container, and the physical resource amount of the container is sent to the resource manager 103. The resource manager 103 allocates resources to the container corresponding to the job according to the physical resource amount of the container, so that the job runs in the container with limited resources. When the type of the job to be executed is a non-container operation type, the job manager 101 obtains the resource group configured by the user through the client 102, converts the configured resource group into the physical resource amount of each working node according to the number of working nodes and the operation strategy, and sends the physical resource amount of each node to the resource manager 103. The resource manager 103 allocates resources to each node according to the node corresponding to the job and the physical resource amount of each node, so that the job runs on the node with limited resources.
[0052] Figure 2 A flow chart of a resource scheduling method provided by an embodiment of the present application is shown. Figure 1 Based on the embodiment shown, Figure 2 As shown, with the job manager as the execution subject, the method of this embodiment may include the following steps:
[0053] S101. Obtain a job to be executed, and determine whether the job type is a container operation type.
[0054] A job is a user-defined logical unit that completes a certain task. The job to be executed may include one or more jobs, and multiple jobs may include a group of jobs with similar functions, similar frequencies, or dependencies.
[0055] After obtaining the job to be executed, the job manager determines whether the job type is a container operation type. A container operation type job may be, for example, a command type job, which may include, for example, python, perl, shell, executable java or other executable commands. A non-container operation type job may be, for example, a job that depends on the spark type of the operation engine.
[0056] After determining whether the type of the job is a container operation type, if the type of the job is a container operation type, executing step S102, if the type of the job is a non-container operation type, executing step S103.
[0057] S102: Convert the configured resource group into the physical resource amount of the container, so that the job runs in the container with limited resources.
[0058] When the job type is the container running type, the job manager obtains the resource group configured by the user from the client and converts the configured resource group into the physical resource amount of the container. The configured resource group can include the maximum resource amount of the job. The user only needs to set the maximum resource amount of the job without paying attention to resource parameters, etc., which improves the user experience. The maximum resource amount can be represented by DCU, which is used to abstract the unit of container resources. The maximum resource amount can be, for example, 9DCU, and 1DCU represents 1core 4G. The maximum resource amount includes the maximum CPU resource amount and the maximum memory resource amount. The physical resource amount of the container can include the resource amount requested by the container job and the maximum memory resource, etc.
[0059] After obtaining the maximum CPU resource amount of the job set by the user, the job manager converts the maximum CPU resource class into the container job application resource amount, so that the job runs in a container with limited resources. The container job application resource amount can be understood as the running resources required to run the job in the container.
[0060] because
[0061] For jobs running in containers, each job runs in an independent pod. A pod consists of one or more containers, that is, a pod is a collection of containers. The setting of physical resources for a single job mainly involves setting the ratio of the limit and request of the CPU used, as well as the ratio of the limit and request of the memory. Request refers to the amount of resources required to meet the scheduling algorithm of the resource manager, that is, the amount of resources for the container job. The resource manager may include Kubernetes, then limit refers to the amount of resources required to meet the scheduling algorithm of Kubernetes. Request is the upper limit of resources used by a single job, that is, the maximum amount of resources. The applicant has found that when the resources of a single Kubernetes node cannot meet the requirements for creating a pod, pod expulsion may occur, and pods with inconsistent memory limit and request ratios are preferentially expelled. Therefore, only the maximum CPU resource amount in the maximum resource amount can be converted, and the maximum memory resource amount is not converted.
[0062] As an implementation method, the maximum CPU resource amount can be compressed according to a preset ratio to convert the maximum CPU resource amount into the resource amount requested by the container job. The preset ratio can be flexibly set as needed. For example, for a shell job, when the maximum resource amount set by the user is 9DCU, the preset ratio can be 0.5, and the non-container job resource amount obtained after converting the maximum CPU resource amount is 4.5DCU.
[0063] S103: Convert the configured resource group into the physical resource amount of each working node according to the number of working nodes and the operation strategy, so that the job runs on the node with limited resources.
[0064] When the job type is non-container operation type, the job manager obtains the resource group configured by the user from the client and converts it into the physical resource amount of each worker node according to the number of worker nodes and the operation strategy. The configured resource group can include the maximum resource amount, the number of worker nodes and the operation strategy. The maximum resource amount can be represented by CU, which is used to abstract the unit of non-container resources. The maximum resource amount can be 3CU, for example, 1CU represents 1core 4G. The maximum resource amount includes the maximum CPU resource amount and the maximum memory resource amount. The number of worker nodes refers to the number of nodes that can execute jobs in the cluster managed by the resource manager, and the number of worker nodes does not exceed the number of worker nodes in the cluster. The working mode includes CPU priority or memory priority. When CPU priority is set, the number of tasks (executors) that can be started by each worker node is (N-1) / number of worker nodes, where N represents the maximum CPU resource amount set by the user, and the number of tasks represents the number of tasks that can be run simultaneously on each node, but the memory value of each task is fixed to 4G. When memory priority is set, each worker node starts a task, and the memory used by the task is ((N-1)×4) / number of worker nodes. Generally, CPU priority can be used for CPU-intensive spark jobs, and memory priority can be used for memory-intensive spark jobs. In addition, since the master node occupies at least 1CU of resources, the maximum resource amount set by the user is at least 3CU.
[0065] The maximum amount of resources in the configured resource group is converted into the amount of physical resources for each working node according to the number of working nodes and the operation strategy in the configured resource group. The amount of physical resources includes the master node resources and the working node resources. For spark type jobs (spark on yarn), its physical resources are mainly composed of two parts: one is the master node resources, namely the spark driver resources, and the other is the working node resources, namely the spark worker resources. The working node resources can include CPU resources and memory resources.
[0066] For example, non-container running jobs are jobs whose running engine depends on Spark. Their physical resources include master node resources (spark drive) and worker node resources (spark worker). The maximum resource amount N = master node resources + worker node resources * number of worker nodes. The maximum resource amount includes the maximum CPU resource amount and the maximum memory resource amount. The maximum CPU resource amount = master node CPU resource amount + worker node CPU resource amount * number of worker nodes, and the maximum memory resource amount = master node memory resource amount + worker node memory resource amount * number of worker nodes. The master node resources can be fixed to 1CU (1core 4G), and the number of worker nodes does not exceed the number of worker nodes in the yarn cluster, which can be set by the user.
[0067] As an implementation method, if the working mode includes CPU priority, the maximum CPU resource is N CU, and the number of tasks that can be started by each working node can be (N-1) / number of working nodes. For example, N is 9, the number of working nodes is 2, and the number of tasks that can be started by each working node is 2. Then, according to the maximum CPU resource, the number of working nodes, and the CPU resource of the master node, the CPU resource of a single node is calculated to be 8G. Since a single node can start two tasks, the CPU resource of a single task started on a single node is 4G. For details, please refer to Table 1. When the job to be executed includes multiple jobs, the task can be understood as a job, and when the job to be executed includes one job, the task can be understood as a part of the job.
[0068] As an implementation method, if the working mode includes memory priority, the memory resource amount of a single node is calculated based on the memory resource amount of a single task, the number of working nodes, and the memory resource amount of the master node. For example, if the maximum memory resource amount is 9CU, the CPU resource amount of the master node is 1CU=4G, and the number of working nodes is 4, then the memory resource amount of a single task = ((9-1)×4) / 4=8, as shown in Table 1.
[0069] Table 1
[0070]
[0071]
[0072] The resource scheduling method provided in this application reasonably encapsulates container-running type jobs and non-container-running type jobs, and provides different resource allocation methods according to job types, thereby reasonably setting container resources and non-container resources to improve resource utilization. In addition, container-running type jobs can be conveniently run in a container environment, and non-container-running type jobs can be conveniently run on each node.
[0073] Figure 3 FIG. 1 is a flow chart of a resource scheduling method provided by an embodiment of the present application. Figure 3 As shown, with the resource manager as the execution subject, the method of this embodiment may include the following steps:
[0074] S201, receiving the physical resource amount of the container or the physical resource amount of each working node sent by the job manager;
[0075] The physical resource amount of the container is obtained by converting the resource group configured for the job when the job manager determines that the job type is the container running type;
[0076] The amount of physical resources of each working node is obtained by converting the resource group configured for the job according to the number of working nodes and the running strategy when the job manager determines that the job type is a non-container running type.
[0077] S202: Allocate resources to the job to be executed according to the physical resource amount of the container or allocate resources to the job to be executed according to the physical resource amount of each working node.
[0078] When the job manager sends the amount of physical resources of the container, the resource manager allocates resources to the container according to the amount of physical resources of the container, so that the job runs in the container with limited resources. When the job manager sends the amount of physical resources of each working node, the resource manager allocates resources to each node according to the amount of physical resources of each working node, so that the job runs on each node with limited resources.
[0079] As an implementation method, the resource manager includes Kubernetes and yarn. Kubernetes is used to allocate execution resources to the container corresponding to the job according to the physical resource amount of the container when the job manager sends the physical resource amount of the container, and yarn is used to allocate resources to each working node according to the physical resource amount of each working node when the job manager sends the physical resource amount of each node, so that the job runs on the node with limited resources.
[0080] The resource scheduling method provided by the present application is that the resource manager allocates resources according to the physical resource amount of the container or the physical resource amount of each node, reasonably sets the container resources and non-container resources, and improves the resource utilization rate. It also enables container-type jobs to be conveniently run in the container environment, and non-container-type jobs to be conveniently run in the corresponding nodes.
[0081] Figure 4 A schematic diagram of the structure of a resource scheduling device provided in an embodiment of the present application is shown. Figure 4 As shown, the resource scheduling device 10 of this embodiment is used to implement the operation corresponding to the job manager in any of the above method embodiments. The resource scheduling device 10 of this embodiment includes:
[0082] A first acquisition module 11, used to acquire a job to be executed;
[0083] A determination module 12 is used to determine whether the type of the job is a container operation type;
[0084] The second acquisition module 13 is used to convert the configured resource group into the physical resource amount of the container when the job type is a container running type, so that the job can run in the container with limited resources; and if the configured resource group is converted into the physical resource amount of each working node according to the number of working nodes and the running strategy, so that the job can run on the node with limited resources.
[0085] The resource scheduling device 10 provided in the embodiment of the present application can execute the above method embodiment. Its specific implementation principle and technical effects can be found in the above method embodiment, and this embodiment will not be repeated here.
[0086] Figure 5 FIG. 1 shows a hardware structure diagram of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 30 is used to implement the operations corresponding to the electronic device in any of the above method embodiments. The electronic device 30 of this embodiment may include: a memory 31, a processor 32 and a communication interface 33.
[0087] The memory 31 is used to store computer instructions. The memory 31 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk.
[0088] The processor 32 is used to execute the computer instructions stored in the memory to implement the resource scheduling method in the above embodiment. For details, please refer to the relevant description in the above method embodiment. The processor 32 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.
[0089] Optionally, the memory 31 may be independent or integrated with the processor 32 .
[0090] The communication interface 33 may be connected to the processor 32. The processor 32 may control the communication interface 33 to implement the functions of receiving and sending information signals.
[0091] The electronic device provided in this embodiment can be used to execute the above-mentioned resource scheduling method, and its implementation method and technical effect are similar, which will not be described in detail in this embodiment.
[0092] The present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the methods provided in the various embodiments described above.
[0093] The present application also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. At least one processor of a device can read the computer instructions from the computer-readable storage medium, and at least one processor executes the computer instructions so that the device implements the methods provided in the various embodiments described above.
[0094] An embodiment of the present application also provides a chip, which includes a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory, so that a device equipped with the chip executes the methods described in the various possible implementation modes above.
[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A resource scheduling method, characterized in that: The method is used in a job manager and comprises: Obtain a job to be executed, and determine whether the type of the job is a container running type; wherein the job manager is used to process container running type jobs and non-container running type jobs; If so, converting the configured resource group into the physical resource amount of the container so that the job runs in the container with limited resources; If not, the configured resource group is converted into the physical resource amount of each working node according to the number of working nodes and the operation strategy, so that the job is run on the node with limited resources.
2. The method according to claim 1, characterized in that The configured resource group includes a maximum CPU resource amount; The step of converting the configured resource group into the physical resource amount of the container specifically includes: The maximum CPU resource amount is converted into a container job application resource amount.
3. The method according to claim 2, characterized in that The conversion of the maximum CPU resource amount into the resource amount requested by the container job specifically includes: The maximum CPU resource amount is compressed according to a preset ratio and converted into the resource amount requested for the container job.
4. The method according to claim 2, characterized in that: The configured resource group includes a maximum memory resource amount, and the operation strategy includes: CPU priority or memory priority; If the operation strategy includes CPU priority, the resource group to be configured is converted into the amount of physical resources of each working node according to the number of working nodes and the operation strategy, specifically including: Calculate the number of tasks to be run by a single working node based on the maximum CPU resource, the number of working nodes, and the CPU resource of the master node; The CPU resource amount of each task on each working node is calculated based on the number of tasks running on each working node, the number of working nodes, the CPU resource amount of a single task, and the CPU resource amount of the master node. The memory of the single task is fixed at 4G.
5. The method according to claim 4, characterized in that If the operation strategy includes memory priority, the resource group to be configured is converted into the amount of physical resources of each working node according to the number of working nodes and the operation strategy, specifically including: The memory resource amount of each working node is calculated according to the memory resource amount of a single task, the number of working nodes and the memory resource amount of the master node, and the CPU of the single task is fixed to 1 core.
6. The method according to claim 1, characterized in that Determining whether the type of the job is a container operation type specifically includes: Determine whether the type of the job is a command line type, where the command line type includes: python, perl, shell or java.
7. A resource scheduling method, characterized in that: The method is used in a resource manager, comprising: Receiving the amount of physical resources of the container or the amount of physical resources of each working node sent by the job manager; wherein the job manager is used to process jobs of the container running type and jobs of the non-container running type; Allocate resources to the container corresponding to the job to be executed according to the amount of physical resources of the container, so that the job runs in the container with limited resources; or allocate resources to the node corresponding to the job to be executed according to the amount of physical resources of each working node, so that the job runs on the node with limited resources; The physical resource amount of the container is obtained by converting the resource group configured for the job when the job manager determines that the type of the job is a container running type; The amount of physical resources of each working node is obtained by converting the resource group configured for the job according to the number of working nodes and the running strategy when the job manager determines that the type of the job is a non-container running type.
8. The method according to claim 7, characterized in that The resource managers include Kubernetes and yarn; The Kubernetes is used to allocate execution resources to the container corresponding to the job according to the physical resource amount of the container; The yarn is used to allocate execution resources to the node corresponding to the job according to the physical resource amount of each working node.
9. A resource scheduling device, characterized in that: The device is used to implement the operation of the job manager, and the device includes: A first acquisition module, used to acquire a job to be executed; A determination module, used to determine whether the type of the job is a container operation type; wherein the job manager is used to process container operation type jobs and non-container operation type jobs; The second acquisition module is used to convert the configured resource group into the physical resource amount of the container when the type of the job is a container running type, so that the job can run in the container with limited resources; and if the configured resource group is converted into the physical resource amount of each working node according to the number of working nodes and the running strategy, so that the job can run on the node with limited resources.
10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the resource scheduling method according to any one of claims 1 to 7 or the resource scheduling method according to claim 8 or 9.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed by a processor, are used to implement the resource scheduling method according to any one of claims 1 to 7 or the resource scheduling method according to claim 8 or 9.
