Resource allocation method, device, electronic device and storage medium

Abstract

The present application relates to a resource allocation method, device, electronic device and storage medium, wherein the method comprises: in the case of monitoring the occurrence of a task addition event, obtaining a task configuration file of a target task for containerized deployment corresponding to the task addition event; pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupancy of the application instance contained in the target task indicated in the task configuration file, so that the resources of the resource occupancy in the available resources of the target node are pre-occupied by the application instance; in the case of successful pre-scheduling of the application instances, scheduling the application instance to the target node. The present application first determines that the application instances in the target task can be pre-scheduled before actually scheduling each application instance, which can effectively improve the efficiency of resource utilization and improve the throughput of the resource pool composed of each candidate node for tasks.

Description

Technical Field

[0001] The present application relates to the technical field of resource scheduling, and in particular to a resource allocation method, device, electronic device and storage medium. Background Art

[0002] In Kubernetes (a container group management system, referred to as K8s), the most basic operating unit is Pod, which can contain multiple containers, but generally speaking, a Pod contains only one container.

[0003] Distributed tasks in Kubernetes refer to a task Job (a job is a one-time task that can run multiple containers in parallel. When the task is completed, it will automatically exit and the cluster will not wake it up again) that contains multiple Pods. Multiple Pods perform calculations separately and then merge the results of the calculations according to a certain protocol. Distributed tasks can break through the limitations of single-machine performance and allow multiple machines to serve the entire calculation, which can speed up the calculation.

[0004] In the related art, when a user creates a Job through Kubernetes, and the resources in the current resource pool can only enable some Pods under this Job to be scheduled and run, the Job will fail to start, and the resources in the resource pool already held by the Job cannot be used by other Jobs; in addition, when there are multiple Jobs to be started, and the resources in the current resource pool can only enable all Pods under one of the Jobs to be scheduled, each Job will hold some Pod resources and will not release them, resulting in a situation where each Job cannot be started; and the above situations will lead to a waste of resources.

[0005] There is no effective solution to the technical problem of resource waste during resource allocation in related technologies. Summary of the invention

[0006] In order to solve the technical problem of resource waste during resource allocation in the related art, the present application provides a resource allocation method, device, electronic device and storage medium.

[0007] In a first aspect, an embodiment of the present application provides a resource allocation method, including:

[0008] In the case of monitoring the occurrence of a task adding event, obtaining a task configuration file of a target task to be deployed in a container corresponding to the task adding event, wherein the task adding event is used to indicate resource allocation for the target task;

[0009] According to the resource occupancy of the application instance included in the target task indicated in the task configuration file, pre-scheduling the application instance to a target node in at least one candidate node, so that resources of the resource occupancy amount in the available resources of the target node are pre-occupied by the application instance, wherein the available resource amount of the available resources is greater than or equal to the resource occupancy amount;

[0010] When the application instances are all pre-scheduled successfully, the application instances are scheduled to the target node, wherein the pre-scheduling success is used to indicate that after pre-scheduling each of the application instances, the amount of remaining resources of each of the candidate nodes is greater than or equal to 0.

[0011] Optionally, as in the above method, before obtaining the task configuration file of the target task to be containerized and deployed corresponding to the task new adding event, the method further includes:

[0012] Obtaining the task configuration file created by the target object;

[0013] The task configuration file is stored in a target database, wherein the task adding event is storing the task configuration file in the target database.

[0014] Optionally, as in the aforementioned method, the step of obtaining a task configuration file of a target task corresponding to the task newly added event includes:

[0015] The task configuration file corresponding to the task new adding event is obtained from the target database.

[0016] Optionally, as in the aforementioned method, the number of the application instances is multiple, and the number of the candidate nodes is multiple;

[0017] The step of pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupation amount of the application instance included in the target task indicated in the task configuration file, so that resources of the resource occupation amount in the available resources of the target node are pre-occupied by the application instance comprises:

[0018] In the case where the sum of the available resource amounts of each candidate node is greater than or equal to the sum of the resource occupancy amounts of each application instance, the multiple application instances are pre-scheduled to at least one target node among the multiple candidate nodes in turn, so that the resources of the available resources of the target node with the resource occupancy amount are pre-occupied by the application instance, wherein one application instance is scheduled to one target node.

[0019] Optionally, as in the aforementioned method, pre-scheduling the plurality of application instances to at least one target node among the plurality of candidate nodes in sequence according to the resource occupation of each application instance and the available resources of the target node comprises:

[0020] Determine the target application instance to be processed currently according to the scheduling order corresponding to each of the application instances;

[0021] Determine a first target node to which the target application instance is pre-scheduled, wherein the first target node is a node among the plurality of candidate nodes, whose available resources are greater than the resource occupation of the target application instance and whose available resources are the least;

[0022] The target application instance is pre-scheduled to the first target node.

[0023] Optionally, as in the foregoing method, after pre-scheduling the target application instance to the first target node, the method further includes:

[0024] The available resource amount of the first target node is updated using the difference between the available resource amount of the first target node and the resource occupation amount of the target application instance.

[0025] Optionally, as in the foregoing method, before scheduling the application instance to the target node, the method further includes:

[0026] The correspondence between the application instance and the target node is stored in the target storage system, so that after the target node detects that it is bound to the application instance, it creates a target container according to the application instance and starts the target container to run the application instance.

[0027] In a second aspect, an embodiment of the present application provides a resource allocation device, including:

[0028] A file acquisition module is used to acquire a task configuration file of a target task to be deployed in a container corresponding to the task addition event when a task addition event is detected, wherein the task addition event is used to indicate resource allocation for the target task;

[0029] A pre-scheduling module, configured to pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupancy of the application instance included in the target task indicated in the task configuration file, so that resources of the resource occupancy in the available resources of the target node are pre-occupied by the application instance, wherein the available resource amount of the available resources is greater than or equal to the resource occupancy;

[0030] A scheduling determination module is used to schedule the application instances to the target node when all the application instances are pre-scheduled successfully, wherein the pre-scheduling success is used to indicate that after pre-scheduling each of the application instances, the amount of remaining resources of each of the candidate nodes is greater than or equal to 0.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0032] The memory is used to store computer programs;

[0033] The processor is used to implement any of the methods described above when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium includes a stored program, wherein the program executes a method as described in any of the preceding items when executed.

[0035] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0036] The method provided in the embodiment of the present application can avoid the situation where, when the application instances are directly scheduled, some application instances in the target task cannot be scheduled because the current candidate nodes cannot provide sufficient resources to the target task, and thus the target task cannot be executed. At the same time, since the scheduled application instances in the target task continue to occupy resources without releasing them, other tasks may easily fail to execute due to the inability to obtain sufficient resources. Furthermore, the method in this embodiment can effectively improve the efficiency of resource utilization and improve the throughput of tasks of the resource pool composed of each candidate node. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1A flowchart of a resource allocation method provided in an embodiment of the present application;

[0040] Figure 2 A flowchart of a resource allocation method provided in another embodiment of the present application;

[0041] Figure 3 A flowchart of a resource allocation method provided in another embodiment of the present application;

[0042] Figure 4 A flowchart of a resource allocation method provided for an application example of this application;

[0043] Figure 5 A block diagram of a resource allocation device provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] In the related technology, Kubernetes provides Job (a collection of one or more Pods that manages work-class containers) for containerized deployment of computing tasks. When a user creates a Job through Kubernetes, the Pods under this Job may be scheduled or not. For example: a Job1 is associated with 10 Pods, of which 6 Pods can be scheduled and run, and the other 4 are in a waiting state. However, when using distributed tasks to run deep learning training, that is, distributed training (generally speaking, deep learning requires computing a large amount of training data, and a single machine cannot meet the requirements, so distributed training is the mainstream method of deep learning), all 10 Pods must be running before training can be performed, and the other 4 Pods must be started before they can continue to run. Undoubtedly, before the other 4 tasks are run, this Job1 not only holds the resources of 6 Pods but also waits for the resources of 4 tasks to be released. This leads to a waste of resources.

[0047] In another case, if there are 4 resources in the resource pool, and there are Job2 and Job3, and there are 4 Pods under Job2 and Job3 respectively, in the related technology, it will happen that Job2 obtains the resources of 1 Pod, and Job3 obtains the resources of 3 Pods. At this time, Job2 and Job3 cannot start normally, but the resources are occupied. If the remaining resources are not increased, these two tasks will never start normally.

[0048] From this, we can see that if there is a new scheduling method for resource calls in related technologies that can solve such scheduling dilemmas, the throughput of the entire system will be greatly improved.

[0049] In order to solve the above technical problems, according to one aspect of the embodiments of the present application, a resource allocation method is provided. Optionally, in this embodiment, the above attribute information acquisition method can be applied to a hardware environment composed of a terminal and a server. The server is connected to the terminal through a network, and can be used to provide services for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0050] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0051] The resource allocation method of the embodiment of the present application can be executed by a server, a terminal, or a server and a terminal together. The resource allocation method of the embodiment of the present application can be executed by a terminal or a client installed thereon.

[0052] Taking the resource allocation method in this embodiment executed by the server as an example, Figure 1 A resource allocation method provided in an embodiment of the present application includes steps S101 to S103 as described below:

[0053] Step S101, when a task adding event is monitored, a task configuration file of a target task to be deployed in a container corresponding to the task adding event is obtained, wherein the task adding event is used to indicate resource allocation for the target task.

[0054] The resource allocation method of the present application is applied to application scenarios where resources need to be allocated according to tasks so that tasks can obtain the resources required for operation; for example, in Kubernetes, the scenario of resource allocation for Job, in addition, it can also be other scenarios. Taking the scenario of resource allocation for containerized deployed Job (i.e., target task) in Kubernetes as an example, for a newly added JobI, since it is necessary to determine how many resources to allocate to the JobI, it is necessary to obtain the task configuration file of the JobI.

[0055] The task configuration file may be a configuration file required for creating the target task JobI. The task configuration file includes all Pods in the target task JobI and the resources required by each Pod. The types of resources may include but are not limited to: storage resources, memory resources, and CPU resources, etc. The task configuration file may be a file created by a user in a specific storage location. After the task configuration file is created, it can be considered that a task addition event has occurred to trigger resource allocation for the target task corresponding to the task configuration file.

[0056] For example, you can monitor at a preset frequency (for example, once every 5 seconds) whether there is a newly generated or newly written task configuration file. When you monitor that a task configuration file is newly written to a storage location, it is determined that a task addition event has occurred, and then the latest written task configuration file is read.

[0057] Step S102, according to the resource occupancy of the application instance contained in the target task indicated in the task configuration file, pre-schedule the application instance to a target node in at least one candidate node, so that the resources with the resource occupancy in the available resources of the target node are pre-occupied by the application instance, wherein the available resource amount of the available resources is greater than or equal to the resource occupancy.

[0058] After obtaining the task configuration file, since the task configuration file corresponds to the target task and contains all Pods in the target task JobI and the resources required by each Pod, and each Pod is an application instance, all application instances included in the target task and the resource usage of each application instance can be determined based on the task configuration file.

[0059] The resource usage can be the sum of the resources required by each application instance Pod in the target task. For example, assuming that the resources required by each application instance Pob included in the target task JobI are consistent, and the resources required by each Pod are recorded as one unit resource (for example, 2GB memory-10 CPU computing units-20GB storage resources), the resource usage of the target task JobI is 4 unit resources. In general, a Job includes at least one Pod, and each Pod needs to be assigned to the same processing node Node to run.

[0060] A candidate node can be a node used to provide computing resources. The candidate node may include one or more. A resource pool can be composed of multiple candidate nodes. The computing resources that each candidate node can provide are limited. Therefore, the amount of available resources in the candidate node will decrease as the number of running tasks increases. Among them, the target node is the node to which an application instance in the target task is scheduled. Furthermore, when it is not determined that all application instances in the target task can be allocated corresponding resources, the application instance can be pre-scheduled only to the target node among at least one candidate node, so that the resources occupied by the target node's available resources are pre-occupied by the application instance; that is, the resources occupied by the available resources on the target node are only temporarily pre-occupied by the application instance, and the resources occupied by the available resources on the target node are not used to run the application instance. In this way, the same resource can be avoided from being repeatedly allocated to multiple application instances; if all application instances included in the final target task can be pre-scheduled to a certain target node, the application instance can occupy the corresponding pre-occupied resources; if at least one application instance included in the final target task cannot be pre-scheduled to any candidate node, all application instances of the target task are released from pre-occupying the corresponding resources, so that the pre-occupied resources are released and used to be pre-allocated to application instances of other tasks.

[0061] For example, when there is only one application instance and three candidate nodes, first determine the resource usage L1 of the application instance, and then determine the available resources L2, L3, and L4 of each candidate node; finally, based on the quantitative relationship between L1 and L2, L3, and L4, select the candidate node with available resources greater than L1 as the target node.

[0062] Step S103, when all application instances are pre-scheduled successfully, the application instances are scheduled to the target node, wherein the pre-scheduling success is used to indicate that after each application instance is pre-scheduled, the amount of remaining resources of each candidate node is greater than or equal to 0.

[0063] Successful pre-scheduling means that after the application instance in the target task is scheduled to the target node corresponding to the application instance, the remaining amount of resources in each target node is still greater than or equal to 0; that is, successful pre-scheduling will not result in a situation where the amount of resources required by the application instance scheduled to the target node is greater than its own available resources; further, in all pre-scheduling corresponding to the target task, each resource can only be occupied by one pre-scheduled application instance.

[0064] When each application instance is pre-scheduled successfully, it means that the target task can be executed by one target node or a combination of multiple target nodes. Therefore, each application instance is scheduled to the target node corresponding to each application instance.

[0065] For example, when there are two application instances A and B for the target task and there are three candidate nodes a, b, and c, when it is determined according to the method in the aforementioned step S102 that application instance A can be pre-scheduled to candidate node a (that is, the available resources of candidate node a are greater than or equal to the resource occupancy of application instance A), and application instance B can be pre-scheduled to candidate node b (that is, the available resources of candidate node b are greater than or equal to the resource occupancy of application instance B), application instance A is scheduled to candidate node a and application instance B is scheduled to candidate node b to complete the resource scheduling of the target task.

[0066] Through the method in this embodiment, it is first determined that all application instances in the target task can be pre-scheduled before each application instance is actually scheduled. This can avoid the situation where, when the application instance is directly scheduled, some application instances in the target task cannot be scheduled because the current candidate node cannot provide sufficient resources to the target task, and thus the target task cannot be executed. At the same time, since the scheduled application instances in the target task continue to occupy resources without releasing them, it is easy for other tasks to fail to execute due to inability to obtain sufficient resources. Furthermore, the method in this embodiment can effectively improve the efficiency of resource utilization and can improve the throughput of tasks for the resource pool composed of each candidate node.

[0067] like Figure 2 As shown, in some embodiments, as in the aforementioned method, before the step S101 acquires the task configuration file of the target task for containerized deployment corresponding to the task addition event, the method further includes the following steps S201 and S202:

[0068] Step S201, obtaining a task configuration file created by a target object.

[0069] The target object may be a configuration person who needs to obtain computing services for the target task, and the task configuration file is a file used by the target object to create the target task.

[0070] Optionally, you need to create a new CRD (Custom Resource Definition) resource in Kubernetes to manage the various Pods below, because the native Job of Kubernetes cannot satisfy the gangscheduler. You can create a new resource IQJob (i.e., task profile). Compared with the Job in the prior art, IQJob adds a PodSetuuid field. Compared with the Job task profile, the PodSetuuid field can define the UUID (Universally Unique Identifier) ​​of the target task, the number of Pods, and the amount of resources required for each Pod.

[0071] Step S202, storing the task configuration file in the target database, wherein the task adding event is storing the task configuration file in the target database.

[0072] The target database may be a database for storing task configuration files. After the task configuration files are stored in the target database, the database may be monitored through a preset query interface at a preset monitoring frequency. When it is confirmed that a new task configuration file has been added to the target database, it is determined that a new task event has occurred.

[0073] For example, after the Kubernetes API server obtains the IQJob (i.e., the task configuration file), it will record it in etcd (i.e., the target database, etcd is a highly available Key / Value storage system, mainly used for shared configuration and service discovery). Through the list_and_watch interface, you can find that an IQJob has been added to etcd and get the entire content of this IQJob.

[0074] Through the method in this embodiment, a newly added task configuration file can be obtained in real time, thereby improving the timeliness of task processing.

[0075] In some embodiments, as in the aforementioned method, the step S101 of obtaining the task configuration file of the target task corresponding to the task adding event includes:

[0076] Obtain a task configuration file corresponding to the task new adding event from the target database.

[0077] It can be seen from steps S201 and S202 of the aforementioned embodiment that the task configuration file is stored in the target database; therefore, the task configuration file can be obtained from the target database.

[0078] For example, based on a task addition event, it can be determined that a new task configuration file has been written into the target database. Then, the latest written file can be queried through the writing time of each file in the database, and it can be confirmed that the file is the task configuration file corresponding to the task addition time.

[0079] Through the method in this embodiment, a newly added task configuration file in the target database can be quickly obtained, so that resources can be quickly allocated to the target task based on the task configuration file at a later time.

[0080] In some embodiments, as in the aforementioned method, the number of application instances is multiple, and the number of candidate nodes is multiple;

[0081] The step S102 pre-schedules the application instance to a target node in at least one candidate node according to the resource occupation of the application instance included in the target task indicated in the task configuration file, so that the resources of the available resources of the target node with the resource occupation are pre-occupied by the application instance, including:

[0082] When the sum of available resources of each candidate node is greater than or equal to the sum of resource occupancy of each application instance, multiple application instances are pre-scheduled to at least one target node among multiple candidate nodes in turn according to the resource occupancy of each application instance, so that the resources with the resource occupancy in the available resources of the target node are pre-occupied by the application instance, wherein one application instance is scheduled to one target node.

[0083] The sum of available resources can be obtained by accumulating available resources of each candidate node; the sum of resource occupancy can be obtained by accumulating resource occupancy of each application instance of the target task, and the resource occupancy of each application instance can be determined based on the configuration content corresponding to each application instance in the task configuration file. Since the target task can only be executed after each application instance in it can be scheduled to the target node, it is necessary to perform pre-scheduling only when the sum of available resources of each candidate node is greater than or equal to the sum of resource occupancy of each application instance.

[0084] In the case where the sum of the available resources of the candidate nodes is greater than or equal to the sum of the resource occupancy of each application instance, the target node to which each application instance is pre-scheduled can be determined based on the relationship between the resource occupancy of each application instance and the available resources of each candidate node, so that the resources of the resource occupancy in the available resources of the target node are pre-occupied by the application instance, and the resources of the resource occupancy in the available resources on the target node are only temporarily pre-occupied by the application instance, and the resources of the resource occupancy in the available resources on the target node are not used to run the application instance. In this way, the same resource can be avoided from being repeatedly allocated to multiple application instances. Furthermore, an application instance only needs to be scheduled to one target node, while a target node can be scheduled with multiple application instances.

[0085] For example, after determining the resource usage of each application instance Pod, each application instance Pod can be judged in turn, and the candidate node Node to which each application instance Pod can be pre-scheduled among all candidate nodes Node is used as the target node, and a corresponding relationship between the application instance Pod and the target node is established. Optionally, the resource usage of each application instance Pod can be compared with the available resources of each candidate node Node in turn. When it is determined that the available resources of the candidate node Node1 are greater than or equal to the resource usage of the application instance Pod, the application instance Pod is scheduled to the candidate node Node1 as the target node.

[0086] Through the method in this embodiment, the application instances can be scheduled according to the resource occupancy of each application instance when the sum of the available resources of each candidate node is greater than or equal to the sum of the resource occupancy of each application instance. This can avoid the situation where all application instances cannot be pre-scheduled due to the sum of the available resources being less than the sum of the resource occupancy, and can effectively improve the efficiency of pre-scheduling.

[0087] like Figure 3 As shown, in some embodiments, as in the aforementioned method, the steps pre-schedule multiple application instances to at least one target node among multiple candidate nodes in sequence according to the resource occupancy of each application instance and the available resources of the target node, including the following steps S301 to S303:

[0088] Step S301 : determining the target application instance to be processed currently according to the scheduling order corresponding to each application instance.

[0089] The scheduling order may be a predetermined order for indicating the order in which each application instance is scheduled, and each application instance has a corresponding scheduling order. Optionally, the scheduling order may be manually set in advance, or may be obtained by sorting according to other strategies, for example: according to the resource usage of each application instance, when the resource usage includes multiple types of resources (for example, memory, CPU, GPU, disk, etc.), each application instance may be sorted by determining the single resource usage of the resource type used for sorting, and the scheduling order corresponding to the application instance may be obtained. In general, each scheduling order is arranged in order.

[0090] After the current application instance is pre-scheduled, the current application instance to be processed can be determined based on the scheduling order of the previous application instance and recorded as the target application instance.

[0091] Step S302, determining a first target node to which the target application instance is pre-scheduled, wherein the first target node is a node among multiple candidate nodes, whose available resources are greater than the resource occupation of the target application instance and whose available resources are the least.

[0092] The candidate nodes can be sorted in order from small to large according to the available resource amount, and compared with the resource occupancy of the target application instance in sequence according to the sorting. When it is determined that a certain available resource amount meets the resource occupancy of the target application instance, the first target node with the least available resource amount and the largest available resource amount can be screened out.

[0093] Step S303: pre-schedule the target application instance to the first target node.

[0094] After determining the first target node that meets the conditions in step S302, the target application instance can be pre-scheduled to the first target node so that the resource occupation amount of the target application instance occupies the corresponding resource amount on the first target node.

[0095] Through the method in this embodiment, by selecting the node with the least available resources and the amount of available resources greater than the resource occupancy of the target application instance as the first target node, the remaining amount of resources of the candidate node can be minimized after the target application instance is scheduled, which can effectively reduce resource waste.

[0096] In some embodiments, as in the aforementioned method, after the target application instance is pre-scheduled to the first target node, the method further includes step S401 as described below:

[0097] Step S401: using the difference between the available resource amount of the first target node and the resource occupancy amount of the target application instance, the available resource amount of the first target node is updated.

[0098] The available resource amount of the first target node is the available resource amount before the target application instance is pre-scheduled to the first target node; and the difference is the available resource amount of the first target node after the target application instance is pre-scheduled to the first target node.

[0099] After obtaining the difference, if it is necessary to allocate an application instance to the first target node later, it is necessary to determine whether the first target node meets the requirements of the application instance to be allocated based on the difference. Therefore, it is necessary to update the available resources of the first target node through the difference. Otherwise, even if the pre-scheduling is successful, the pre-scheduled application instance cannot be actually scheduled to the first target node during the actual scheduling due to insufficient available resources.

[0100] In some embodiments, as in the above method, before scheduling the application instance to the target node, the method further includes the following step S501:

[0101] Step S501, storing the correspondence between the application instance and the target node in the target storage system, so that after the target node monitors that it is bound to the application instance, it creates a target container according to the application instance and starts the target container to run the application instance.

[0102] The target storage system may be etcd as described in the aforementioned embodiment, and the correspondence between the application instance and the target node may be written into etcd through the K8s scheduler.

[0103] The kubelet running on the target node can monitor the application instance pod bound to itself through the list and watch interface, then create a target container based on the application instance and start the target container to run the application instance.

[0104] Through the method in this embodiment, each target node can start the target container to run the application instance according to the corresponding relationship.

[0105] like Figure 4 The following is an application example of the method in any of the above embodiments:

[0106] 1. Receive the IQJob (i.e., task configuration file) created by the user. The format of the parameters in the IQJob is consistent with the format of creating a Job (i.e., the target task).

[0107] 2. After receiving the request, the K8s API server (which provides HTTP Rest interfaces such as add, delete, modify, check, and watch for various K8s resource objects (Pod, RC, Service, etc.), and is the data bus and data center of the entire system) will record it in etcd (i.e., the target database).

[0108] 3. IQJob Controller can use the list_and_watch interface (an interface used to monitor etcd) to discover that an IQJob has been added to etcd, and then obtain the entire content of the IQJob from etcd.

[0109] 4. First, query the K8s API server to obtain the resource usage of all current Nodes (i.e., candidate nodes).

[0110] 5. According to the content of the Pod (i.e., application instance) in IQJob, initialize PodSet (used to determine whether the sum of the available resources of each candidate node is greater than or equal to the sum of the resource usage of each application instance). After determining whether the sum of the available resources of each candidate node is greater than or equal to the sum of the resource usage of each application instance, determine whether each Pod can be scheduled to the Node.

[0111] 6. If possible, it will be scheduled through the gang scheduler (a kube-batch job may have multiple Pods, and these Pods must either all be executed or none of them will be executed), and the K8s Scheduler will be requested to do the actual scheduling.

[0112] 7. K8s Scheduler writes the scheduling information, that is, the correspondence between Pod and Node, into etcd (that is, the target storage system consistent with the target database).

[0113] 8. At this time, the kubelet running on the Node (the main "node agent" running on each Node) listens to etcd through the list and watch interface, obtains the Pod bound to itself according to the corresponding relationship, and then starts the container to run the Pod bound to itself.

[0114] like Figure 5 As shown, according to an embodiment of another aspect of the present application, a resource allocation device is also provided, including:

[0115] The file acquisition module 1 is used to acquire the task configuration file of the target task to be deployed in containerization corresponding to the task addition event when monitoring the occurrence of the task addition event, wherein the task addition event is used to indicate resource allocation for the target task;

[0116] A pre-scheduling module 2 is used to pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupancy of the application instance included in the target task indicated in the task configuration file, so that the resources of the resource occupancy in the available resources of the target node are pre-occupied by the application instance, wherein the available resource amount of the available resources is greater than or equal to the resource occupancy;

[0117] The scheduling determination module 3 is used to schedule the application instances to the target node when all application instances are pre-scheduled successfully, wherein the pre-scheduling success is used to indicate that after pre-scheduling each application instance, the amount of remaining resources of each candidate node is greater than or equal to 0.

[0118] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0119] In some embodiments, the aforementioned apparatus further comprises:

[0120] An acquisition module, used to acquire a task configuration file created by a target object;

[0121] The file storage module is used to store the task configuration file in the target database, wherein the task new adding event is to store the task configuration file in the target database.

[0122] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0123] In some embodiments, as in the aforementioned device, the file acquisition module 1 is used to acquire a task configuration file corresponding to a task new adding event from a target database.

[0124] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0125] In some embodiments, as in the aforementioned apparatus, the number of application instances is multiple, and the number of candidate nodes is multiple;

[0126] The pre-scheduling module 2 is used to pre-scheduling multiple application instances to at least one target node among the multiple candidate nodes in sequence when the sum of the available resource amounts of the candidate nodes is greater than or equal to the sum of the resource occupancy amounts of the application instances, so that the resources with the resource occupancy amount in the available resources of the target node are pre-occupied by the application instances, wherein one application instance is scheduled to one target node.

[0127] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0128] In some embodiments, as in the aforementioned device, the pre-scheduling module 2 includes:

[0129] A first determining unit, configured to determine a target application instance to be processed currently according to a scheduling order corresponding to each application instance;

[0130] A second determination unit is used to determine a first target node to which the target application instance is pre-scheduled, wherein the first target node is a node among multiple candidate nodes, whose available resource amount is greater than the resource occupation amount of the target application instance and whose available resource amount is the least;

[0131] The pre-scheduling unit is used to pre-scheduling the target application instance to the first target node.

[0132] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0133] In some embodiments, the aforementioned apparatus further comprises:

[0134] The updating module is used to update the available resource amount of the first target node by using the difference between the available resource amount of the first target node and the resource occupation amount of the target application instance.

[0135] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0136] In some embodiments, the aforementioned apparatus further comprises:

[0137] The corresponding relationship storage module is used to store the corresponding relationship between the application instance and the target node in the target storage system, so that after the target node monitors that it is bound to the application instance, it creates a target container according to the application instance and starts the target container to run the application instance.

[0138] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0139] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 6 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0140] Memory 1503, used for storing computer programs;

[0141] The processor 1501 is used to implement the steps of the above method embodiment when executing the program stored in the memory 1503.

[0142] The bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0143] The communication interface is used for communication between the above electronic device and other devices.

[0144] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0145] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0146] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0147] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0148] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A resource allocation method, characterized in that: include: In the case of monitoring the occurrence of a task adding event, obtaining a task configuration file of a target task to be deployed in a container corresponding to the task adding event, wherein the task adding event is used to indicate resource allocation for the target task; According to the resource occupation amount of the application instance included in the target task indicated in the task configuration file, the application instance is pre-scheduled to a target node in at least one candidate node, so that resources of the resource occupation amount in the available resources of the target node are pre-occupied by the application instance, wherein the available resource amount of the available resources is greater than or equal to the resource occupation amount, and the pre-occupancy means that the resources are temporarily occupied by the application instance, and the resources of the resource occupation amount in the available resources on the target node are not used to run the application instance; In the case where the application instances are all pre-scheduled successfully, scheduling the application instances to the target node, wherein the pre-scheduling success is used to indicate that after pre-scheduling each of the application instances, the amount of remaining resources of each of the candidate nodes is greater than or equal to 0; In the case that at least one application instance included in the target task cannot be pre-scheduled to any candidate node, all application instances of the target task are enabled to release the pre-occupied corresponding resources, so that the pre-occupied resources are released.

2. The method according to claim 1, characterized in that Before obtaining the task configuration file of the target task to be containerized and deployed corresponding to the task new adding event, the method further includes: Obtaining the task configuration file created by the target object; The task configuration file is stored in a target database, wherein the task adding event is storing the task configuration file in the target database.

3. The method according to claim 2, characterized in that The step of obtaining a task configuration file of a target task corresponding to the task newly added event includes: The task configuration file corresponding to the task new adding event is obtained from the target database.

4. The method according to claim 1, characterized in that The number of the application instances is multiple, and the number of the candidate nodes is multiple; The step of pre-scheduling the application instance to a target node in at least one candidate node according to the resource occupation amount of the application instance included in the target task indicated in the task configuration file, so that resources of the resource occupation amount in the available resources of the target node are pre-occupied by the application instance comprises: In the case where the sum of the available resource amounts of each candidate node is greater than or equal to the sum of the resource occupancy amounts of each application instance, the multiple application instances are pre-scheduled to at least one target node among the multiple candidate nodes in turn, so that the resources of the available resources of the target node with the resource occupancy amount are pre-occupied by the application instance, wherein one application instance is scheduled to one target node.

5. The method according to claim 4, characterized in that The pre-scheduling the plurality of application instances to at least one target node among the plurality of candidate nodes in sequence includes: Determine the target application instance to be processed currently according to the scheduling order corresponding to each of the application instances; Determine a first target node to which the target application instance is pre-scheduled, wherein the first target node is a node among the plurality of candidate nodes, whose available resources are greater than the resource occupation of the target application instance and whose available resources are the least; The target application instance is pre-scheduled to the first target node.

6. The method according to claim 5, characterized in that After pre-scheduling the target application instance to the first target node, the method further includes: The available resource amount of the first target node is updated using the difference between the available resource amount of the first target node and the resource occupation amount of the target application instance.

7. The method according to any one of claims 1 to 6, characterized in that Before scheduling the application instance to the target node, the method further includes: The correspondence between the application instance and the target node is stored in the target storage system, so that after the target node detects that it is bound to the application instance, it creates a target container according to the application instance and starts the target container to run the application instance.

8. A resource allocation device, characterized in that: include: A file acquisition module is used to acquire a task configuration file of a target task to be deployed in a container corresponding to the task addition event when a task addition event is detected, wherein the task addition event is used to indicate resource allocation for the target task; a pre-scheduling module, configured to pre-scheduling the application instance to a target node among at least one candidate node according to the resource occupation amount of the application instance included in the target task indicated in the task configuration file, so that resources of the resource occupation amount in the available resources of the target node are pre-occupied by the application instance, wherein the available resource amount of the available resources is greater than or equal to the resource occupation amount, and the pre-occupancy means that the resources are temporarily occupied by the application instance, and the resources of the resource occupation amount in the available resources on the target node are not used to run the application instance; A scheduling determination module, configured to schedule the application instances to the target node if all the application instances are pre-scheduled successfully, wherein the pre-scheduling success is used to indicate that after pre-scheduling each of the application instances, the amount of remaining resources of each of the candidate nodes is greater than or equal to 0; It also includes a module for releasing the pre-occupied resources of all application instances of the target task when at least one application instance included in the target task cannot be pre-scheduled to any candidate node.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 7 when executed.

Images