Container scheduling method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a container scheduling method and device, electronic equipment and a medium, and relates to the technical field of computers. The method comprises the following steps: constructing a container utility function and a load balancing objective function, and determining a pheromone updating strategy of a multi-objective ant colony algorithm according to the container utility function and the load balancing objective function; the container utility function indicates the utility of distributing the target container to the target computing node, the utility of distributing the target container to the target computing node is in direct proportion to the number of to-be-read data in the target computing node, and the load balancing target function indicates the balancing degree of distributing the target container to the target computing node; and determining a container scheduling scheme according to the initial parameters of the multi-target ant colony algorithm and a pheromone updating strategy. According to the method, the container is preferentially deployed on the computing node storing the to-be-read data, and meanwhile, the balanced distribution of the container is ensured, so that the running speed of the task in the container can be accelerated, and network IO is avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a container scheduling method, device, electronic equipment and medium. Background Art

[0002] Containers are a lightweight and portable virtualization technology used to encapsulate applications and all their dependencies and configurations so that they can run in different computer environments. YARN (Yet Another Resource Negotiator) is a resource scheduling component in the Hadoop (distributed system infrastructure) ecosystem, used for resource management and job scheduling, and providing server computing resources for computing programs. In YARN, applications perform work through containers running on the underlying operating system. For application jobs, the speed of reading data or files from the local node is higher than the speed of reading from the remote machine. However, in the current container scheduling method, container allocation is relatively random, and local read-write priority strategies are rarely considered. It is difficult to avoid unnecessary network IO (network IO refers to the process of data interaction through the network), which makes it difficult to effectively improve the efficiency of job operation in the container. Summary of the invention

[0003] In view of this, the embodiments of the present invention provide a container scheduling method, device, electronic device and medium, which give priority to localization when allocating containers to ensure the efficiency of task running in the container. At the same time, in order to prevent the problem of excessive concentration of container allocation, the problem is circumvented by designing an optimization goal of node load balancing.

[0004] In a first aspect, an embodiment of the present invention provides a container scheduling method, including:

[0005] Constructing a container utility function and a load balancing objective function, and determining the pheromone update strategy of the multi-objective ant colony algorithm according to the container utility function and the load balancing objective function; wherein the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node, and the load balancing objective function is used to indicate the degree of balance of allocating the target container to the target computing node;

[0006] Initialize the initial parameters of the multi-objective ant colony algorithm;

[0007] A container scheduling scheme is determined according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy.

[0008] Optionally, determining the container scheduling scheme based on the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy includes: establishing resource constraint rules for containers and computing nodes; determining the container scheduling scheme based on the initial parameters of the multi-objective ant colony algorithm, the pheromone update strategy and the resource constraint rules.

[0009] Optionally, the method further includes:

[0010] According to the container operation diagram of the job to be executed, the containers are divided into MapNode type and ReduceNode type; MapNode type containers are containers before shuffle occurs, and ReduceNode type containers are containers after shuffle occurs;

[0011] The container utility function is constructed, including:

[0012] Constructing a first utility function and constructing a second utility function; the first utility function is used to indicate the utility of allocating a container of a MapNode type to a target computing node, and the second utility function is used to indicate the utility of allocating a container of a ReduceNode type to a target computing node;

[0013] The pheromone update strategy of the multi-objective ant colony algorithm is determined according to the container utility function and the load balancing objective function, including: using the first utility function and the load balancing objective function as a first pheromone update strategy for determining a scheduling scheme for a container of the MapNode type, and using the second utility function and the load balancing objective function as a second pheromone update strategy for determining a scheduling scheme for a container of the ReduceNode type.

[0014] Optionally, determining the container scheduling scheme according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy includes:

[0015] Determine a first scheduling scheme for a container of type MapNode according to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy;

[0016] A scheduling scheme for a ReduceNode type container is determined according to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy and the first scheduling scheme.

[0017] Optionally, determining a first scheduling scheme for a container of type MapNode according to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy includes:

[0018] Constructing a first node candidate set, the first node candidate set including a first computing node storing the to-be-read data indicated by the to-be-executed job and a second computing node not storing the to-be-read data;

[0019] According to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy, determine the first target computing node for deploying the MapNode type container from the first node candidate set to determine a first scheduling scheme for the MapNode type container;

[0020] The step of determining a scheduling scheme for a ReduceNode type container according to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes:

[0021] Constructing a second node candidate set, where the second node candidate set includes a third computing node deployed with the first target computing node and a container of the MapNode type not deployed;

[0022] According to the initial parameters of the multi-objective ant colony algorithm and the second pheromone update strategy, a second target computing node for deploying the ReduceNode type container is determined from the second node candidate set to determine a second scheduling scheme for the ReduceNode type container.

[0023] Optionally, determining a first scheduling scheme for a container of type MapNode according to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy includes:

[0024] In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first computing nodes in the first node candidate set, and the remaining ants are randomly assigned to the second computing nodes in the first node candidate set;

[0025] The step of determining a scheduling scheme for a ReduceNode type container according to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes:

[0026] In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

[0027] Optionally, in the first iteration of the multi-objective ant colony algorithm, allocating some ants to the first computing nodes in the first node candidate set includes:

[0028] In the first iteration of the multi-objective ant colony algorithm, for each first computing node, it is determined whether to allocate ants to the first computing node according to the proportion of the to-be-read data blocks included in the first computing node.

[0029] In a second aspect, an embodiment of the present invention provides a container scheduling device, including:

[0030] A construction module is used to construct a container utility function and a load balancing objective function, and determine the pheromone update strategy of the multi-objective ant colony algorithm according to the container utility function and the load balancing objective function; wherein the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node, and the load balancing objective function is used to indicate the degree of balance of allocating the target container to the target computing node;

[0031] Initialization module, used to initialize the initial parameters of the multi-objective ant colony algorithm;

[0032] The determination module is used to determine the container scheduling scheme according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy.

[0033] Optionally, the determination module is further used to: establish resource constraint rules for containers and computing nodes; and determine a container scheduling solution based on initial parameters of the multi-objective ant colony algorithm, the pheromone update strategy and the resource constraint rules.

[0034] Optionally, the device further includes a container division module, which is used to: divide the containers into MapNode type and ReduceNode type according to the container operation diagram of the job to be executed; the MapNode type container is the container before the shuffle occurs, and the ReduceNode type container is the container after the shuffle occurs;

[0035] The construction module is used to: construct a first utility function and a second utility function; the first utility function is used to indicate the utility of allocating a container of the MapNode type to a target computing node, and the second utility function is used to indicate the utility of allocating a container of the ReduceNode type to a target computing node; the first utility function and the load balancing objective function are used as a first pheromone update strategy for determining a scheduling scheme for a container of the MapNode type, and the second utility function and the load balancing objective function are used as a second pheromone update strategy for determining a scheduling scheme for a container of the ReduceNode type.

[0036] Optionally, the determination module is used to: determine a first scheduling scheme for a MapNode type container based on the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy; determine a scheduling scheme for a ReduceNode type container based on the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy and the first scheduling scheme.

[0037] Optionally, the determining module is used to:

[0038] Constructing a first node candidate set, the first node candidate set including a first computing node storing the to-be-read data indicated by the to-be-executed job and a second computing node not storing the to-be-read data;

[0039] According to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy, determine the first target computing node for deploying the MapNode type container from the first node candidate set to determine a first scheduling scheme for the MapNode type container;

[0040] Constructing a second node candidate set, where the second node candidate set includes a third computing node deployed with the first target computing node and a container of the MapNode type not deployed;

[0041] According to the initial parameters of the multi-objective ant colony algorithm and the second pheromone update strategy, a second target computing node for deploying the ReduceNode type container is determined from the second node candidate set to determine a second scheduling scheme for the ReduceNode type container.

[0042] Optionally, the determining module is used to:

[0043] In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first computing nodes in the first node candidate set, and the remaining ants are randomly assigned to the second computing nodes in the first node candidate set;

[0044] The step of determining a scheduling scheme for a ReduceNode type container according to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes:

[0045] In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

[0046] Optionally, the determination module is used to: during the first iteration of the multi-objective ant colony algorithm, for each first computing node, determine whether to assign ants to the first computing node according to the proportion of the to-be-read data blocks included in the first computing node.

[0047] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the container scheduling method provided by any embodiment of the present invention when executing the program stored in the memory.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the container scheduling method provided by any embodiment of the present invention.

[0049] The technical solution provided by the embodiment of the present invention brings at least the following beneficial effects:

[0050] The container scheduling method provided by the embodiment of the present invention uses the container utility function indicating the allocation of the target container to the target computing node and the load balancing objective function indicating the degree of balance of the target container allocation to the target computing node as the pheromone update strategy of the multi-objective ant colony algorithm, and uses the pheromone update strategy to determine the optimal container scheduling method. The method uses the container utility function to preferentially decompose the application task to the computing node containing the data to be read, which is convenient for quickly reading the locally stored data from the computing node, can avoid network IO, and speed up the task running speed in the container; at the same time, by introducing the load balancing objective function, the problem of relatively concentrated container allocation that may be caused by only considering local read priority is solved, so that the container allocation is more balanced, and the unreasonable allocation is avoided, which causes a few computing nodes to be abnormally busy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0052] Figure 1 A schematic diagram showing a flow chart of a container scheduling method according to an embodiment of the present invention is shown;

[0053] Figure 2 A schematic diagram showing a flow chart of a container scheduling method according to another embodiment of the present invention;

[0054] Figure 3 A schematic diagram showing the structure of a container scheduling device according to an embodiment of the present invention is shown;

[0055] Figure 4A schematic diagram showing the structure of a container scheduling and analysis system according to an embodiment of the present invention is shown;

[0056] Figure 5 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0057] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0058] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0059] Figure 1 FIG. 2 is a flow chart showing a method for scheduling containers according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] Step S101: construct a container utility function and a load balancing objective function, and determine a pheromone update strategy of a multi-objective ant colony algorithm according to the container utility function and the load balancing objective function.

[0061] Step S102: Initialize the initial parameters of the multi-objective ant colony algorithm.

[0062] Step S103: Determine a container scheduling solution based on the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy.

[0063] Among them, the container utility function is used to indicate the utility of allocating the target container to the target computing node. The utility can be used to measure the effect of deploying the target container to the target computing node, such as reducing network IO and improving the operating efficiency of the container. Optionally, the greater the utility of deploying a container to a computing node, the better the effect of deploying the container to the computing node, such as being able to effectively reduce network IO and improve the operating efficiency of the container. In this embodiment, in order to reduce network IO, local reading of data files or data blocks is given priority to avoid pulling data files or data blocks from remote storage nodes. Therefore, when the container is deployed on a computing node that stores data blocks to be read, the utility is greater, and the utility of allocating the container to the computing node is proportional to the number of data to be read included in the computing node. Optionally, the utility of deploying the container on the node can be measured based on the number of data files or data blocks contained in the computing node.

[0064] The load balancing objective function is used to indicate the degree of balance in allocating the target container to the target computing node, and is used to measure the degree of balance in container allocation. The container scheduling method of the embodiment of the present invention deploys containers based on the strategy of prioritizing local reads, which easily leads to the concentration of containers on several specific computing nodes, thereby causing the IO and computing load of the specific computing node to be abnormally high, causing abnormal reading and writing of other files allocated on the computing node, affecting the stable operation of the distributed storage system. To avoid this problem, this embodiment controls the deployment of containers through the load balancing objective function, thereby ensuring that the load is evenly distributed to each computing node.

[0065] The multi-objective ant colony algorithm in this embodiment is intended to solve the optimization problem of maximizing the container utility function and minimizing the load balancing objective function. Among them, the Ant Colony Optimization (ACO) is an optimization algorithm that simulates the foraging behavior of ants and is mainly used to solve combinatorial optimization problems. The ant colony algorithm finds the optimal solution by simulating the behavior of ants. Specifically, when ants move forward on a path, they will choose the path they want to take based on the pheromones left by the ants that walked in front. The probability of choosing a path is proportional to the intensity of the secretions on the path. Therefore, the collective behavior of a group composed of a large number of ants actually constitutes a positive feedback phenomenon of learning information: the more ants walk along a path, the greater the possibility that the ants behind will choose the path. This mechanism enables the ant colony algorithm to find the optimal solution by imitating the behavior of ants. The multi-objective ant colony algorithm combines the ant colony algorithm and the multi-objective optimization algorithm, and aims to solve optimization problems with multiple objectives.

[0066] Optionally, the initial parameters of the multi-objective ant colony algorithm include but are not limited to: the number of ants (antNum), the pheromone matrix (pheromoneMatrix), the critical number of ants that adopt the random allocation strategy in one iteration (criticalPointMatrix), the proportion of pheromone increase q when the ant passes through each path, the proportion of pheromone attenuation p after each iteration, and the number of algorithm iterations numIter. Among them, the pheromone matrix is ​​a two-dimensional matrix used to record the pheromone concentration of container i assigned to the computing node a. The value of each element of the pheromone matrix is ​​1 at the time of initialization. During the iteration process of the ant colony algorithm, some ants need to follow the allocation strategy with the highest pheromone concentration, and the other part of the ants need to follow the random allocation strategy to find a new local optimal solution. The number of ants that adopt the random allocation strategy is controlled by criticalPointMatrix. For example, if the number of ants is set to 10, then in each iteration, 10 ants complete the allocation of all tasks. And the allocation process is carried out in the order of ant numbers from small to large (ants are numbered from 0). If criticalPointMatrix[0]=5, when assigning the 0th task, ants numbered 0 to 5 will assign tasks based on the pheromone concentration (i.e., assign the task to the node with the highest pheromone concentration in the row for processing), and ants numbered 6 to 9 will use random assignment (i.e., randomly assign the task to any node for processing).

[0067] According to the container utility function and load balancing objective function, the pheromone update strategy of the multi-objective ant colony algorithm is determined to control the update of the pheromone concentration. When the ant colony algorithm completes an iteration, the pheromone matrix needs to be updated, including:

[0068] 1) Reduce the concentration of all pheromones in the pheromone matrix by p% to simulate the volatilization of pheromones.

[0069] 2) Determine the optimal allocation scheme in this iteration based on the container utility function and the load balancing objective function, and increase the pheromone concentration corresponding to the scheme by q%. In each iteration of the multi-objective ant colony algorithm, AntNum ants will generate AntNum container scheduling schemes, and determine the path with the maximum container utility and the most balanced container allocation based on the container utility function and the load balancing objective function, and increase the pheromone concentration of this path by q%.

[0070] The container scheduling scheme provided by the embodiment of the present invention uses the container utility function and the load balancing objective function as the pheromone update strategy to control the update of the pheromone of each iteration of the ant colony algorithm, thereby determining the optimal container deployment scheme. Among them, the container utility function can ensure the priority data localization reading, and allocate the container to the computing node where the data to be read is located as much as possible, so that the application reads data from the local node as much as possible, and avoids accessing data from the remote node, thereby achieving the purpose of reducing network IO and improving job operation efficiency; the load balancing objective function can ensure that the load is evenly distributed to each computing node, avoiding system instability due to the high load of a specific computing node.

[0071] In an optional embodiment, the container scheduling method further includes: establishing resource constraint rules for containers and computing nodes, and determining a container scheduling solution based on initial parameters of a multi-objective ant colony algorithm, a pheromone update strategy, and the resource constraint rules.

[0072] In order to ensure successful container deployment during the container allocation process, it is necessary to ensure that the resource specifications of a single container are less than the available resources of each computing node in the current node set. For example, the resource specifications of a single container (such as the number of CPU cores and memory specifications) are less than the minimum value of the available resources of the computing nodes in the current node set. In an optional embodiment, in order to facilitate the rapid allocation of multiple containers, a fixed container resource specification can be set. On the other hand, from the perspective of the computing node, the available resources of the computing node (also referred to as the remaining allocatable resources) should be greater than the resource specifications of the container. For example, the number of available CPU cores and memory of the computing node need to be greater than the number of CPU cores and memory required by the container. Only when this condition is met can the container be allocated on the computing node.

[0073] In an embodiment of the present invention, multiple containers are allowed to be allocated to the same computing node. When using the multi-objective ant colony algorithm to determine the optimal container deployment solution, it is necessary to ensure that the resource constraint rules of the container and the computing node are met, so as to ensure successful deployment of the container.

[0074] Figure 2 FIG. 2 is a flow chart of a container scheduling method provided by another embodiment of the present invention. Figure 2 As shown, the method includes:

[0075] Step S201: According to the container operation diagram of the job to be executed, the containers are divided into MapNode type and ReduceNode type; the MapNode type container is the container before the shuffle occurs, and the ReduceNode type container is the container after the shuffle occurs.

[0076] In an optional implementation scenario, a large-scale data processing engine such as Spark can be used to process the jobs to be executed. The Spark computing engine completes the conversion between RDDs (Resilient Distributed Datasets) based on dependencies. In the processing of the Spark computing engine, the computing nodes of each stage only process part of the data of the task. If the next stage needs to rely on all the calculation results of the previous stage, all the calculation results of the previous stage need to be reintegrated and classified, that is, the data on all nodes in the cluster need to be reintegrated and classified through shuffle.

[0077] In this embodiment, containers are divided based on whether shuffle occurs, and are divided into: MapNode type and ReduceNode type. MapNode type containers are containers before shuffle occurs, and ReduceNode type containers are containers after shuffle occurs. For MapNode type containers, the container will focus on reading the original data in the storage system, reading and processing these original files or data blocks in parallel, and need to be passed to downstream nodes after processing. The main performance consumption of this type of container is in the reading of original data files. Since the network IO performance overhead is much higher than local IO, this type of container will give priority to local IO. For ReduceNode type containers, they mainly receive data read or processed from MapNode type containers, and their executable tasks include repartitioning, aggregation, association and other operations. The performance overhead of this type of container is mainly the IO overhead during the shuffle process.

[0078] In an optional embodiment, the Spark computing engine can be used to generate a container operation diagram that is interdependent in the job, and the containers can be divided according to the container operation diagram. For example, the container operation diagram is presented in the form of a directed acyclic graph (DAG). Each node in the directed acyclic graph represents a container. By analyzing the graph, the containers therein are divided into two categories based on whether shuffle occurs as the boundary. At the same time, based on the directed acyclic head, the number and time of shuffles in the job execution process can be identified, thereby identifying the number of times and the timing of calling the multi-objective ant colony algorithm to determine the container scheduling solution.

[0079] When modeling the allocation and scheduling of container sets before and after shuffle, it is necessary to consider the state transformation path of the container before and after shuffle, and determine the state related to container scheduling based on the state transformation path of the container. Optionally, the state transformation path of the container before and after shuffle is as follows: (1) Data_Input intensive state; (2) CPU intensive state; (3) shuffle_Output intensive state; (4) shuffle_Input intensive state; (5) CPU intensive state; (6) Data_Ouput intensive state. In the above state transformation, states (1), (2), and (3) belong to the states of MapNode type containers, and states (4), (5), and (6) belong to the states of ReduceNode type containers. Among the five states, the CPU intensive state is optimized and has a low correlation with the scheduling and allocation of the container. Its state transformation occurs after the IO of data loading, and its performance is related to the number of CPU cores allocated to the container. Since the data writing process has no correlation with the allocation and scheduling of containers, the Data_Output intensive state and the shuffle_Output intensive state do not need to be adjusted and optimized. Therefore, we can consider optimizing the Data_Input intensive state of MapNode type containers and the shuffle_Input intensive state of ReduceNode type containers.

[0080] Step S202: construct a first utility function and a second utility function. The first utility function is used to indicate the utility of allocating a container of the MapNode type to a target computing node. The second utility function is used to indicate the utility of allocating a container of the ReduceNode type to a target computing node.

[0081] In step S201, it is determined that the Data_Input intensive state of the MapNode type container can be optimized, and the data reading phase of the MapNode type container can be optimized. The data that needs to be read in the data reading phase is raw data, which has a large amount of data, rich and complex data types, and high data writing pressure. For MapNode type containers, the locations where the containers are allocated are usually coupled with the nodes of the distributed storage system. Therefore, in order to avoid reading data files or data blocks from distributed remote storage nodes, reduce network reading pressure, and improve container reading efficiency, priority is given to local reading of files, that is, deploying containers on computing nodes that store data to be read. Therefore, the utility of deploying a MapNode type container on a computing node can be represented by the number of waiting-to-read data (files to be read or data blocks to be read) contained in the computing node. The more data to be read contained in the computing node, the greater the utility of deploying the container on the computing node. Optionally, the first utility function corresponding to the MapNode type container is shown in the following formula (1):

[0082] C i (k) = count k (m) (1)

[0083] C i (k) represents the utility of allocating container i of type MapNode to computing node k, count k (m) indicates that the number of data to be read (eg, files to be read or data blocks to be read) in computing node k is m.

[0084] For the n MapNode type containers to be scheduled, the scheduling scheme must maximize the utility of each container, that is, maximize the following first objective function:

[0085]

[0086] Among them, t1 represents the target value of the scheduling scheme for n MapNode type containers. The larger t1 is, the better the utility of the container scheduling scheme is, and the better the effect of the scheme is.

[0087] In step S201, it is determined that the shuffle_Input intensive state of the ReduceNode type container can be optimized, that is, the shuffle_Input reading phase of the ReduceNode type container can be optimized. The core of this process is to optimize the cross-container data transmission in the shuffle. Since cross-container data transmission is divided into two categories, one is the transmission across containers within the same computing node, which is essentially local IO; the other is the transmission of data across computing nodes to remote containers, which is essentially network IO. The local IO speed of different containers within the same computing node is much higher than the network IO. Therefore, in the optimization process, it is necessary to consider coupling the ReduceNode type container with the MapNode type container and deploying it on the same computing node. Therefore, the utility of deploying the ReduceNode type container on the computing node can be represented by the number of connections between the ReduceNode type container and the MapNode type container in the same computing node. The larger the number of connections between the ReduceNode type container and the MapNode type container in the same computing node, the greater the utility of deploying the ReduceNode type container on the computing node. Optionally, the second utility function corresponding to the ReduceNode type container is shown in the following formula (3):

[0088] C j (k) = count k (i,s) (3)

[0089] C i(k) represents the utility of allocating container j of type ReduceNode to computing node k, count k (i, s) indicates that the number of connections between the MapNode type container deployed in the computing node k and the ReduceNode type container j is s.

[0090] Optionally, one or more containers of the MapNode type may be deployed in the same computing node, and one or more containers of the ReduceNode type may also be deployed. In this case, the number of connections between the ReduceNode type container and the MapNode type container in the same computing node is the total number of connections between the ReduceNode type container and the MapNode type container. As an example, if one MapNode type container and one ReduceNode type container are deployed in the same computing node, the number of connections between the ReduceNode type container and the MapNode type container in the computing node is 1. If one MapNode type container and two ReduceNode type containers are deployed in the same computing node, the number of connections between the ReduceNode type container and the MapNode type container in the computing node is 2. If two MapNode type containers and two ReduceNode type containers are deployed in the same computing node, the number of connections between the ReduceNode type container and the MapNode type container in the computing node is 4.

[0091] For the z ReduceNode type containers to be allocated, the scheduling scheme needs to maximize the utility of each container, that is, maximize the following objective function:

[0092]

[0093] Wherein, t2 represents the target value of the scheduling scheme of the z ReduceNode type containers. The larger the target value, the better the utility of the z ReduceNode type containers.

[0094] Step S203: construct a load balancing objective function. The load balancing objective function is used to indicate the degree of balance in allocating target containers to target computing nodes.

[0095] In step S202, the optimization of Data_Input and Shuffle_Input using local read is given priority, but this will lead to another serious problem. That is, in order to maximize the above objective function, the allocated MapNode type containers and ReduceNode type containers will be concentrated on several specific computing nodes, which may easily lead to the problem of relatively concentrated container allocation. The IO and computing load of a specific computing node may be abnormally high, causing abnormal reading and writing of other files allocated on the computing node, affecting the stable operation of the system. Based on this, the problem of uneven node load caused by overly concentrated container distribution must be eliminated. A load balancing objective function can be introduced here to measure the degree of balance of container allocation. As an example, the load balancing objective function is shown in the following formula (5):

[0096]

[0097] Among them, L p represents the load of the pth computing node, and L represents the average load of P computing nodes. In order to ensure that the load is evenly distributed to each computing node, the above objective function needs to be minimized.

[0098] Step S204: Establish resource constraint rules for containers and computing nodes.

[0099] In order to ensure successful container deployment during container allocation, it is necessary to ensure that the resource specifications of a single container are smaller than the available resources of each computing node in the current node set. For example, the resource specifications of a single container (such as the number of CPU cores and memory specifications) are smaller than the minimum value of the available resources of the computing nodes in the current node set. That is, the following constraints need to be met:

[0100]

[0101] Core i Indicates the number of CPU cores required for container i, Mem i Indicates the memory specification required by container i. Indicates the minimum number of available CPU cores in node set A. Indicates the minimum value of available memory in node set A.

[0102] In an optional embodiment, to facilitate rapid allocation of multiple containers, a fixed container resource specification may be set.

[0103] On the other hand, from the perspective of the computing node, the available resources (also called the remaining allocatable resources) of the computing node to which the container is to be allocated should be greater than the resource specifications of the container. For example, the number of available CPU cores and memory of the computing node must be greater than the number of CPU cores and memory required by the container. Only when this condition is met can the container be allocated on the computing node. For example, for a computing node a, the condition for allocating container i on the node is:

[0104]

[0105] Core_Remain a Indicates the number of available CPU cores of computing node a, Mem_Remain a Indicates the available memory of computing node a.

[0106] In an embodiment of the present invention, multiple containers are allowed to be allocated to the same computing node. When using the multi-objective ant colony algorithm to determine the optimal container deployment solution, it is necessary to ensure that the resource constraint rules of the container and the computing node are met, so as to ensure successful deployment of the container.

[0107] Step S205: Initialize the initial parameters of the multi-objective ant colony algorithm.

[0108] In this embodiment, the goal of the multi-objective ant colony algorithm is: given |I| containers and |A| nodes, where each container i∈I and node a∈A, establish a container-to-node allocation or mapping relationship I→A, allowing multiple containers to be allocated to one node. The initial parameters of the multi-objective ant colony algorithm include but are not limited to: the number of ants (antNum), the pheromone matrix (pheromoneMatrix), the critical number of ants using a random allocation strategy in one iteration (criticalPointMatrix), the proportion of pheromone increase q when the ant passes through each path, the proportion of pheromone attenuation p after each iteration, and the number of algorithm iterations numIter.

[0109] Step S206: Using the first utility function and the load balancing objective function as the pheromone first update strategy for determining the scheduling scheme of the MapNode type container, and determining the first scheduling scheme of the MapNode type container according to the initial parameters of the multi-objective ant colony algorithm, the pheromone first update strategy and the resource constraint rules.

[0110] Optionally, considering that the cluster size of computing nodes is generally large, if all computing nodes are used as candidate nodes, it will take a lot of time for the multi-objective ant colony algorithm to calculate the optimal container scheduling solution. Therefore, the complexity and time of the algorithm search can be reduced by narrowing the node candidate set to improve efficiency. Since the allocation of MapNode type containers needs to give priority to reading data from the local area, the nodes containing the data to be read are selected from all computing nodes as candidate set A. At the same time, in order to ensure that the container allocation will not be too concentrated on a specific node, computing nodes that do not contain data to be read are randomly selected and added to candidate set A. Therefore, an embodiment of the present invention also includes: constructing a first node candidate set, the first node candidate set includes a first computing node that stores data to be read indicating a job to be executed and a second computing node that does not store data to be read; according to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy, the first target computing node for deploying a MapNode type container is determined from the first node candidate set to determine the first scheduling solution for the MapNode type container.

[0111] In the first iteration of the multi-objective ant colony algorithm, in order to get close to the optimal scheduling solution and accelerate the iteration convergence speed, some ants (for example, 50% of the ants) can be assigned to the first computing node in the first node candidate set, and the remaining ants can be randomly assigned to the second computing node in the first node candidate set. More preferably, when some ants are assigned to the first computing node, it can be determined whether to assign the ant to the computing node based on the number of data blocks to be read contained in the first computing node. If the first computing node contains a large number of data blocks to be read, the ant is preferentially assigned to the computing node. Optionally, it can be determined whether to assign the ant to the computing node based on the proportion of data blocks to be read contained in each first computing node. Optionally, ants can be assigned based on the probability of the ant being assigned to the first computing node. The probability of ant t being assigned to the first computing node a is calculated according to the following formula:

[0112]

[0113] p(t,a) represents the probability that ant t is assigned to computing node a, u(a) represents the number of data blocks to be read contained in the first computing node a, ∑ a∈A μ(a) represents the total amount of data blocks to be read.

[0114] In each subsequent iteration of the multi-objective ant colony algorithm, ants are assigned as follows: According to the critical number criticalPointMatrix, the ants that follow the highest pheromone distribution and the randomly assigned ants are determined. For the ants that need to follow the highest pheromone distribution, the ants are assigned to the nodes with the highest pheromone concentration in this row according to the current pheromone matrix. For the ants that need to follow the random distribution, they are randomly assigned to any computing node in this row.

[0115] In each iteration of the multi-objective ant colony algorithm, the pheromone matrix is ​​updated once each iteration is completed. The process of updating the pheromone matrix includes:

[0116] 1) Reduce the concentration of all pheromones in the pheromone matrix by p% to simulate the volatilization of pheromones.

[0117] 2) Determine the optimal allocation scheme in this iteration according to the container utility function and the load balancing objective function, and increase the pheromone concentration corresponding to the scheme by q%. Specifically, the objective function value of each ant is calculated according to the first objective function and the load balancing objective function. Optionally, the original size and distribution of the data of the first objective function and the load balancing objective function can be balanced by regularization parameters λ1 and λ2, that is, the objective function value of each ant is calculated according to the following formula (9):

[0118] T=λ1t1+λ2t3 (9)

[0119] In each iteration, AntNum ants will generate AntNum allocation schemes, find the path that satisfies the maximum objective function according to formula (9), and increase the pheromone concentration of this path.

[0120] Step S206: Using the second utility function and the load balancing objective function as the pheromone second update strategy for determining the scheduling scheme of the ReduceNode type container, the scheduling scheme of the ReduceNode type container is determined according to the initial parameters of the multi-objective ant colony algorithm, the pheromone second update strategy, the resource constraint rules and the first scheduling scheme.

[0121] For ReduceNode type containers, ReduceNode type containers are preferentially deployed on computing nodes containing MapNode containers to reduce network IO. For example, the computing nodes where MapNode type containers are located can be used as search candidate set A, and nodes that do not contain MapNode type containers are randomly added to candidate set A. This ensures that ReduceNode type containers can also maximize the use of localized computing results of MapNode. Therefore, an embodiment of the present invention also includes: constructing a second node candidate set, the second node candidate set includes a third computing node deployed with a first target computing node and a third computing node not deployed with a MapNode type container; according to the initial parameters of the multi-objective ant colony algorithm and the second pheromone update strategy, the second target computing node for deploying the ReduceNode type container is determined from the second node candidate set to determine a second scheduling scheme for the ReduceNode type container.

[0122] In the first iteration of the multi-objective ant colony algorithm, in order to get close to the optimal scheduling plan and accelerate the iteration convergence speed, some ants (for example, 50% of the ants) can be assigned to the first target computing node in the second node candidate set, and the remaining ants can be randomly assigned to the third computing node in the second node candidate set.

[0123] In each subsequent iteration of the multi-objective ant colony algorithm, ants are assigned as follows: According to the critical number criticalPointMatrix, the ants that follow the highest pheromone distribution and the randomly assigned ants are determined. For the ants that need to follow the highest pheromone distribution, the ants are assigned to the nodes with the highest pheromone concentration in this row according to the current pheromone matrix. For the ants that need to follow the random distribution, they are randomly assigned to any computing node in this row.

[0124] In each iteration of the multi-objective ant colony algorithm, the pheromone matrix is ​​updated once each iteration is completed. The process of updating the pheromone matrix includes:

[0125] 1) Reduce the concentration of all pheromones in the pheromone matrix by p% to simulate the volatilization of pheromones.

[0126] 2) Determine the optimal allocation scheme in this iteration according to the container utility function and the load balancing objective function, and increase the pheromone concentration corresponding to the scheme by q%. Specifically, calculate the objective function value of each ant according to the second objective function and the load balancing objective function. Optionally, the original size and distribution of the data of the first objective function and the load balancing objective function can be balanced by regularization parameters λ3 and λ2, that is, the objective function value of each ant is calculated according to the following formula (10):

[0127] T=λ3t2+λ2t3 (10)

[0128] In each iteration, AntNum ants will generate AntNum allocation schemes, find the path that satisfies the maximum objective function according to formula (10), and increase the pheromone concentration of this path.

[0129] The container scheduling method provided in the embodiment of the present invention divides the containers into two types, MapNode and ReduceNode, based on the boundary before and after the shuffle. For the allocation of MapNode type containers, it is necessary to give priority to local files to be read. The allocation of ReduceNode type containers depends on the distribution of existing MapNode type containers. Therefore, for MapNode type containers and ReduceNode type containers, their allocation needs to be divided into two stages. First, the scheduling calculation of MapNode type containers is completed, and then the scheduling calculation of ReduceNode type containers is completed. Stage 1 is used to perform MapNode type container scheduling calculations, and the algorithm optimization objectives are formulas (2) and (5). Stage 2 is used to perform ReduceNode type container scheduling solution calculations, and the nodes allocated with MapNode type containers in stage 1 are taken as part of the candidate set, and the algorithm optimization objectives are changed to formulas (4) and (5). Both stages use the above multi-objective ant colony algorithm to iteratively search for the optimal value. The embodiment of the present invention divides the containers into MapNode type and ReduceNode type based on the boundary before and after the shuffle, and adopts a method of allocating MapNode type containers first and then allocating ReduceNode type containers, dynamically completing the two types of allocation in a two-stage manner, splitting the problem to be solved into two-stage local search problems, and taking into account both algorithm performance and search efficiency. When allocating the two types of containers, the embodiment of the present invention reduces the number of nodes that the ant colony algorithm needs to search by narrowing the candidate set of nodes to be allocated, thereby reducing the computational complexity. Furthermore, during the first iteration of the multi-objective ant colony algorithm, the embodiment of the present invention reduces the iterative search time by initializing an allocation plan that is close to the target, which facilitates the algorithm to approach the optimal solution faster.

[0130] In an optional embodiment, in order to improve the efficiency of container scheduling and avoid spending a lot of time waiting for the scheduling plan before the task starts, a pre-training method can be adopted, that is, the number and specifications of one or more containers can be predetermined to calculate the container allocation plan of the corresponding number and specifications, and pre-stored in the system. For example, a resident memory thread pool is started, and a thread is submitted in the thread pool for real-time extraction of the distribution of each file and its file data block in the distributed storage system. The thread can obtain information such as file block size and block distribution by establishing an RPC connection (Remote Procedure Call) with the NameNode node (NameNode maintains the metadata of the file system, including the directory tree of the file system, the location information of files and blocks, etc.). In addition, a thread is submitted in the thread pool to obtain the used and remaining CPU and memory resource metadata of the computing node. The thread obtains the above information by establishing an RPC with the ResourceManager. The metadata obtained for the thread can be persisted and saved in the kv (key-value) storage system. For the files read in the metadata, the optimal container scheduling plan is calculated for each file. That is, for a file, different container specifications (CPU, memory) and container quantities are pre-set, and the multi-objective ant colony algorithm is used under the current computing node resource distribution to calculate the allocation schemes of MapNode type containers and ReduceNode type containers respectively, and save them persistently in the kv storage system. It should be noted that if the computing node resources change, the container scheduling scheme of the file in the current environment must be modified. When the container needs to be started, the relevant scheduling plan can be directly queried and quickly returned to the ResourceManager (resource manager, responsible for resource allocation of the entire cluster, used to quickly start the relevant container). For example, when the client needs to initiate a container scheduling request, it issues a scheduling request specifying the container specification and the number of containers. In response to the scheduling request, the kv storage will be queried. If there is a relevant scheduling scheme, the container scheduling scheme will be handed over to the client, which is convenient for quickly querying the pre-stored scheme and reducing the waiting time before the task is executed. If there is no relevant scheduling scheme in the kv storage, the multi-objective ant colony algorithm is used under the current computing node resource distribution according to the container specifications and container capacity to calculate the allocation schemes of MapNode type containers and ReduceNode type containers respectively.

[0131] The container scheduling scheme provided by the embodiment of the present invention uses the container utility function and the load balancing objective function as the pheromone update strategy to control the update of the pheromone of each iteration of the ant colony algorithm, thereby determining the optimal container deployment scheme. Among them, the container utility function can ensure the priority data localization reading, and allocate the container to the computing node where the data to be read is located as much as possible, so that the application reads data from the local node as much as possible, and avoids accessing data from the remote node, thereby achieving the purpose of reducing network IO and improving job operation efficiency; the load balancing objective function can ensure that the load is evenly distributed to each computing node, avoiding system instability due to the high load of a specific computing node.

[0132] Figure 3 FIG. 1 shows a schematic diagram of the structure of a container scheduling device provided by an embodiment of the present invention. Figure 3 As shown, the container scheduling device 300 includes:

[0133] A construction module 301 is used to construct a container utility function and a load balancing objective function, and determine the pheromone update strategy of the multi-objective ant colony algorithm according to the container utility function and the load balancing objective function; wherein the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the load balancing objective function is used to indicate the degree of balance of allocating the target container to the target computing node;

[0134] Initialization module 302, used to initialize initial parameters of the multi-objective ant colony algorithm;

[0135] The determination module 303 is used to determine the container scheduling solution according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy.

[0136] Optionally, the determination module is also used to: establish resource constraint rules for containers and computing nodes; and determine a container scheduling solution based on initial parameters of the multi-objective ant colony algorithm, a pheromone update strategy, and resource constraint rules.

[0137] Optionally, the device further includes a container division module, which is used to: divide the containers into MapNode type and ReduceNode type according to the container operation diagram of the job to be executed; the MapNode type container is the container before the shuffle occurs, and the ReduceNode type container is the container after the shuffle occurs;

[0138] The construction module is used to: construct a first utility function and a second utility function; the first utility function is used to indicate the utility of allocating a container of the MapNode type to a target computing node, and the second utility function is used to indicate the utility of allocating a container of the ReduceNode type to a target computing node; the first utility function and the load balancing objective function are used as a pheromone first update strategy for determining a scheduling scheme for a container of the MapNode type, and the second utility function and the load balancing objective function are used as a pheromone second update strategy for determining a scheduling scheme for a container of the ReduceNode type.

[0139] Optionally, the determination module is used to: determine the first scheduling scheme for MapNode type containers based on the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy; determine the scheduling scheme for ReduceNode type containers based on the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy and the first scheduling scheme.

[0140] Optionally, determine whether the module is used to:

[0141] Constructing a first node candidate set, the first node candidate set including a first computing node storing data to be read indicated by a job to be executed and a second computing node not storing data to be read;

[0142] According to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy, a first target computing node for deploying a container of the MapNode type is determined from the first node candidate set to determine a first scheduling scheme for the container of the MapNode type;

[0143] Construct a second node candidate set, where the second node candidate set includes a third computing node deployed with the first target computing node and a container of the MapNode type not deployed;

[0144] According to the initial parameters of the multi-objective ant colony algorithm and the second pheromone update strategy, a second target computing node for deploying a ReduceNode type container is determined from the second node candidate set to determine a second scheduling scheme for the ReduceNode type container.

[0145] Optionally, determine whether the module is used to:

[0146] In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first computing node in the first node candidate set, and the remaining ants are randomly assigned to the second computing node in the first node candidate set;

[0147] According to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy and the first scheduling scheme, a scheduling scheme for a ReduceNode type container is determined, including:

[0148] In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

[0149] Optionally, the determination module is used to: during the first iteration of the multi-objective ant colony algorithm, for each first computing node, determine whether to assign ants to the first computing node according to the proportion of the to-be-read data blocks included in the first computing node.

[0150] The above device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the container scheduling method provided by the embodiment of the present invention.

[0151] Optionally, an embodiment of the present invention further provides a container scheduling system 500 , including a resource analysis module 501 , a modeling and calculation module 502 , and a query response module 503 .

[0152] Resource analysis module 501: This module extracts the distribution of each file and its file data blocks in the distributed storage system, and the used and remaining resources of each computing node in real time, to provide basic data for scheduling modeling. The execution process of this module is: start a resident memory thread pool to obtain metadata information of scheduling resources. Submit a thread in the thread pool to obtain block metadata information. The thread can obtain information such as file block size and block distribution by establishing an RPC connection with the NameNode node. In addition, submit a thread in the thread pool to obtain the used and remaining CPU and memory resource metadata of the computing node. The thread obtains the above information by establishing an RPC with the ResourceManager. The metadata obtained for the thread can be persisted in the kv storage system.

[0153] Modeling and calculation module 502: Use the container scheduling method provided by the embodiment of the present invention to perform modeling and calculation of container scheduling. In order to avoid spending a lot of time waiting for the scheduling plan before the task starts, it is recommended to adopt a pre-training method, that is, the number and specifications of one or more containers can be determined in advance to calculate the container allocation plan of the corresponding number and specifications, and pre-store it in the system. The execution process of this module is: for the files read from the metadata, the optimal container scheduling plan is calculated for each file. That is, for a file, different container specifications (CPU, memory) and container numbers are predetermined, and the container scheduling method provided by the embodiment of the present invention is used to calculate the allocation plan of MapNode and ReduceNode respectively under the current computing node resource distribution status, and persist it in the kv storage system. It should be noted that if the computing node resources change, the container scheduling plan of the file in the current environment needs to be modified, and the method proposed in this article is also used, but the previous optimal solution is still used when the solution is initialized, and then the new optimal solution is quickly calculated by this method.

[0154] Query response module 403: used to respond to users. When it is necessary to start a container, the relevant scheduling plan can be directly queried and quickly returned to the ResourceManager. The execution process of this module is: when the client needs to initiate a container scheduling request, it sends a scheduling request to the module specifying the container specification and the number of containers. The query response module will query the kv storage, and if there is a relevant scheduling plan, the container scheduling plan will be handed over to the client. If there is no relevant scheduling plan in the kv storage, the container specification and container capacity data will be submitted to the modeling and calculation module to calculate the container scheduling plan under the specification limit, and the calculation results will be submitted to the client.

[0155] Processor 601, communication interface 602, memory 603 and communication bus 604, wherein processor 601, communication interface 602, memory 603 communicate with each other via communication bus 604.

[0156] Memory 603, used for storing computer programs;

[0157] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0158] Constructing a container utility function and a load balancing objective function, and determining the pheromone update strategy of the multi-objective ant colony algorithm according to the container utility function and the load balancing objective function; wherein the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node, and the load balancing objective function is used to indicate the degree of balance of allocating the target container to the target computing node;

[0159] Initialize the initial parameters of the multi-objective ant colony algorithm;

[0160] A container scheduling scheme is determined according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy.

[0161] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0162] The communication interface is used for communication between the above terminal and other devices.

[0163] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0164] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0165] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the bandwidth audit method described in any one of the above embodiments.

[0166] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the bandwidth audit method described in any one of the above embodiments.

[0167] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0168] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0169] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A container scheduling method, characterized in that: include: Constructing a container utility function and a load balancing objective function, and determining the pheromone update strategy of the multi-objective ant colony algorithm according to the container utility function and the load balancing objective function; wherein the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node, and the load balancing objective function is used to indicate the degree of balance of allocating the target container to the target computing node; Initialize the initial parameters of the multi-objective ant colony algorithm; A container scheduling scheme is determined according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy.

2. The method according to claim 1, characterized in that The step of determining a container scheduling scheme according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy includes: Establish resource constraint rules for containers and computing nodes; A container scheduling scheme is determined according to the initial parameters of the multi-objective ant colony algorithm, the pheromone update strategy and the resource constraint rule.

3. The method according to claim 1, characterized in that The method further comprises: According to the container operation diagram of the job to be executed, the containers are divided into MapNode type and ReduceNode type; MapNode type containers are containers before shuffle occurs, and ReduceNode type containers are containers after shuffle occurs; The container utility function is constructed, including: Constructing a first utility function and constructing a second utility function; the first utility function is used to indicate the utility of allocating a container of a MapNode type to a target computing node, and the second utility function is used to indicate the utility of allocating a container of a ReduceNode type to a target computing node; The pheromone update strategy of the multi-objective ant colony algorithm is determined according to the container utility function and the load balancing objective function, including: using the first utility function and the load balancing objective function as a first pheromone update strategy for determining a scheduling scheme for a container of the MapNode type, and using the second utility function and the load balancing objective function as a second pheromone update strategy for determining a scheduling scheme for a container of the ReduceNode type.

4. The method according to claim 3, characterized in that The step of determining a container scheduling scheme according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy includes: Determine a first scheduling scheme for a container of type MapNode according to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy; A scheduling scheme for a ReduceNode type container is determined according to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy and the first scheduling scheme.

5. The method according to claim 4, characterized in that The step of determining a first scheduling scheme for a container of a MapNode type according to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy includes: Constructing a first node candidate set, the first node candidate set including a first computing node storing the to-be-read data indicated by the to-be-executed job and a second computing node not storing the to-be-read data; According to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy, determine the first target computing node for deploying the MapNode type container from the first node candidate set to determine a first scheduling scheme for the MapNode type container; The step of determining a scheduling scheme for a ReduceNode type container according to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes: Constructing a second node candidate set, where the second node candidate set includes a third computing node deployed with the first target computing node and a container of the MapNode type not deployed; According to the initial parameters of the multi-objective ant colony algorithm and the second pheromone update strategy, a second target computing node for deploying the ReduceNode type container is determined from the second node candidate set to determine a second scheduling scheme for the ReduceNode type container.

6. The method according to claim 5, characterized in that The step of determining a first scheduling scheme for a container of a MapNode type according to the initial parameters of the multi-objective ant colony algorithm and the first pheromone update strategy includes: In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first computing nodes in the first node candidate set, and the remaining ants are randomly assigned to the second computing nodes in the first node candidate set; The step of determining a scheduling scheme for a ReduceNode type container according to the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes: In the first iteration of the multi-objective ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

7. The method according to claim 6, characterized in that In the first iteration of the multi-objective ant colony algorithm, allocating some ants to the first computing node in the first node candidate set includes: In the first iteration of the multi-objective ant colony algorithm, for each first computing node, it is determined whether to allocate ants to the first computing node according to the proportion of the to-be-read data blocks included in the first computing node.

8. A container scheduling device, characterized in that: include: A construction module is used to construct a container utility function and a load balancing objective function, and determine the pheromone update strategy of the multi-objective ant colony algorithm according to the container utility function and the load balancing objective function; wherein the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node, and the load balancing objective function is used to indicate the degree of balance of allocating the target container to the target computing node; Initialization module, used to initialize the initial parameters of the multi-objective ant colony algorithm; The determination module is used to determine the container scheduling scheme according to the initial parameters of the multi-objective ant colony algorithm and the pheromone update strategy.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images