Virtual machine placement method, computer device and computer program product

Abstract

The present application is applicable to the field of computer technology, and provides a virtual machine placement method, computer equipment and computer program product, including: for a virtual machine placement task associated with a data center cluster, obtaining the placement attributes of a virtual machine to be placed; based on the topological structure of the data center cluster, constructing a three-channel matrix for characterizing the current resource state of the data center cluster; based on the three-channel matrix and the placement attributes of the virtual machine to be placed, executing a Markov decision process adapted to the virtual machine placement task to generate a target placement strategy; based on the target placement strategy, placing the virtual machine to be placed in the data center cluster. The present application can improve the overall communication quality of the data center by deploying virtual machines with topology awareness.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a virtual machine placement method, a computer device, and a computer program product. Background Art

[0002] With the development of cloud computing, virtual machine (VM) scheduling has become increasingly important. Cloud service providers use VMs to meet users' computing needs, but there are many challenges in allocating VMs to suitable servers. Efficient scheduling can not only improve processing capabilities, but also increase profits. However, as the scale and complexity of services grow, new challenges also arise: users now tend to submit a group of VMs that follow a specific service level agreement (SLA) as a service group. These VMs may need to be deployed in a decentralized manner to ensure high availability, or deployed in a centralized manner to reduce communication latency. The structure of cloud data centers has become more complex, including a multi-level topology consisting of network domains, racks, etc., and multiple non-uniform memory access (NUMA) architectures are widely used. This requires scheduling strategies that can adapt to the needs of VMs of different sizes. The "pay as you go" business model makes it difficult to predict future requests based on historical data, so the system must respond quickly to the need to create or delete VMs.

[0003] Deep reinforcement learning (DRL) provides a new approach to solve these problems, which can automatically optimize resource scheduling. However, among the existing methods, some placement strategies ignore the complexity of the actual topological structure of the data center, resulting in the mismatch between the placement of virtual machines and the actual placement scenario. Some placement strategies often only focus on resource utilization and ignore service quality, resulting in a serious decline in the service quality of the data center after the virtual machine is placed. Summary of the invention

[0004] In view of this, embodiments of the present application provide a virtual machine placement method, a computer device, and a computer program product, which can improve the overall service quality of a data center by deploying virtual machines with topology awareness.

[0005] A first aspect of an embodiment of the present application provides a virtual machine placement method, including:

[0006] For a virtual machine placement task associated with a data center cluster, obtain placement attributes of the virtual machine to be placed;

[0007] Based on the topological structure of the data center cluster, construct a three-channel matrix for characterizing the current resource status of the data center cluster;

[0008] Based on the three-channel matrix and the placement attributes of the virtual machine to be placed, executing a Markov decision process adapted to the virtual machine placement task to generate a target placement strategy;

[0009] Based on the target placement strategy, the virtual machine to be placed is placed in the data center cluster.

[0010] In an implementation of the first aspect, constructing a three-channel matrix for characterizing a current resource state of the data center cluster based on the topological structure of the data center cluster includes:

[0011] Based on the cluster constraints and the topological structure of the data center cluster, a first matrix representing the availability of memory nodes, a second matrix representing the server allocation status, and a third matrix representing the network domain allocation are respectively constructed;

[0012] The first matrix, the second matrix and the third matrix are used as a three-channel matrix for characterizing the current resource status of the data center cluster, and the first matrix, the second matrix and the third matrix have the same matrix size.

[0013] In an implementation of the first aspect, obtaining a placement attribute of the virtual machine to be placed includes:

[0014] Receiving a placement request for a virtual machine to be placed in a data center cluster;

[0015] The placement request is parsed to obtain placement attributes of the virtual machine to be placed, wherein the placement attributes include a virtual machine identifier, a service group number to which the virtual machine belongs, and resource units required for placing the virtual machine.

[0016] In an implementation of the first aspect, the step of executing a Markov decision process adapted to the virtual machine placement task based on the three-channel matrix and the placement attributes of the virtual machine to be placed to generate a target placement strategy includes:

[0017] Performing feature extraction on the three-channel matrix to obtain topological features of the data center cluster;

[0018] Flatten the topological feature, and concatenate it with the attribute vector of the placement attribute of the virtual machine to be placed, to obtain a concatenated feature vector;

[0019] A Markov decision process adapted to the virtual machine placement task is executed for the concatenated feature vector to generate a target placement strategy.

[0020] In an implementation of the first aspect, the Markov decision process adapted to the virtual machine placement task based on the concatenated feature vector generates a target placement strategy, including:

[0021] Performing action value evaluation on the concatenated feature vector through a duel network to obtain an action value vector, each element of which represents the value of placing the virtual machine to be placed in each memory node;

[0022] Based on the action value vector, action mask screening is performed to obtain a target placement strategy that meets the cluster constraint conditions.

[0023] In an implementation of the first aspect, the duel network includes a state value subnetwork and an action advantage subnetwork, and the duel network is used to perform action value evaluation on the concatenated feature vector to obtain an action value vector, including:

[0024] Through the state value sub-network, the state value of the concatenated feature vector is evaluated to obtain the state value;

[0025] Performing action advantage evaluation on the concatenated feature vector through the action advantage sub-network to obtain action advantage;

[0026] Based on the state value and the action advantage, an action value vector is obtained.

[0027] In an implementation of the first aspect, the method further includes: determining a policy reward for executing a placement strategy for the virtual machine to be placed based on a reward function of the Markov decision process, wherein the reward function is determined based on a communication quality of each service group in the data center cluster;

[0028] If the placement strategy does not meet the adjustment target of maximizing the strategy return, the placement strategy of the virtual machine to be placed is dynamically adjusted until the adjusted placement strategy meets the adjustment target, and the adjusted placement strategy is used as the target placement strategy.

[0029] In an implementation of the first aspect, the method further includes:

[0030] For each service group of the data center cluster, determining a communication distance between any two virtual machines in the service group;

[0031] Determine the reciprocal of the communication distance as the communication speed between the two virtual machines;

[0032] A weighted sum is taken of the communication speeds between all virtual machines in the targeted service group to obtain the communication quality of the targeted service group.

[0033] A second aspect of an embodiment of the present application provides a virtual machine placement device, including:

[0034] An acquisition module, used for acquiring placement attributes of a virtual machine to be placed for a virtual machine placement task associated with a data center cluster;

[0035] A construction module, configured to construct a three-channel matrix for characterizing a current resource status of the data center cluster based on a topological structure of the data center cluster;

[0036] A generation module, configured to execute a Markov decision process adapted to the virtual machine placement task based on the three-channel matrix and the placement attributes of the virtual machine to be placed, and generate a target placement strategy;

[0037] A placement module is used to place the virtual machine to be placed in the data center cluster based on the target placement strategy.

[0038] A third aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements the method described in the first aspect above.

[0039] A fourth aspect of the embodiments of the present application provides a computer program product, including a computer program, which, when executed, executes the method described in the first aspect above.

[0040] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0041] In the first aspect of the embodiment of the present application, a three-channel matrix is ​​used to represent the resource usage of the cluster and the distribution of the virtual machines of each service group. Based on the three-channel matrix and the placement attributes of the virtual machines to be placed, a Markov decision process for the virtual machine placement task is solved to obtain a target placement strategy. Finally, the target placement strategy is executed in the data center cluster to complete the virtual machine placement task. In this way, the target placement strategy is implemented through topology awareness, the communication delay and spatial fragmentation within the service group in the data center are reduced, and the overall communication quality of the data center is improved.

[0042] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a schematic diagram of the implementation process of the virtual machine placement method provided in the embodiment of the present application;

[0045] Figure 2 This is an example diagram of the topological structure of a data center cluster provided in an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of a three-channel matrix of resource states provided in an embodiment of the present application;

[0047] Figure 4 This is an example diagram of a method for constructing a three-channel matrix of resource status provided in an embodiment of the present application;

[0048] Figure 5 is a flow chart of a method for generating a target placement strategy provided in an embodiment of the present application;

[0049] Figure 6 is a flow chart of a method for executing a Markov decision process provided by an embodiment of the present application;

[0050] Figure 7 is a flow chart of a method for determining an action value vector provided in an embodiment of the present application;

[0051] Figure 8 It is a flow chart of a placement strategy generation method based on a reward function provided in an embodiment of the present application;

[0052] Fig. 9 is a flow chart of a virtual machine placement method based on deep reinforcement learning provided in an embodiment of the present application;

[0053] Fig.10 is a schematic diagram of a virtual machine placement device provided in an embodiment of the present application;

[0054] Fig.11 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0056] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0057] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0058] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0059] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0060] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] In one embodiment, Figure 1 As shown, a virtual machine placement method is provided, and the method is applied to a computer device (deploying a reinforcement learning agent) as an example for description, including the following steps S101 to S104:

[0062] Step S101, for a virtual machine placement task associated with a data center cluster, obtain placement attributes of a virtual machine to be placed.

[0063] First, the data center cluster is explained. The data center cluster includes at least one network domain Network, each network domain includes at least one NUMA architecture server, each server includes at least one NUMA node, each NUMA node includes at least one resource unit (also called resource slot), and the resource slot status includes idle or occupied, and occupied means being used by a virtual machine. Each virtual machine can occupy at least one resource slot. There is a strict subordinate relationship between the network domain, server and NUMA node.

[0064] For example, Figure 2 The topology of the data center cluster shown in the figure includes a service cluster C0 and two network domains D1 and D2. The network domain D1 includes two servers S1 and S2. The network domain D2 includes two servers S3 and S4. Each server includes two NUMA nodes.

[0065] Secondly, the virtual machine placement task is explained. The virtual machine placement task refers to the implementation of a virtual machine placement strategy in a data center with a multi-dimensional topology structure consisting of network domains, servers, and NUMA nodes to improve the overall service quality of the data center and meet users' needs for computing efficiency.

[0066] In the application, the placement attributes of the virtual machine to be placed are obtained by receiving the placement request of the virtual machine. The virtual machine placement attributes include the virtual machine ID, the service group number to which it belongs, and the number of resource units required to be occupied in the memory node NUMA. The placement attributes are represented in vector form, such as the placement attribute {11, 1, 2} in vector form, where 11 represents the virtual machine ID, 1 represents the service group number to which the virtual machine to be placed belongs, and 2 represents the number of resource units required to be occupied in the memory node. It should be noted that there is a communication requirement between virtual machines in the same service group, so they should be placed as close as possible to reduce communication delay and improve service quality.

[0067] Step S102: constructing a three-channel matrix for characterizing the current resource status of the data center cluster based on the topological structure of the data center cluster.

[0068] In the application, in order to fully describe the topological structure of the data center cluster, a multi-channel matrix can be used to represent the distribution and resource usage of the service group. The number of channels of the multi-channel matrix is ​​determined by the type of hardware in the data center cluster. The data center cluster in this embodiment includes three types of hardware: network domain, server and memory node, and the number of channels is three, that is, a three-channel matrix. The three-channel matrix includes the state matrix D of the network domain, the state matrix S of the server and the state matrix N of the memory node. The data center cluster is represented as c:D×S×N×d, which means that the data center cluster is divided into D network domains, each network domain contains S NUMA architecture servers, each server contains N NUMA nodes (N=2 is fixed in this article), and d represents the resource feature dimension of each NUMA node, including two indicators of CPU and memory. Among them, the state matrix D of the network domain represents the network domain allocation of the cluster, the state matrix S of the server represents the server allocation in the cluster, and the state matrix N of the memory node represents the availability of the memory node in the cluster. The availability of the memory node refers to whether the resource slot of the memory node is occupied by the virtual machine. The matrix sizes of the three state matrices in the three-channel matrix are the same. The three-channel matrix can clearly show the distribution and resource usage of hardware resources (network domains, servers, NUMA) in the data center cluster.

[0069] For example, Figure 3 As shown, a 2×3×2 cluster returns the resource distribution status, where the N matrix represents the VM distribution of all NUMA nodes and resource slots in the cluster, and the S matrix and the D matrix are normalized. The state matrix N of the NUMA node is an L×U matrix, where L is the unit length of the maximum resource that can be placed in a NUMA (that is, the number of resource slots included in a single NUMA), and U is the total number of NUMA nodes in the data center cluster. Each element of the state matrix N corresponds to a resource slot of a specific NUMA, and indicates whether the current state of the resource slot is idle or being used by a virtual machine. For example, when a virtual machine VM that requires 2 resource units (slots) is created in a specified NUMA, the two elements of the NUMA (column) in the state matrix N with a value of 0 (in idle state) are assigned a value of w, indicating a transition from idle to being used. Among them, w is the service group number to which the virtual machine VM belongs. The server state matrix S represents the server allocation in the cluster,

[0070] Step S103 , based on the three-channel matrix and the placement attributes of the virtual machine to be placed, a Markov decision process adapted to the virtual machine placement task is executed to generate a target placement strategy.

[0071] In applications, the virtual machine placement task is essentially a decision-making process for virtual machine placement under uncertain data center cluster states. It has the Markov property and can be modeled as a Markov decision process (MDP) and solved through a preset reinforcement learning method to obtain a virtual machine placement strategy. By constructing an MDP for virtual machine placement tasks, the uncertainty in virtual machine placement can be effectively dealt with, especially when facing dynamically changing loads, and a flexible and efficient virtual machine placement strategy can be provided.

[0072] It should be noted that the solution process of the virtual machine placement task needs to meet the cluster constraints of the data center. The cluster constraints include space constraints. Consistency Constraints Time Constraints and service group constraints Among them, the space constraint It is required that at any time t, the remaining amount of resources of the NUMA node cannot be negative. The creation and deletion of virtual machines must be performed in the same location. Time constraints The online characteristics of the problem are described: once a virtual machine placement or deletion request does not receive a response (i.e., no server has sufficient resources to process the virtual machine placement request), the request is terminated and stopped. In addition, since the virtual machine placement strategy is service-oriented, virtual machines only communicate within the service group, and there is no communication requirement between service groups. The calculation of communication quality depends on the correct grouping of virtual machines. Therefore, the service group constraint is introduced. Service group constraints require that virtual machines must be correctly assigned to specified service groups when placed, and the calculation of communication efficiency must be based on the grouping records of the service group to ensure compliance with actual topology communication rules.

[0073] Based on the above description of cluster constraints, the entire virtual machine placement task solution process can be expressed as the following constrained optimization problem:

[0074]

[0075]

[0076] Among them, x is a decision variable, which represents the virtual machine placement strategy. Qos(x) represents the quality of service (QoS) of the entire data center cluster after the virtual machine placement strategy is executed. The service quality of the data center cluster is characterized by the communication quality of the data center cluster, that is, the virtual machine placement strategy is adjusted with the goal of maximizing the communication quality of the cluster.

[0077] In the application, since it is very difficult to directly solve the constrained optimization problem, a Markov decision process (MDP) adapted to the virtual machine placement task is constructed, and the deep reinforcement learning (DRL) framework is used to solve it.

[0078] The relevant content of Markov decision process is explained. Markov decision process MDP is a mathematical framework used to describe the process in which an agent (Agent) interacts with the environment through a series of actions (Action) in an environment to maximize the cumulative reward (Reward).

[0079] Based on the above description, the agent of the virtual machine placement task is used to determine the target placement strategy for the virtual machine to be placed based on the cluster state of the data center cluster and the placement attributes of the virtual machine to be placed. The environment corresponding to the MDP of the virtual machine placement task is represented by the cluster state of the data center cluster. The action of the MDP is the virtual machine placement operation. The reward mechanism of the MDP is designed to encourage the placement of virtual machines in the same service group as close as possible, while reasonably placing different service groups to improve resource utilization, increase the number of placements, and thus maximize the sum of the communication qualities of all service groups in the data center. Based on the reward mechanism, the immediate reward for executing the placement strategy (determined based on the reward function) is set to the communication quality of the service group to which the virtual machine to be placed belongs. The strategy return of the MDP of the virtual machine placement task is the sum of the communication qualities of all service groups in the data center cluster after the execution of a single placement strategy.

[0080] In the application, in order to improve the perception ability of the topological structure of the data center, a convolutional neural network can be introduced to extract the topological features of the current state of the data center cluster. Specifically, the three-channel matrix of the cluster state is first input into the convolutional neural network CNN for feature extraction (convolution and pooling operations), and then the topological features output by the convolutional network are flattened and connected with the placement attributes of the virtual machine represented by the vector to generate a concatenated feature vector, which integrates the topological structure information of the data center cluster and the placement attributes of the virtual machine. The concatenated feature vector is used as the input information of the Markov process to solve the placement strategy for the virtual machine to be placed through the preset reinforcement learning method of the Markov process.

[0081] In the application, the preset reinforcement learning method for solving the Markov process adapted to the virtual machine placement task can be a deep reinforcement learning method based on the Rainbow paradigm. This method integrates dual DQN (DDQN), dueling network Dueling Network and noise network Noisy Network. Among them, the introduction of Dueling Network can improve the performance and learning efficiency of the model when processing redundant actions by decomposing the Q value into state value and advantage value; at the same time, the addition of NoisyNetwork can replace the epsilon-greedy strategy with learnable noise, thereby enhancing the exploration ability and reducing the dependence on hyperparameters.

[0082] Dueling Network consists of two independent sub-networks: the state value network (V network) and the advantage function network (A network). The V network is used to estimate the state value (State Value), and the other is used to estimate the advantage function (Advantage Function). The dueling network can separate the state value and the action advantage, and output the action value vector Q. Q is a vector, and each element in the vector represents the action value of placing the virtual machine on each NUMA in the data center cluster. The action mask is used in the Noisy Network to filter legal actions to ensure that the selected action meets the above cluster constraints required when placing virtual machines in the data center cluster, so as to avoid unreasonable placement operations for virtual machines.

[0083] In the application, in the process of solving the MDP for virtual machine placement, the reward function design aims to encourage "compact" placement of virtual machines in the same service group by optimizing the placement strategy of the virtual machines to reduce internal communication delays. In addition, by rationally allocating the resources of different service groups, the overall resource utilization and placement efficiency are improved, thereby maximizing the overall communication efficiency of the data center. Based on this, the optimization goal (i.e., adjustment goal) in the MDP process is set to maximize the sum of the communication qualities of all service groups in the data center cluster. The communication quality of a service group is the weighted sum of the communication speeds between all virtual machines in the service group. The communication speed between any two virtual machines in the same service group is determined based on the communication distance. The communication distance is related to the hardware topology distance of the data center cluster. The communication speed is inversely proportional to the communication distance. In the application, virtual machine a is defined to another virtual machine a in the same service group. j The communication speed is the reciprocal of the topological space distance between the two.

[0084] In the application, the reward function in the MDP process of setting the virtual machine placement task is as follows:

[0085]

[0086] Where k is the number of service groups in the data center cluster. i is the communication quality of the ith service group, which is represented by the weighted sum of the communication speeds between all virtual machines in the ith service group. Virtual machines in the same service group can communicate with each other, but virtual machines in different service groups cannot communicate with each other. In the virtual machine placement scenario, the policy return of the data cluster after executing each placement strategy is equal to the sum of the communication speeds within all service groups in the data center cluster. The optimal placement strategy can maximize the policy return, which represents the optimal overall service quality of the data center cluster. Q i The calculation formula is as follows:

[0087]

[0088] Among them, ∑ m≠j V(a j ,a m ), represents any virtual machine a in a service group j With other virtual machines a in the same service group i m The sum of the communication speeds.

[0089] Step 104: placing the virtual machine to be placed in the data center cluster based on the target placement strategy.

[0090] In the application, the target placement strategy is executed in the data center cluster to complete the placement operation for the virtual machine to be placed, and obtain the target placement position of the virtual machine to be placed in the data center cluster.

[0091] In this embodiment, a three-channel matrix is ​​used to represent the resource usage of the cluster and the distribution of the virtual machines in each service group. The Markov decision process for the virtual machine placement task is solved based on the three-channel matrix and the placement attributes of the virtual machines to be placed to obtain a target placement strategy. Finally, the target placement strategy is executed in the data center cluster to complete the virtual machine placement task. In this way, the target placement strategy is implemented through topology awareness, the communication delay and spatial fragmentation within the service group in the data center are reduced, and the overall communication quality of the data center is improved.

[0092] In one embodiment, Figure 4 As shown, the implementation process of step S102 includes the following steps S201 to S202:

[0093] Step S201, based on cluster constraints and the hardware topology of the data center cluster, construct a first matrix representing the availability of memory nodes, a second matrix representing server allocation, and a third matrix representing network domain allocation.

[0094] In the application, the cluster state of the data center cluster is represented by a three-channel matrix, which includes a first matrix representing the availability of memory nodes, a second matrix representing the server allocation status, and a third matrix representing the network domain allocation.

[0095] Step S202: The first matrix, the second matrix and the third matrix are used as a three-channel matrix for characterizing the cluster state of the data center cluster, and the matrix sizes of the first matrix, the second matrix and the third matrix are the same.

[0096] In this embodiment, the cluster state of the data center cluster is represented by a three-channel matrix, which can improve the perception capability of topology information.

[0097] In one embodiment, the implementation process of step S101 is as follows: receiving a placement request for a virtual machine to be placed in a data center cluster; parsing the placement request to obtain placement attributes of the virtual machine to be placed, the placement attributes including the virtual machine identifier, the service group number to which the virtual machine belongs, and the resource units required to place the virtual machine.

[0098] In the application, the placement attributes of the virtual machine to be placed are obtained through a placement request for the virtual machine, which includes the virtual machine identifier of the virtual machine to be placed, the service group number to which the virtual machine belongs, and the resource units required to place the virtual machine.

[0099] In one embodiment, Figure 5 As shown, the implementation process of step S103 includes the following steps S301 to S303. Among them:

[0100] Step S301, extracting features from the three-channel matrix to obtain topological features of the data center cluster.

[0101] In the application, the three-channel matrix represents the resource status of the data center cluster, that is, the resource usage of the data center cluster and the distribution of virtual machines in each service group. The topological features of the data center cluster can be extracted from the three-channel matrix with the help of the feature extraction capability of the convolutional neural network. Specifically, the convolutional neural network CNN can be introduced to extract the features of the three-channel matrix to obtain the topological features of the data center cluster, thereby enhancing the perception of the topological structure of the data center cluster.

[0102] Step S302 : Flatten the topological features and concatenate them with the attribute vector of the placement attribute of the virtual machine to be placed to obtain a concatenated feature vector.

[0103] In the application, the topological features are flattened, and the flattened topological features are concatenated with the attribute vector of the placement attribute of the virtual machine to be placed to obtain a concatenated feature vector.

[0104] Step S303 , generating a target placement strategy based on the concatenated feature vector and a Markov decision process adapted to the virtual machine placement task.

[0105] In the application, the Markov decision process can be implemented based on the deep reinforcement learning method. Among them, the deep reinforcement learning method can be a reinforcement learning method based on the Rainbow paradigm to implement the Markov decision process for the virtual machine placement task. This reinforcement learning method is to add the Dueling Network and the Noisy Network on the basis of DDQN. Among them, the introduction of Dueling Network can evaluate the action value in the placement strategy by separating the state value function and the advantage function in the Markov decision process. Specifically, the state value of the virtual machine placement strategy is determined by separating the state value function, and the advantage value of the placement strategy is determined by the advantage function. The action value of the placement strategy is determined based on the state value and advantage value of the placement strategy. The action value corresponding to the placement strategy is an m-dimensional vector, where m is the number of memory node NUMAs in the topological structure of the data center cluster, and each element in the vector represents the action value of placing the virtual machine on the NUMA.

[0106] After the output Q value of the duel network, the action mask is used to filter the legal actions through the Noisy Network to ensure that the placement actions in the placement strategy comply with the cluster environment constraints, thereby avoiding unreasonable operations and invalid resource selection.

[0107] In this embodiment, based on the topological features of the topological structure including the data center cluster and the placement attributes of the virtual machines to be placed, a Markov decision process is executed to obtain a target placement strategy, which can achieve the optimal placement strategy through topology awareness.

[0108] In one embodiment, Figure 6 As shown, the implementation process of step S303 includes the following steps S401 to S402. Among them:

[0109] Step S401, performing action value evaluation on the concatenated feature vector through a duel network to obtain an action value vector, wherein each element of the action value vector represents the value of placing the virtual machine to be placed in each memory node.

[0110] In the application, the Markov decision process for the virtual machine placement task can be implemented by a reinforcement learning method based on the Rainbow paradigm. The reinforcement learning method is to add a dueling network and a noise network to DDQN. Based on the Dueling Network, the action value in the placement strategy is evaluated, and the obtained action value is an m-dimensional vector. The Dueling Network includes two independent sub-networks: the state value network (V network) and the advantage function network (A network). The V network is used to estimate the state value (State Value), and the A network is used to estimate the action advantage. The dueling network can separate the state value and the action advantage, and output the action value vector Q. Q is a vector, and each element in the vector represents the action value of placing the virtual machine on each NUMA in the data center cluster.

[0111] Step S402: Based on the action value vector, action mask screening is performed to obtain a target placement strategy that meets the cluster constraint conditions.

[0112] In the application, the reinforcement learning method also includes the Noisy Network. The Noisy Network uses an action mask to screen legal actions to ensure that the selected actions meet the above cluster constraints required when placing virtual machines in the data center cluster, so as to avoid unreasonable placement operations for virtual machines.

[0113] In this embodiment, by using a reinforcement learning method based on the Rainbow paradigm, especially combining a dueling network and a noisy network, the performance of the virtual machine placement task can be effectively improved.

[0114] In one embodiment, Figure 7 As shown, the implementation process of step S401 includes the following steps S501 to S503:

[0115] Step S501, through the state value sub-network, the state value of the spliced ​​feature vector is evaluated to obtain the state value.

[0116] In the application, the concatenated feature vector is input into the state-value subnetwork for processing to estimate the value of the cluster state represented by the three-channel matrix.

[0117] Step S502: Perform action advantage evaluation on the concatenated feature vector through the action advantage sub-network to obtain the action advantage.

[0118] In the application, the concatenated feature vector is input into the action advantage sub-network for processing to estimate the action advantage of performing the virtual machine placement operation in the cluster state represented by the three-channel matrix.

[0119] Step S503, obtaining an action value vector based on the state value and the action advantage.

[0120] In the application, the Dueling DQN evaluates the value Q of the action through the state value sub-network V(s; θ) and the advantage function sub-network D(s, a; θ). The calculation of the Q value can be used using the formula:

[0121]

[0122] In this embodiment, the Q value can be decomposed into state value and action advantage through the state value sub-network and the action advantage sub-network, thereby improving the performance and learning efficiency of the model when processing redundant actions.

[0123] In one embodiment, Figure 8 As shown, the implementation process of step S401 includes the following steps S601 to S602. Among them:

[0124] Step S601, based on a reward function, determining a policy reward for executing a placement strategy for a virtual machine to be placed, wherein the reward function is determined based on the communication quality of each service group in the data center cluster.

[0125] In applications, virtual machine placement is service-oriented. Only virtual machines in the same service group can communicate, and there is no communication between different service groups. Therefore, the purpose of virtual machine placement is to place virtual machines in the same service group as close as possible, and at the same time, to reasonably place different service groups to improve resource utilization, increase the number of placements, and thus maximize the sum of the communication qualities of all service groups in the data center. Therefore, the reward function of the Markov decision process that matches the virtual machine placement task is determined based on the communication quality of each service group in the data center cluster. The communication quality of each service group refers to the sum of the communication speeds between all virtual machines in the service group. Among them, the formula of the reward function is as follows:

[0126]

[0127] Where k is the number of service groups in the data center cluster. i is the communication quality of the ith service group, which is represented by the weighted sum of the communication speeds between all virtual machines in the group.

[0128] In the application, the strategy reward after executing each placement strategy is determined by the above reward function. It can be determined that the strategy reward is equal to the sum of the communication speeds in all service groups in the data center cluster after full placement.

[0129] Step S602: If the placement strategy does not meet the adjustment target of maximizing the strategy return, the placement strategy of the virtual machine to be placed is dynamically adjusted until the adjusted placement strategy meets the adjustment target, and the adjusted placement strategy is used as the target placement strategy.

[0130] In the application, the dynamic adjustment in the Markov decision process is carried out with the goal of maximizing the return. In the virtual machine placement task, the essence of maximizing the return is to maximize the cumulative reward, that is, to achieve the sum of the communication quality of all service groups in the data center cluster. Therefore, if the placement strategy does not meet the adjustment goal of maximizing the strategy return, the placement strategy of the virtual machine to be placed is dynamically adjusted until the adjusted placement strategy meets the adjustment goal, and the adjusted placement strategy is used as the target placement strategy.

[0131] In this embodiment, the placement strategy of the virtual machine can be adjusted through the reward function to maximize the strategy reward.

[0132] In one embodiment, the implementation process of determining the communication quality of each service group is as follows: for each service group in the data center cluster, determine the communication distance between any two virtual machines in the service group; determine the reciprocal of the communication distance as the communication speed between the two virtual machines; and perform weighted summation of the communication speeds between all virtual machines in the service group to obtain the communication quality of the service group.

[0133] In the application, the communication quality of the service group is determined by the communication speed between all virtual machines in the service group. The communication speed between any two virtual machines in the same service group is determined based on the communication distance between the two virtual machines, which refers to the topological space distance between the two virtual machines. The reciprocal of the communication distance is determined as the communication speed between the two virtual machines. The communication speed between all virtual machines in the service group is weighted and summed to obtain the communication quality of the service group. The calculation formula for the communication quality of a single service group is as follows:

[0134]

[0135] Where j is the index of the current virtual machine in service group i, n i is the number of virtual machines in service group i, V(a j ,a m ) indicates virtual machine a j With other virtual machines a in service group i m On this basis, the reward function can be designed as the sum of the communication rates between virtual machine j and other virtual machines in the service group to which it belongs when placing virtual machine j. Specifically, the reward function r j The calculation formula is as follows:

[0136]

[0137] The communication distance is often inversely proportional to the communication speed. Define virtual machine a j To another virtual machine a in the same group m The communication speed is the inverse of the topological space distance between the two, that is, V(a j ,a m ) is calculated as follows:

[0138]

[0139] Therefore, for virtual machine a j After executing the placement strategy, the immediate feedback (i.e. reward) is r j :

[0140]

[0141] In this embodiment, the reward mechanism determined based on the communication quality of the group to which the virtual machine belongs can encourage each service group to be placed reasonably, thereby significantly improving the communication quality of the data center.

[0142] For example, the network model result of solving the Markov decision process corresponding to the virtual machine placement task is as follows: Fig. 9 As shown in the figure, the process of solving the virtual machine placement task and obtaining the virtual machine placement strategy by using the reinforcement learning method based on the Rainbow paradigm. In the application, the reinforcement learning model based on the Rainbow paradigm includes the convolutional neural network CNN, the dueling network and the noise network. By introducing a CNN neural network model to perform topological feature extraction of the topological structure of the data center cluster, the introduction of the Dueling Network can decompose the Q value into a state value and an advantage value, thereby improving the performance and learning efficiency of the model when processing redundant actions. At the same time, by adding the Noisy Network, the epsilon-greedy strategy can be replaced by learnable noise, thereby enhancing the exploration ability and reducing the dependence on hyperparameters. The combination of these two technologies can significantly improve the stability and exploration efficiency of DDQN.

[0143] First, the three-channel matrix C that represents the cluster state t <D|S|N> The input is fed into the convolutional neural network (CNN) for feature extraction to enhance the perception of the topological structure of the data center cluster. The output topological features and placement attributes are then flattened and concatenated into a concatenated feature vector. The concatenated feature vector is then fed into the Dueling DQN architecture, which evaluates the value of the action through the state value subnetwork V(s; θ) and the action advantage subnetwork D(s, a; θ).

[0144] In this embodiment, a deep reinforcement learning algorithm with CNN is proposed based on the Rainbow paradigm, and CNN is used to enhance the perception of topology. Dueling Network can decompose Q value into state value and advantage value, improving the performance and learning efficiency of the model when dealing with redundant actions; Noisy Network can replace the epsilon-greedy strategy with learnable noise, thereby enhancing exploration ability and reducing dependence on hyperparameters. The combination of these two networks can significantly improve the stability and exploration efficiency of DDQN.

[0145] In summary, in the embodiment of the present application, a method for placing virtual machines in a data center with a multi-dimensional topology is used, taking into account service groups and multi-NUMA architecture servers, a state representation form is cleverly designed, and a convolutional neural network is introduced to significantly enhance the perception of the cluster topology structure; for the improvement of communication quality, the design of the topology distance cost in the industry is referred to, and the reward mechanism for training the reinforcement learning agent strategy is redesigned according to the formalized multi-constraint optimization formula; by introducing the idea of ​​dynamic programming, the problem scenario is modeled as a Markov decision MDP process, and a deep reinforcement learning algorithm with CNN is proposed based on the Rainbow paradigm to solve the problem, which can solve the problem well. The reasonable allocation of resources is achieved, and the user service quality is effectively improved.

[0146] The present application also provides a virtual machine placement device for executing the steps in the above virtual machine placement method embodiment. Fig.10 As shown, the virtual machine placement device 900 provided in the embodiment of the present application includes:

[0147] The acquisition module 910 is used to acquire the placement attributes of the virtual machines to be placed for the virtual machine placement tasks associated with the data center cluster.

[0148] The construction module 920 is used to construct a three-channel matrix for representing the current resource status of the data center cluster based on the topological structure of the data center cluster.

[0149] The generating module 930 is used to execute a Markov decision process adapted to the virtual machine placement task based on the three-channel matrix and the placement attributes of the virtual machine to be placed, and generate a target placement strategy.

[0150] The placement module 940 is used to place the virtual machines to be placed in the data center cluster based on the target placement strategy.

[0151] In one embodiment, the construction module is also used to construct a first matrix characterizing the availability of memory nodes, a second matrix characterizing the server allocation status, and a third matrix characterizing the network domain allocation based on cluster constraints and the topology of the data center cluster; the first matrix, the second matrix and the third matrix are used as a three-channel matrix for characterizing the current state of the data center cluster, and the matrix sizes of the first matrix, the second matrix and the third matrix are the same.

[0152] In one embodiment, the acquisition module is also used to receive a placement request for the virtual machine to be placed in the data center cluster; parse the placement request to obtain the placement attributes of the virtual machine to be placed, the placement attributes including the virtual machine identifier, the service group number to which the virtual machine belongs, and the resource units required to place the virtual machine.

[0153] In one embodiment, the generation module is also used to extract features from the three-channel matrix to obtain topological features of the data center cluster; flatten the topological features and splice them with the attribute vector of the placement attributes of the virtual machine to be placed to obtain a spliced ​​feature vector; for the spliced ​​feature vector, execute a Markov decision process adapted to the virtual machine placement task to generate a target placement strategy.

[0154] In one embodiment, the generation module is also used to perform action value evaluation on the spliced ​​feature vector through a duel network to obtain an action value vector, each element of the action value vector represents the value of placing the virtual machine to be placed in each memory node; based on the action value vector, action mask screening is performed to obtain a target placement strategy that meets the cluster constraint conditions.

[0155] In one embodiment, the generation module is also used to evaluate the state value of the spliced ​​feature vector through the state value sub-network to obtain the state value; evaluate the action advantage of the spliced ​​feature vector through the action advantage sub-network to obtain the action advantage; and obtain the action value vector based on the state value and the action advantage.

[0156] In one embodiment, the generation module is also used to determine the policy reward of executing the placement strategy of the virtual machine to be placed based on the reward function of the Markov decision process; if the placement strategy does not meet the adjustment goal of maximizing the policy reward, the placement strategy of the virtual machine to be placed is dynamically adjusted until the adjusted placement strategy meets the adjustment goal, and the adjusted placement strategy is used as the target placement strategy.

[0157] In one embodiment, the generation module is also used to determine, for each service group in the data center cluster, the communication distance between any two virtual machines in the service group; determine the reciprocal of the communication distance as the communication speed between the two virtual machines; and perform weighted summation of the communication speeds between all virtual machines in the service group to obtain the communication quality of the service group.

[0158] In applications, each module in the virtual machine placement device may be a software program module, or may be implemented by different logic circuits integrated in a processor, or may be implemented by multiple distributed processors.

[0159] Fig.11 A schematic diagram of the structure of a computer device for a virtual machine placement method provided in an embodiment of the present application. Fig.11 As shown, the computer device 1000 of this embodiment includes: at least one processor 1010 ( Fig.11 Only one is shown in the figure) a processor, a memory 1020, and a computer program 1030 stored in the memory 1020 and executable on the at least one processor 1010, and when the processor 1010 executes the computer program 1030, the steps in any of the above-mentioned exception handling method embodiments are implemented.

[0160] The computer device 1000 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art will appreciate that Fig.11 The computer device 1000 is merely an example and does not constitute a limitation on the computer device 1000 . The computer device 1000 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device 1000 may also include input and output devices, network access devices, etc.

[0161] The processor 1010 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0162] In some embodiments, the memory 1020 may be an internal storage unit of the computer device 1000, such as a hard disk or memory of the computer device 1000. In other embodiments, the memory 1020 may also be an external storage device of the computer device 1000, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1000. Further, the memory 1020 may also include both an internal storage unit of the computer device 1000 and an external storage device. The computer device 1000 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory 1020 may also be used to temporarily store data that has been output or is to be output.

[0163] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0164] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0165] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0166] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0169] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0170] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A virtual machine placement method, characterized in that the method include: For a virtual machine placement task associated with a data center cluster, obtain placement attributes of the virtual machine to be placed; Based on the topological structure of the data center cluster, construct a three-channel matrix for characterizing the current resource status of the data center cluster; Based on the three-channel matrix and the placement attributes of the virtual machine to be placed, executing a Markov decision process adapted to the virtual machine placement task to generate a target placement strategy; Based on the target placement strategy, the virtual machine to be placed is placed in the data center cluster.

2. The virtual machine placement method according to claim 1, characterized in that: The three-channel matrix for characterizing the current resource status of the data center cluster is constructed based on the topological structure of the data center cluster, including: Based on the cluster constraints and the topological structure of the data center cluster, a first matrix representing the availability of memory nodes, a second matrix representing the server allocation status, and a third matrix representing the network domain allocation are respectively constructed; The first matrix, the second matrix and the third matrix are used as a three-channel matrix for characterizing the current resource status of the data center cluster, and the first matrix, the second matrix and the third matrix have the same matrix size.

3. The virtual machine placement method according to claim 1, characterized in that: The step of obtaining the placement attributes of the virtual machine to be placed includes: Receiving a placement request for a virtual machine to be placed in a data center cluster; The placement request is parsed to obtain placement attributes of the virtual machine to be placed, wherein the placement attributes include a virtual machine identifier, a service group number to which the virtual machine belongs, and resource units required for placing the virtual machine.

4. The virtual machine placement method according to claim 1, characterized in that: The step of executing a Markov decision process adapted to the virtual machine placement task based on the three-channel matrix and the placement attributes of the virtual machine to be placed to generate a target placement strategy includes: Performing feature extraction on the three-channel matrix to obtain topological features of the data center cluster; Flatten the topological feature, and concatenate it with the attribute vector of the placement attribute of the virtual machine to be placed, to obtain a concatenated feature vector; A Markov decision process adapted to the virtual machine placement task is executed for the concatenated feature vector to generate a target placement strategy.

5. The virtual machine placement method according to claim 4, characterized in that: The Markov decision process based on the concatenated feature vector and adapted to the virtual machine placement task generates a target placement strategy, including: Performing action value evaluation on the concatenated feature vector through a duel network to obtain an action value vector, each element of which represents the value of placing the virtual machine to be placed in each memory node; Based on the action value vector, action mask screening is performed to obtain a target placement strategy that meets the cluster constraint conditions.

6. The virtual machine placement method according to claim 4, characterized in that: The duel network includes a state value sub-network and an action advantage sub-network. The duel network is used to evaluate the action value of the spliced ​​feature vector to obtain an action value vector, including: Through the state value sub-network, the state value of the concatenated feature vector is evaluated to obtain the state value; Performing action advantage evaluation on the concatenated feature vector through the action advantage sub-network to obtain action advantage; Based on the state value and the action advantage, an action value vector is obtained.

7. The virtual machine placement method according to claim 1, characterized in that: The method further comprises: Determining a policy reward for executing the placement strategy of the virtual machine to be placed based on a reward function of the Markov decision process, wherein the reward function is determined based on the communication quality of each service group in the data center cluster; If the placement strategy does not meet the adjustment target of maximizing the strategy return, the placement strategy of the virtual machine to be placed is dynamically adjusted until the adjusted placement strategy meets the adjustment target, and the adjusted placement strategy is used as the target placement strategy.

8. The virtual machine placement method according to claim 7, characterized in that: The method further comprises: For each service group of the data center cluster, determining a communication distance between any two virtual machines in the service group; Determine the reciprocal of the communication distance as the communication speed between the two virtual machines; A weighted sum is taken of the communication speeds between all virtual machines in the targeted service group to obtain the communication quality of the targeted service group.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the computer device is caused to implement the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method according to any one of claims 1 to 8 to be performed.

Images