Training method of value model and related device

Abstract

The application provides a value model training method and related equipment. The method comprises: sampling placement records to obtain a transition state sequence and a reward sequence, wherein the transition state sequence records the state of a cloud platform at each time from placing a first virtual machine in an initial state to placing a first unplaceable virtual machine; the reward sequence records the reward obtained at each time from placing the first virtual machine in the initial state to placing the first unplaceable virtual machine; determining a value sequence according to the reward sequence and a reward determination model, wherein the elements in the value sequence and the elements in the transition state sequence have a one-to-one correspondence; wherein the reward determination model considers both the past obtained reward and the future obtainable reward; and training the value model using the transition state sequence and the value sequence.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method for training a value model and related equipment. Background Technology

[0002] Carriers always hope to create more virtual machines (VMs) on a cloud platform with the same physical resources (e.g., CPU, memory, and network resources) to increase their revenue. Ideally, the VMs should just exhaust all the physical resources of the cloud platform, maximizing the carrier's profits.

[0003] However, in actual use, the specifications of physical hosts in cloud platforms (including CPU size, memory size, network bandwidth, etc.) can vary, and the specifications of virtual machines that users expect to create (including CPU size, memory size, network bandwidth, etc.) can also vary. As a result, cloud platforms inevitably generate fragmentation, leading to a decrease in the utilization rate of cloud platform resources. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a training method and related equipment for a value model, which can effectively reduce the fragmentation rate.

[0005] Firstly, a method for training a value model is provided, the method comprising:

[0006] The placement records are sampled to obtain a transition state sequence and a reward sequence. The placement records are the records of the cloud platform from the initial state where the first virtual machine is placed until the first virtual machine that cannot be placed is encountered. The transition state sequence records the state of the cloud platform at each time point from the initial state where the first virtual machine is placed until the first virtual machine that cannot be placed is encountered. The reward sequence records the rewards obtained by the cloud platform at each time point from the initial state where the first virtual machine is placed until the first virtual machine that cannot be placed is encountered.

[0007] The value sequence is determined based on the reward sequence and the reward determination model, wherein there is a one-to-one correspondence between the elements in the value sequence and the elements in the transition state sequence; wherein the reward determination model considers both past rewards and future rewards.

[0008] The value model is trained using transition state sequences and value sequences.

[0009] In some possible designs, the reward determination model can be represented as:

[0010]

[0011] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(s,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a).

[0012] In some possible designs, the placement records are sampled to obtain a sequence of transition states and a sequence of rewards, including:

[0013] By sampling the placement records, a state sequence is obtained:

[0014] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,S T

[0015] The state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π.t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R T R2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

[0016] In some possible designs, the reward includes any one of the following: the number of virtual machines in the cloud platform, the price of the virtual machines placed in the cloud platform, the incremental resource allocation rate of the cloud platform, and the number of physical hosts required.

[0017] In some possible designs, the value model employs a deep neural network or a convolutional neural network.

[0018] Secondly, a value prediction method is provided, the method comprising:

[0019] Obtain the current status of the cloud platform, wherein the status is used to represent the resource allocation status of each physical host in the cloud platform;

[0020] The state is input into the value model to obtain the value corresponding to the state. The value model is trained using known states and corresponding known values. The known states and corresponding known values ​​are obtained by inputting the transition state sequence and reward sequence from the state sequence obtained by sampling the placement records into the reward determination model.

[0021] In some possible designs, the reward determination model can be represented as:

[0022]

[0023] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(s,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a).

[0024] In some possible designs, the state sequence is as follows:

[0025] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,S T

[0026] The state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R TR2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

[0027] In some possible designs, the reward includes any one of the following: the number of virtual machines in the cloud platform, the price of the virtual machines placed in the cloud platform, the incremental resource allocation rate of the cloud platform, and the number of physical hosts required.

[0028] In some possible designs, the value model employs a deep neural network or a convolutional neural network.

[0029] Thirdly, a training device is provided, which includes a sampling module, a reward determination model, and a training module.

[0030] The sampling module is used to sample the placement records to obtain a transition state sequence and a reward sequence. The placement record is the record of the cloud platform from the initial state of placing the first virtual machine to the first virtual machine that cannot be placed. The transition state sequence records the state of the cloud platform at each moment from the initial state of placing the first virtual machine to the first virtual machine that cannot be placed. The reward sequence records the rewards obtained by the cloud platform at each moment from the initial state of placing the first virtual machine to the first virtual machine that cannot be placed.

[0031] The reward determination model is used to determine the value sequence based on the reward sequence, wherein there is a one-to-one correspondence between the elements in the value sequence and the elements in the transition state sequence; wherein the reward determination model considers both past rewards and future rewards.

[0032] The training module is used to train the value model using the transition state sequence and the value sequence.

[0033] In some possible designs, the reward determination model can be represented as:

[0034]

[0035] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(s,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a).

[0036] In some possible designs, the sampling module is used to sample the placement records to obtain a state sequence:

[0037] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,S T

[0038] The state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R TR2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

[0039] In some possible designs, the reward includes any one of the following: the number of virtual machines in the cloud platform, the price of the virtual machines placed in the cloud platform, the incremental resource allocation rate of the cloud platform, and the number of physical hosts required.

[0040] In some possible designs, the value model employs a deep neural network or a convolutional neural network.

[0041] Fourthly, a prediction device is provided, the prediction device including an acquisition module and a prediction module.

[0042] The acquisition module is used to acquire the current status of the cloud platform, wherein the status represents the resource allocation status of each physical host in the cloud platform.

[0043] The prediction module is used to input the transition state into the value model to obtain the value corresponding to the state. The value model is trained using known states and corresponding known values. The known states and corresponding known values ​​are obtained by inputting the transition state sequence and reward sequence from the state sequence obtained by sampling the placement records into the reward determination model.

[0044] In some possible designs, the reward determination model can be represented as:

[0045]

[0046] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(S,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a).

[0047] In some possible designs, the state sequence is as follows:

[0048] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,S T

[0049] The state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R T R2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1The reward received.

[0050] In some possible designs, the reward includes any one of the following: the number of virtual machines in the cloud platform, the price of the virtual machines placed in the cloud platform, the incremental resource allocation rate of the cloud platform, and the number of physical hosts required.

[0051] In some possible designs, the value model employs a deep neural network or a convolutional neural network.

[0052] Fifthly, a training device is provided, including a memory and a processor, the processor executing a program in the memory to perform the method as described in any of the first aspects.

[0053] A sixth aspect provides a prediction device including a memory and a processor, the processor executing a program in the memory to perform the method as described in any of the second aspects.

[0054] A seventh aspect provides a readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method as described in either the first or second aspect.

[0055] Eighthly, a computer program product is provided, wherein when the computer program product is read and executed by a terminal device, the method described in either the first or second aspect will be executed.

[0056] The above scheme uses sampling and combines past and future rewards to dynamically weigh the final effect of each scheduling strategy and learn the optimal scheduling strategy. Compared with traditional heuristic algorithms, it is more intelligent and flexible and can effectively reduce the fragmentation rate when placement fails. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0058] Figure 1 This is a schematic diagram illustrating the creation of virtual machines on physical resources of a cloud platform, as described in this application.

[0059] Figures 2A-2B This is a comparative diagram illustrating the creation of the same virtual machines on the same cloud platform's physical resources using different scheduling methods.

[0060] Figure 3 This is a flowchart illustrating a resource scheduling method provided in this application;

[0061] Figure 4This is a structural diagram of a value model provided in this application;

[0062] Figure 5 The method provided in this application is implemented using a deep neural network. Figure 4 A schematic diagram of the value model shown;

[0063] Figure 6 This is a flowchart illustrating a value prediction method provided in this application;

[0064] Figure 7 This is a flowchart illustrating a training method for a value model provided in this application;

[0065] Figure 8 This is a schematic diagram of the structure of an identification device provided in this application;

[0066] Figure 9 This is a chip hardware structure diagram provided in this application;

[0067] Figure 10 This is a schematic diagram of the structure of a cloud system provided in this application;

[0068] Figure 11 This is a schematic diagram of a system architecture provided in this application;

[0069] Figure 12 This is a schematic diagram of the structure of a value prediction system provided in this application. Detailed Implementation

[0070] Operators always hope to create more virtual machines (VMs) on a cloud platform with the same physical resources (e.g., CPU, memory, and network resources) to increase their revenue. Ideally, the VMs should just exhaust all the physical resources of the cloud platform, maximizing the operator's profit. For example, suppose the cloud platform's physical resources are a resource pool built from three physical hosts: host 1 has a 27u CPU and 54GB of memory; host 2 has an 18u CPU and 36GB of memory; and host 3 has a 27u CPU and 54GB of memory. The user needs to create 24 VMs with 3u CPUs and 6GB of memory each on the cloud platform's physical resources. When using... Figure 1 When created in the manner shown, physical host 1 creates 9 virtual machines with 3u CPUs and 6GB of memory, physical host 2 creates 6 virtual machines with 3u CPUs and 6GB of memory, and the 9 virtual machines with 3u CPUs and 6GB of memory are exactly consumed by the user-created virtual machines, thus achieving the ideal state and maximizing the operator's benefits.

[0071] In practical applications, the specifications of physical hosts in a cloud platform (including CPU size, memory size, and network bandwidth, etc.) can vary widely, as can the specifications of virtual machines (including CPU size, memory size, and network bandwidth, etc.) that users expect to create. This inevitably leads to fragmentation on the cloud platform. However, the fragmentation rate of a cloud platform varies significantly depending on the scheduling methods employed. The following will combine... Figure 2A The scheduling method shown and Figure 2B The scheduling method shown is explained in detail.

[0072] like Figure 2A As shown, assume physical host 1 has a 27u CPU and 54GB of memory; physical host 2 has an 18u CPU and 36GB of memory; and physical host 3 has a 27u CPU and 54GB of memory. A user creates a virtual machine (VM1) with an 18u CPU and 36GB of memory on physical host 1, a virtual machine (VM2) with a 12u CPU and 24GB of memory on physical host 2, and seven virtual machines (VMs 3 through 10) with 3u CPUs and 6GB of memory each on physical host 3. When the user wants to create another virtual machine (VM11) with a 12u CPU and 24GB of memory on the cloud platform, they find that it cannot be created on any physical host. At this point, physical host 1 has fragments with a 9u CPU and 18GB of memory, physical host 2 has fragments with a 6u CPU and 12GB of memory, and physical host 3 has fragments with a 6u CPU and 12GB of memory. Therefore, the total fragmentation of physical hosts 1 through 3 reaches a CPU of 21u and a memory of 42GB.

[0073] like Figure 2B As shown, assume physical host 1 has a 27u CPU and 54GB of memory; physical host 2 has an 18u CPU and 36GB of memory; and physical host 3 has a 27u CPU and 54GB of memory. A user creates one virtual machine (VM 1) with an 18u CPU and 36GB of memory, and three virtual machines (VMs 3 through 5) with 3u CPUs and 6GB of memory each on physical host 1. On physical host 2, a user creates one virtual machine (VM 2) with a 12u CPU and 24GB of memory, and two virtual machines (VMs 6 and 7) with 3u CPUs and 6GB of memory each. On physical host 3, a user creates three virtual machines (VMs 8 through 10) with 3u CPUs and 6GB of memory each. When the user wants to create another virtual machine (VM 11) with an 18u CPU and 36GB of memory on the cloud platform, they can do so on physical host 3. At this point, physical host 1, physical host 2, and physical host 3 have no fragmentation. Therefore, the total fragmentation of physical hosts 1 through 3 is 0.

[0074] Obviously, when adopting Figure 2BThe scheduling method shown results in a significantly lower fragmentation rate for the cloud platform compared to when using [other scheduling methods]. Figure 2A The scheduling method shown represents the fragmentation rate of the cloud platform. This is because when the initial virtual machine creation fails, each physical host often has some remaining fragments. Although the fragments on a single physical host are insufficient to meet the resource requirements of the virtual machine, consolidating these fragments will leave more resources to meet the virtual machine's needs, allowing for continued virtual machine creation and thus reducing the fragmentation rate.

[0075] Understandable. Figure 2A and Figure 2B This is merely an illustration of a specific implementation; in practical applications, Figure 2A and Figure 2B The number of physical hosts, the CPU size of each physical host, the memory size of each physical host, the number of virtual machines, the CPU size of each virtual machine, and the memory size of each virtual machine can also be other values; no specific limitations are specified here. Additionally, the above... Figure 2A and Figure 2B When scheduling, physical resources only consider two dimensions: CPU resources and memory resources. In practical applications, more or fewer dimensions can be considered, but no specific limit is made here.

[0076] The scheduling method plays a decisive role in the fragmentation rate. Therefore, finding a resource scheduling method that can effectively reduce the fragmentation rate and increase the operator's revenue is a very important issue.

[0077] See Figure 3 , Figure 3 This is a flowchart illustrating a resource scheduling method provided in this application. Figure 3 As shown, the resource scheduling method in this embodiment includes:

[0078] S101: Receive a scheduling request sent by a user, wherein the scheduling request is used to request the creation of a target virtual machine on the cloud platform;

[0079] S102: Obtain the resource status of each physical host in the cloud platform, wherein the resource status is used to describe the total amount of resources and the amount of remaining resources of the physical host.

[0080] S103: Select multiple physical hosts to be selected based on the filtering criteria, wherein the filtering criteria include one or more of the remaining resources of the physical host, the affinity of the scheduling request, the anti-affinity of the scheduling request, etc.

[0081] S104: Each candidate physical host is scored according to a scoring method to obtain a score for each candidate physical host relative to the target virtual machine. The scoring method includes FirstFit, BestFit, Cosine Similarity, and Perpendicular Distance algorithms, among others.

[0082] S105: Select the target host from the plurality of candidate physical hosts according to the selection method.

[0083] S106: Create the target virtual machine on the target host.

[0084] In this specific implementation, the scoring method employs a greedy algorithm. That is, when solving the problem, it always makes the choice that seems best at the moment, without considering the overall optimal solution or being unaware of potential future situations. Therefore, the greedy algorithm only provides a locally optimal solution. Here, the scoring method can be a cosine algorithm, Best Fit algorithm, perpendicular distance algorithm, Arc-flow algorithm, simulated annealing algorithm, genetic algorithm, or even integer programming, etc. Assuming only considering the two physical resource dimensions of CPU and memory, taking the cosine algorithm as an example, the score of the candidate physical host j relative to the target virtual machine i... for:

[0085]

[0086] Where i represents the target virtual machine, j represents the candidate physical host, and RC i Represents the CPU size required to create the target virtual machine, UC i Indicates the CPU size already allocated to the candidate physical host j, TC i RM represents the total CPU size of the candidate physical host j. i UM represents the amount of memory required to create the target virtual machine. i TM represents the amount of memory already allocated and used by the candidate physical host j. i This represents the total memory size of the candidate physical host j. Assuming the target virtual machine is created from candidate physical host j, the CPU allocation rate of candidate physical host j is... This represents the memory allocation rate of the candidate physical host j, assuming the target virtual machine is created by the candidate physical host j.

[0087] It can be seen that the cosine similarity algorithm only considers the consistency of resource consumption. In actual placement, it will produce a certain degree of uniform placement effect. However, the cosine similarity algorithm only makes the best choice at the moment, but does not consider the overall optimality, thus leading to a high fragmentation rate. Especially when trying to create a large-scale target virtual machine, it is prone to creation failure and high fragmentation rate.

[0088] In the specific implementation of this embodiment, the selection method typically involves choosing the candidate physical host with the highest score as the target host. However, to ensure rapid response when multiple tenants simultaneously submit virtual machine creation requests, cloud platforms usually employ a distributed scheduling approach. This means multiple schedulers simultaneously send scheduling requests to different tenants, but maintain the same resource pool. Therefore, if the physical host with the highest score is always selected for target virtual machine creation, if different tenants request to create multiple target virtual machines of the same specifications at any given time, multiple schedulers may compete for the same physical host. This results in only one tenant successfully creating the target virtual machine, while other tenants are unable to create their target virtual machines and must resend scheduling requests, reducing the operational efficiency of the cloud platform.

[0089] Since the above scoring methods are not ideal, this application aims to propose another scoring method that can effectively reduce the fragmentation rate.

[0090] Assuming there are n potential physical hosts in the cloud platform, the set of potential physical hosts in the cloud platform can be denoted as A{a1,a2,…a2}. n Let s be the current resource allocation status of the n candidate physical hosts in the cloud platform, and let S be the set of all possible states of the n candidate physical hosts in the cloud platform, where s∈S. If the target virtual machine is created on candidate physical host a1, then the state of the n candidate physical hosts in the cloud platform is s1; if the target virtual machine is created on candidate physical host a2, then the state of the n candidate physical hosts in the cloud platform is s2; ...; if the target virtual machine is created on candidate physical host a... n If created in the cloud platform, the status of the n candidate physical hosts is denoted as s. n Furthermore, s1, s2, ..., s n All belong to set S.

[0091] To facilitate understanding, a simple example will be used below. Assume there are two candidate physical hosts, a1 and a2, in the cloud platform. Candidate physical host a1 has two allocation states: (1) 0u CPU and 0G memory have been allocated, denoted as a1(0,0); (2) 2u CPU and 2G memory have been allocated, denoted as a1(2,2). Candidate physical host a2 has two allocation states: (1) 0u CPU and 0G memory have been allocated, denoted as a2(0,0); (2) 2u CPU and 2G memory have been allocated, denoted as a2(2,2). Therefore, all possible states of the two candidate physical hosts in the cloud platform are denoted as the set S{[a1(0,0),a2(0,0)],[a1(2,2),a2(0,0)],[a1(0,0),a2(2,2)],[a1(2,2),a2(2,2)]}. The current allocation status of the two candidate physical hosts in this cloud platform is denoted as s[a1(0,0),a2(0,0)]. If the target virtual machine (CPU 2u, memory 2G) is created on candidate physical host a1, then the status of the n candidate physical hosts in this cloud platform is denoted as s1[a1(2,2),a2(0,0)]. If the target virtual machine (CPU 2u, memory 2G) is created on candidate physical host m2, then the status of the two candidate physical hosts in this cloud platform is denoted as s2[a1(0,0),a2(2,2)]. It should be understood that the above examples are for illustrative purposes only and should not constitute specific limitations.

[0092] In one specific embodiment, such as Figure 4 As shown, the states s1, s2, ..., s n Input the trained value model into each state s1, s2, ..., s. n The values ​​y1, y2, ..., y n The higher the value, the higher the score. The value model can be implemented using deep neural networks (DNN) or convolutional neural networks (CNN), etc., without specific limitations here.

[0093] In a specific embodiment, the trained value model can be represented as:

[0094] y i =value(s) i )

[0095] Among them, s i For the i-th state, s i ∈(s1,s2,…s n ). y iFor state s i The corresponding value, value(), represents the mapping relationship between states and values. value() can be obtained through training with a large number of states and their corresponding known values.

[0096] For simplicity, the following detailed explanations will use a deep neural network as an example to illustrate the value model. Figure 5 As shown, a deep neural network includes an input layer, a hidden layer, and an output layer.

[0097] With state s i For example, state s i Let the states be s1, s2, ..., s n Let i be any state in the array, where i is a free variable, i is a positive integer, and 1≤i≤n.

[0098] Input layer:

[0099] Assume the input to the input layer is state s i The output is equal to the input, meaning no processing is performed on the input. For simplicity, this assumes the input layer does not perform any processing; however, in practice, normalization and other processing can be applied to the input layer, which is not specifically limited here.

[0100] Hidden layer:

[0101] The state s output from the input layer i Assuming there are a total of L (L≧2) hidden layers as input to the hidden layers, let Z be the input to the hidden layers. l This represents the output of the l-th layer. When l = 1, Z 1 =s i Where 1≤l≤L, then the relationship between the l-th layer and the (l+1)-th layer is:

[0102] a l+1 =W l Z l +b l

[0103] Z l+1 =f l+1 (a l+1 )

[0104] Among them, W l Let b be the weight vector of the l-th layer. l Let a be the bias vector of the l-th layer. l+1 f is the intermediate vector of the (l+1)th layer. l +1 Z is the activation function for the (l+1)th layer. l+1This represents the hidden layer result of layer l+1. The activation function can be any of the following: sigmoid function, hyperbolic tangent function, ReLU function, ELU (Exponential Linear Units) function, etc.

[0105] Output layer:

[0106] For batch training / prediction, assume the output of layer L is Z. L Specifically, (Q1, Q2, ..., Q) S If the state is s, then the state is s. i Value y i It can be calculated using the softmax function:

[0107]

[0108] Among them, y i It can be state s i The value of Q, where e is a natural number. j The output Z of the Lth layer L The j-th element in Q, where j is a free variable. i The output Z of the Lth layer L The i-th element in the expression represents the set of all possible states of all candidate physical hosts in the cloud platform. It's understood that the above example uses the softmax function for illustration; however, in practical applications, functions such as the logistic function can also be used, without specific limitations here.

[0109] The preceding content detailed the process of identifying the value of a state using a trained value model. However, before identification, the untrained value model needs to be trained to obtain a trained value model. The following section will explain in detail how to train an untrained value model.

[0110] (1) Obtain a large number of known states and corresponding large number of known values ​​as a training set.

[0111] (2) Input a large number of known states and a large number of known values ​​from the training set into the untrained value model for repeated training until the value model can correctly identify the state.

[0112] For a single training iteration, because the output of the value model should be as close as possible to the desired predicted value, a known state can be input into the value model to obtain a predicted value for that known state. The value corresponding to that known state, converged through the Bellman equation, is then taken as the desired target value. The current value model's predicted value is compared to the desired target value, and the weight vector of each layer of the value model is updated based on the difference between the two (of course, there is usually an initialization process before the first update, i.e., pre-configuring parameters for each layer in the value model). For example, if the value model's predicted value is too high, the weight vector is adjusted to predict a lower value, and this adjustment continues until the value model can predict the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the value model becomes a process of minimizing this loss as much as possible.

[0113] In one specific embodiment, the training set can be obtained as follows: a database records a large number of placement records from the initial state of the cloud platform, up to the point where no virtual machine can be placed. Of course, to enrich the data in the database, a generator can also be used to generate new placement records based on the patterns of historical scheduling requests (such as distribution ratio, time interval, lifecycle, etc.) and store them in the database. Monte Carlo sampling is performed multiple times on the records in the database to obtain multiple transition state sequences and corresponding reward sequences. These multiple transition state sequences and corresponding reward sequences are then input into the reward determination model to obtain a large number of known states and corresponding known values ​​as the training set. The following explanation uses the Bellman equation as an example of the reward determination model.

[0114]

[0115] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(s,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a). It can be understood that the reward here is illustrated using the size of the virtual machine placed on the cloud platform as an example. In practical applications, the reward can also be the price of the virtual machine placed on the cloud platform, the increase in the resource allocation rate of the cloud platform, the number of physical hosts required, etc., but no specific limitation is made here.

[0116] Taking the k-th Monte Carlo sampling based on policy π as an example, a complete state sequence will be obtained:

[0117] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,S T

[0118] The aforementioned state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S TTo place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R T R2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

[0119] The following section will detail how to process the reward sequence R2, R3, ..., R of the k-th Monte Carlo sampling. t+1 ,…,R T Substituting into the Bellman equation, we can calculate the states S1, S2, ..., S t ,…,S T Each has its own value. Among them, all ownership values ​​of the value network are initialized with regularized randomness.

[0120] (1) R T Input the Bellman equation to calculate the state S T-1 Corresponding value

[0121] G T-1 =R T

[0122] N(S T-1 A T-1 ) = T

[0123]

[0124] in, The state S obtained from the (k-1)th Monte Carlo sampling T-1 The corresponding value.

[0125] (2) R T-1 and R T Input the Bellman equation to calculate the state S T-2 The corresponding value.

[0126] G T-2 =R T +γR T-1

[0127] N(S T-2 A T-2 ) = T-1

[0128]

[0129] in, The state S obtained from the (k-1)th Monte Carlo sampling T-2 The corresponding value.

[0130] …;

[0131] (Tt) will R t+1 ,R t+2 ,R t+3 ,…,R T Input the Bellman equation to calculate the state S t The corresponding value.

[0132] G t =R t+1 +γR t+2 +γ 2 R t+3 +…+γ T-t-1 R T

[0133] N(S t A t )=N(S t+1 A t+1 )-1

[0134]

[0135] in, The state S obtained from the (k-1)th Monte Carlo sampling t The corresponding value.

[0136] …;

[0137] (T-1) Let R2, R3, ..., R t+1 ,…,R T The value corresponding to state S1 is calculated by inputting the Bellman equation.

[0138] G1=R2+γR3+γ 2 R4+…+γ T-2 R T

[0139] N(S1,A1)=N(S2,A2)⁻¹

[0140]

[0141] in, Let S be the value corresponding to state S1 obtained from the (k-1)th Monte Carlo sampling.

[0142] Therefore, the known state S can be obtained. Tand the corresponding known value R T Given state S T-1 and the corresponding known value Given state S T-2 and the corresponding known value …, known state S t and the corresponding known value …, known state S1 and its corresponding known value Etc. are used as training samples in the training set.

[0143] For ease of understanding, the above example illustrates training the value network in batches after obtaining a batch of training samples. In practical applications, the value network can also be trained immediately after obtaining a single training sample. For example, after obtaining a known state S... T and the corresponding known value R T Then, immediately use the known state S T and the corresponding known value R T Train the value network, and then compute the known state S. T-1 and the corresponding known value Then use the known state S T-1 and the corresponding known value The value network is trained, and so on, without specific limitations here.

[0144] The aforementioned scheme employs Monte Carlo sampling and Markov Decision Process (MDP) evaluation, dynamically weighing the final effect of each scheduling strategy to learn the optimal one. Compared to traditional heuristic algorithms, it is more intelligent and flexible, effectively reducing fragmentation rates during placement failures. Experimental data comparison shows that the fragmentation rate of traditional heuristic algorithms is 15%, while the fragmentation rate of the proposed method is only 5%, representing a significant reduction. Furthermore, the state space in cloud resource scheduling scenarios is more complex, reaching… Among them, R i Here, N represents the capacity of a single resource dimension on a physical host, N is the total number of resource dimensions, and H is the total number of physical hosts. For example, assuming a data center has 1000 hosts, each with 92u CPU and 324GB of memory, the search space would be (92×324). 1000 The memory of a typical physical host is far from sufficient to store such a large state space, and simply expanding storage resources is not a fundamental solution. Therefore, a deep neural network is introduced to transform the massive action-value function table into a value model. Thus, during storage, only the weights of the value model need to be stored, greatly reducing the storage space requirements.

[0145] To avoid the value model getting stuck in local optima or resource conflicts in distributed parallel scheduling scenarios (where multiple schedulers compete for the same resource block, resulting in only one success and the others failing), we optimize by setting an exploration rate ε∈[0,1]. This ensures that each scheduling decision has a certain probability of escaping the current optimal solution and selecting from several candidate physical hosts with higher scores. The formal representation is as follows:

[0146]

[0147] Using this selection strategy π(a|s) will not greedily seek the current best solution, but will assume that all physical hosts with higher rankings have the potential to be placed.

[0148] See Figure 6 , Figure 6 This is a flowchart illustrating a value prediction method provided in this application. The value prediction method of this embodiment includes:

[0149] S201: Obtain the current state of the cloud platform, wherein the state is used to represent the resource allocation status of each physical host in the cloud platform.

[0150] In one specific embodiment, the current state of the cloud platform is one of the set of all possible states of all candidate physical hosts in the cloud platform.

[0151] In one specific embodiment, there are n candidate physical hosts in the cloud platform. Therefore, the set of n candidate physical hosts in the cloud platform can be denoted as A{a1,a2,…a2}. n Let s be the current resource allocation status of the n candidate physical hosts in the cloud platform, and let S be the set of all possible states of the n candidate physical hosts in the cloud platform, where s∈S. If the target virtual machine is created on candidate physical host a1, then the state of the n candidate physical hosts in the cloud platform is s1; if the target virtual machine is created on candidate physical host a2, then the state of the n candidate physical hosts in the cloud platform is s2; ...; if the target virtual machine is created on candidate physical host a... n If created in the cloud platform, the status of the n candidate physical hosts is denoted as s. n Furthermore, s1, s2, ..., s n All belong to set S.

[0152] In a specific embodiment, assume there are two candidate physical hosts, a1 and a2, in the cloud platform. Candidate physical host a1 has two allocation states: (1) CPU has been allocated 0u and memory has been allocated 0G, denoted as a1(0,0); (2) CPU has been allocated 2u and memory has been allocated 2G, denoted as a1(2,2). Candidate physical host a2 has two allocation states: (1) CPU has been allocated 0u and memory has been allocated 0G, denoted as a2(0,0); (2) CPU has been allocated 2u and memory has been allocated 2G, denoted as a2(2,2). Therefore, all possible states of the two candidate physical hosts in the cloud platform are denoted as the set S{[a1(0,0),a2(0,0)],[a1(2,2),a2(0,0)],[a1(0,0),a2(2,2)],[a1(2,2),a2(2,2)]}. The current allocation status of the two candidate physical hosts in this cloud platform is denoted as s[a1(0,0),a2(0,0)]. If the target virtual machine (CPU 2u, memory 2G) is created on candidate physical host a1, then the status of the n candidate physical hosts in this cloud platform is denoted as s1[a1(2,2),a2(0,0)]. If the target virtual machine (CPU 2u, memory 2G) is created on candidate physical host m2, then the status of the two candidate physical hosts in this cloud platform is denoted as s2[a1(0,0),a2(2,2)]. It should be understood that the above examples are for illustrative purposes only and should not constitute specific limitations.

[0153] S202: Input the state into the value model to obtain the value corresponding to the state, wherein the value model is obtained by training using known states and corresponding known values, and the known states and corresponding known values ​​are obtained by inputting the transition state sequence and reward sequence from the state sequence obtained by sampling the placement record into the reward determination model.

[0154] In one specific embodiment, such as Figure 4 As shown, the states s1, s2, ..., s n Input the trained value model into each state s1, s2, ..., s. n The values ​​y1, y2, ..., y n The higher the value, the higher the score. The value model can be DNN or CNN, etc., without specific limitations here.

[0155] In a specific embodiment, the trained value model can be represented as:

[0156] y i =value(s) i )

[0157] Among them, si For the i-th state, s i ∈(s1,s2,…s n ). y i For state s i The corresponding value, value(), represents the mapping relationship between states and values. value() can be obtained through training with a large number of states and their corresponding known values.

[0158] For simplicity, the following detailed explanations will use a deep neural network as an example to illustrate the value model. Figure 5 As shown, a deep neural network includes an input layer, a hidden layer, and an output layer.

[0159] With state s i For example, state s i Let the states be s1, s2, ..., s n Let i be any state in the array, where i is a free variable, i is a positive integer, and 1≤i≤n.

[0160] Input layer:

[0161] Assume the input to the input layer is state s i The output is equal to the input, meaning no processing is performed on the input. For simplicity, this assumes the input layer does not perform any processing; however, in practice, normalization and other processing can be applied to the input layer, which is not specifically limited here.

[0162] Hidden layer:

[0163] The state s output from the input layer i Assuming there are a total of L (L≧2) hidden layers as input to the hidden layers, let Z be the input to the hidden layers. l This represents the output of the l-th layer. When l = 1, Z 1 =s i Where 1≤l≤L, then the relationship between the l-th layer and the (l+1)-th layer is:

[0164] a l+1 =W l Z l +b l

[0165] Z l+1 =f l+1 (a l+1 )

[0166] Among them, W l Let b be the weight vector of the l-th layer. l Let a be the bias vector of the l-th layer. l+1 f is the intermediate vector of the (l+1)th layer. l +1Z is the activation function for the (l+1)th layer. l+1 This represents the hidden layer result of layer l+1. The activation function can be any of the following: sigmoid function, hyperbolic tangent function, ReLU function, ELU (Exponential Linear Units) function, etc.

[0167] Output layer:

[0168] For batch training / prediction, assume the output of layer L is Z. L Specifically, (Q1, Q2, ..., Q) S If the state is s, then the state is s. i Value y i It can be calculated using the softmax function:

[0169]

[0170] Among them, y i It can be state s i The value of Q, where e is a natural number. j The output Z of the Lth layer L The j-th element in Q, where j is a free variable. i The output Z of the Lth layer L The i-th element in the expression represents the set of all possible states of all candidate physical hosts in the cloud platform. It's understood that the above example uses the softmax function for illustration; however, in practical applications, functions such as the logistic function can also be used, without specific limitations here.

[0171] In a specific embodiment, the reward determination model may employ the Bellman equation and related variations thereof, etc., without specific limitations here. The following explanation will use the Bellman equation as an example to illustrate the reward determination model.

[0172]

[0173] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(S,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a). It can be understood that the reward here is illustrated using the size of the virtual machine placed on the cloud platform as an example. In practical applications, the reward can also be the price of the virtual machine placed on the cloud platform, the increase in the resource allocation rate of the cloud platform, the number of physical hosts required, etc., but no specific limitation is made here.

[0174] See Figure 7 , Figure 7 This is a flowchart illustrating a training method for a value model provided in this application. The value prediction method of this embodiment includes:

[0175] S301: Sample the placement records to obtain the transition state sequence and the reward sequence.

[0176] In one specific embodiment, the placement record refers to the record from the initial state of the cloud platform placing the first virtual machine until the first virtual machine that could not be placed. The database records a large number of placement records from the initial state of the cloud platform placing the first virtual machine until the first virtual machine that could not be placed. Of course, to enrich the data in the database, a generator can also be used to generate some new placement records based on the patterns of historical scheduling requests (such as distribution ratio, time interval, lifecycle, etc.) and store them in the database.

[0177] In one specific embodiment, the transition state sequence records the state of the cloud platform at each moment from the initial state where the first virtual machine is placed until the first virtual machine that cannot be placed.

[0178] In one specific embodiment, the reward sequence records the rewards obtained by the cloud platform at each moment from the initial state when the first virtual machine is placed until the first virtual machine that cannot be placed.

[0179] In one specific embodiment, the state and reward sequences can be obtained by performing Monte Carlo sampling multiple times on the records in the database, thereby obtaining multiple transition state sequences and corresponding reward sequences. Taking the k-th Monte Carlo sampling based on policy π as an example, a complete state sequence will be obtained:

[0180] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,ST

[0181] The aforementioned state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R T R2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

[0182] S302: Determine the value sequence based on the reward sequence and the reward determination model, wherein there is a one-to-one correspondence between the elements in the value sequence and the elements in the transition state sequence; the reward determination model considers both past rewards and future rewards.

[0183] In a specific embodiment, the reward determination model may employ the Bellman equation and related variations thereof, etc., without specific limitations here. The following explanation will use the Bellman equation as an example to illustrate the reward determination model.

[0184]

[0185] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(s,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a). It can be understood that the reward here is illustrated using the size of the virtual machine placed on the cloud platform as an example. In practical applications, the reward can also be the price of the virtual machine placed on the cloud platform, the increase in the resource allocation rate of the cloud platform, the number of physical hosts required, etc., but no specific limitation is made here.

[0186] Taking the k-th Monte Carlo sampling based on policy π as an example, a complete state sequence will be obtained:

[0187] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,S T

[0188] The aforementioned state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S TS1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R T R2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

[0189] The following section will detail how to process the reward sequence R2, R3, ..., R of the k-th Monte Carlo sampling. t+1 ,…,R T Substituting into the Bellman equation, we can calculate the states S1, S2, ..., S t ,…,S T Each has its own value. Among them, all ownership values ​​of the value network are initialized with regularized randomness.

[0190] (1) R T Input the Bellman equation to calculate the state S T-1 Corresponding value

[0191] G T-1 =R T

[0192] N(S T-1 A T-1 ) = T

[0193]

[0194] in, The state S obtained from the (k-1)th Monte Carlo sampling T-1 The corresponding value.

[0195] (2) R T-1 and R T Input the Bellman equation to calculate the state S T-2 The corresponding value.

[0196] G T-2 =RT +γR T-1

[0197] N(S T-2 A T-2 ) = T-1

[0198]

[0199] in, The state S obtained from the (k-1)th Monte Carlo sampling T-2 The corresponding value.

[0200] …;

[0201] (Tt) will R t+1 ,R t+2 ,R t+3 ,…,R T Input the Bellman equation to calculate the state S t The corresponding value.

[0202] G t =R t+1 +γR t+2 +γ 2 R t+3 +…+γ T-t-1 R T

[0203] N(S t A t )=N(S t+1 A t+1 )-1

[0204]

[0205] in, The state S obtained from the (k-1)th Monte Carlo sampling t The corresponding value.

[0206] …;

[0207] (T-1) Let R2, R3, ..., R t+1 ,…,R T The value corresponding to state S1 is calculated by inputting the Bellman equation.

[0208] G1=R2+γR3+γ 2 R4+…+γ T-2 R T

[0209] N(S1,A1)=N(S2,A2)⁻¹

[0210]

[0211] S303: Train the value model using the transition state sequence and the value sequence.

[0212] See Figure 8 , Figure 8 This is a schematic diagram of the structure of a prediction device provided in this application. The prediction device of this embodiment includes: one or more first processors 210, samplers 220, communication interfaces 230, external memory 240, second processors 250, and a bus 260. The first processors 210, samplers 220, communication interfaces 230, external memory 240, and second processors 250 can be connected via the bus 260.

[0213] The first processor 210 includes one or more general-purpose processors, which can be any type of device capable of processing electronic instructions, including a central processing unit (CPU), microprocessor, microcontroller, main processor, controller, and ASIC (Application Specific Integrated Circuit), etc. The first processor 210 executes various types of digital storage instructions, such as software or firmware programs stored in memory 230, which enables the identification device to provide a wide range of services. For example, the first processor 210 can execute programs or process data to perform at least a portion of the methods discussed herein.

[0214] Sampler 220 can perform sampling, for example, Monte Carlo sampling, etc.

[0215] The communication interface 230 can be a wired interface (e.g., an Ethernet interface) for communicating with other identification devices or users.

[0216] External storage 240 may include volatile memory, such as random access memory (RAM); it may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); and it may include combinations of the above types of memory. External storage 240 may store program code and program data.

[0217] The first processor 210 executes the following steps by calling program code stored in external memory 240:

[0218] The first processor 210 calls the sampler 220 to obtain the current state of the cloud platform, wherein the state is used to represent the resource allocation state of each physical host in the cloud platform;

[0219] The first processor 210 calls the second processor 250 to input the state into the value model, thereby obtaining the value corresponding to the state. The value model is obtained by training using known states and corresponding known values. The known states and corresponding known values ​​are obtained by inputting the transition state sequence and reward sequence from the state sequence obtained by sampling the placement record into the reward determination model.

[0220] For the sake of simplicity, this embodiment does not provide a detailed explanation of the concepts of state and value model, nor does it provide a detailed explanation of how to obtain the state, how to predict value through the value model, or how to train the value model, etc. For details, please refer to [link / reference needed]. Figures 1 to 5 And related descriptions.

[0221] See Figure 9 , Figure 9 This is a chip hardware structure diagram provided in this application. Figure 8 The second processor shown can be Figure 9 Implemented in the chip shown, or in other words, Figure 6 Some steps in the value prediction method shown can be performed in [the following text is incomplete and likely refers to a different method]. Figure 9 The chip shown is used for implementation. The following explanation will use a Neural Processing Unit (NPU) as an example. The NPU is mounted as a coprocessor on the host CPU, which allocates tasks. The core of the NPU is the arithmetic circuit, which uses controller 304 to control the arithmetic circuit 303 to retrieve matrix data from memory and perform multiplication operations.

[0222] In some implementations, the arithmetic circuit 303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0223] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 208.

[0224] Unified memory 306 is used to store input and output data. Weight data is directly accessed via the Direct Memory Access Controller (DMAC) 305 and transferred to the weight memory 302. Input data is also transferred to unified memory 306 via DMAC.

[0225] BIU stands for Bus Interface Unit 310, which is used for interaction between the AXI bus and the DMAC and the instruction fetch buffer 509.

[0226] The Bus Interface Unit (BIU) 310 is used by the instruction fetch memory 309 to fetch instructions from external memory, and also by the memory access controller 305 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0227] The DMAC is mainly used to move input data from external memory DDR to unified memory 306, or to weight data to weight memory 302, or to input data to input memory 301.

[0228] The vector computation unit has over 307 processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computation in non-convolutional / FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0229] In some implementations, the vector computation unit 307 can store the processed output vector into a unified buffer 306. For example, the vector computation unit 307 can apply a nonlinear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 303, for example, for use in subsequent layers of a neural network.

[0230] The instruction fetch buffer 309 connected to the controller 304 is used to store the instructions used by the controller 304;

[0231] Unified memory 306, input memory 301, weighted memory 302, and instruction fetch memory 309 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0232] in, Figure 4 The operations of each layer in the value model shown can be performed by the weight memory 302 or the vector calculation unit 307.

[0233] See Figure 10 , Figure 10 This is a schematic diagram of the structure of a cloud system provided in this application. The prediction device 410 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The prediction device 410 can be deployed on a single physical site or distributed across multiple physical sites. The prediction device 410 can use data from the data storage system 450 or call program code from the data storage system 450. Figure 6 The value prediction method shown, specifically, the prediction device 410 can perform the following steps:

[0234] The prediction device 410 is used to obtain the current state of the cloud platform, wherein the state is used to represent the resource allocation state of each physical host in the cloud platform;

[0235] The prediction device 410 is used to input the state into a value model to obtain the value corresponding to the state. The value model is trained using known states and corresponding known values. The known states and corresponding known values ​​are obtained by inputting the transition state sequence and reward sequence from the state sequence obtained by sampling the placement record into a reward determination model.

[0236] Users can interact with the prediction device 410 by operating their respective user devices (e.g., local device 401 and local device 402). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0237] Each user's local device can interact with the prediction device 410 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0238] In another implementation, one or more aspects of the prediction device 410 may be implemented by each local device. For example, local device 401 may provide local data or feedback calculation results to the prediction device 410.

[0239] It should be noted that all the functions of the prediction device 410 can also be implemented by a local device. For example, local device 401 implements the functions of the prediction device 410 and provides services to its own users, or provides services to the users of local device 402.

[0240] For the sake of simplicity, this embodiment does not provide a detailed explanation of the concepts of state and value model, nor does it provide a detailed explanation of how to obtain the state, how to predict value through the value model, or how to train the value model, etc. For details, please refer to [link / reference needed]. Figures 1 to 5 And related descriptions.

[0241] See Figure 11 , Figure 11 This is a schematic diagram of a system architecture provided in this application. The data acquisition device 660 collects placement records and stores them in the database 630. The training device 620 generates a value prediction model 610 based on the placement records in the database 630. The following describes in more detail how the training device 620 obtains a value prediction model 601 based on the placement records. The value prediction model 601 can predict states, thereby determining the value of the state. Specifically,

[0242] The training device 620 samples the placement records to obtain a transition state sequence and a reward sequence. The placement records are the records of the cloud platform from the initial state when the first virtual machine is placed until the first virtual machine that cannot be placed. The transition state sequence records the state of the cloud platform at each moment from the initial state when the first virtual machine is placed until the first virtual machine that cannot be placed. The reward sequence records the rewards obtained by the cloud platform at each moment from the initial state when the first virtual machine is placed until the first virtual machine that cannot be placed.

[0243] The training device 620 determines the value sequence based on the reward sequence and the reward determination model, wherein there is a one-to-one correspondence between the elements in the value sequence and the elements in the transition state sequence; wherein the reward determination model considers both past rewards and future rewards.

[0244] The training device 620 uses the transition state sequence and the value sequence to train the value model, thereby generating a trained value model 601.

[0245] During training, since we want the output of the value model to be as close as possible to the actual value we want to predict, we can compare the current value model's predicted value with the actual target value, and then update the weight vector of each layer of the neural network in the value model based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the value model).

[0246] The value model obtained from training device 620 can be applied to different systems or devices. Figure 11 In the process, the execution device 610 is equipped with an I / O interface 612 for data interaction with external devices. The "user" can input data to the I / O interface 612 through the client device 640.

[0247] The execution device 610 can call data, code, etc. in the data storage system 650, and can also store data, instructions, etc. in the data storage system 650.

[0248] The calculation module 611 uses the value model 601 to process the input state, thereby obtaining the value corresponding to the state.

[0249] This also includes functional association module 613 and functional association module 614.

[0250] Finally, the I / O interface 612 returns the processing result to the client device 640 for the user.

[0251] At a deeper level, the training device 620 can generate corresponding value models 601 based on different data for different objectives, in order to provide users with better results.

[0252] exist Figure 11In the illustrated scenario, the user can manually specify the data input to execution device 610, for example, by operating through the interface provided by I / O interface 612. Alternatively, client device 640 can automatically input data to I / O interface 612 and obtain results. If client device 640 requires user authorization to automatically input data, the user can set appropriate permissions within client device 640. The user can view the results output by execution device 601 on client device 640, which can be presented through display, sound, animation, or other specific methods. Client device 640 can also act as a data acquisition terminal, storing the collected training data into database 630.

[0253] It is worth noting that Figure 11 This is merely a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 11 In this context, the data storage system 650 is an external memory relative to the execution device 610. In other cases, the data storage system 650 may also be placed within the execution device 610.

[0254] For the sake of simplicity, this embodiment does not provide a detailed explanation of the concepts of state and value model, nor does it provide a detailed explanation of how to obtain the state, how to predict value through the value model, or how to train the value model, etc. For details, please refer to [link / reference needed]. Figures 1 to 5 And related descriptions.

[0255] See Figure 12 , Figure 12 This is a schematic diagram of the structure of a value prediction system provided in this application. The value prediction system of this embodiment may include a prediction device and a training device. The prediction device and the training device can communicate with each other, and the training device and the prediction device can be integrated or separate devices; no specific limitation is made here.

[0256] like Figure 12 As shown, the prediction device 710 may include an acquisition module 711 and a prediction module 712. Wherein,

[0257] The acquisition module 711 is used to acquire the current status of the cloud platform, wherein the status is used to represent the resource allocation status of each physical host in the cloud platform;

[0258] The prediction module 712 is used to input the state into the value model to obtain the value corresponding to the state. The value model is trained using known states and corresponding known values. The known states and corresponding known values ​​are obtained by inputting the transition state sequence and reward sequence from the state sequence obtained by sampling the placement record into the reward determination model.

[0259] like Figure 12 As shown, the training device 720 includes a sampling module 721, a reward determination model 722, and a training module 723.

[0260] The sampling module 721 is used to sample the placement records to obtain a transition state sequence and a reward sequence. The placement record is the record of the cloud platform from the initial state of placing the first virtual machine to the first virtual machine that cannot be placed. The transition state sequence records the state of the cloud platform at each moment from the initial state of placing the first virtual machine to the first virtual machine that cannot be placed. The reward sequence records the rewards obtained by the cloud platform at each moment from the initial state of placing the first virtual machine to the first virtual machine that cannot be placed.

[0261] The reward determination model 722 is used to determine the value sequence based on the reward sequence, wherein there is a one-to-one correspondence between the elements in the value sequence and the elements in the transition state sequence; wherein the reward determination model considers both past rewards and future rewards.

[0262] The training module 723 is used to train the value model using the transition state sequence and the value sequence.

[0263] In some possible designs, the reward determination model can be represented as:

[0264]

[0265] Here, the specification of the virtual machine to be created is denoted as vector f, and the set of all virtual machine specifications is F, i.e., f∈F; the current allocation state of the physical hosts in the cloud platform is denoted as s, and the state space of all physical hosts is set S, i.e., s∈S; the set of physical hosts in the cloud platform can be denoted as M, and the set of candidate physical hosts is A. The currently selected candidate physical host is a, a∈A, and the scheduling policy is the function π: F×S→A, where π is a probability distribution function. The set of all scheduling policies can be denoted as Π, where π∈Π. When the candidate physical host a=π(s,f) is selected according to policy π to deploy the virtual machine to be created, the reward for this selection is... The reward can be the size of the virtual machine. The state s' is determined by s, f, and a, denoted as s′=Ω(s,f,a). The reward decay factor is denoted as γ∈[0,1]. E() is the function for calculating the expectation, and the function for the value under policy π is denoted as Q. π (s,a).

[0266] In some possible designs, the state sequence is as follows:

[0267] S1,A1,R2,S2,A2,R3,…,S t A t ,R t+1 ,…,A T-1 ,R T ,S T

[0268] The state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions. The above state sequence includes the transition state sequence S1, S2, ..., S... t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The resulting states. The above state sequence includes the reward sequence R2, R3, ..., R. t+1 ,…,R T R2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

[0269] In some possible designs, the reward includes any one of the following: the number of virtual machines in the cloud platform, the price of the virtual machines placed in the cloud platform, the incremental resource allocation rate of the cloud platform, and the number of physical hosts required.

[0270] In some possible designs, the value model employs a deep neural network or a convolutional neural network.

[0271] For the sake of simplicity, this embodiment does not provide a detailed explanation of the concepts of state and value model, nor does it provide a detailed explanation of how to obtain the state, how to predict value through the value model, or how to train the value model, etc. For details, please refer to [link / reference needed]. Figures 1 to 5 And related descriptions.

[0272] The aforementioned scheme employs Monte Carlo sampling and Markov Decision Process (MDP) evaluation, dynamically weighing the final effect of each scheduling strategy to learn the optimal one. Compared to traditional heuristic algorithms, it is more intelligent and flexible, effectively reducing fragmentation rates during placement failures. Experimental data comparison shows that the fragmentation rate of traditional heuristic algorithms is 15%, while the fragmentation rate of the proposed method is only 5%, representing a significant reduction. Furthermore, the state space in cloud resource scheduling scenarios is more complex, reaching… Among them, R i Here, N represents the capacity of a single resource dimension on a physical host, N is the total number of resource dimensions, and H is the total number of physical hosts. For example, assuming a data center has 1000 hosts, each with 92u CPU and 324GB of memory, the search space would be (92×324). 1000 The memory of a typical physical host is far from sufficient to store such a large state space, and simply expanding storage resources is not a fundamental solution. Therefore, a deep neural network is introduced to transform the massive action-value function table into a value model. Thus, during storage, only the weights of the value model need to be stored, greatly reducing the storage space requirements.

[0273] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, storage disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for training a value model, characterized in that, The method comprises: sampling placement records to obtain a transition state sequence and a reward sequence, wherein the placement records are records of a cloud platform placing a first virtual machine from an initial state to a first unable-to-place virtual machine, the transition state sequence records states of the cloud platform at each time point from placing the first virtual machine from the initial state to the first unable-to-place virtual machine, and the states are used to indicate allocation states of resources of each physical host in the cloud platform; and the reward sequence records rewards obtained at each time point from placing the first virtual machine from the initial state to the first unable-to-place virtual machine, and the rewards include any one of a quantity of virtual machines in the cloud platform, a price of a placed virtual machine in the cloud platform, an increase of a resource allocation rate of the cloud platform, and a quantity of physical hosts to be used. determining a value sequence according to the reward sequence and a reward determination model, wherein elements in the value sequence and elements in the transition state sequence have a one-to-one correspondence relationship; and the reward determination model considers both past obtained rewards and future obtainable rewards. training a value model using the transition state sequence and the value sequence.

2. The method of claim 1, wherein, The reward determination model can be represented as: Wherein, the specification of the virtual machine to be created is a vector f, the collection of the specification of all virtual machines is F, that is, f∈F; the current allocation state of the physical host in the cloud platform is recorded as s, and the state space of all is a collection S, that is, s∈S; the collection of the physical host in the cloud platform can be recorded as M, and the collection of the candidate physical host to be selected is A, The currently selected candidate physical host is a, a∈A, the scheduling strategy is a function π: F×S→A, π is a probability distribution function, and the collection constituted by all scheduling strategy spaces can be recorded as Π, π∈Π; when the candidate physical host a is selected according to the strategy π to deploy the virtual machine to be created, the reward of the selected is The reward can be the specification size of the virtual machine, the state s' is determined by s, f and a together, recorded as s′=Ω(s,f,a), the reward decay factor is recorded as γ∈[0,1], E() is a function for calculating the expectation, and the function for the value under the strategy π is recorded as Q π (s,a).

3. The method of claim 1, wherein, sampling placement records to obtain a transition state sequence and a reward sequence comprises: sampling placement records to obtain a state sequence comprises: S1, A1, R2, S2, A2, R3,..., S t ,A t ,R t+1 ,…,A T-1 ,R T ,S T Wherein, the state sequence includes action sequence A1, A2, …, A t T-1 A1 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m1 under the policy π, A2 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m2 under the policy π, A t is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m t under the policy π, A T-1 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m T-1 under the policy π, the state sequence includes transition state sequence S1, S2, …, S t T S1 is an initial state of the cloud platform under the policy π, S2 is a state obtained after the cloud platform places the first virtual machine a1 under the policy π, S t is a state obtained after the cloud platform places the t-1th virtual machine a t-1 under the policy π, S T is a state obtained after the cloud platform places the last virtual machine a T-1 under the policy π, the state sequence includes reward sequence R2, R3, …, R t+1 T R2 is a reward obtained after the cloud platform places the first virtual machine m1 under the policy π, R3 is a reward obtained after the cloud platform places the second virtual machine m2 under the policy π, R t+1 is a reward obtained after the cloud platform places the tth virtual machine m t+1 under the policy π, R T is a reward obtained after the cloud platform places the last virtual machine m T-1 under the policy π.​​​ 4. The method according to any one of claims 1 to 3, characterized in that, The rewards include any one of a quantity of virtual machines in the cloud platform, a price of a placed virtual machine in the cloud platform, an increase of a resource allocation rate of the cloud platform, and a quantity of physical hosts to be used.

5. The method according to any one of claims 1 to 3, characterized in that, The value model adopts a deep neural network or a convolutional neural network.

6. A value prediction method characterized by, The method comprises: obtaining a current state of a cloud platform, wherein the state is used to indicate allocation states of resources of each physical host in the cloud platform; inputting the state into a value model to obtain a value corresponding to the state, wherein the value model is obtained by training using known states and corresponding known values, the known states are obtained from a transition state sequence in a state sequence obtained by sampling placement records, the placement records are records of the cloud platform placing a first virtual machine from an initial state to a first unable-to-place virtual machine, the transition state sequence records states of the cloud platform at each time point from placing the first virtual machine from the initial state to the first unable-to-place virtual machine; and the known values are obtained by inputting a reward sequence into a reward determination model, the reward sequence records rewards obtained at each time point from placing the first virtual machine from the initial state to the first unable-to-place virtual machine, the rewards include any one of a quantity of virtual machines in the cloud platform, a price of a placed virtual machine in the cloud platform, an increase of a resource allocation rate of the cloud platform, and a quantity of physical hosts to be used, and the reward determination model considers both past obtained rewards and future obtainable rewards.

7. The method of claim 6, wherein, The reward determination model can be represented as: wherein the specification of the virtual machine to be created is a vector f, the set of all virtual machine specifications is F, i.e. f∈F; the current allocation state of the physical hosts in the cloud platform is denoted by s, the set of all state spaces is S, i.e. s∈S; the set of physical hosts in the cloud platform can be denoted by M, the set of candidate physical hosts is A, the currently selected candidate physical host is a, a∈A, the scheduling strategy is a function π: F x S→A, π is a probability distribution function, the set of all scheduling strategy spaces can be denoted by Π, π∈Π; when the candidate physical host a = π(s, f) is selected according to the strategy π to deploy the virtual machine to be created, the reward of the selection is The reward can be the size of the specification of the virtual machine, the state s' is determined by s, f, and a together, denoted as s' = Ω(s, f, a), the reward decay factor is denoted as γ∈[0, 1], E() is a function for calculating the expectation, and the function for the value under the strategy π is denoted as Q π (s, a).

8. The method of claim 6, wherein, The state sequence is S1, A1, R2, S2, A2, R3,..., S t ,A t ,R t+1 ,…,A T-1 ,R T ,S T Wherein, the state sequence comprises action sequence A1, A2, …, A t T-1 A1 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m1 under the policy π, A2 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m2 under the policy π, A t is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m t under the policy π, A T-1 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m T-1 under the policy π, the state sequence comprises transition state sequence S1, S2, …, S t T S1 is an initial state of the cloud platform under the policy π, S2 is a state obtained after the cloud platform places the first virtual machine a1 under the policy π, S t is a state obtained after the cloud platform places the t-1th virtual machine a t-1 under the policy π, S T is a state obtained after the cloud platform places the last virtual machine a T-1 under the policy π, the state sequence comprises reward sequence R2, R3, …, R t+1 T R2 is a reward obtained after the cloud platform places the first virtual machine m1 under the policy π, R3 is a reward obtained after the cloud platform places the second virtual machine m2 under the policy π, R t+1 is a reward obtained after the cloud platform places the tth virtual machine m t+1 under the policy π, R T is a reward obtained after the cloud platform places the last virtual machine m T-1 under the policy π.​​​ 9. The method according to any one of claims 6 to 8, characterized in that, The reward includes any one of a number of virtual machines in the cloud platform, a price of a virtual machine placed in the cloud platform, an increment of a resource allocation rate of the cloud platform, and a number of physical hosts to be used.

10. The method of any of claims 6 to 8, wherein, The value model adopts a deep neural network or a convolutional neural network.

11. A training apparatus, characterized by The training device includes a sampling module, a reward determination model, and a training module, The sampling module is configured to sample placement records to obtain a transition state sequence and a reward sequence, wherein the placement records are records of placing a first virtual machine from an initial state of the cloud platform to a first unplaceable virtual machine, the transition state sequence records states of the cloud platform at each time from placing the first virtual machine from the initial state of the cloud platform to the first unplaceable virtual machine, the state is used to indicate an allocation state of resources of each physical host in the cloud platform, the reward sequence records rewards obtained at each time from placing the first virtual machine from the initial state of the cloud platform to the first unplaceable virtual machine, and the reward includes any one of a number of virtual machines in the cloud platform, a price of a virtual machine placed in the cloud platform, an increment of a resource allocation rate of the cloud platform, and a number of physical hosts to be used. The reward determination model is configured to determine a value sequence according to the reward sequence, wherein elements in the value sequence and elements in the transition state sequence have a one-to-one correspondence relationship, and the reward determination model considers both past rewards and future rewards. The training module is configured to train the value model by using the transition state sequence and the value sequence.

12. The training device of claim 11, wherein, The reward determination model can be represented as: wherein the specification of the virtual machine to be created is a vector f, the set of all virtual machine specifications is F, i.e. f∈F; the current allocation state of the physical hosts in the cloud platform is denoted by s, the set of all state spaces is S, i.e. s∈S; the set of physical hosts in the cloud platform can be denoted by M, the set of candidate physical hosts is A, The currently selected candidate physical host is a, a∈A, the scheduling strategy is a function π: F × S → A, π is a probability distribution function, the set of all scheduling strategy spaces can be denoted by Π, π∈Π; when the candidate physical host a = π(s, f) is selected according to the strategy π to deploy the virtual machine to be created, the reward of the selection is The reward can be the size of the specification of the virtual machine, the state s' is determined by s, f, and a together, denoted as s' = Ω(s, f, a), the reward decay factor is denoted as γ∈[0, 1], E() is a function for calculating the expectation, and the function for the value under the strategy π is denoted as Q π (s, a).

13. The training apparatus of claim 11, wherein, The sampling module is configured to sample placement records to obtain a state sequence: S1, A1, R2, S2, A2, R3,..., S t ,A t ,R t+1 ,…,A T-1 ,R T ,S T The state sequence includes action sequences A1, A2, ..., A t ,…,A T-1 A1 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m1, and A2 is the action of selecting a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m2. t To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m t The action, A T-1 To select a candidate physical host from the set A of candidate physical hosts under policy π to place the first virtual machine m T-1 The actions, the above state sequence includes the transition state sequence S1, S2, ..., S t ,…,S T S1 represents the initial state of the cloud platform under policy π, and S2 represents the state obtained after placing the first virtual machine a1 on the cloud platform under policy π. t To place the (t-1)th virtual machine a on the cloud platform under policy π t-1 The obtained state, S T To place the last virtual machine a on the cloud platform under Policy π T-1 The obtained states, the above state sequence includes the reward sequence R2, R3, ..., R t+1 ,…,R T R2 is the reward obtained after placing the first virtual machine m1 on the cloud platform under policy π, and R3 is the reward obtained after placing the second virtual machine m2 on the cloud platform under policy π. t+1 To place the t-th virtual machine m on the cloud platform under policy π t+1 The reward received, R T To place the last virtual machine m on the cloud platform under strategy π T-1 The reward received.

14. Training apparatus according to any of claims 11 to 13, characterized in that The reward includes any one of a number of virtual machines in the cloud platform, a price of a virtual machine placed in the cloud platform, an increment of a resource allocation rate of the cloud platform, and a number of physical hosts to be used.

15. The training apparatus of any one of claims 11 to 13, wherein, The value model adopts a deep neural network or a convolutional neural network.

16. A prediction device, comprising: The prediction device includes an acquisition module and a prediction module, The acquisition module is configured to acquire a current state of the cloud platform, wherein the state is used to indicate an allocation state of resources of each physical host in the cloud platform. The prediction module is configured to input the state into a value model to obtain a value corresponding to the state, wherein the value model is trained using known states and corresponding known values, the known states are obtained from a transition state sequence in a state sequence obtained by sampling a placement record, the placement record is a record of placing a first virtual machine from an initial state to a first unplaceable virtual machine by the cloud platform, and the transition state sequence records states of the cloud platform at each time point from placing the first virtual machine from the initial state to the first unplaceable virtual machine; and the known values are obtained by inputting a reward sequence into a reward determination model, the reward sequence records rewards obtained at each time point from placing the first virtual machine from the initial state to the first unplaceable virtual machine by the cloud platform, and the rewards include any one of a number of virtual machines in the cloud platform, a price of a placed virtual machine in the cloud platform, a resource allocation rate increment of the cloud platform, and a number of physical hosts to be used, and the reward determination model considers both past rewards and future rewards.

17. The prediction device of claim 16, wherein, The reward determination model can be represented as: wherein the specification of the virtual machine to be created is a vector f, the set of all virtual machine specifications is F, i.e. f∈F; the current allocation state of the physical hosts in the cloud platform is denoted by s, the set of all state spaces is S, i.e. s∈S; the set of physical hosts in the cloud platform can be denoted by M, the set of candidate physical hosts is A, The currently selected candidate physical host is a, a∈A, the scheduling strategy is a function π: F × S → A, π is a probability distribution function, the set of all scheduling strategy spaces can be denoted by Π, π∈Π; when the candidate physical host a = π(s, f) is selected according to the strategy π to deploy the virtual machine to be created, the reward of the selection is The reward can be the size of the specification of the virtual machine, the state s' is determined by s, f, and a together, denoted as s' = Ω(s, f, a), the reward decay factor is denoted as γ∈[0, 1], E() is a function for calculating the expectation, and the function for the value under the strategy π is denoted as Q π (s, a).

18. The prediction device of claim 16, wherein, The state sequence is: S1, A1, R2, S2, A2, R3,..., S t ,A t ,R t+1 ,…,A T-1 ,R T ,S T Wherein, the state sequence comprises action sequence A1, A2, …, A t T-1 A1 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m1 under the policy π, A2 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m2 under the policy π, A t is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m t under the policy π, A T-1 is an action of selecting a candidate physical host from the set A of candidate physical hosts to place the first virtual machine m T-1 under the policy π, the state sequence comprises transition state sequence S1, S2, …, S t T S1 is an initial state of the cloud platform under the policy π, S2 is a state obtained after the cloud platform places the first virtual machine a1 under the policy π, S t is a state obtained after the cloud platform places the t-1th virtual machine a t-1 under the policy π, S T is a state obtained after the cloud platform places the last virtual machine a T-1 under the policy π, the state sequence comprises reward sequence R2, R3, …, R t+1 T R2 is a reward obtained after the cloud platform places the first virtual machine m1 under the policy π, R3 is a reward obtained after the cloud platform places the second virtual machine m2 under the policy π, R t+1 is a reward obtained after the cloud platform places the tth virtual machine m t+1 under the policy π, R T is a reward obtained after the cloud platform places the last virtual machine m T-1 under the policy π.​​​ 19. The prediction device according to any one of claims 16 to 18, characterized by, The rewards include any one of a number of virtual machines in the cloud platform, a price of a placed virtual machine in the cloud platform, a resource allocation rate increment of the cloud platform, and a number of physical hosts to be used.

20. The prediction device according to any one of claims 16 to 18, wherein The value model adopts a deep neural network or a convolutional neural network.

21. A training apparatus, characterized by A computer readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5.

22. A prediction device, comprising: A computer readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 6 to 10.

23. A readable storage medium characterized by, Instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 10.

Images