Excessive subscription reinforcement learning device

Abstract

A computing system includes one or more processing devices that train an oversubscription reinforcement learner at least in part by receiving a computing resource usage trajectory. At the oversubscription reinforcement learner, the training further includes generating a prototype based at least in part on the computing resource usage trajectory. The training also includes generating an excess subscription rate based at least in part on the prototype. The training also includes outputting a prototype feedback query and / or an excess subscription rate feedback query. The training also includes receiving prototype feedback input and / or excess subscription rate feedback input. Based at least in part on the computing resource usage trajectory, the prototype, and the prototype feedback input and / or the excess subscription rate feedback input, the training further includes computing an objective function value and training an excess subscription reinforcement learner based at least in part on the objective function value.

Description

Background Art

[0001] The term "oversubscription" is used to describe a scenario in which a system provides more resources or services to a user or entity than its available capacity, assuming that not all users will utilize the allocated capacity simultaneously or completely. In cloud services, cloud providers frequently oversubscribe their computing resources in order to allow a larger proportion of those computing resources to be utilized. Thus, oversubscription can allow cloud service providers to utilize unused capacity and can operate their data centers more efficiently.

[0002] Designing an oversubscription strategy raises challenges related to overshooting and undershooting predicted computing resource utilization. Once a system is oversubscribed, overload as well as underutilization may occur at any time. Predicting user demand and utilization behavior with the right granularity and cadence is often difficult. Aggressive oversubscription strategies unfairly treat an indeterminate number of users who cannot access resources, a condition known as overload. On the other hand, conservative oversubscription strategies may result in resources and capacity going unused, leading to inefficient resource usage or waste. Summary of the invention

[0003] A computing system is provided, the computing system including one or more processing devices for training an oversubscription reinforcement learner during a training phase. Training the oversubscription reinforcement learner includes receiving a plurality of computing resource usage traces. Training the oversubscription reinforcement learner also includes generating a plurality of prototypes encoding corresponding prototype traces at the oversubscription reinforcement learner based at least in part on the plurality of computing resource usage traces. Training the oversubscription reinforcement learner also includes generating an oversubscription rate based at least in part on the plurality of prototypes. Training the oversubscription reinforcement learner also includes outputting a prototype feedback query associated with a prototype in the plurality of prototypes to a user interface, and / or outputting an oversubscription rate feedback query indicating an oversubscription rate to the user interface. Training the oversubscription reinforcement learner also includes receiving a prototype feedback input via the user interface in response to outputting the prototype feedback query, and / or receiving an oversubscription rate feedback input via the user interface in response to outputting the oversubscription rate feedback query. Based at least in part on the plurality of computing resource usage traces, the plurality of prototypes, and the prototype feedback input and / or the oversubscription rate feedback input, training the oversubscription reinforcement learner also includes calculating an objective function value. The oversubscription reinforcement learner is trained based at least in part on the objective function value.

[0004] This summary is provided to introduce in simplified form a selection of concepts further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A computing system including a plurality of nodes according to an example embodiment is schematically described.

[0006] Figure 2 Schematically shows the Figure 1 A node of an example, multiple virtual machines are executed at the node and oversubscription is performed at the node.

[0007] Figure 3 Schematically shows the Figure 1 An example computing system of when an oversubscription reinforcement learner is executed at one or more processing devices.

[0008] Figure 4 Schematically shown in additional detail according to Figure 3 Example of an oversubscription reinforcement learner when an oversubscription rate is generated.

[0009] Figure 5 Schematically shows the Figure 3 Example of an oversubscription reinforcement learner when the prototype feedback input and the oversubscription rate feedback input are used to compute the objective function value.

[0010] Figure 6 Schematically shows the Figure 5 Objective function of the oversubscription reinforcement learner for an example.

[0011] Figure 7 Schematically shows the Figure 6 Calculation of a representative capacity term included in the objective function of an example.

[0012] Figure 8 Schematically shows the Figure 6 An example of the computation of the interpretable term included in the objective function.

[0013] Fig. 9 Schematically shows the Figure 6 Example of calculation of the diversity term included in the objective function.

[0014] Fig.10 Schematically shows the Figure 3Example of a selection of prototypes for prototype feedback query generation.

[0015] Fig.11 Shown according to Figure 3 An example prototype feedback query that can be presented to a user at a graphical user interface.

[0016] Fig.12 Shown according to Figure 3 An example oversubscription rate feedback query that may be presented to a user at a graphical user interface.

[0017] Fig.13A Schematically shows the Figure 5 An example of processing of a merged input received as prototype feedback input.

[0018] Fig. 13B Schematically shows the Figure 5 An example of processing of a segmented input received as a prototype feedback input.

[0019] Fig.14 Schematically shows the Figure 3 An example computing system during an inference phase performed after a training phase.

[0020] Fig.15A Shown according to Figure 1 A flowchart of an example method for oversubscribing a reinforcement learner that can be performed at a computing system.

[0021] Fig. 15B shows that in some examples may be performed during training of an oversubscribed reinforcement learner Fig.15A Additional steps of the method.

[0022] Fig. 15C It shows that it can be Fig. 15B Additional steps of the method are performed in the example shown for generating multiple trajectory embedding vectors.

[0023] Fig.15D shows that a method may be implemented to use imitation learning during the training of an oversubscribed reinforcement learner Fig.15A Additional steps of the method.

[0024] Fig.15E shows that can be performed during the inference phase Fig.15A Additional steps of the method.

[0025] Fig.16 Shows where you can instantiate Figure 1 A schematic diagram of an example computing environment for a computing system of FIG. DETAILED DESCRIPTION

[0026] The problem of generating an oversubscription strategy is solved by the system and method discussed herein. Using such a system and method, an oversubscription strategy can be generated to have a low risk of overload and a high level of resource utilization.

[0027] Previous oversubscription approaches typically have a low level of generalizability across different scenarios where oversubscription can be performed in cloud computing environments. For example, in these previous works, the problem of selecting an oversubscription rate in cloud computing has been formulated as a variation of the online bin packing problem with constraints. Such approaches solve resource allocation strategies instead of designing oversubscription policies. Other existing works focus on using migration to alleviate overload situations. However, general oversubscription policies with the competing goals of efficiently utilizing unused capacity while reducing overload risk have not been fully developed.

[0028] The optimal oversubscription problem is proposed in this paper as a sequential decision making problem with constraints or resource limitations. Predicting future utilization behavior based on historical observations through traditional supervised learning methods is insufficient because such methods are unaware of the interaction between users and their environment. In methods such as constrained reinforcement learning that aim to solve Markov decision processes (MDPs) with constraints, balancing different possible competing objectives is challenging. Typically, since the problem does not have a convex solution space, such methods cannot be guaranteed to converge to the optimal solution. Moreover, in constrained reinforcement learning, it is difficult to design an ideal set of constraints, and even difficult to design a reasonable learning environment with correct feedback / rewards.

[0029] The oversubscription reinforcement learner that solves the above problems is discussed below. Instead of traditional reinforcement learning (RL) methods, imitation learning (IL) can be used to solve the MDP constraint problem where the expert policy satisfies the constraints. In the prototype imitation learning method (PROTOHAIL) discussed below, the reinforcement learner learns to take actions selected based on a set of learned prototypes. Prototypes are data instances that represent equivalence classes of expert trajectories. Facilitated by the interpretability of prototype IL, human-in-the-loop (Human-in-the-loop) training is used to guide the model closer to the realization of the goal of the oversubscription problem. This human-in-the-loop training method can allow the oversubscription reinforcement learner to generate appropriate oversubscription rates even when the utilization data used in training is noisy, incomplete, or sparse.

[0030] Figure 1 and Figure 2 Provides context for the problem of setting oversubscription rates in cloud computing environments. Figure 1Schematically depicts a computing system 10 including a plurality of nodes 11 according to an example. Figure 1 As depicted, nodes 11 are multiple networked physical computing devices, which may be located, for example, in a data center. Each of nodes 11 includes one or more processing devices 12. For example, one or more processing devices 12 may include one or more central processing unit (CPU) cores 12A. One or more processing devices 12 may additionally or alternatively include one or more additional processing devices 12B, such as one or more graphics processing units (GPUs), field programmable gate arrays (FPGAs), dedicated hardware accelerators, and / or other types of processing devices 12. Each of nodes 11 also includes a memory 14 communicatively coupled to one or more processing devices 12. Memory 14 may, for example, include one or more volatile memory devices and / or one or more non-volatile memory devices.

[0031] Figure 1 The computing system 10 shown in the example of also includes one or more input devices 16 and one or more output devices 18 that are communicatively coupled to the plurality of nodes 11. The one or more input devices 16 may include, for example, a keyboard, a mouse, a touch screen, a microphone, an optical sensor, and / or one or more other types of input devices. The one or more output devices 18 may include, for example, a display, a speaker, a tactile feedback device, and / or one or more other types of output devices. Figure 1 In the example of the embodiment of the present invention, the one or more output devices 18 include a display. The one or more input devices 16 and the one or more output devices 18 are communicatively coupled to the one or more processing devices 12, so that a graphical user interface (GUI) 30 that allows a user to interact with the one or more processing devices 12 is displayed at the one or more output devices 18 and is configured to receive input via the one or more input devices 16.

[0032] In some examples, one or more input devices 16 and one or more output devices 18 displaying GUI 30 are located at a client computing device 17, which is included in computing system 10 and coupled to multiple nodes 11 via network 19. Client program 31 executed at client computing device 17 is configured to send input data to one or more nodes 11 and receive output data from one or more nodes 11. In addition, client program 31 is configured to receive user input and transmit output to the user via GUI 30. In other examples, GUI 30 can be implemented at a node 11 of multiple nodes 11 that is locally coupled to one or more input devices 16 and one or more output devices 18. In some examples, one or more input devices 16 and one or more output devices 18 can also be located in multiple physical computing devices.

[0033] exist Figure 1 In the example of FIG. 1 , multiple virtual machines (VMs) 20 are executed at the depicted node 11. Each of the VMs 20 can utilize one or more CPU cores 12A and / or one or more additional processing devices 12B as computing resources. In addition, each of the VMs 20 utilizes a corresponding portion of an allocated memory (configuration memory) 14A. The network bandwidth used during communication between the VMs 20 and the client computing device 17 over the network 19 can also be a computing resource utilized by the VMs 20.

[0034] In a cloud computing setting, processing device, memory, and network bottlenecks may occur, among which processor bottleneck is usually the most common computing resource bottleneck. Figure 1 The computing system 10 is shown when a processing device bottleneck occurs at the node 11. The processing bottleneck causes the memory 14 of the node 11 to not be fully allocated to the plurality of VMs 20, thereby leaving unutilized stranded memory 14B.

[0035] Figure 2 The underutilization of memory at node 11 is shown in more detail. Figure 2 In the example of FIG. 1 , three VMs 20A, 20B, and 20C are executed at a node 11. A CPU core 12A of the node 11 is fully allocated to the VMs 20A, 20B, and 20C, and a portion of the memory 14 that is not allocated to the VMs 20A, 20B, and 20C is reserved as a retained memory 14B. A portion of a physical CPU that is allocated to a VM 20 is called a virtual CPU (vCPU).

[0036] Figure 2 Also shown is node 11 when oversubscription has been performed, so that an additional VM 20D is executed at node 11. Although the vCPU shares nominally allocated to VMs 20A, 20B, and 20C have the same amount as the vCPU shares that would be allocated without oversubscription, the vCPU shares actually used by VMs 20A, 20B, and 20C are less than the nominal amount. Therefore, in addition to VMs 20A, 20B, and 20C, an additional VM 20D is executed on CPU core 12A of node 11. Memory is also allocated to VM 20D, thereby reducing the amount of retained memory 14B. Accordingly, Figure 2 Demonstrated the increased efficiency in computing resource utilization achieved using oversubscription.

[0037] The oversubscription rate for node 11 can be adjusted dynamically, which is discussed further below. For different computing workloads hosted on VM 20, vCPU usage (usage, utilization rate) changes over time according to different patterns. For example, in areas with similar time zones, services like email and work-related software show daily and weekly patterns. Such services usually receive peak traffic during the daytime of weekdays, making vCPU usage higher during these time periods and lower at night and weekends. On the other hand, services that provide social media and video game applications show different vCPU usage patterns, where peak usage occurs during the user's free time. Other non-user-oriented services that run regularly, such as monitoring and maintenance services, sometimes do not show daily or weekly patterns, but show patterns caused by the underlying configuration set by the service team. The multiple vCPU usage patterns of different services encourage the adaptive oversubscription of the vCPU of the VM used for such services. For example, during the predicted low vCPU usage period, the service may be oversubscribed.

[0038] Now go to Figure 3 , illustrates the computing system 10 when training an example oversubscription reinforcement learner 50 during a training phase. The oversubscription reinforcement learner 50 is trained to output an oversubscription rate 60. Training the oversubscription reinforcement learner 50 includes receiving a plurality of computing resource usage traces 40 at one or more processing devices 12. Figure 3 As shown in the example of , the computing resource usage trajectory 40 is a sequence of state-action pairs, each state-action pair including a state 42 and a corresponding action 44. The state-action pair is associated with a corresponding time step (timestep) 46. The state 42 indicates the computing resource usage level at the time step 46, and the action 44 indicates the oversubscription rate 60. The computing resource usage trajectory 40 can be expressed as τ = {(s1, a1, ..., s T ,a T )}, where T is the time horizon, is the state at time t, and is the action at time t.

[0039] The technique presented in this paper for generating an oversubscription rate of 60 utilizes prototype learning. In prototype learning, a machine learning model compares new inputs to prototypes that serve as exemplar situations. Machine learning models trained using prototype learning may exhibit inherent interpretability due to the model's reliance on a small number of prototypes for its behavior. Thus, a policy learned by a reinforcement learner can be interpreted by a user in terms of prototypes. Figure 3In the example of , training the oversubscription reinforcement learner 50 includes generating a plurality of prototypes 52 encoding corresponding prototype trajectories 54 at the oversubscription reinforcement learner 50. The prototypes 52 are generated based at least in part on the plurality of computing resource usage trajectories 40. Figure 3 In the example of FIG. 1 , one or more processing devices 12 may use vector The prototype is generated in the form of 52. k Indexed by k={1,2,…,K}; K<<T.

[0040] The one or more processing devices 12 are also configured to generate an oversubscription rate 60 based at least in part on the plurality of prototypes 52 . Figure 4 The oversubscription reinforcement learner 50 is schematically shown in more detail when generating an oversubscription rate 60 according to an example. Figure 4 In the example of , the oversubscription reinforcement learner 50 includes a trajectory encoder 70 that receives a plurality of computing resource usage trajectories 40. The trajectory encoder 70 may be a transform encoder, a long short-term memory (LSTM) encoder, or some other type of sequence encoder model. At the trajectory encoder 70, one or more processing devices 12 generate a plurality of trajectory embedding vectors 72 corresponding to the computing resource usage trajectories 40. The trajectory encoder 70 converts the function Applied to calculate resource usage trajectory τ t To generate an m-dimensional trajectory embedding vector h t =f(τ t ). The trajectory embedding vector h t and the prototype vector p k Each has the same length m.

[0041] like Figure 4 As shown, the oversubscription reinforcement learner 50 further includes a prototype similarity module 74, which is configured to receive the plurality of trajectory embedding vectors 72. At the prototype similarity module 74, for each trajectory embedding vector h t The one or more processing devices 12 are further configured to calculate the trajectory embedding vector h t With each prototype p k The similarity level between the prototype similarity module 74 can be, for example, the L2 norm. Alternatively, the prototype similarity module 74 can use some other similarity metric, such as the L1 norm. The vector of similarity values ​​calculated at the prototype similarity module 74 can be expressed as P = [sim (f (τ t ),p1),…,sim(f(τ t ),p K )].

[0042] The oversubscription reinforcement learner 50 also includes a policy layer 76, where the product of the prototype similarity vector P and the weight vector 78 is calculated. The weight vector 78 can be expressed as w=[w1, ..., w k ]. Therefore, the oversubscription rate 60 is calculated as a=wP. Therefore, at the oversubscription reinforcement learner 50 , a corresponding oversubscription rate 60 is calculated for each of the computing resource usage traces 40 .

[0043] In a human-in-the-loop (HITL) approach utilized when training the oversubscription reinforcement learner 50, the one or more processing devices 12 generate a prototype feedback query 56 associated with a prototype 52 in the plurality of prototypes 52. The one or more processing devices 12 also output the prototype feedback query 56 to a user interface, which is Figure 3 30 in the example of . At the GUI 30, the user may select a prototype feedback input 58. In response to outputting the prototype feedback query 56, one or more processing devices 12 may accordingly receive the prototype feedback input 58 via the user interface.

[0044] The one or more processing devices 12 also generate an oversubscription rate feedback query 62 indicating the oversubscription rate 60. The oversubscription rate feedback query 62 is output to the user interface together with or separately from the prototype feedback query 56. The user can select an oversubscription rate feedback input 64 at the GUI 30 for transmission to the one or more processing devices 12. Thus, the one or more processing devices 12 can receive the oversubscription rate feedback input 64 via the user interface in response to outputting the oversubscription rate feedback query 62. As discussed in further detail below, when calculating the objective function value 68, the prototype feedback input 58 and the oversubscription rate feedback input 64 can be used as inputs to the objective function 66. Thus, the user feedback shapes the oversubscription policy generated at the oversubscription reinforcement learner 50 during the training process.

[0045] The one or more processing devices 12 calculate an objective function value 68 based at least in part on the plurality of computing resource usage trajectories 40, the plurality of prototypes 52, the prototype feedback input 58, and the oversubscription rate feedback input 64. The one or more processing devices 12 then train the oversubscription reinforcement learner 50 based at least in part on the objective function value 68. In some examples, the objective function 66 is a loss function for which the one or more processing devices 12 are configured to estimate a minimum. Alternatively, the objective function 66 may be a reward function for which the one or more processing devices 12 are configured to estimate a maximum. During training of the oversubscription reinforcement learner 50, the one or more processing devices 12 may modify the weights of the trajectory encoder 70 and the policy layer 76 using a stochastic gradient descent algorithm that receives the objective function value 68 as an input.

[0046] Figure 5The prototype feedback input 58 is shown in more detail. Figure 5 As shown in the example of , the prototype feedback input 58 can be an approval input 58A, a disapproval input 58B, a merge input 58C, a split input 58D, or an update input 58E. Alternatively, the user can provide a null input, thereby skipping the step of providing the prototype feedback input 58 for the prototype 52. The prototype feedback input 58 can be used to calculate a scaling factor 69 applied to the terms of the objective function 66, such as Figure 5 For example, as discussed in further detail below, the number of approval inputs 58A and disapproval inputs 58 received in the prototype feedback input 58 can be used to calculate one or more of the scaling factors 69. The oversubscription reinforcement learner 50 can merge two prototypes 52 in response to receiving the merge input 55C, and can split the prototype 52 into multiple prototypes 52 in response to receiving the split input 58D. The user can also edit one or more parameters of the prototype 52 by providing an update input 58E.

[0047] Figure 5 In addition, an oversubscription rate feedback input 64 is shown in detail. The oversubscription rate feedback input 64 can be an approval input 64A or a disapproval input 64B. The number of approval inputs 64 and disapproval inputs 64B received in the oversubscription rate feedback input 64 can also be used to calculate one or more scaling factors 69.

[0048] In addition to using the prototype 52, Figure 3The oversubscription reinforcement learner 50 also utilizes imitation learning. However, in some scenarios, previous IL methods may not be suitable for the oversubscription problem. Learning decision-making strategies or prediction models may be challenging in the presence of (1) system noise (sample / feedback sparsity / noise, delayed signals, cognitive biases, suboptimal trajectories, etc.), (2) non-stationarity, and / or (3) unsafe exploration with high cost safety issues. Moreover, in complex decision problems, appropriate reward design may be difficult to handle. Therefore, fully data-driven learning may be risky. Even when learning from demonstrations, such as in reverse RL, offline RL, or previous forms of IL, trajectories may still be noisy, and imperfect human guidance may also lead to errors. Some related methods utilize prior knowledge, such as value-based priors or preference-based priors on the decision space, while other methods include constraints on imitation targets based on domain knowledge or encoded knowledge as reward shaping functions. In addition, some methods use statistical models as priors, such as probabilistic model-based imitation learning for handling uncertainty in trajectories. Examples of existing interactive HITL simulation frameworks include Guided Behavior Cloning, DAGGER, and HgDAGGER. However, such HITL simulation frameworks have simple and inefficient feedback elicitation mechanisms and do not support multi-level feedback.

[0049] When training the oversubscription reinforcement learner 50, imitation learning can be performed using data provided by experts. Figure 5 As shown in the example of , training the oversubscription reinforcement learner 50 also includes receiving one or more user-provided computing resource usage trajectories 80. The one or more user-provided computing resource usage trajectories 80 may each include a plurality of user-provided state-action pairs associated with corresponding time steps. In some examples, the one or more user-provided computing resource usage trajectories 80 may be received via a user interface. The user may, for example, select one or more user-provided computing resource usage trajectories 80 from a set of historical computing resource usage trajectory data as examples of patterns that frequently occur in historical data.

[0050] The one or more processing devices 12 also perform imitation learning of the plurality of prototypes 52 based at least in part on the one or more user-provided computing resource usage traces 80. The one or more user-provided computing resource usage traces 80 may be used when computing one or more terms 67 of the objective function 66, as will be discussed in further detail below.

[0051] We now provide a formal description of prototype imitation learning. Prototype imitation learning is a type of imitation learning that learns by comparing the generated prototypes 52 with the ones from the expert policy (in Figure 5In the example of FIG. 8 , the computing resources provided by the user are aligned with the reference prototypes (prototype trajectories) of the trajectories 80 to make decisions. Specifically, each prototype 52 can be represented by a prototype pattern received from an expert. Prototype imitation learning learns a metric space where decision making can be performed by calculating the distance to the prototype strategy of the expert.

[0052] Assume that τ={(s1,a1,…,s T ,a T )} is a computing resource usage trace 80 provided by the user, where T is the time range, is the state at time t, and is the action at time t. The goal of prototype imitation learning is to learn a representative prototype p of an equivalence class that interpretably represents patterns in computational resource usage data. k Therefore, the prototype p k It can be used as a reference for decision making and for analogical explanation of computing resource usage trends. When a new input state is received, relative to the prototype p in the learned latent space, k The similarity of each representative trajectory of the input state is measured. The prediction of the new action can then be derived and explained by the closest prototype trajectory ( Figure 3 In the example above the oversubscription rate is 60).

[0053] In the oversubscription problem, the state space is factored using a hybrid feature vector that includes temporal features. The action / decision space is a continuous space of possible oversubscription rates 60. Therefore, the oversubscription problem may be too complex to be solved using direct behavioral cloning. Instead, trajectories are embedded into a latent space of equivalence classes or prototypes, and approximate symmetries between trajectory patterns are exploited.

[0054] Prototype imitation learning can be performed in three main stages. The first stage is the prototype discovery stage, where trajectories are divided into K groups. From the trajectory groups, the oversubscription reinforcement learner 50 learns the prototype projection trajectory p k , where p k is an m-dimensional vector. The second stage is the action strategy learning stage, in which the action strategy is compared with similar prototypes p in multiple states. k The third stage is the feedback stage, where the oversubscribed reinforcement learner 50 receives feedback from the human-machine loop to evaluate the prototype p k and get feedback on the level of overload risk caused by the strategy.

[0055] Figure 6 The objective function 66 of the oversubscription reinforcement learner 66 according to one example is schematically shown in more detail. Figure 6The objective function 66 shown in the example of is the loss function that the oversubscription reinforcement learner 50 is trained to approximately minimize. Figure 6 In the example of , the objective function 66 includes a representative capacity term 67A, an interpretability term 67B, a diversity term 67C, a behavior cloning term 67D, and an adversarial imitation learning term 67E. Prototype discovery can be performed using the representative capacity term 67A and the diversity term 67C, and prototype projection can be performed using the interpretability term 67B. The behavior cloning term 67D and the adversarial imitation learning term 67E can be used to implement imitation learning. The objective function 66 can be a weighted sum of multiple terms 67, each of which has an associated hyperparameter weight.

[0056] Figures 7 to 10 The calculation of the terms of the objective function 66 in some examples is schematically shown. Figure 7 As shown in the example of , the representative capacity term 67A may be proportional to the sum of the distances 90A between the plurality of prototypes 52 and the corresponding closest trajectory embedding vectors 72 to those prototypes 52. The representative capacity term 67A may be given by the following equation: When the representative capacity term 67A is included in the objective function 66 , the trajectory embedding vectors 72 are grouped into K trajectory clusters 92 in the embedding space around the corresponding prototype 52 .

[0057] Figure 8 The calculation of the explainability term 67B according to one example is schematically shown. Figure 8 In the example of FIG. 5 , one or more processing devices 12 group the plurality of trajectory embedding vectors 72 into a plurality of trajectory clusters 92 corresponding to the plurality of prototypes 52. The trajectory clusters 92 may be trajectory clusters caused by the representative capacity terms 67A, such as Figure 7 When the oversubscription reinforcement learner 50 is trained using the trajectory clusters 92, the prototypes 52 can be matched with the representative components of the trajectory clusters 92. In the example of training the oversubscription reinforcement learner 50 using both the representative capacity term 67A and the explainability term 67B, as shown in FIG. Figure 6 As shown in the example of , one or more processing devices 12 may iteratively adjust the clustering structure of the trajectory embedding vector 72 used to calculate the interpretability term 67B.

[0058] exist Figure 8 In the example of , the explainability term 67B is proportional to the sum of the distances 90 between the plurality of prototypes 52 and the respective closest trajectory embedding vectors 72 within the corresponding trajectory clusters 92 associated with those prototypes 52. The explainability term 67B may be given by the following equation: In the above equation, τk is closest to p k The distance 90B is the L2 norm, and the interpretability term 67B is calculated as the average of the minimum distances. Therefore, the interpretability term 67B can allow each prototype 52 to be simulated as a trajectory embedding vector 72 of the real-world computing resource usage trajectory 40.

[0059] Fig. 9 The calculation of the diversity term 67C included in the objective function 66 is schematically shown. Fig. 9 In the example of , the diversity term 67C is proportional to the sum of the maximum distances 90C between pairs 94 of prototypes 52. The diversity term 67C may be given by the following equation: In the above equation, distance 90C is the L2 norm, and diversity term 67C is calculated as the average of the maximum distances 90C between prototypes 52 included in pair 94. Diversity term 67C penalizes prototypes 52 that are close to each other, thereby allowing the oversubscription reinforcement learner 50 to model a wider range of computing resource usage trajectories 40 using prototypes 52.

[0060] Return to Figure 6 In the example of , imitation learning is implemented at least in part using the behavior cloning term 67D and the adversarial imitation learning term 67E included in the objective function 66. In the equations for the behavior cloning term 67D and the adversarial imitation learning term 67E discussed below, π(a|τ t ,P) is a policy layer 76 that learns to take actions aligned with the prototype 52. The policy layer 76 can be represented as follows: In the above equation, is the SoftMax layer, and is the negative Euclidean distance indicating the distance between the trajectory embedding vector 72 and the prototype 52.

[0061] The behavior clone term 67D measures the extent to which the oversubscription rate 60 calculated at the oversubscription reinforcement learner 50 at each time step simulates the user-provided oversubscription rate included in the user-provided computing resource usage trace 80. The behavior clone term 67D may be given by the following equation: In the above equation, τ E is the computing resource usage trace 80 provided by the user, and π E is the policy of the user-provided computing resource usage trajectory 80. Therefore, the oversubscription reinforcement learner 50 can learn to imitate the computing resource usage trajectory 80 provided by the user.

[0062] When performing adversarial imitation learning, the oversubscription reinforcement learner 50 may learn to reduce the Jensen-Shannon (JS) divergence between the distribution of computing resource usage traces generated by the oversubscription reinforcement learner 50 and the distribution of computing resource usage traces 80 provided by the user. The adversarial imitation learning term 67E may be given by the following equation: Where D KL is the Kullback-Leibler divergence, is the distribution of state-action pairs with policy π, and γ is the discount factor. Accordingly, including the adversarial imitation learning term 67E in the objective function can cause the oversubscription reinforcement learner 50 to generate an oversubscription rate 60 having a distribution similar to the distribution of oversubscription rates included in the user-provided computing resource usage trace 80.

[0063] By combining the above discussed item 67, Figure 6 The overall objective function 66 shown in can be given by: In the above equation, w1,w2,w3,w4∈[0,1] are hyperparameters used to balance the weights of the loss terms. Can be equal to or their linear combinations.

[0064] The policy π can be reinterpreted as a quadratic model. P is represented as the similarity vector [sim(f(τ t ),p1),…,sim(f(τ t ),p K )]. The policy π can be rewritten in quadratic form as follows: in is a fully connected layer with only linear operators. Then π can be further rewritten as: where b i =1,…,K are the values ​​of the linear neurons in the fully connected layer. The terms of π can be converted into the following linear form: On the right side of the above equation, the first term is f(τ t ), the second term can be regarded as where w k =2b k p k , and the third term can be viewed as f(τ t ) in the constant term.

[0065] Starting from the quadratic rewriting of the policy π, the actions can be interpreted as t ) is the sum of K quadratic functions with the same sign of quadratic coefficients. Therefore, the action and f(τ t ) can be decomposed into at most two fragments, and in each fragment, the action and f(τ t ) is monotonic. Using the above observations, the policy π can be more easily interpreted.

[0066] Return to Figure 5 , further discussing in detail the incorporation of user feedback into the objective function 66. In the field of VM oversubscription, system noise is prevalent in the computing resource usage trace 40. This system noise may lead to suboptimal prototype embeddings, prototype selections, and final policies. To address the system noise, the oversubscription reinforcement learner 50 utilizes active feedback to improve the policies learned on the prototypes 52. Human feedback is received in the following contexts: (1) feedback on the prototypes, including the quality of the prototype embeddings and the prototype alignment and diversity, and (2) feedback on the risk level of the proposed oversubscription rate 60.

[0067] To elicit relevant knowledge from human feedback, sometimes it is not sufficient to perform self-generated queries on human feedback at fixed or random intervals, which may actually be counterproductive in some scenarios. Generating learned queries at relevant points may instead allow useful knowledge to be obtained. When determining the point at which the human-machine loop is prompted to provide feedback, a relevant prototype 52 and a predicted oversubscription rate 60 are identified.

[0068] Fig.10 The selection of a prototype 52 for prototype feedback query generation is schematically illustrated. Fig.10 The prototype 52 in the example is associated with a trajectory cluster 92 including a plurality of trajectory embedding vectors 72. The one or more processing devices 12 may calculate a cluster entropy 96 for the trajectory cluster 92 associated with the prototype 52. The one or more processing devices 12 may also output a prototype feedback query 56 associated with the prototype 52 in response to determining that the cluster entropy 96 is greater than a predefined uncertainty threshold 98.

[0069] Alternatively, the one or more processing devices 12 may select a prototype 52 as the subject of a prototype feedback query 56 based at least in part on an average distance 99 between the prototype 52 and the trajectory embedding vector 72 included in the trajectory cluster 92 associated with the prototype 52. The one or more processing devices 12 may determine that the average distance 99 is included in a predetermined number of highest average distances among the plurality of prototypes 52. The one or more processing devices 12 may also output a prototype feedback query 56 associated with the prototype 52 in response to the determination. Thus, the prototype 52 having the highest n average distances may be selected for the prototype feedback query, where n is a predetermined number.

[0070] Further details regarding the generation of the prototype feedback query 56 are provided below. Can include sets of prototypes and the set of trajectories plus predictions for which they request feedback gather Actions in is the oversubscription rate 60 at the subsequent time step. Figure 5 In the example, with one or more prototypes p q Related prototype feedback query 56 and a collection of one or more trajectories plus predictions The related oversubscription rate feedback query 62 is separately output to the GUI 30. In other examples, the prototype feedback query 56 and the oversubscription rate feedback query 62 may be output to the GUI 30 in combination for presentation to the user.

[0071] In the above expression for query q, the set p of prototypes q Can be p q =p μ ∩p D Given, where p μ is defined as: In the above equation, Tr is the uncertainty threshold. The uncertainty is the clustering entropy μ of the prototype p. D The prototypes include the top-n average distances to the trajectory embedding vector h and are given as follows: In the above equation, Represents "model".

[0072] The set is determined using both the prediction uncertainty and the overload risk calculated via differential entropy (continuous value) If the predicted oversubscription is less than the expected actual usage Then the predicted oversubscription is an overload risk.

[0073] Fig.11 An example prototype feedback query 56 is shown that may be presented to a user at the GUI 30. Fig.11 In the example of , the prototype feedback query 56 includes a prototype visualization 57 of the prototype p0. The example prototype feedback query 56 also includes an indicator that the prototypes p0 and p5 behave unstable, indicating that these prototypes are included in the high uncertainty prototype set p μ In addition, the prototype feedback query 56 includes an indicator that the prototype pairs (p0, p2), (p1, p4), and (p2, p5) are redundant. Redundant prototype pairs can be identified, for example, by calculating distances 90C in the embedding space between pairs 94 of the prototypes 52. In such an example, for some predetermined number of pairs n, pairs 94 with the bottom n distances 90C can be indicated in the prototype feedback query 56 as potentially redundant.

[0074] Fig.11 The example prototype feedback query 56 also includes an approval feedback interface element 59A, a disapproval feedback interface element 59B, a merge feedback interface element 59C, a split feedback interface element 59D, and an update feedback interface element 59E. By selecting the approval feedback interface element 59A, the disapproval feedback interface element 59B, the merge feedback interface element 59C, the split feedback interface element 59D, or the update feedback interface element 59E, the user can enter the approval input 58A, the disapproval input 58B, the merge input 58C, the split input 58D, or the update input 58E as the prototype feedback input 58, respectively.

[0075] Fig.12 An example oversubscription rate feedback query 62 that may be displayed at the GUI 30 is shown. Fig.12 The example oversubscription rate feedback query 62 shows the oversubscription rate 60 for the computing resource usage trajectory 40 as a function of time and a trajectory prediction visualization 63 of the computing resource usage trajectory 40. The oversubscription rate feedback query 62 also includes an approval feedback interface element 65A and a disapproval feedback interface element 65B, via which a user can enter an approval input 64A or a disapproval input 64B, respectively, as an oversubscription rate feedback input 64.

[0076] Return to Figure 6 , the calculation of the representative capacity term scaling factor 69A, the interpretability term scaling factor 69B, and the diversity scaling factor 69C are now discussed in more detail. Approval input 58A and disapproval input 58B can be indicated as: (↑ / ↓=+1 / -1), respectively. For a given prototype p j ∈p q , current cumulative feedback Similarly, for the trajectory-action pair indicated in the oversubscription rate feedback query 62,

[0077] Fig.13A The processing of the merged input 58C received as the prototype feedback input 58 is schematically illustrated. In response to receiving the merged input 58C as the prototype feedback input 58, the one or more processing devices 12 generate a merged prototype 53 based at least in part on the prototype 52 indicated in the prototype feedback query 56 and the additional prototype 52. j and p k When merged, the embedding of the merged prototype 53 can be calculated as p j and p k The average value of the embeddings.

[0078] Fig. 13B The processing of the segmentation input 58D received as the prototype feedback input 58 is schematically illustrated. In response to receiving the segmentation input 58D as the prototype feedback input 58, the one or more processing devices 12 generate a first segmentation prototype 55A and a second segmentation prototype 55B based at least in part on the prototype 52. k When segmented, the first segmented prototype 55A and the second segmented prototype 55B may each have a trajectory given by: In the above equation, τ k belongs to p k τ′ in the above equation model can be selected by re-clustering the trajectory τ. For example, performing re-clustering can include evaluating the argTop-n function in the above equation with respect to the value of the distance measure, thereby selecting the trajectory τ whose corresponding trajectory embedding vector f(τ) has the top n distances from its corresponding prototype 52 for the trajectory τ. Then, one or more processing devices 12 can re-cluster the trajectory τ by defining a new cluster including the output of the argTop-n function.

[0079] Using the exponential suggestion gate, the HITL feedback included in the prototype feedback input 58 and the oversubscription rate feedback input 64 is incorporated into the objective function 66, and the term 67 of the objective function 66 is controlled and scaled via the exponential suggestion gate. The suggestion potential selectively changes the objective function value 68 based on the context and user feedback. Therefore, the suggestion potential can drive training towards model parameters that are more closely aligned with the user's goals. The suggestion potential gate is a product term with the exponential form in is the current cumulative feedback, and 0 is the neutral feedback. Because Φ is exponential, Yes A function that scales to [-1, +1]. Thus, unbounded values ​​of Φ are avoided.

[0080] Utilizing Prototypes Feedback and actions available on Feedback and prototypes The objective function 66 of the oversubscription reinforcement learner 50 can be modified as follows: The proposed potential may be to amplify or reduce the correlation term based on the feedback obtained67. In the above equation, for τ i , p j ,p k The dependencies are merged into While the hyperparameters w1, w2, w3, w4 are static hyperparameters and control the relative contributions of term 67, the suggestion potential dynamically controls the contribution to the objective function value 68 for a given prototype and prediction context. Thus, the suggestion potential can allow the oversubscription reinforcement learner 50 to dynamically navigate the objective function laterally toward parameter values ​​that more accurately meet the user's goals.

[0081] Fig.14 1 shows the computing system 10 during the inference phase performed after the training phase. In the inference phase, the oversubscription reinforcement learner 50 is used to dynamically set the inference time oversubscription rate 120 in real time. Fig.14 As shown in the example of , one or more processing devices 12 receive inference time computing resource usage data 110. The inference time computing resource usage data 110 may include a plurality of inference time states 112 paired with corresponding inference time actions 114 and occurring at corresponding inference time steps 116. The inference time states 112 may be prior inference time computing resource usage levels, and the inference time actions 114 may be prior inference time oversubscription rates.

[0082] The one or more processing devices 12 set an inference time oversubscription rate 120 at the oversubscription reinforcement learner 50 based at least in part on the inference time computing resource usage data 110. The one or more processing devices 12 may set the inference time oversubscription rate 120 during the inference phase without receiving further feedback from the user. Alternatively, in some examples, one or more phases of additional training with feedback queries and inputs may be performed after deployment to correct for distribution shift.

[0083] The one or more processing devices 12 also allocate computing resources to the plurality of virtual machines 20 specified by the inference time oversubscription rate 120. Thus, the one or more processing devices 12 may allocate vCPU, memory, network bandwidth, or some other computing resource such that the total amount of the computing resource allocated to the plurality of VMs 20 has the specified inference time oversubscription rate 120.

[0084] Fig.15A A flow diagram of an example method 200 for oversubscribing a reinforcement learner performed at a computing system is shown. Fig.15A Steps performed during a training phase of training an oversubscription reinforcement learner are shown. At step 202, method 200 includes receiving a plurality of computing resource usage traces. The computing resource usage traces may each include a plurality of state-action pairs associated with respective time steps. Each state in the state-action pair may be a computing resource usage level, and each action may be an oversubscription rate.

[0085] At step 204, method 200 also includes generating a plurality of prototypes at the oversubscription reinforcement learner. The plurality of prototypes encodes corresponding prototype trajectories and is generated based at least in part on a plurality of computing resource usage trajectories. Each of the prototypes may represent a corresponding cluster of computing resource usage trajectories and may be expressed as a vector in an embedding space.

[0086] At step 206, method 200 also includes generating an oversubscription rate based at least in part on the plurality of prototypes. The oversubscription rate can be associated with a computing resource such as a vCPU, memory utilization, or network bandwidth. In some examples, the oversubscription rate is specific to a node included in the plurality of nodes in the computing system.

[0087] At step 208, the method 200 further includes outputting a prototype feedback query and an oversubscription rate feedback query to a user interface. The prototype feedback query and the oversubscription rate feedback query are prompts for feedback from the human-machine loop. The prototype feedback query is associated with a prototype in the plurality of prototypes, and the oversubscription rate feedback query indicates an oversubscription rate.

[0088] At step 210, the method 200 further includes receiving a prototype feedback input via a user interface in response to outputting the prototype feedback query. The prototype feedback input may be an approval input, a disapproval input, a merge input, a split input, or an update input. Additionally, at step 212, the method 200 further includes receiving an oversubscription rate feedback input via a user interface in response to outputting the oversubscription rate feedback query. The oversubscription rate feedback input may be an approval input or a disapproval input.

[0089] At step 214, method 200 also includes calculating an objective function value based at least in part on multiple computing resource usage traces, multiple prototypes, prototype feedback inputs, and oversubscription rate feedback inputs. The objective function value can be the value of a loss function that the oversubscription reinforcement learner is trained to approximately minimize, or the value of a reward function that the oversubscription reinforcement learner is trained to approximately maximize. In some examples, the objective function may include a representative capacity term and an interpretability term, as described below. The objective function may additionally or alternatively include a diversity term, which may be proportional to the sum of the maximum distances between prototype pairs. In some examples, one or more imitation learning terms may also be included in the objective function.

[0090] At step 216, method 200 also includes training an oversubscription reinforcement learner based at least in part on the objective function value. For example, the oversubscription reinforcement learner can be trained via stochastic gradient descent.

[0091] Fig. 15B Additional steps of method 200 that may be performed in some examples during a training phase are shown. At step 218, method 200 may also include receiving a plurality of computing resource usage trajectories at a trajectory encoder included in the oversubscription reinforcement learner. The trajectory encoder may be, for example, a transform encoder or an LSTM encoder. Method 200 may also include, at step 220, generating a plurality of trajectory embedding vectors corresponding to the computing resource usage trajectories. The trajectory embedding vectors may each have the same number of elements as each prototype.

[0092] In execution Fig. 15B In an example of the step of, multiple trajectory embedding vectors can be utilized when calculating the value of the objective function. For example, the objective function can include a representative capacity term that is proportional to the sum of the distances between multiple prototypes and the corresponding closest trajectory embedding vectors to those prototypes.

[0093] Fig. 15C It shows that it can be Fig. 15B Additional steps of method 200 performed in an example of generating a plurality of trajectory embedding vectors are shown. In such an example, calculating the value of the objective function may also include, at step 222, grouping the plurality of trajectory embedding vectors into a plurality of trajectory clusters corresponding to the plurality of prototypes. In such an example, the objective function may include an interpretable term that is proportional to the sum of the distances between the plurality of prototypes and the corresponding closest trajectory embedding vectors within the corresponding trajectory clusters associated with those prototypes.

[0094] The clustering of trajectory embedding vectors may be further used to determine when to output a prototype feedback query to a user. In such an example, at step 224, method 200 may further include outputting a prototype feedback query in response to determining that the cluster entropy of the trajectory cluster associated with the prototype is greater than a predefined uncertainty threshold. Additionally or alternatively, at step 226, method 200 may further include outputting a prototype feedback query in response to determining that the average distance between the prototype and the trajectory embedding vector included in the trajectory cluster associated with the prototype is included in a predetermined number of highest average distances between multiple prototypes. Thus, the corresponding prototype feedback query may be selectively output for a subset of prototypes, rather than requesting user feedback for each prototype.

[0095] Fig.15D Additional steps of method 200 that may be performed to use imitation learning during training of an oversubscription reinforcement learner are shown. At step 228, method 200 may also include receiving one or more user-provided computing resource usage trajectories via a user interface. At step 230, method 200 may also include performing imitation learning of multiple prototypes based at least in part on the one or more user-provided computing resource usage trajectories. For example, imitation learning may be implemented at least in part using a behavioral cloning term included in the objective function. Additionally or alternatively, imitation learning may be implemented at least in part using an adversarial imitation learning term included in the objective function. Thus, the oversubscription reinforcement learner may learn to imitate an expert-specified oversubscription rate selected for the corresponding trajectory.

[0096] Fig.15E Additional steps of method 200 are shown that may be performed during the inference phase after training the oversubscription reinforcement learner. At step 232, method 200 may also include receiving inference time computing resource usage data. At step 234, method 200 may also include setting an inference time oversubscription rate at the oversubscription reinforcement learner based at least in part on the inference time computing resource usage data. At step 236, method 200 may also include allocating computing resources to a plurality of virtual machines specified by the inference time oversubscription rate. Accordingly, the oversubscription reinforcement learner may dynamically control the oversubscription rate during real-time operation of the VMs.

[0097] We now discuss the experimental results of comparing the oversubscription reinforcement learner 50 (PROTOHAIL) with the prior oversubscription rate setting method. The oversubscription reinforcement learner is trained using historical vCPU utilization data and historical oversubscription rate data collected in a cloud computing environment. The two metrics of oversubscription policy performance used in the experiments are the percentage of hot nodes (the percentage of nodes with CPU utilization of 85% or higher) and the number of remaining cores (the number of additional CPU cores that would become available through oversubscription if the policy was implemented).

[0098] PROTOHAIL is compared to other oversubscription methods, including grid search, moving average, deep deterministic policy gradient (DDPG) reinforcement learning, behavior cloning, generative adversarial imitation learning (GAIL), and dataset aggregation (Dagger) imitation learning (with 20 human-guided time steps). In addition, PROTOHAIL is tested without HITL feedback. The oversubscription method is tested on a test set of historical vCPU utilization data and historical oversubscription rate raw data.

[0099] The following table summarizes the experimental results: method Hot Nodes Remaining core Grid Search 0％ 7450 Moving Average 1.39％ 7628 DDPG 1.47％ 5030 Behavioral cloning 1.19％ 7870 GAIL 1.2％ 6980 Dagger (20 time steps) 0.96％ 7938 PROTOHAIL (no HITL) 0％ 8153 PROTOHAIL 0％ 8161 As shown in the table above, both PROTOHAIL and PROTOHAIL without human feedback achieved 0% hot nodes. PROTOHAIL achieved the highest number of remaining cores among the methods tested, while PROTOHAIL without human feedback achieved the second highest number of remaining cores. It is worth noting that the PROTOHAIL results were achieved using a machine learning architecture that was developed to be human-interpretable. Therefore, PROTOHAIL does not exhibit a trade-off between interpretability and performance as occurs in many machine learning systems, but instead shows an increase in both interpretability and performance relative to prior machine learning methods such as DDPG, behavioral cloning, GAIL, and Dagger.

[0100] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0101] Fig.16 A non-limiting embodiment of a computing system 300 is schematically shown that can implement one or more of the above methods and processes. The computing system 300 is shown in simplified form. The computing system 300 can embody the above and in Figure 1 The computing system 10 shown in FIG. The computing system 300 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices (such as smart watches and head-mounted augmented reality devices).

[0102] The computing system 300 includes a logic processor 302, a volatile memory 304, and a non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or Fig.16 Other components not shown.

[0103] Logical processor 302 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve a desired result.

[0104] The logical processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The various components of the logical processor may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured with a cloud computing configuration. In this case, these virtualized aspects are run on different physical logical processors of various different machines.

[0105] The non-volatile storage device 306 includes one or more physical devices configured to hold instructions executable by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 306 may be transformed, for example, to hold different data.

[0106] The non-volatile storage device 306 may include a removable and / or built-in physical device. The non-volatile storage device 306 may include an optical memory, a semiconductor memory, and / or a magnetic memory, or other mass storage device technology. The non-volatile storage device 306 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It should be understood that the non-volatile storage device 306 is configured to retain instructions even when the non-volatile storage device 306 is powered off.

[0107] Volatile memory 304 may include a physical device that includes random access memory. Volatile memory 304 is typically used by logical processor 302 to temporarily store information during the processing of software instructions. It should be understood that when volatile memory 304 is powered off, volatile memory 304 typically does not continue to store instructions.

[0108] Aspects of the logic processor 302, volatile memory 304, and non-volatile storage device 306 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), programmed and application specific integrated circuits (PASIC / ASIC), programmed and application specific standard products (PSS / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0109] The terms "module," "program," and "engine" may be used to describe aspects of the computing system 300 that are typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves a transformation process that specifically configures the processor to perform that function. Thus, a module, program, or engine may be instantiated using portions of volatile memory 304 via a logical processor 302 executing instructions held by a non-volatile storage device 306. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may cover individual or groups of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0110] When included, the display subsystem 308 can be used to present a visual representation of the data held by the non-volatile storage device 306. The visual representation can take the form of a graphical user interface (GUI). As the methods and processes described herein change the data held by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of the display subsystem 308 can also be transformed to visually represent the changes in the underlying data. The display subsystem 308 can include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logical processor 302, the volatile memory 304, and / or the non-volatile storage device 306 in a shared housing, or such a display device can be a peripheral display device.

[0111] When included, the input subsystem 310 may include or interface with one or more user input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0112] When included, the communication subsystem 312 can be configured to communicatively couple the various computing devices described herein to each other and to communicatively couple to other devices. The communication subsystem 312 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem can allow the computing system 300 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0113] The following paragraphs discuss several aspects of the present disclosure. According to one aspect of the present disclosure, a computing system is provided, the computing system including one or more processing devices, which train an oversubscription reinforcement learner during a training phase at least in part by receiving a plurality of computing resource usage trajectories. Training the oversubscription reinforcement learner also includes generating a plurality of prototypes encoding corresponding prototype trajectories at the oversubscription reinforcement learner based at least in part on the plurality of computing resource usage trajectories. Based at least in part on the plurality of prototypes, the training oversubscription reinforcement learner also includes generating an oversubscription rate. The training oversubscription reinforcement learner also includes outputting a prototype feedback query associated with a prototype in the plurality of prototypes and / or an oversubscription rate feedback query indicating an oversubscription rate to a user interface. The training oversubscription reinforcement learner also includes receiving a prototype feedback input via the user interface in response to outputting the prototype feedback query, and / or receiving an oversubscription rate feedback input via the user interface in response to outputting the oversubscription rate feedback query. Based at least in part on the plurality of computing resource usage trajectories, the plurality of prototypes, and the prototype feedback input and / or the oversubscription rate feedback input, the training oversubscription reinforcement learner also includes calculating an objective function value. The oversubscription reinforcement learner is trained based at least in part on the objective function value. The above features can have a technical effect of efficiently allocating computing resources via oversubscription while avoiding overload. In addition, the oversubscription reinforcement learner can generate oversubscription rates in a human-interpretable manner.

[0114] According to this aspect, during the inference phase, one or more processing devices may also receive inference time computing resource usage data. Based at least in part on the inference time computing resource usage data, one or more processing devices may also set an inference time oversubscription rate at the oversubscribed reinforcement learner. One or more processing devices may also allocate computing resources to a plurality of virtual machines specified by the inference time oversubscription rate. The above features may have a technical effect of efficiently allocating computing resources via oversubscription while avoiding overload.

[0115] According to this aspect, training the oversubscription reinforcement learner may further include receiving, via a user interface, one or more user-provided computing resource usage traces, and performing imitation learning of the plurality of prototypes based at least in part on the one or more user-provided computing resource usage traces. The above features may have the technical effect of generating an oversubscription policy that more closely reflects a policy generated by a human expert.

[0116] According to this aspect, imitation learning may be implemented at least in part with a behavior cloning term included in the objective function.The above features may have the technical effect of generating an oversubscription policy that more closely reflects a policy generated by a human expert.

[0117] According to this aspect, imitation learning may be implemented at least in part using an adversarial imitation learning term included in the objective function.The above features may have the technical effect of generating an oversubscription policy that more closely reflects a policy generated by a human expert.

[0118] According to this aspect, the oversubscription reinforcement learner may include a trajectory encoder that receives a plurality of computing resource usage trajectories and generates a plurality of trajectory embedding vectors corresponding to the computing resource usage trajectories. The above features may have a technical effect of converting the computing resource usage trajectories into a form that can be more easily processed at the oversubscription reinforcement learner.

[0119] According to this aspect, the objective function includes a representative capacity term proportional to the sum of distances between the plurality of prototypes and the corresponding closest trajectory embedding vectors to those prototypes. The above features may have the technical effect of training an oversubscribed reinforcement learner to generate prototypes that accurately model the computational resource usage trajectories used as training data.

[0120] According to this aspect, one or more processing devices group the plurality of trajectory embedding vectors into a plurality of trajectory clusters corresponding to the plurality of prototypes. The objective function may include an interpretability term proportional to the sum of distances between the plurality of prototypes and the respective closest trajectory embedding vectors within the corresponding trajectory clusters associated with those prototypes. The above features may have a technical effect of training an oversubscribed reinforcement learner to generate human interpretable prototypes.

[0121] According to this aspect, one or more processing devices may output a prototype feedback query in response to determining that the cluster entropy of the trajectory cluster associated with the prototype is greater than a predefined uncertainty threshold, or determining that the average distance between the prototype and the trajectory embedding vector included in the trajectory cluster associated with the prototype is included in a predetermined number of highest average distances among multiple prototypes. The above features may have a technical effect of selectively requesting human feedback related to the prototype.

[0122] According to this aspect, the objective function may include a diversity term proportional to the sum of the maximum distances between pairs of prototypes.The above features may have the technical effect of training an oversubscribed reinforcement learner to generate prototypes that model a wider range of potential inputs.

[0123] According to this aspect, the prototype feedback input may be an approval input, a disapproval input, a merge input, a split input, or an update input. The above features may have the technical effect of allowing the human-machine loop to provide various different types of prototype feedback inputs.

[0124] According to this aspect, the prototype feedback input may be a merge input. In response to receiving the prototype feedback input, one or more processing devices may generate a merged prototype based at least in part on the prototype and the additional prototype. The above features may have a technical effect of allowing the human-machine loop to merge prototypes when the human-machine loop determines that the prototypes are sufficiently similar to each other.

[0125] According to this aspect, the prototype feedback input may be a segmentation input. In response to receiving the prototype feedback input, one or more processing devices may generate a first segmentation prototype and a second segmentation prototype based at least in part on the prototype. The above features may have a technical effect of allowing the human-machine loop to segment the prototype when the human-machine loop determines that the prototype models multiple different categories of trajectories.

[0126] According to this aspect, when calculating the objective function value, one or more processing devices may apply a corresponding plurality of scaling factors to the plurality of terms of the objective function based at least in part on the prototype feedback input and / or the oversubscription rate feedback input. The above features may have the technical effect of incorporating human feedback into the calculation of the value of the objective function.

[0127] According to another aspect of the present disclosure, a method for an oversubscription reinforcement learner executed at a computing system is provided. The method includes receiving a plurality of computing resource usage trajectories. At the oversubscription reinforcement learner, the method also includes generating a plurality of prototypes encoding corresponding prototype trajectories based at least in part on the plurality of computing resource usage trajectories. The method also includes generating an oversubscription rate based at least in part on the plurality of prototypes. The method also includes outputting a prototype feedback query associated with a prototype in the plurality of prototypes to a user interface, and / or an oversubscription rate feedback query indicating an oversubscription rate. The method also includes receiving a prototype feedback input via the user interface in response to outputting the prototype feedback query, and / or receiving an oversubscription rate feedback input via the user interface in response to outputting the oversubscription rate feedback query. Based at least in part on the plurality of computing resource usage trajectories, the plurality of prototypes, and the prototype feedback input and / or the oversubscription rate feedback input, the method also includes calculating an objective function value. The method also includes training the oversubscription reinforcement learner based at least in part on the objective function value. The above features may have a technical effect of effectively allocating computing resources via oversubscription while avoiding overload. In addition, the oversubscription reinforcement learner may generate an oversubscription rate in a human-interpretable manner.

[0128] According to this aspect, during the inference phase, the method may further include receiving inference time computing resource usage data. Based at least in part on the inference time computing resource usage data, the method may further include setting an inference time oversubscription rate at the oversubscribed reinforcement learner. The method may further include allocating computing resources to a plurality of virtual machines specified by the inference time oversubscription rate.

[0129] According to this aspect, the method may also include receiving, via a user interface, one or more user-provided computing resource usage traces. The method may also include performing imitation learning of multiple prototypes based at least in part on the one or more user-provided computing resource usage traces. The above features may have the technical effect of generating an oversubscription strategy that more closely reflects a strategy generated by a human expert.

[0130] According to this aspect, the method may further include receiving a plurality of computing resource usage trajectories at a trajectory encoder included in the oversubscribed reinforcement learner. The method may further include generating a plurality of trajectory embedding vectors corresponding to the computing resource usage trajectories. The above features may have a technical effect of converting the computing resource usage trajectories into a form that can be more easily processed at the oversubscribed reinforcement learner.

[0131] According to this aspect, the prototype feedback input may be an approval input, a disapproval input, a merge input, a split input, or an update input. The above features may have the technical effect of allowing the human-machine loop to provide various different types of prototype feedback inputs.

[0132] According to another aspect of the present disclosure, a computing system is provided, the computing system including one or more processing devices, which train an oversubscription reinforcement learner at least in part by receiving a plurality of computing resource usage trajectories during a training phase. Training the oversubscription reinforcement learner also includes receiving computing resource usage trajectories provided by one or more users via a user interface. Training the oversubscription reinforcement learner also includes receiving a plurality of computing resource usage trajectories at a trajectory encoder and generating a plurality of trajectory embedding vectors corresponding to the computing resource usage trajectories. Training the oversubscription reinforcement learner also includes generating a plurality of prototypes encoding corresponding prototype trajectories based at least in part on the plurality of computing resource usage trajectories. Training the oversubscription reinforcement learner also includes grouping the plurality of trajectory embedding vectors into a plurality of trajectory clusters corresponding to the plurality of prototypes. Based at least in part on the plurality of prototypes, training the oversubscription reinforcement learner also includes generating an oversubscription rate. Training the oversubscription reinforcement learner also includes calculating an objective function value of an objective function, which includes a representative capacity term proportional to the sum of the distances between the plurality of prototypes and the corresponding closest trajectory embedding vectors to those prototypes. The objective function also includes an interpretability term proportional to the sum of distances between the plurality of prototypes and the corresponding closest trajectory embedding vectors within the corresponding trajectory clusters associated with those prototypes. The objective function also includes a diversity term proportional to the sum of the maximum distances between pairs of prototypes. The objective function also includes one or more imitation learning terms computed using the trajectories based at least in part on one or more user-provided computing resources. The oversubscribed reinforcement learner is trained based at least in part on the objective function value.

[0133] As used herein, "and / or" is defined as an inclusive OR, specified by the following truth table: A B A∨B real real real real Fake real Fake real real Fake Fake Fake

[0134] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. The specific routine or method described herein can represent one or more of any number of processing strategies. Thus, the various actions shown and / or described can be performed in the order shown and / or described, in other orders, in parallel or omitted. Likewise, the order of the above process can be changed.

[0135] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system comprising: One or more processing devices that train an oversubscription reinforcement learner during a training phase at least in part by: receiving a plurality of computing resource usage traces; generating, at the oversubscribed reinforcement learner, a plurality of prototypes encoding respective prototype trajectories based at least in part on the plurality of computing resource usage trajectories; generating an oversubscription rate based at least in part on the plurality of prototypes; Output to the user interface: a prototype feedback query associated with a prototype of the plurality of prototypes; and / or an oversubscription rate feedback query indicating the oversubscription rate; receiving prototype feedback input via the user interface in response to outputting the prototype feedback query, and / or receiving oversubscription rate feedback input via the user interface in response to outputting the oversubscription rate feedback query; calculating an objective function value based at least in part on the plurality of computing resource usage traces, the plurality of prototypes, and the prototype feedback input and / or the oversubscription rate feedback input; as well as The oversubscription reinforcement learner is trained based at least in part on the objective function value.

2. The computing system of claim 1, wherein: During the inference phase, the one or more processing devices also: receiving inference time computing resource usage data; setting an inference time oversubscription rate at the oversubscribed reinforcement learner based at least in part on the inference time computational resource usage data; as well as The computing resources are allocated to a number of virtual machines specified by the inference time oversubscription rate.

3. The computing system of claim 1 , wherein training the oversubscription reinforcement learner further comprises: receiving, via the user interface, computing resource usage traces provided by one or more users; as well as Imitation learning of the plurality of prototypes is performed based at least in part on the computing resource usage traces provided by the one or more users. 4 . The computing system of claim 3 , wherein the imitation learning is implemented at least in part using a behavior cloning term included in the objective function.

5. The computing system of claim 3, wherein the imitation learning is implemented at least in part using an adversarial imitation learning term included in the objective function.

6. The computing system of claim 1 , wherein the oversubscription reinforcement learner comprises a trajectory encoder, wherein: receiving the plurality of computing resource usage traces; and A plurality of trajectory embedding vectors corresponding to the computing resource usage trajectories are generated.

7. The computing system of claim 6, wherein the objective function comprises a representative capacity term that is proportional to the sum of distances between the plurality of prototypes and corresponding closest trajectory embedding vectors to the plurality of prototypes.

8. The computing system of claim 6, wherein: The one or more processing devices grouping the plurality of trajectory embedding vectors into a plurality of trajectory clusters corresponding to the plurality of prototypes; and The objective function includes an interpretable term that is proportional to a sum of distances between the plurality of prototypes and respective closest trajectory embedding vectors within corresponding trajectory clusters associated with the plurality of prototypes.

9. The computing system of claim 8, wherein the one or more processing devices output the prototype feedback query in response to: determining that a cluster entropy of the cluster of trajectories associated with the prototype is greater than a predefined uncertainty threshold; or It is determined that an average distance between the prototype and the trajectory embedding vector included in the trajectory cluster associated with the prototype is included in a predetermined number of highest average distances among the plurality of prototypes.

10. The computing system of claim 1, wherein the objective function includes a diversity term that is proportional to the sum of the maximum distances between pairs of the prototypes.

11. The computing system of claim 1, wherein the prototype feedback input is an approval input, a disapproval input, a merge input, a split input, or an update input.

12. The computing system of claim 1, wherein: The prototype feedback input is a merged input; and In response to receiving the prototype feedback input, the one or more processing devices generate a merged prototype based at least in part on the prototype and the additional prototype.

13. The computing system of claim 1, wherein: The prototype feedback input is a segmentation input; as well as In response to receiving the prototype feedback input, the one or more processing devices generate a first segmented prototype and a second segmented prototype based at least in part on the prototype.

14. The computing system of claim 1 , wherein when calculating the objective function value, the one or more processing devices apply a corresponding plurality of scaling factors to a plurality of terms of the objective function based at least in part on the prototype feedback input and / or the oversubscription rate feedback input.

15. A method for oversubscribing a reinforcement learner performed at a computing system, the method comprising: receiving a plurality of computing resource usage traces; generating, at the oversubscribed reinforcement learner, a plurality of prototypes encoding respective prototype trajectories based at least in part on the plurality of computing resource usage trajectories; generating an oversubscription rate based at least in part on the plurality of prototypes; Output to the user interface: a prototype feedback query associated with a prototype of the plurality of prototypes; and / or an oversubscription rate feedback query indicating the oversubscription rate; receiving prototype feedback input via the user interface in response to outputting the prototype feedback query, and / or receiving oversubscription rate feedback input via the user interface in response to outputting the oversubscription rate feedback query; calculating an objective function value based at least in part on the plurality of computing resource usage traces, the plurality of prototypes, and the prototype feedback input and / or the oversubscription rate feedback input; as well as The oversubscription reinforcement learner is trained based at least in part on the objective function value.

Images