Dynamic Resource Reservation Using Reinforcement Learning

Abstract

A method and system for reserving resources includes determining a state of the distributed computing system based on resource needs of applications executing on the distributed computing system and system resource constraints (302). A behavior is determined using the state of the distributed computing system as an input to a trained reinforcement learning model (304). A resource request (306) is issued for the application to reserve resources based on the behavior.

Description

[Technical field]

[0001] Related Application Information This application claims priority to U.S. Patent Application No. 63 / 345,350, filed May 24, 2022, each of which is incorporated by reference in its entirety. This application claims priority to U.S. Patent Application No. 18 / 322,102, filed May 23, 2023, each of which is incorporated by reference in its entirety. [Background technology]

[0002] The present invention relates to resource allocation, and more particularly to a reinforcement learning system that dynamically adapts resource reservations. 2. Description of Related Art

[0003] In complex network systems such as IoT systems that contain many devices and sensors, resource needs can change dynamically. As a result, static allocation of resources such as computing and network resources may result in reserved resources being underutilized for a long period of time. Summary of the Invention

[0004] A method for reserving resources includes determining a state of a distributed computing system based on resource needs of applications executing on the distributed computing system and system resource constraints. An action is determined using the state of the distributed computing system as an input to a trained reinforcement learning model. A resource request is issued for an application to reserve resources based on the action.

[0005] A system for reserving resources includes a hardware processor and a memory that stores a computer program that, when executed, causes the hardware processor to determine actions using a state of the distributed computing system as inputs to a trained reinforcement learning model, and to issue resource requests for applications to reserve resources.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0007] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0008] [Figure 1] 1 is a diagram of a distributed computing system with dynamic resource reservation of computing and network resources in accordance with one embodiment of the present invention.

[0009] [Diagram 2] 1 is a block / flow diagram of a method for dynamically reserving computing and network resources in a distributed computing system according to one embodiment of the present invention.

[0010] [Diagram 3] 2 is a block / flow diagram of a method for determining resources to reserve for an application in a distributed computing system according to one embodiment of the present invention.

[0011] [Figure 4] FIG. 1 is a block diagram of a resource orchestrator system configured to dynamically reserve computing and network resources for applications in a distributed computing system according to one embodiment of the present invention.

[0012] [Diagram 5] 1 is a block diagram of a computer processing system capable of acting as a resource orchestrator for dynamic resource reservation according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] To accommodate dynamic changes in resource requirements and nonlinear coupling relationships between compute and network resource needs, reinforcement learning can be used to automatically optimize the resource requests of an application. For example, application-specific compute and network coupling relationships can be automatically captured by a reinforcement learning model to enable resource allocation options to better utilize reserved resources. Accurate and dynamically changing resource allocation can minimize resource underutilization and improve the actual compute and network resource capacity of the system. Although compute and network resources are specifically discussed and described herein, it should be understood that additional or alternative resources may be used.

[0014] Referring to FIG. 1, a multi-tier edge cloud environment is shown. The environment includes various Internet of Things (IoT) devices 102, including, for example, mobile devices such as phones and tablets, connected vehicles, sensors such as cameras, and various other distributed devices. The IoT devices 102 communicate with wireless access points 104, such as base stations or cellular access nodes. This enables the IoT devices 102 to communicate with edge nodes 106, which perform local processing and communicate with further computational resources in the cloud 108. Exemplary applications of such an environment include autonomous driving, smart manufacturing, telemedicine, augmented or virtual reality, and other suitable purposes or functions.

[0015] The wireless access points 104 may represent any suitable form of network resource, for example constituting a mobile telephony / data network such as a 5G network, a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN). A single cloud networking system may use multiple different networking configurations. These different network configurations have different capacity and service level guarantees. Different network functions at different tiers, for example 5G connections between IoT devices 102 and edge nodes 106, MANs between distributed edge nodes 106, WANs between edge nodes 106 and cloud 108, and changing network conditions further enhance the dynamic network environment. Although wireless access is specifically envisioned, it should be understood that the present principles may also be applied to IoT devices 102 communicating with a distributed computing system via a wired communication medium.

[0016] The resource orchestrator 110 communicates with the compute and network infrastructure and has a global view of the application components running in the various compute tiers. Resource reservation is performed by the resource orchestrator 110 and is responsible for ensuring that multiple applications running across the system have access to the resources they need to operate. Resource reservation requests may be sent by the resource orchestrator 110 to the underlying compute and network infrastructure components.

[0017] An application may reserve a certain amount of resources, such as network bandwidth or processing time, to meet specific performance needs. Network bandwidth may be allocated by network infrastructure devices such as routers and switches that implement software-defined networking and traffic shaping. Computational resources may be allocated by allocating processor cores, scheduling individual processor cores, or instantiating and de-allocating processing resources in response to resource reservation requests. Infrastructure providers may charge applications based on these reservations. The more resources reserved, the higher the cost. Minimizing required reservations reduces the operating costs of an application.

[0018] Neither the infrastructure capacity nor the environment remains fixed during the lifetime of an application. For example, as the number and type of applications run by the infrastructure change, the resources required by those applications also change. For example, the resources required by a video analytics application may change as the scene viewed by a video camera changes. This causes the amount of network and compute resources required by the application to fluctuate, and resources must be over-provisioned with static, one-time, fixed reservations to enable the application to operate under dynamically changing conditions.

[0019] For example, an object detection pipeline may include a video camera acting as an IoT sensor, performing object detection and issuing an alert when a particular type of object is detected. The video camera communicates with an edge node 106 that performs the object detection, and an application in the cloud 108 may include a database of alert information and issue alerts in response to detected objects. During the execution of the object detection application, network resources are required to communicate the video stream from the video camera, and computational resources are required to analyze the frames of the video stream to detect objects.

[0020] The network bandwidth required for a video stream from a camera can vary over time. For example, a video stream might use a lower bitrate when there is little change between frames, and a higher bitrate when there is a lot of change. An example of a low-changing environment would be a nighttime scene with little activity, as opposed to a view of the same scene captured during the day with lots of pedestrians. By learning and leveraging such periodic variations, the reserved network resources can be dynamically adjusted.

[0021] Depending on the network bandwidth usage, the computational resources required to process the video stream also change. In this example, the larger the volume of video data, the more processing power is required to process that data. Similarly, by identifying the relationship between network resource usage and computational resource usage, reservations of one type of resource can be changed in response to changes in the other type of resource.

[0022] In some cases, available network and compute resources may not be sufficient to meet the resource needs of all applications. Changes in infrastructure conditions can directly impact the resource reservations that applications can make. If one resource is in short supply due to a lack of available resources, changes can be made to another resource reservation to reflect its reduced capacity. For example, if the amount of bandwidth is limited, the computational resource reservation may be reduced to reflect reduced data throughput. The situation is also reversed; that is, if there is a limit on the amount of available computational resources, the network resource reservation may be reduced accordingly.

[0023] Therefore, resource reservation for different applications can be achieved by slicing. The set of computing nodes is denoted here as M, and the set of different resource types is denoted here as

number

number

number

[0024] An application can be modeled as a collection of functions or microservices and the interconnections that represent the data dependencies between the functions. A graph G=(V,E) represents an application, where V represents the set of functions in the application and E represents the interconnections, such as data dependencies, between the functions. The terms R v =(ω v ,C v ) represents the requirements for the function v. Here, ω v and C. v =(core min,v ,tier v ) are the network and computation needs of the application, respectively. min,v is the minimum number of cores required for function v. v specifies the computational requirements of a function v, where tier v is the infrastructure layer in which to place the function v, if there is such a requirement.

[0025] Since resource demands fluctuate over time, dynamic resource allocation can be used to address the adjustment of resource usage or placement decisions while considering the binding relationships between resources. To this end, a reinforcement learning-based orchestration system can automatically derive resource binding relationships periodically and select optimal actions using, for example, a state-action-reward-state-action reinforcement learning model. The available amount of resource t on node m is quantized into L levels and aggregated.

number

number

number

number

[0026] The state of the system at time step i is the tuple

number

[0027] The actions may include effective resource reservations that determine the amount of a particular resource that infrastructure node hosting and application microservices can consume. The set of actions may include A=5|V| actions that capture five possible actions for each function. These actions include increasing compute resource reservation, decreasing compute resource reservation, increasing network resource reservation, decreasing network resource reservation, and an action that indicates no change in reservation.

[0028] The reward function is used as feedback to the reinforcement learning system to improve performance based on the reward value after an action is executed. To increase the likelihood of a good action, the reward function generates a positive reward. The reward value is defined as:

number

number

[0029] 2, an overview of the implementation of a reinforcement learning model for resource reservation is shown. Block 202 deploys the model to a live system, which in block 204 monitors the state of the system and its relationship to resource needs.

[0030] During operation, the reinforcement learning model is used to allocate resources to applications at block 206. The trained model calculates rewards for the applications based on application-specific measurements. For example, the applications may reserve computing resources from the edge nodes 106 or cloud 108, or reserve network resources from the wireless access nodes 104. The applications may determine the state of the system, for example, based on their own internal state or information collected from the network. For example, the internal state information may include the instantaneous computing and network needs of the system itself, and the external state information may include information about the availability of network and computing resources, such as the current available capacity of the nodes in the system. Block 208 trains a reinforcement learning model that responds to the correlation between resource reservation and the state of the system based on the ongoing monitoring, for example, using a state-action-reward-state-action model. Processing returns to block 204, where the updated system state is monitored, and new resource reservations at block 204 are repeatedly performed to respond to the changing conditions of the system.

[0031] 3, additional details regarding determining resource reservations 206 are shown. Block 302 determines relevant state information, such as internal and external state information. This state is used as input to a trained reinforcement learning model, and effective actions that maximize a reward function are output in block 306, and a reward calculation is performed in block 305. The reward function can also be used to improve the trained model. For example, the reward function may represent a performance metric of an application, while the effective actions represent a set of resource reservations that are satisfied by the system and take into account the relationships between various resource constraints.

[0032] Block 306 issues a reservation request, which may include, for example, reserving computational resources from the edge node 106 or cloud 108 and network resources from the access node 104. Once the reservation is completed, block 308 waits until the next iteration begins, which may be defined by a predetermined period of time, for example a predetermined number of seconds, minutes, or hours, after which processing returns to block 302 to determine an updated state.

[0033] In some cases, the next iteration may be determined by a detected change in internal or external conditions. For example, block 308 may detect a change in resources required by an application or receive information about a change in resource capacity of the system. If this occurs before a predetermined period of time has elapsed, an early update may be triggered. In some cases, the next iteration may be triggered only by a detected change in conditions, in some cases, the next iteration may be triggered only periodically, in some cases, the next iteration may be triggered by a combination of both a change in conditions and periodically, and in some cases, the next iteration may be triggered by one or more other factors.

[0034] 4, additional details are provided regarding the resource orchestrator 110. Although the resource orchestrator 110 is shown and described herein as a standalone hardware device, it should be understood that the functions of the resource orchestrator 110 may be performed within the edge node 106 or within the cloud 108.

[0035] The resource orchestrator 110 includes a hardware processor 402 and a memory 404. The network interface 406 may provide communication with other elements of the computing system, including the edge nodes 106, the cloud computing elements 108, the network infrastructure, and the device 102. The resource orchestrator 110 may include a number of functional modules, which may be implemented as software stored in the memory 404 and executed by the hardware processor 402. In some cases, one or more of the functional modules may be implemented as separate hardware components. In some cases, one or more of the functional modules may be implemented in a separate hardware device and communicate with the resource orchestrator 110.

[0036] The status monitor 408 receives information from the network interface 406, which may include information regarding the capacity of respective computing and / or network resources from the components of the distributed computing system. The status monitor 408 may also monitor the resource requirements of one or more applications running within the distributed computing system.

[0037] The monitored state information is used by model training 410 to identify how parameters of resource reservation model 412 need to be modified to improve future performance. The monitored state information may also be used as input to resource reservation model 412 to identify actions to perform, for example, in the form of a set of resource reservation requests related to the operation of the application. Resource reservation unit 414 uses network interface 406 to send resource reservation requests to components of the distributed computing system. For example, the resource reservation requests may include requests for computing resources from edge nodes 106 and cloud 108, and may also include requests for network resources from the network infrastructure.

[0038] As shown in FIG. 5, computing device 500 illustratively includes a processor 510, an input / output subsystem 520, a memory 530, a data storage device 540, and a communication subsystem 550, and / or other components and devices typically found in a server or similar computing device. Computing device 500 may, in other embodiments, include other or additional components such as typically found in a server computer (e.g., various input / output devices). Furthermore, in some embodiments, one or more of the illustrative components may be incorporated in or otherwise form part of another component. For example, memory 530, or portions thereof, may be incorporated in processor 510 in some embodiments.

[0039] The processor 510 may be embodied as any type of processor capable of performing the functions described herein, and may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0040] Memory 530 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, memory 530 may store various data and software used during operation of computing device 500, such as an operating system, applications, programs, libraries, and drivers. Memory 530 is communicatively coupled to processor 510 via I / O subsystem 520 and may be embodied as circuits and / or components to facilitate input / output operations with processor 510, memory 530, and other components of computing device 500. For example, I / O subsystem 520 may be embodied as or may otherwise include a memory controller hub, an input / output control hub, a platform controller hub, integrated control circuits, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems to facilitate input / output operations. In some embodiments, the I / O subsystem 520 may form part of a system on a chip (SOC) and be integrated with the processor 510, memory 530, and other components of the computing device 500 on a single integrated circuit chip.

[0041] The data storage device 540 may be embodied as any type of device or device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. The data storage device 540 may store program code 540A for performing resource reservation based on monitored system state information, program code 540B for determining reward values ​​for certain actions based on monitored system state information, and / or program code 540C for training a reinforcement learning model based on monitored system state information. The communication subsystem 550 of the computing device 500 may be embodied as any network interface controller or other communication circuitry, device, or collection thereof that may enable communication between the computing device 500 and other remote devices over a network. The communication subsystem 550 may be configured to effectuate such communication using any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.).

[0042] As shown, computing device 500 may also include one or more peripheral devices 560. Peripheral devices 560 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 560 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0043] Of course, computing device 500 may include other elements (not shown) or omit certain elements, as would be readily appreciated by one of ordinary skill in the art. For example, various other sensors, input devices, and / or output devices may be included in computing device 500 depending on the particular implementation of the same, as would be readily appreciated by one of ordinary skill in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Additionally, additional processors, controllers, memories, and the like may be utilized in a variety of configurations. These and other variations of processing system 500 will be readily contemplated by one of ordinary skill in the art in view of the teachings of the present invention provided herein.

[0044] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements, in a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0045] The embodiments may include a computer program product accessible from a computer usable or computer readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer usable or computer readable medium may include any apparatus that stores, communicates, propagates, or transports a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium. The medium may include a computer readable storage medium such as a semiconductor or solid state memory, a magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk.

[0046] Each computer program may be tangibly stored on a machine-readable storage medium or device (e.g., a program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention may also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0047] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0048] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0049] As employed herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be included in a central processing unit, an image processing unit, and / or a separate processor or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.). In some embodiments, the hardware processor subsystem may include one or more memories that may be on-board or off-board, or may be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0050] In one embodiment, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code to achieve a particular result.

[0051] In other embodiments, the hardware processor subsystem may include dedicated, dedicated circuitry to perform one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0052] These and other variations of hardware processor subsystems are also contemplated in accordance with embodiments of the present invention.

[0053] In the specification, references to "one embodiment" or "one embodiment" of the invention, as well as other variations, mean that the particular features, structures, characteristics, etc. described in connection with the embodiment are included in at least one embodiment of the invention. Thus, the phrases "in one embodiment" or "in one embodiment" appearing in various places in this specification, as well as any other variations, do not necessarily all refer to the same embodiment. However, given the teachings of the invention provided herein, it should be understood that features of one or more embodiments can be combined.

[0054] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one of," such as "A and / or B," "at least one of A and B," will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as there are listed items.

[0055] The foregoing is understood in all respects to be illustrative and illustrative, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Various other feature combinations could be made by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is as set forth in the appended claims.

Claims

1. 1. A computer-implemented method for reserving resources, comprising: determining (302) a state of a distributed computing system based on resource needs of applications executing on the distributed computing system and system resource constraints; Using the state of the distributed computing system as an input to a trained reinforcement learning model to determine an action (304); and the application issuing (306) a resource request to reserve resources.

2. 2. The method of claim 1 , The method further comprising iterating between determining a state of the distributed computing system, determining the resource requests, and issuing the resource requests in response to satisfying an iteration criterion.

3. 3. The method of claim 2, The method, wherein the recurrence criteria includes the passage of a predetermined period of time.

4. 3. The method of claim 2, The method, wherein the iteration criteria includes a change in a state of the distributed computing system.

5. 2. The method of claim 1 , The method, wherein the reinforcement learning model includes a reward function based on a performance metric of the application.

6. 6. The method of claim 5, The reward function is the reward value d i and the reward value d i teeth, [0010] is defined as p i is the performance metric of the application at time step i, and α and β t is a parameter that characterizes the tradeoff between computational resources, network resources, and performance metrics; [0025] is the amount of resource t at time step i of node m allocated to function v.

7. 6. The method of claim 5, The method, wherein the reward function includes constraints to enforce resource capacity limits.

8. 2. The method of claim 1 , The method, wherein the resource request includes a request for a computational resource and a request for a network resource.

9. 2. The method of claim 1 , The trained reinforcement learning model characterizes the coupling relationships between computational resources and network resources.

10. 2. The method of claim 1 , The method, wherein the state of the distributed computing system includes current computing and network resource usage and capacity of the distributed computing system.

11. 1. A system for reserving resources, comprising: A hardware processor (402); When executed, the hardware processor: determining (302) a state of a distributed computing system based on resource needs of applications executing on the distributed computing system and system resource constraints; using the state of the distributed computing system as an input to a trained reinforcement learning model to determine an action (304); and a procedure (306) for the application to issue a resource request to reserve a resource.

12. 12. The system of claim 11, The computer program further causes the hardware processor to iterate between determining a state of the distributed computing system, determining the resource requests, and issuing the resource requests in response to satisfying an iteration criterion.

13. 13. The system of claim 12, The recurrence criteria includes the passage of a predetermined period of time.

14. 13. The system of claim 12, The iteration criteria includes a change in state of the distributed computing system.

15. 12. The system of claim 11, The system, wherein the reinforcement learning model includes a reward function based on a performance metric of the application.

16. 16. The system of claim 15, The reward function is the reward value d i and the reward value d i teeth, [0030] is defined as p i is the performance metric of the application at time step i, and α and β t is a parameter that characterizes the tradeoff between computational resources, network resources, and performance metrics; [0045] is the amount of resource t at time step i of node m allocated to function v in the system.

17. 16. The system of claim 15, The system wherein the reward function includes constraints to enforce resource capacity limits.

18. 12. The system of claim 11, The system, wherein the resource requirements include a computational resource requirement and a network resource requirement.

19. 12. The system of claim 11, The system, wherein the trained reinforcement learning model characterizes the coupling relationships between computational resources and network resources.

20. 12. The system of claim 11, The state of the distributed computing system includes current computing and network resource usage and capacity of the distributed computing system.

Images