System for controlling dual-level cloud resources for real-time services and method for controlling the same
Patent Information
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-08-12
Smart Images
Figure R1020250197811_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a load balancing system. More specifically, it relates to a dual-level cloud resource control system for real-time services and a method for controlling the same. Background Technology
[0002] Recently, real-time services requiring low-latency characteristics, such as video streaming, volumetric video, augmented reality (AR), virtual reality (VR), and cloud gaming, have been rapidly increasing in cloud environments. For these services, the quality of the user experience relies heavily on Quality of Service (QoS) metrics such as latency, throughput, and frame rate (FPS).
[0003] Kubernetes (K8s), a representative cloud platform, is utilized as a core technology of modern cloud infrastructure by deploying and running applications in Pod units and automating deployment, management, disaster recovery, and auto-scaling. In particular, Kubernetes is combined with Open RAN (Open Radio Access Network) technology, which emerged in the 6G environment, and is being utilized as a platform that simultaneously runs RAN functions and user applications on O-Cloud based on General Purpose Servers (COTS).
[0004] Meanwhile, load balancing and resource scaling are representative technologies for ensuring application performance in a cloud environment. Load balancing is a technology that distributes incoming service requests to multiple servers or worker nodes, while resource scaling is a technology that dynamically increases or decreases resources, such as CPU and memory, allocated to an application. Kubernetes provides VPA (Vertical Pod Autoscaler) and HPA (Horizontal Pod Autoscaler) to automate vertical scaling and horizontal scaling.
[0005] However, these existing cloud infrastructure technologies have limitations in accurately reflecting the QoS requirements of real-time services, and more intelligent control methods are required, especially in Open RAN environments and next-generation 6G services that require low latency and high reliability characteristics.
[0006] However, conventional technology has the following structural limitations.
[0007] First is the problem of QoS non-cognitive distribution in existing load balancers.
[0008] Conventional load balancers typically use simple distribution algorithms, such as Round Robin and Least Connection, to distribute requests on a server-by-server basis. While this approach is advantageous for evenly distributing load among servers, it has limitations in that it fails to consider the performance requirements (QoS), resource demands, and latency sensitivity of each application. Consequently, in real-time services such as video streaming, AR / VR, and cloud gaming, increased service latency, frame loss, and degraded user experience may occur.
[0009] Second, there is the issue of limiting resource types in Kubernetes autoscaling.
[0010] Kubernetes VPAs and HPAs are designed by default to perform resource scaling centered on CPU and memory resources. However, in actual real-time applications, various resources such as GPUs, disk I / O, and network bandwidth (including cache and network bandwidth) have a direct impact on performance. Conventional technologies face the problem that it is difficult or only possible to provide integrated and precise control over these heterogeneous resources, making it difficult to reliably guarantee the performance of complex real-time services.
[0011] Third, there is a lack of intelligent performance control technology optimized for Open RAN and 6G environments.
[0012] In an Open RAN environment, RAN functions and user applications run on the same O-Cloud, and demands for network slicing, ultra-low latency communication (URLLC), and massive connectivity (mMTC) occur simultaneously. However, conventional load balancing and resource control technologies fail to provide a structure that optimizes in real-time by simultaneously considering the performance requirements of RAN components and user applications. Consequently, there is a limitation in that it is difficult to reliably guarantee the high-reliability, ultra-low latency service quality required in 6G and Open RAN environments.
[0013] Therefore, in the case of conventional technology, there were problems that caused inconvenience to users due to ① the problem of QoS non-cognitive distribution of existing load balancers, ② the problem of resource type limitations in Kubernetes autoscaling, and ③ the lack of intelligent performance control technology optimized for Open RAN and 6G environments. Prior art literature
[0014] Korean Registered Patent Publication No. 10-2609406 B2 (Publication Date: Dec. 04, 2023) The problem to be solved
[0015] The embodiment disclosed in this disclosure aims to provide an intelligent load balancing system that predicts the likelihood of QoS satisfaction for each worker node based on the application characteristics and performance requirements of a client and the real-time resource status of a plurality of worker nodes, and distributes service requests to the optimal node according to the prediction result.
[0016] The embodiment disclosed in this disclosure aims to provide a load balancing system that enables load balancing that considers service quality and cost simultaneously by exchanging performance and price information through a negotiation channel between a client and a load balancer and allowing the client to directly select one of a plurality of service selection options.
[0017] The embodiment disclosed in this disclosure aims to provide a load balancing system capable of stably maintaining application performance even under load fluctuation conditions by continuously monitoring application performance after service execution and dynamically adjusting CPU, GPU, memory, and network resources.
[0018] The problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below.
[0019] The problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem
[0020] A load balancing system according to the present disclosure comprises: a load balancer that, in accordance with a connection request from a client, creates a data channel for transmitting and receiving application data and a negotiation channel for exchanging service performance requirements and pricing information, distributes the service request to worker nodes corresponding to service selection options selected by the client through the negotiation channel, and controls each of the plurality of worker nodes to perform an application service corresponding to the service request through the data channel; a metric collector that collects in real time the usage status of at least one resource among CPU, GPU, memory, and network usage rates and performance information of an application currently running from each of the plurality of worker nodes; a database that stores the information collected by the metric collector; and an LB controller that executes a QoS prediction model to predict expected performance (QoS) for each of the plurality of worker nodes using the resource status information of each of the plurality of worker nodes stored in the database and the application characteristics and performance requirements of the client received through the negotiation channel as inputs, generates a plurality of service selection options based on the QoS prediction result, and controls the load balancer to provide the generated plurality of selection options to the client through the negotiation channel. and includes a resource controller that continuously monitors the performance of the application during the execution of the application service and dynamically adjusts the allocation of at least one of the CPU, GPU, memory, and network resources according to the monitoring results.
[0021] In addition, the above QoS prediction model uses the resource status information of the worker nodes and the application characteristics and performance requirements of the client as input, and predicts the expected response time or processing performance of each worker node based on a Multi-Layer Perceptron (MLP).
[0022] In addition, the LB controller receives application characteristic information of the client from the load balancer and provides the application characteristic information, the system metric, and the application performance information, along with the system metric and application performance information collected by the metric collector and stored in the database, as input data for the QoS prediction model to predict the QoS for each of the plurality of worker nodes.
[0023] In addition, the load balancer exchanges service performance requirements and pricing information in the negotiation channel in at least one of TCP-based negotiation, HTTP-based negotiation, and user-defined protocol-based negotiation.
[0024] In addition, the load balancer filters worker nodes that satisfy minimum performance conditions among the plurality of worker nodes, and determines the priority of the service selection options by scoring the filtered worker nodes based on performance, cost, and resource availability.
[0025] In addition, the load balancer inserts a negotiation header into the data stream transmitted and received from the client via a library hooking method and transmits it, and upon receiving the data stream, processes the negotiation header and application data separately.
[0026] In addition, the load balancer transmits a QoS option header containing the predicted QoS and price information of each of the plurality of worker nodes received from the LB controller to the client through the negotiation channel.
[0027] In addition, the resource controller determines the resource allocation amount using a reinforcement learning-based learning model.
[0028] In addition, the resource controller identifies an application using an application identifier (App ID) transmitted from the load balancer and automatically selects a set of resources to be controlled among CPU, memory, disk, network, GPU, and Pods according to the type of the identified application.
[0029] In addition, the resource controller further includes a reinforcement learning controller, and the reinforcement learning controller performs learning using a reward function, and the reward function is implemented by the following mathematical formula.
[0030]
[0031] The above represents the weight for QoS satisfaction, and the above represents the weight for the resource usage penalty, and the above refers to the performance satisfaction indicator, and the above represents the total resource utilization rate.
[0032] A load balancing control method according to the present disclosure comprises: a load balancer creating a data channel for transmitting and receiving application data and a negotiation channel for exchanging service performance requirements and pricing information in response to a connection request from a client; a metric collector collecting, in real time, the usage status of at least one resource among CPU, GPU, memory, and network usage and performance information of an application running from each of the plurality of worker nodes; a database storing the collected resource usage status and performance information; an LB controller executing a QoS prediction model that predicts expected performance (QoS) for each of the plurality of worker nodes using the resource status information of each of the plurality of worker nodes stored in the database and the application characteristics and performance requirements of the client received through the negotiation channel as inputs; an LB controller generating a plurality of service selection options based on the QoS prediction result; an LB controller controlling the load balancer to provide the generated plurality of service selection options to the client through the negotiation channel; and a load balancer distributing the service request to a worker node corresponding to the service selection option selected by the client through the negotiation channel. The method includes the step of the selected worker node performing an application service through the data channel; and the step of the resource controller continuously monitoring the performance of the application during service execution and dynamically adjusting the allocation of at least one of the CPU, GPU, memory, and network resources according to the monitoring result.
[0033] Additionally, it further includes a step of predicting the expected response time or processing performance of each worker node based on a Multi-Layer Perceptron (MLP), using the resource status information of the worker nodes and the application characteristics and performance requirements of the client as input.
[0034] Additionally, the method further includes the step of receiving application characteristic information including application characteristic information of the client from the load balancer; and the step of predicting QoS for each of the plurality of worker nodes by providing the application characteristic information, the system metric, and the application performance information, together with the system metric and application performance information collected by the metric collector and stored in the database, as input data for the QoS prediction model.
[0035] Additionally, the method further includes the step of exchanging service performance requirements and pricing information in at least one of TCP-based negotiation, HTTP-based negotiation, and user-defined protocol-based negotiation in the aforementioned negotiation channel.
[0036] Additionally, the method further includes the step of filtering worker nodes that satisfy minimum performance conditions among the plurality of worker nodes; and the step of determining the priority of the service selection option by scoring the filtered worker nodes based on performance, cost, and resource availability.
[0037] Additionally, the method further includes the step of inserting a negotiation header into a data stream transmitted or received from a client by a library hooking method and transmitting it; and the step of distinguishing and processing the negotiation header and application data when receiving the data stream.
[0038] In addition, it further includes the step of transmitting a QoS option header containing the predicted QoS and price information of each of the plurality of worker nodes received from the LB controller to the client through the negotiation channel.
[0039] In addition, it further includes the step of determining the resource allocation amount using a reinforcement learning-based learning model.
[0040] Additionally, it further includes the step of identifying an application using an application identifier (App ID) transmitted from a load balancer; and the step of automatically selecting a set of control target resources among CPU, memory, disk, network, GPU, and Pod according to the type of the identified application.
[0041] In addition, it further includes a step of performing learning using a reward function, and the said reward function is implemented by the following mathematical formula.
[0042]
[0043] The above represents the weight for QoS satisfaction, and the above represents the weight for the resource usage penalty, and the above refers to the performance satisfaction indicator, and the above represents the total resource utilization rate.
[0044] In addition to this, a computer program stored on a computer-readable recording medium may be further provided to carry out a method for implementing the present disclosure.
[0045] In addition, a computer-readable recording medium for recording a computer program for executing a method for implementing the present disclosure may be further provided. Effects of the invention
[0046] According to the present invention, since a worker node is selected by predicting the likelihood of QoS satisfaction at the time of a service request, there is an effect of reducing latency and stably guaranteeing the Quality of Service (QoS) compared to the existing static round-robin method.
[0047] According to the present invention, by distributing requests while precisely reflecting the resource utilization rate of each server, it is possible to prevent the use of servers beyond what is necessary, thereby enabling the expectation of a reduction in the total number of servers, energy savings, and reduced cloud infrastructure operating costs.
[0048] According to the present invention, traffic can be flexibly redistributed even when an overload occurs on a specific node, and resources can be dynamically adjusted even after the service is executed, so the service stability and availability can be continuously secured even in the event of a failure or sudden load change.
[0049] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing
[0050] FIG. 1 is a configuration diagram of a load balancing system according to the present disclosure. FIG. 2 is a flowchart illustrating a load balancing control method according to the present disclosure. FIG. 3 is a drawing illustrating the core concept of the present invention according to the present disclosure. FIG. 4 is a diagram illustrating the collection of server metrics according to the present disclosure. FIG. 5 is a diagram illustrating an RDMA-based multi-client processing structure of the present invention according to the present disclosure. FIG. 6 is a diagram illustrating an embodiment of the MLP-based QoS prediction of the present invention according to the present disclosure. FIG. 7 is a diagram illustrating policy-based load balancing through filtering and scoring according to the present disclosure. FIG. 8 is a diagram illustrating negotiation-based load balancing of the present invention according to the present disclosure. FIG. 9 is a diagram illustrating the structure of the application QoS request information header of the present invention according to the present disclosure. FIG. 10 is a diagram illustrating the structure of the QoS option header message of the present invention according to the present disclosure. FIG. 11 is a diagram illustrating the structure of a QoS option selection header message of the present invention according to the present disclosure. FIG. 12 is a diagram illustrating the TCP protocol of the present invention according to the present disclosure. FIG. 13 is a drawing illustrating the HTTP protocol of the present invention according to the present disclosure. FIG. 14 is a diagram illustrating the App ID-based resource control structure of the present invention according to the present disclosure. FIG. 15 is a diagram illustrating the reinforcement learning-based VPA and HPA linked resource control structure of the present invention according to the present disclosure. FIG. 16 is a diagram illustrating a reinforcement learning resource control structure based on a reward function according to the present disclosure. FIG. 17 is a drawing illustrating a method for controlling the number of Pods of the present invention according to the present disclosure. FIG. 18 is a diagram illustrating a method for controlling a CPU, MEMORY, DISK, and NETWORK according to the present disclosure. Specific details for implementing the invention
[0051] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.
[0052] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.
[0053] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.
[0054] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.
[0055] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.
[0056] Singular expressions include plural expressions unless there is an obvious exception in the context.
[0057] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.
[0058] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.
[0059] In this specification, the present invention may be implemented not only as a server system but also as various devices capable of performing computational processing and providing results to a user. For example, the present invention may include a computer, a server device, and a portable terminal, or may take the form of any one of them.
[0060] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.
[0061] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.
[0062] The above portable terminal may include, for example, all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).
[0063] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.
[0064] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined operation rules or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using a number of training data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.
[0065] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a Deep Neural Network (DNN), such as a Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the examples mentioned above.
[0066] The processor can create a neural network, train (or learn) a neural network, perform operations based on received input data, generate an information signal based on the results of the operation, or retrain the neural network.
[0067] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), ESN (Echo It will be understood by a person skilled in the art that any neural network may be included, but is not limited to, State Network, Deep Residual Network, Differential Neural Computer, Neural Turning Machine, Capsule Network, Kohonen Network, and Attention Network.
[0068] According to an exemplary embodiment of the present disclosure, the processor comprises a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4 for Natural Language Processing, Visual Analytics, Visual Understanding, Video Synthesis for Vision Processing, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization for ResNet Data Intelligence, Various artificial intelligence structures and algorithms, such as recommendation and data creation, may be used, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.
[0069] FIG. 1 is a configuration diagram of a load balancing system according to the present disclosure.
[0070] Referring to FIG. 1, the load balancing system (1000) includes a load balancing device (100), a client (200), worker nodes (301, 302, 303, 304), an application, a system, and a node resource pool.
[0071] Node resource pools include GPUs, memory, and CPUs.
[0072] The load balancing device (100) includes a load balancer (110), an LB controller (120), a resource controller (130), a metric collector (140), a database (150), and a QoS prediction model (160).
[0073] The load balancer (110) creates a data channel for transmitting and receiving application data and a negotiation channel for exchanging service performance requirements and price information in response to a connection request from the client (200), distributes the service request to worker nodes corresponding to the service selection option selected by the client through the negotiation channel, and controls each of the plurality of worker nodes (301-304) to perform an application service corresponding to the service request through the data channel.
[0074] The LB controller (120) executes a QoS prediction model (160) that predicts the expected performance (QoS) for each of the plurality of worker nodes (301-304) using resource status information of each of the plurality of worker nodes (301-304) stored in the database (150) and application characteristics and performance requirements of the client (200) received through the negotiation channel as inputs, generates a plurality of service selection options based on the QoS prediction results, and controls the load balancer (110) to provide the generated plurality of selection options to the client through the negotiation channel.
[0075] The resource controller (130) continuously monitors the performance of the application during the execution of the application service and dynamically adjusts the allocation of at least one of the CPU, GPU, memory, and network resources according to the monitoring results.
[0076] The metric collector (140) collects, in real time, the usage status of at least one resource among CPU, GPU, memory, and network usage and performance information of the running application from each of the plurality of worker nodes (301-304).
[0077] The database (150) stores information collected by the metric collector (140).
[0078] The QoS prediction model (160) predicts the expected performance (QoS) for each of the plurality of worker nodes (301-304) by taking as input the resource status information of each of the plurality of worker nodes (301-304) stored in the database (150) and the application characteristics and performance requirements of the client (200) received through the negotiation channel.
[0079] The client (200), as the service user, transmits a request to execute an application, provides its application characteristics and QoS requirements to the load balancer, and performs the role of receiving application services from a selected server based on the negotiation result.
[0080] Multiple worker nodes (301 to 304) are execution entities that actually process service requests distributed by the load balancer (110), and each worker node functions as a computing node where an application runs.
[0081] Each worker node performs application services using CPU, memory, GPU, and network resources, and resource usage status and application performance information during the service execution process are collected in real time through a metric collector (140).
[0082] In addition, the plurality of worker nodes are dynamically adjusted in resource allocation amount by the control of the resource controller (130), and operate so that the processing load is balancedly distributed according to the control policy of the QoS prediction model (160) and the LB controller (120).
[0083] The application is an executable program that runs on multiple worker nodes (301 to 304) and performs actual service functions in response to service requests from a client (200).
[0084] The system is a physical or virtual computing environment in which an application is executed, and provides an execution platform including hardware resources such as a CPU, memory, storage device, network and GPU.
[0085] The above QoS prediction model (160) takes the resource status information of the worker nodes and the application characteristics and performance requirements of the client as inputs, and predicts the expected response time or processing performance of each worker node based on a Multi-Layer Perceptron (MLP).
[0086] The LB controller (120) receives application characteristic information of the client from the load balancer (110) and provides the application characteristic information, the system metric, and the application performance information, along with the system metric and application performance information collected by the metric collector (140) and stored in the database (150), as input data to the QoS prediction model (160) to predict the QoS for each of the plurality of worker nodes.
[0087] A detailed explanation of this is shown in Fig. 6.
[0088] The load balancer (110) exchanges service performance requirements and price information in the negotiation channel in at least one of TCP-based negotiation, HTTP-based negotiation, and user-defined protocol-based negotiation.
[0089] A detailed explanation of this is given in Figures 12 and 13.
[0090] The load balancer (110) filters worker nodes that satisfy minimum performance conditions among the plurality of worker nodes, and scores the filtered worker nodes based on performance, cost, and resource availability to determine the priority of the service selection options.
[0091] A detailed explanation of this is given in Fig. 7.
[0092] The load balancer (110) inserts a negotiation header into a data stream transmitted and received from a client (200) by a library hooking method and transmits it, and when receiving the data stream, it distinguishes and processes the negotiation header and application data.
[0093] The load balancer (110) transmits a QoS option header containing the predicted QoS and price information of each of the plurality of worker nodes received from the LB controller (120) to the client (200) through the negotiation channel.
[0094] A detailed explanation of this is shown in Fig. 8.
[0095] The resource controller (130) determines the resource allocation amount using a reinforcement learning-based learning model.
[0096] A detailed explanation of this is shown in Fig. 15.
[0097] The resource controller (130) identifies an application using an application identifier (App ID) transmitted from the load balancer (110) and automatically selects a set of resources to be controlled among CPU, memory, disk, network, GPU, and Pod according to the type of the identified application.
[0098] A detailed explanation of this is shown in Fig. 14.
[0099] The above resource controller (130) further includes a reinforcement learning controller, and the reinforcement learning controller performs learning using a reward function.
[0100] The above compensation function is implemented by the following mathematical formula.
[0101]
[0102] The above represents the weight for QoS satisfaction, and the above represents the weight for the resource usage penalty, and the above refers to the performance satisfaction indicator, and the above represents the total resource utilization rate.
[0103] A detailed explanation of this is shown in Fig. 16.
[0104] At least one component may be added or removed in response to the performance of the components illustrated in FIG. 1. Additionally, it will be readily understood by those skilled in the art that the relative positions of the components may be changed in response to the performance or structure of the system.
[0105] Meanwhile, each component illustrated in Fig. 1 refers to a software and / or hardware component such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).
[0106] FIG. 2 is a flowchart illustrating a load balancing control method according to the present disclosure.
[0107] The present invention is performed by the components of a load balancing system (1000).
[0108] A load balancer (110) creates a data channel for transmitting and receiving application data and a negotiation channel for exchanging service performance requirements and price information in response to a connection request from the client (200) (S210).
[0109] A metric collector (140) collects in real time the resource usage status of at least one of CPU, GPU, memory, and network usage and the performance information of the running application from each of the plurality of worker nodes (S220).
[0110] The database (150) stores the collected resource usage status and performance information (S230).
[0111] The LB controller (120) executes a QoS prediction model that predicts the expected performance (QoS) for each of the plurality of worker nodes using the resource status information of each of the plurality of worker nodes stored in the database (150) and the application characteristics and performance requirements of the client received through the negotiation channel as input (S240).
[0112] The LB controller (120) generates a plurality of service selection options based on the above QoS prediction result (S250).
[0113] The LB controller (120) controls the load balancer (110) to provide the generated plurality of service selection options to the client through the negotiation channel (S260).
[0114] The load balancer (110) distributes the service request to a worker node corresponding to the service selection option selected by the client through the negotiation channel (S270).
[0115] The selected worker node performs application services through the data channel (S280).
[0116] A resource controller (130) continuously monitors the performance of the application during service execution and dynamically adjusts the allocation of at least one of the CPU, GPU, memory, and network resources according to the monitoring results (S290).
[0117] FIG. 3 is a drawing illustrating the core concept of the present invention according to the present disclosure.
[0118] With reference to FIG. 3, the core concept of the present invention is explained.
[0119] The present invention is an intelligent load balancing method that predicts and selects a server that matches the performance (QoS) desired by a client (200) in advance, and automatically adjusts server resources during service.
[0120] (1) Client sends order and requirements (S310)
[0121] When a client (200) connects to the server, the load balancer (110) simultaneously creates a data channel and a negotiation channel.
[0122] Here, ① the data channel refers to the actual service data channel, and ② the negotiation channel refers to the channel for conveying requirements such as “I prefer high speed” or “price is important.”
[0123] (2) Real-time collection of the current worker node status (S320)
[0124] At each worker node (server), the metric collector (140) continuously collects data on how much CPU is used, how much GPU is used, memory, network, and actual service speed, and stores it in the DB.
[0125] (3) Predict the matching performance of clients and worker nodes using AI (S330)
[0126] The LB controller (120) combines the server status information stored in the DB and the performance requirements stated by the client (200) with a QoS prediction model (160) (MLP artificial intelligence) to predict performance in advance, saying, “This server has a speed score of X points, that server has a speed score of Y points.”
[0127] (4) Present the predicted performance as selectable options to the client (S340)
[0128] For example, it is as follows.
[0129] Option 1: Fast + Expensive
[0130] Option 2: Medium speed + Medium price
[0131] Option 3: Slow but cheap
[0132] By creating multiple options like this, they are shown directly to the client (200) through a negotiation channel.
[0133] (5) Connect the actual service to the server selected by the client (S350)
[0134] When a client (200) selects an option, the load balancer (110) forwards the request to the worker node corresponding to that option, and then the actual service data is processed as is through the data channel.
[0135] (6) Automatically continue to adjust server resources even during service (S360)
[0136] Even after the service has started, the resource controller (130) continues to monitor whether the actual service speed has slowed down, whether resources are insufficient, or whether too much is being wasted, and automatically increases or decreases CPU, GPU, memory, and network resources.
[0137] FIG. 4 is a diagram illustrating the collection of server metrics according to the present disclosure.
[0138] Referring to FIG. 4 (410), a system metric collection system according to one embodiment of the present invention is configured such that a client-side monitoring tool and a server-side metric collector are each deployed in the form of Kubernetes-based Pods.
[0139] (1) Client-side metric transmission structure
[0140] Multiple client pods equipped with monitoring tools are deployed on the client side, and each client pod connects to the server-side metric collector pod through a Queue Pair (QP) using the RDMA RC (Reliable Connection) method.
[0141] Each of the above client pods sends system metrics (CPU usage, memory usage, GPU usage, and GPU-related metrics) to the server side at regular intervals.
[0142] At this time, the above metric data is transmitted via the RDMA protocol using a kernel bypass method (zero-copy), thereby ensuring lower latency and higher transmission efficiency compared to existing TCP / IP-based transmission.
[0143] (2) Server-side metric reception and processing structure
[0144] A server pod equipped with a metric collector is deployed on the server side, and the server pod receives metric data transmitted from multiple client pods using a shared receive queue (SRQ) and a shared completion queue (SCQ).
[0145] Through this structure, the following effects are achieved for metric traffic flowing in simultaneously from multiple client pods (reduction in receive queue management overhead, improvement in interrupt processing efficiency, and high-speed RDMA-based parallel receive processing).
[0146] (3) Metric Store Update and Prometheus Integration Method
[0147] Metric frames received from the server pod are parsed in real time by an internal parser, and the parsed metric information is accumulated and updated in real time in the metric store inside the server.
[0148] In addition, the above metric collector server internally provides an HTTP endpoint and converts metric data into a format compliant with the Prometheus standard, exposing it externally through the / gpu_metrics path.
[0149] Accordingly, Prometheus can reliably collect metric data by periodically performing HTTP GET requests to the above / gpu_metrics path.
[0150] FIG. 5 is a diagram illustrating an RDMA-based multi-client processing structure of the present invention according to the present disclosure.
[0151] Referring to Fig. 5 (510), an RDMA-based multi-client processing structure will be described.
[0152] The server uses a Shared Completion Queue (SCQ) and a Shared Receive Queue (SRQ) structure to efficiently process RDMA communication data transmitted simultaneously from multiple clients.
[0153] (1) Shared Completion Queue (SCQ)
[0154] First, the Shared Completion Queue (SCQ) integrates and stores all transmit and receive completion events from multiple Queue Pairs (QPs) into a single Queue Queue (CQ), and is configured so that a single polling thread periodically checks the SCQ to process all completion events. This significantly reduces thread creation and context switching overhead compared to the existing method of allocating individual processing threads to each QP.
[0155] (2) Shared Receive Queue (SRQ)
[0156] In addition, the shared receive queue (SRQ) is configured so that multiple QPs jointly use a single receive buffer pool. As a result, there is no longer a need to manage individual receive queues for each connection (QP), which improves memory usage efficiency and enables stable receive processing even in environments with large-scale concurrent connections.
[0157] Meanwhile, when the server receives a completion event from the SCQ, it checks the QP number (qp_num) included in the event and uses it as a key value in a predefined connection map to search for the connection context corresponding to that QP. Through this method of connection demultiplexing, the received data is delivered to the correct application connection.
[0158] Through such a shared CQ, shared SRQ, and QP number-based demultiplexing structure, the present invention can provide high scalability and stable reception performance without increasing the number of threads, even in an environment with thousands or more simultaneous RDMA connections.
[0159] FIG. 6 is a diagram illustrating an embodiment of the MLP-based QoS prediction of the present invention according to the present disclosure.
[0160] Referring to FIG. 6 (610), an embodiment for MLP-based QoS prediction is described.
[0161] The QoS prediction model (160) takes the resource status information of the worker nodes and the application characteristics and performance requirements of the client as inputs and predicts the expected response time or processing performance of each worker node based on a Multi-Layer Perceptron (MLP).
[0162] For example, when a client requests a high-resolution video streaming service,
[0163] - Latency requirement: 30ms or less,
[0164] - Frame rate requirement: 60FPS,
[0165] - Input bitrate: 20Mbps
[0166] It is assumed that performance requirements such as are are conveyed together.
[0167] At this time, the QoS prediction model (160) receives resource status information such as CPU usage, memory usage, GPU availability, and network bandwidth usage collected from multiple worker nodes, along with application characteristic information such as resolution, bitrate, and frame rate.
[0168] The MLP-based QoS prediction model computes the above input values through a multilayer neural network, for example
[0169] - Worker Node 1: Estimated latency 25ms, Estimated FPS 62
[0170] - 2nd Worker Node: Estimated Latency 48ms, Estimated FPS 45
[0171] - 3rd Worker Node: Estimated Latency 28ms, Estimated FPS 58
[0172] Outputs the expected response time or processing performance (QoS) for each worker node as shown.
[0173] Based on the predicted results, the LB controller (120) selects the worker node most likely to satisfy the client's QoS requirements and delivers it to the load balancer (110), and the load balancer (110) distributes service requests to the corresponding worker node.
[0174] The LB controller (120) receives application characteristic information including application characteristic information of the client from the load balancer (110), and provides the application characteristic information, the system metric, and the application performance information, along with the system metric and application performance information collected by the metric collector (140) and stored in the database (150), as input data to the QoS prediction model (160) to predict the QoS for each of the plurality of worker nodes.
[0175] Application feature information is information representing the characteristics of an application requested by a client, and may include, for example, at least one of output resolution, output bitrate, input resolution, and input bitrate.
[0176] System metric information is information indicating the current operating status of each worker node, and may include, for example, at least one of a node ID, CPU utilization, memory utilization, GPU model information, and GPU available resources.
[0177] Application performance information is information representing application execution performance, and may include, for example, at least one of a session identifier (session ID), a process identifier (PID), average latency (avg latency), and average frame rate (avg FPS).
[0178] FIG. 7 is a diagram illustrating policy-based load balancing through filtering and scoring according to the present disclosure.
[0179] Referring to FIG. 7 (710), an example of policy-based load balancing through filtering and scoring will be described.
[0180] The load balancer (110) filters worker nodes that satisfy minimum performance conditions among the plurality of worker nodes and determines the priority of the service selection options by scoring the filtered worker nodes based on performance, cost, and resource availability.
[0181] This explains the load balancing behavior based on QoS prediction-based server selection and negotiation.
[0182] The load balancer (110) controls the selection of a worker node suitable for the service using the output result of the QoS prediction model (160) based on the QoS requirements received from the client, or allows the client to directly select a node through negotiation.
[0183] (1) QoS prediction value generation step
[0184] The load balancer (110) provides, for each worker node, resource status information such as CPU usage, memory usage, GPU usage status, and network load status, and application characteristic information such as the resolution, bitrate, FPS, and latency requirements of the client application as input values for a Multi-Layer Perceptron (MLP) based QoS prediction model.
[0185] The QoS prediction model (160) generates a list of predicted QoS values, such as expected latency, processing performance, or FPS for each worker node, and transmits them to the load balancer (110).
[0186] (2) 1st filtering stage based on QoS requirements (Standalone method)
[0187] The load balancer (110) compares ① the target QoS requested by the client (200) (e.g., maximum latency 30ms, FPS 60 or higher, etc.) and ② the predicted QoS for each node, and filters only the nodes that satisfy the QoS requirements as primary candidate nodes.
[0188] (3) Scoring-based final node selection step
[0189] The following score calculation is performed on the filtered candidate nodes.
[0190] 1) Individual resource usage ratio It is implemented by the following mathematical formula.
[0191]
[0192] i represents the index of the resource type (e.g., CPU, memory, GPU, network, etc.).
[0193] t represents the measurement time.
[0194] represents the actual usage of resource i at time t.
[0195] represents the total available capacity of resource i at time t.
[0196] represents the relative usage rate of resource i (normalized value range 0 to 1).
[0197] In other words, it is a value that quantitatively indicates how much of an individual resource is being used within the whole.
[0198] 2) Average total resource usage rate It is implemented by the following mathematical formula.
[0199]
[0200] m represents the total number of resources under consideration.
[0201] represents the usage ratio of each resource.
[0202] represents the average utilization rate of the total resources of the corresponding node.
[0203] This value is a representative indicator showing how much load a single node is under overall.
[0204] 3) The score for the final node selection is implemented using the following mathematical formula.
[0205]
[0206] represents the margin between the predicted QoS at node i and the client's requested QoS. (e.g., the difference between the allowed latency and the actual predicted latency)
[0207] represents the average utilization rate of the total resources of node i.
[0208] represents the weighting coefficient between QoS satisfaction and resource utilization.
[0209] represents the highest evaluation score of the corresponding node.
[0210] According to the present invention, ① the score increases as the QoS margin increases, and ② the score decreases as the resource utilization rate increases, so ③ nodes that satisfy QoS and are not overloaded can be selected preferentially.
[0211] In summary, ① when using the resource utilization policy, nodes with high resource utilization are assigned higher scores, and ② when using the balancing policy, nodes with low resource utilization are assigned higher scores, so the node with the highest score is selected as the final service node.
[0212] (4) Negotiation-based node selection step
[0213] Based on the above QoS prediction results, the load balancer (110) generates a plurality of QoS option messages (QoS Option) including node ID, expected QoS, amount of resources used, and price information, and transmits them to the client.
[0214] The client (200) selects one of the above options and responds back to the load balancer (110), and the load balancer (110) confirms the node selected by the client (200) as the final service node.
[0215] (5) Connection setup and data relay steps
[0216] After establishing a TCP or HTTP connection with the selected node, the load balancer (110) relays application data (e.g., video stream, HTTP packet, etc.) received from the client (200) to the destination node without changing the content.
[0217] Afterwards, the load balancer (110) transmits a load balancing completion signal (lb signal) to the LB controller to notify that the connection has been successfully established.
[0218] FIG. 8 is a diagram illustrating negotiation-based load balancing of the present invention according to the present disclosure.
[0219] Referring to FIG. 8 (810), an embodiment regarding negotiation header-based communication processing will be described.
[0220] When the load balancer (110) operates in negotiation mode, it can process a data stream in which a negotiation header is inserted during the process of transmitting and receiving data with the client (200) using a library hooking method.
[0221] Specifically, the library hooking described above can be configured to directly intervene in the send / receive function at the client's application library call layer to automatically insert a negotiation header along with the application data, or to separate and recognize it upon reception.
[0222] Negotiation headers are classified into the following three types depending on the message type.
[0223] ① QoS Requirement Header
[0224] The QoS Requirement Header includes characteristic information and performance requirements (QoS) of an application running on a client (200). For example, it may include information such as an application identifier (App ID), latency requirements, throughput requirements, frame rate (FPS), input bitrate, and resolution.
[0225] ② QoS Option Header
[0226] The QoS Option Header includes the QoS performance of each worker node predicted by the LB controller (120) and corresponding pricing information. That is, it includes the expected latency, processing performance, and expected billing information based on the use of the resources for each node, and is provided to the client (200).
[0227] ③ QoS Selection Header
[0228] The QoS Selection Header contains the result of the client (200) selecting one of the QoS Option Headers, and includes the selected application ID and the selected option number (Option number).
[0229] An example of header insertion based on library hooking is described.
[0230] The load balancer (110) inserts a negotiation header into a data stream transmitted and received from a client (200) by a library hooking method and transmits it, and when receiving the data stream, it distinguishes and processes the negotiation header and application data.
[0231] According to the present invention, even if negotiation data and actual application data are transmitted and received simultaneously over a single TCP connection, the load balancer can accurately distinguish them and perform negotiation procedures and service data processing in parallel.
[0232] An example of QoS option transmission implementation is described.
[0233] The load balancer (110) transmits a QoS Option Header containing the predicted QoS and price information of each of the plurality of worker nodes received from the LB controller (120) to the client (200) through the negotiation channel.
[0234] The client (200) compares the performance and price information for each node included in the above QoS Option Header, selects the option it wants, and then sends a response in the form of a QoS Selection Header.
[0235] Accordingly, the load balancer (110) can finally determine the worker node corresponding to the option selected by the client.
[0236] FIG. 9 is a diagram illustrating the structure of the application QoS request information header of the present invention according to the present disclosure.
[0237] Referring to Fig. 9 (910), the structure diagram of the application QoS request information header will be explained.
[0238] The QoS Requirement Header is a core header area containing the performance requirements of the application requested by the client, and includes the following detailed fields.
[0239] ① In the case of App ID, it is a unique identifier used to identify applications running on the client.
[0240] ② In the case of a Feature, it is information expressed in the form of bit values representing additional functional characteristics required depending on the type or operational characteristics of the application.
[0241] ③ In the case of latency, it is the acceptable delay time requirement demanded by the application.
[0242] ④ In the case of throughput, it is a performance requirement based on the throughput (transmission amount) required by the application.
[0243] ⑤ FPS (Frames Per Second) represents the performance requirement for the number of frames per second required in video applications.
[0244] ⑥ Input Bitrate is the bitrate information of the original video that is input into the transcoding application.
[0245] ⑦ In the case of Resolution, it is the resolution information of the original video input from the transcoding application.
[0246] According to the present invention, the QoS requirement message has the advantage of enabling the load balancer and LB controller to perform node selection and resource control optimized for application characteristics by formalizing and transmitting performance requirements such as latency, throughput, FPS, bitrate, and resolution.
[0247] FIG. 10 is a diagram illustrating the structure of the QoS option header message of the present invention according to the present disclosure.
[0248] Referring to Fig. 10 (1010), the structure of the QoS option header message will be described.
[0249] QoS is a core header area containing multiple service selection option information provided to the client (200), and includes the following detailed fields.
[0250] ① Option Count is information indicating the total number of QoS options provided by the load balancer to the client.
[0251] ② Option Length is information indicating the data length (in bits) of each QoS option.
[0252] ③ In the case of App ID, it is an identifier used to identify the application to which the corresponding QoS option applies.
[0253] ④ For Option, it is an option identification number to distinguish each QoS option.
[0254] ⑤ Node ID is the identifier of the candidate worker node to which the corresponding QoS option applies.
[0255] ⑥ In the case of Latency, it represents the predicted application latency (performance metric) at the worker node corresponding to the above Node ID.
[0256] ⑦ Price is information on the predicted service usage cost when selecting the corresponding QoS option, and the unit can be expressed in USD / s.
[0257] According to the present invention, by including node identifiers, latency, and price information, there is an advantage in that it supports a client in selecting an optimal service option by considering performance and cost simultaneously.
[0258] FIG. 11 is a diagram illustrating the structure of a QoS option selection header message of the present invention according to the present disclosure.
[0259] Referring to Fig. 11 (1110), a structure diagram of the QoS option selection header message is illustrated.
[0260] A header area containing key information for identifying the service option selected by the client (200), comprising the following fields.
[0261] The App ID is an identifier for identifying the application to be executed on the client (200).
[0262] The Option Number is number information for identifying the option finally selected by the client (200) among the multiple QoS options provided by the load balancer (110).
[0263] According to the present invention, there is an advantage in that the load balancer can accurately determine the final service node by clearly identifying the application and QoS options selected by the client.
[0264] FIG. 12 is a diagram illustrating the TCP protocol of the present invention according to the present disclosure.
[0265] Referring to Figures 12 and 13, an example of a protocol-specific negotiation method for load balancing is described.
[0266] The load balancer (110) exchanges service performance requirements and price information on the negotiation channel in at least one of TCP-based negotiation, HTTP-based negotiation, and user-defined protocol-based negotiation.
[0267] Referring to Fig. 12 (1210), the operation of TCP protocol negotiation-based QoS load balancing is described.
[0268] (1) TCP connection setup step
[0269] The client (200) establishes a TCP protocol-based connection with the load balancer (110). At this time, the TCP handshake packet is temporarily stored by the load balancer (110).
[0270] (2) QoS request transmission step
[0271] The client (200) initiates negotiation by sending a QoS requirement message containing the characteristics and performance requirements of the application to the load balancer (110) over the TCP connection.
[0272] (3) QoS prediction request step
[0273] The load balancer (110) transmits the application characteristic information included in the QoS requirement message to the LB controller (120).
[0274] (4) QoS prediction step
[0275] The LB controller (120) uses application characteristic information transmitted from the client and system metric information of each worker node stored in the model database (Model DB) as input values for the QoS prediction model to calculate the expected QoS value for each worker node.
[0276] (5) QoS option provision step
[0277] The LB controller (120) transmits QoS option information, including node-specific predicted QoS and price information, to the load balancer (110), and the load balancer (110) transmits this to the client (200) in the form of a negotiation message.
[0278] (6) Service Node Selection Step
[0279] The client (200) selects one of the provided multiple QoS options and responds with an Option select message.
[0280] (7) Connection completion and data transmission steps
[0281] The load balancer (110) identifies the worker node corresponding to the option selected by the client and delivers the temporarily stored TCP handshake packet to the corresponding worker node to complete the TCP connection between the client and the worker node.
[0282] (8) Real-time data relay stage
[0283] Afterwards, the load balancer (110) relays the RTMP-based video streaming data received from the client to the selected worker node.
[0284] (9) Load balancing completion notification step
[0285] The load balancer (110) notifies the LB controller (120) in the form of an lb signal that traffic distribution is complete.
[0286] In the present invention, the RTMP protocol used in video transmission can be cited as a representative example of the TCP protocol.
[0287] FIG. 13 is a drawing illustrating the HTTP protocol of the present invention according to the present disclosure.
[0288] Referring to Fig. 13 (1310), the HTTP protocol negotiation-based QoS load balancing operation will be described.
[0289] (1) Client initial connection phase
[0290] The client (200) performs a TCP handshake with the load balancer (110) to establish a basic communication connection.
[0291] (2) Service request transmission step
[0292] The client (200) sends an HTTP request message corresponding to the video request to the load balancer (110) over the established TCP connection.
[0293] (3) Transmission of QoS requirements and initiation of negotiation
[0294] The client (200) initiates negotiation by sending a QoS requirement message (QoS requirement) for negotiation to the load balancer (110) separately from the HTTP request.
[0295] (4) QoS Requirements Delivery Step
[0296] The load balancer (110) transmits the received QoS requirement message to the LB controller (120).
[0297] (5) QoS prediction execution step
[0298] The LB controller (120) calculates a predicted QoS for each of the multiple nodes by using the received application characteristic information and the system metrics of the Model DB stored in advance as model inputs.
[0299] (6) QoS option provision step
[0300] The LB controller (120) transmits QoS option information, including QoS information, resource status, and price information of each predicted node, to the load balancer (110), and the load balancer (110) transmits this to the client (200) in the form of a QoS option message.
[0301] (7) Option selection step
[0302] The client (200) selects one of the multiple QoS options and sends an Option select message to the load balancer (110) to end the negotiation.
[0303] (8) Step to set up the connection to the target node
[0304] The load balancer (110) identifies the target node corresponding to the selected option and establishes a TCP connection with the target node using the TCP handshake information that was previously temporarily stored.
[0305] (9) HTTP request forwarding step
[0306] The load balancer (110) forwards the HTTP request message received in step (2) as is to the target node, changing only the destination.
[0307] (10) Node response processing step
[0308] The target node generates a response (e.g., a video manifest) to the received HTTP request and sends it to the load balancer (110).
[0309] (11) Client response delivery step
[0310] The load balancer (110) transmits the HTTP response message received from the node to the client (200) as is.
[0311] (12) Application service execution step
[0312] Afterwards, continuous data communication is performed between the client (200) and the target node according to the characteristics of the application.
[0313] (13) Load balancing completion signal transmission step
[0314] The load balancer (110) transmits the fact that load balancing is complete to the LB controller (120) in the form of an lb signal.
[0315] FIG. 14 is a diagram illustrating the App ID-based resource control structure of the present invention according to the present disclosure.
[0316] Referring to Fig. 14 (1410), an App ID-based resource control structure diagram is described.
[0317] The resource controller (130) identifies an application using an application identifier (App ID) transmitted from the load balancer (110) and automatically selects a set of resources to be controlled among CPU, memory, disk, network, GPU, and Pod according to the type of the identified application.
[0318] Specifically, the load balancer (110) extracts an application identifier (App ID) from a QoS requirement header received during the negotiation process and transmits it to the resource controller (130).
[0319] The resource controller (130) identifies the application to be controlled using the received App ID and determines the set of resources to be controlled (Resource Space) corresponding to the application. The resource set includes at least one of CPU, memory, disk, network, GPU, and Pod.
[0320] The resource controller (130) applies different control methods depending on the type of selected resource.
[0321] (1) CPU and memory control - VPA-based vertical expansion
[0322] The resource controller (130) uses a Vertical Pod Autoscaler (VPA) to dynamically scale the computational performance of the application by adjusting the limits of CPU and memory allocated to the container.
[0323] (2) Pod count control - HPA-based horizontal scaling
[0324] The resource controller (130) uses the Horizontal Pod Autoscaler (HPA) to dynamically adjust throughput by increasing or decreasing the number of replicas of the application container.
[0325] (3) Disk and Network Control - OS Level cgroup Control
[0326] The resource controller (130) uses an operating system level control method (OS Controller) to control disk bandwidth using a cgroup file system for disk resources and network bandwidth using cgroup tc for network resources.
[0327] At this time, to identify the container to be controlled, ① collect pod UID, namespace, pod name, and pod class (QoS Class) information using the kubectl command, and ② specify the file system path to be controlled based on the node information where the pod is located.
[0328] (4) GPU resource control - NVIDIA MPS-based control
[0329] The resource controller (130) uses NVIDIA MPS (Multi-Process Service) to control the ratio of GPU memory and computational resources allocated to a specific container, and to do this, modifies the distribution file (application.yaml) of the container to reflect the GPU control settings.
[0330] According to the present invention, there are advantages such as ① the ability to integrally control heterogeneous resources (CPU, memory, disk, network, GPU, pod) according to application characteristics, ② the implementation of precise resource control based on QoS rather than simple auto-scaling, ③ the ability to recover dynamic resources in real-time when performance degradation occurs, and ④ the ability to guarantee high-reliability QoS in cloud and Kubernetes-based real-time service environments.
[0331] FIG. 15 is a diagram illustrating the reinforcement learning-based VPA and HPA linked resource control structure of the present invention according to the present disclosure.
[0332] With reference to FIG. 15 (1510), the operation of a reinforcement learning-based resource control controller will be described.
[0333] The resource controller (130) further includes a reinforcement learning controller (RL Controller).
[0334] The resource controller (130) determines the resource allocation amount using a reinforcement learning-based learning model.
[0335] The reinforcement learning controller receives application performance information and system metrics stored in a database at predetermined intervals. The system metrics include CPU, memory, disk, network, and GPU usage, and the application performance information may include latency, throughput, FPS, etc.
[0336] The RL controller receives the above system metrics and application performance information as a state and determines a resource control action corresponding to each application through a reinforcement learning algorithm.
[0337] The above action includes control values for at least one of CPU, memory, disk, network bandwidth, GPU, and number of pods.
[0338] In addition, the RL controller is configured to perform learning based on a reward function in which the reward increases as resource usage decreases and as QoS requirements are satisfied.
[0339] Accordingly, the RL controller continuously updates the control policy in a direction that simultaneously satisfies resource efficiency and service quality.
[0340] The control result determined by the RL controller is transmitted to at least one of ① a VPA (Vertical Pod Autoscaler) for vertical expansion, ② a HPA (Horizontal Pod Autoscaler) for horizontal expansion, and ③ an operating system (OS) level resource control module, so that the execution resources of the application are dynamically adjusted.
[0341] According to the present invention, intelligent resource control is possible to minimize unnecessary resource usage while preventing performance degradation of the application.
[0342] FIG. 16 is a diagram illustrating a reinforcement learning resource control structure based on a reward function according to the present disclosure.
[0343] Referring to FIG. 16 (1610), an embodiment of performing learning using a reward function will be described.
[0344] The resource controller (130) further includes a reinforcement learning controller.
[0345] The reinforcement learning controller performs learning using a reward function.
[0346] The compensation function is implemented by the following mathematical formula.
[0347] The compensation function is configured to simultaneously reflect resource usage efficiency and QoS satisfaction, and each component of the formula has the following meaning.
[0348] Explain the internal composition of the compensation function.
[0349] 1) Resource Usage Function
[0350] The resource utilization function is implemented by the following mathematical formula.
[0351]
[0352] represents the utilization rate of the i-th resource at time t.
[0353] represents the actual usage of the i-th resource at time t.
[0354] represents the maximum available capacity of the i-th resource.
[0355] For each resource (CPU, Memory, Disk, Network, GPU, etc.), the current usage is normalized and expressed as a ratio value in the range of 0 to 1.
[0356] 2) Average total resource usage rate
[0357] The average total resource usage rate is implemented using the following mathematical formula.
[0358]
[0359] represents the total system resource utilization at time t.
[0360] m represents the number of resource types. (e.g., CPU, Memory, Disk, Network, GPU)
[0361] It refers to the importance weight for each resource.
[0362] represents the utilization rate of the i-th resource.
[0363] According to the present invention, the utilization rate of each resource is weighted by importance weights to evaluate the overall resource burden level of the system as a single value.
[0364] 3) Performance metrics against QoS requirements
[0365] The performance indicator relative to QoS requirements is implemented by the following mathematical formula.
[0366] refers to the target QoS requirements of the application. (e.g., target latency, target throughput, target FPS, etc.)
[0367] refers to actual measured application performance. (e.g., actual latency, actual FPS, etc.)
[0368]
[0369] The above mathematical formula is used as a ratio indicator to show how well actual performance satisfies QoS requirements.
[0370] 4) Final reward function
[0371] The final reward function is implemented by the following mathematical formula.
[0372]
[0373] represents the reinforcement learning reward value at time t.
[0374] represents the weight for QoS satisfaction.
[0375] represents the weight for the resource usage penalty.
[0376] represents a performance satisfaction indicator.
[0377] represents the total resource utilization rate.
[0378] This reward function is designed with two objectives: ① the reward increases as QoS is better satisfied, and ② the reward decreases as more resources are used. In other words, it guides the reinforcement learning agent toward satisfying the maximum QoS with the minimum resources.
[0379] FIG. 17 is a drawing illustrating a method for controlling the number of Pods of the present invention according to the present disclosure.
[0380] Referring to Fig. 17 (1710), a method for controlling the number of Pods is described.
[0381] Increasing the number of Pods allows you to effectively handle situations where the load spikes.
[0382] Control the pod through Kubernetes using the k8s API.
[0383] Information to identify targets to control is collected on each node using Kubernetes API commands.
[0384] Collection of deployment name: kubectl get pod <pod-name> -n <namespace>--show-labels
[0385] Namespace 수집: kubectl get pods -A | grep <pod-name>| awk '{print $1}'
[0386] <x>=> deployment name: (string)
[0387] <y>=> pod's namespace: (length 36 string)
[0388] <value>=> Value to control (number)
[0389] FIG. 18 is a diagram illustrating a method for controlling a CPU, MEMORY, DISK, and NETWORK according to the present disclosure.
[0390] Referring to FIG. 18 (1810), a method of controlling the resource controller (130) will be described.
[0391] (1) Method to control CPU, MEMORY, DISK and Network
[0392] The resource controller (130) dynamically controls the CPU, memory, disk, and network allocated to each pod to ensure the performance of the application running in the Kubernetes environment.
[0393] The resource controller (130) controls the file system of the cgroup by writing the value to be controlled to the corresponding file.
[0394] Information to identify targets to control is collected on each node using Kubernetes API commands.
[0395] Collect pod uid: kubectl get pods -A -o jsonpath='{range .items[]}{.metadata.uid}{"\t"}{.metadata.namespace} / {.metadata.name}{"\n"}{end}'
[0396] Collect pod class: kubectl get pod <pod-name> -n <namespace>-o jsonpath='{.status.qosClass}'
[0397] Collect pod nodes: kubect;
[0398] The resource controller (130) passes the file system path and the value to be controlled to the node where the pod is located, and executes control on that node.
[0399] (2) How to control the GPU
[0400] The Relos controller (130) modifies and applies the deployment file of each pod.
[0401] <x>=> pod class: (burstable, besteffort)
[0402] <y>=> pod uid: (length 36 string)
[0403] <value>=> Value to control (number)
[0404] Control of GPU resources is performed by modifying the Pod deployment file (application.yaml). For example, GPU usage can be controlled by setting environment variables or resource options such as the following.
[0405] Here, resource options include GPU multiprocessing limits and GPU memory limits.
[0406] The resource controller (130) controls GPU resources by modifying and redeploying the distribution file of the corresponding pod.
[0407] (3) Summary of control execution flow
[0408] The resource controller (130) collects Pod UID, QoS class, and node information through the Kubernetes API. CPU, memory, disk, and network resources are directly controlled at the node level using the cgroup file system. GPU resources are controlled via redeployment by modifying the Pod deployment file, which enables immediate resource reallocation and performance recovery in the event of application performance degradation.
[0409] The aforementioned program may include code encoded in a computer language such as C, C++, JAVA, or machine language, which can be read by the computer's processor (CPU) through the computer's device interface, in order for the computer to read the program and execute the methods implemented in the program. Such code may include functional code related to functions that define the necessary functions for executing the methods, and may include control code related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. Additionally, such code may further include memory reference code regarding where (address) additional information or media necessary for the computer's processor to execute the functions should be referenced in the computer's internal or external memory. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to transmit or receive during communication.
[0410] The above-mentioned storage medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the above-mentioned storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the above-mentioned program may be stored on various recording media on various servers that the computer can access, or on various recording media on the user's computer. Additionally, the above-mentioned medium may be distributed across networked computer systems, and computer-readable code may be stored in a distributed manner.
[0411] The steps of the method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, a hard disk, a removable disk, a CD-ROM, or any form of computer-readable recording medium well known in the art to which the present disclosure belongs.
[0412] Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive.
[0413] The various embodiments of the present disclosure are not intended to list all possible combinations but to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.< / value> < / y> < / x> < / namespace> < / pod-name> < / value> < / y> < / x> < / namespace> < / pod-name>
Claims
Claim 1 A load balancing system that receives a service request from a client and distributes the service request to at least one of a plurality of worker nodes, comprising: a load balancer that, in accordance with a connection request from the client, creates a data channel for transmitting and receiving application data and a negotiation channel for exchanging service performance requirements and price information, distributes the service request to a worker node corresponding to a service selection option selected by the client through the negotiation channel, and controls each of the plurality of worker nodes to perform an application service corresponding to the service request through the data channel; a metric collector that collects in real time the resource usage status of at least one of CPU, GPU, memory, and network usage rates and performance information of an application currently running from each of the plurality of worker nodes; a database that stores the information collected by the metric collector; and executes a QoS prediction model that predicts expected performance (QoS) for each of the plurality of worker nodes using the resource status information of each of the plurality of worker nodes stored in the database and the application characteristics and performance requirements of the client received through the negotiation channel as input, generates a plurality of service selection options based on the QoS prediction result, and provides the generated plurality of selection options to the client through the negotiation channel. LB controller controlling a load balancer; and resource controller continuously monitoring the performance of the application during the execution of the application service and dynamically adjusting the allocation of at least one of CPU, GPU, memory, and network resources according to the monitoring result, wherein the resource controller further includes a reinforcement learning controller, and the reinforcement learning controller performs learning using a reward function, and the reward function is implemented by the following mathematical formula. The above represents the weight for QoS satisfaction, and the above represents the weight for the resource usage penalty, and the above refers to the performance satisfaction indicator, and the above A load balancing system that refers to the total resource utilization rate. Claim 2 In claim 1, the load balancing system, wherein the QoS prediction model takes resource status information of the worker nodes and application characteristics and performance requirements of the client as inputs, and predicts the expected response time or processing performance of each worker node based on a Multi-Layer Perceptron (MLP). Claim 3 A load balancing system according to claim 1, wherein the LB controller receives application characteristic information of the client from the load balancer, and provides the application characteristic information, the system metric, and the application performance information together with the system metric and application performance information collected by the metric collector and stored in the database as input data for the QoS prediction model to predict QoS for each of the plurality of worker nodes. Claim 4 In claim 1, the load balancer is a load balancing system that exchanges service performance requirements and pricing information in at least one of TCP-based negotiation, HTTP-based negotiation, and user-defined protocol-based negotiation in the negotiation channel. Claim 5 A load balancing system according to claim 1, wherein the load balancer filters worker nodes satisfying minimum performance conditions among the plurality of worker nodes, and determines the priority of the service selection option by scoring the filtered worker nodes based on performance, cost, and resource availability. Claim 6 A load balancing system according to claim 1, wherein the load balancer inserts a negotiation header into a data stream transmitted and received from a client by a library hooking method and transmits it, and distinguishes and processes the negotiation header and application data when receiving the data stream. Claim 7 A load balancing system according to claim 6, wherein the load balancer transmits a QoS option header containing predicted QoS and price information for each of the plurality of worker nodes received from the LB controller to the client through the negotiation channel. Claim 8 In claim 1, the resource controller is a load balancing system that determines the resource allocation amount using a reinforcement learning-based learning model. Claim 9 A load balancing system according to claim 1, wherein the resource controller identifies an application using an application identifier (App ID) transmitted from the load balancer and automatically selects a set of resources to be controlled among CPU, memory, disk, network, GPU, and Pod according to the type of the identified application. Claim 10 delete Claim 11 A load balancing control method for receiving a service request from a client and distributing the service request to at least one of a plurality of worker nodes, comprising: a step in which a load balancer creates a data channel for transmitting and receiving application data and a negotiation channel for exchanging service performance requirements and pricing information in accordance with a connection request from the client; a step in which a metric collector collects in real time the resource usage status of at least one of CPU, GPU, memory, and network usage and performance information of an application running from each of the plurality of worker nodes; a step in which a database stores the collected resource usage status and performance information; a step in which an LB controller executes a QoS prediction model that predicts expected performance (QoS) for each of the plurality of worker nodes using the resource status information of each of the plurality of worker nodes stored in the database and the application characteristics and performance requirements of the client received through the negotiation channel as inputs; a step in which an LB controller generates a plurality of service selection options based on the QoS prediction result; a step in which an LB controller controls the load balancer to provide the generated plurality of service selection options to the client through the negotiation channel; and a step in which the load balancer sends the service request to the worker node corresponding to the service selection option selected by the client through the negotiation channel. The method comprises: a step of distributing; a step in which the selected worker node performs an application service through the data channel; and a step in which a resource controller continuously monitors the performance of the application during service execution and dynamically adjusts the allocation of at least one of the CPU, GPU, memory, and network resources according to the monitoring result, and further comprises a step of performing learning using a reward function, wherein the reward function is implemented by the following mathematical formula. The above represents the weight for QoS satisfaction, and the above represents the weight for the resource usage penalty, and the above refers to the performance satisfaction indicator, and the above A load balancing control method that refers to the total resource utilization rate. Claim 12 A load balancing control method according to claim 11, further comprising the step of predicting the expected response time or processing performance of each worker node based on a Multi-Layer Perceptron (MLP) using resource status information of the worker nodes and application characteristics and performance requirements of the client as inputs. Claim 13 A load balancing control method according to claim 11, further comprising: receiving application characteristic information including application characteristic information of the client from the load balancer; and providing the application characteristic information, the system metric, and the application performance information, together with the system metric and application performance information collected by the metric collector and stored in the database, as input data for the QoS prediction model to predict the QoS for each of the plurality of worker nodes. Claim 14 A load balancing control method according to claim 11, further comprising the step of exchanging service performance requirements and price information in at least one of TCP-based negotiation, HTTP-based negotiation, and user-defined protocol-based negotiation in the negotiation channel. Claim 15 A load balancing control method according to claim 11, further comprising: a step of filtering worker nodes satisfying minimum performance conditions among the plurality of worker nodes; and a step of determining the priority of the service selection option by scoring the filtered worker nodes based on performance, cost, and resource availability. Claim 16 A load balancing control method according to claim 11, further comprising: a step of inserting a negotiation header into a data stream transmitted and received from a client by a library hooking method and transmitting it; and a step of distinguishing and processing the negotiation header and application data when receiving the data stream. Claim 17 A load balancing control method according to claim 16, further comprising the step of transmitting a QoS option header containing predicted QoS and price information for each of the plurality of worker nodes received from the LB controller to the client through the negotiation channel. Claim 18 A load balancing control method according to claim 11, further comprising the step of determining the resource allocation amount using a reinforcement learning-based learning model. Claim 19 A load balancing control method according to claim 11, further comprising: a step of identifying an application using an application identifier (App ID) transmitted from the load balancer; and a step of automatically selecting a set of resources to be controlled among CPU, memory, disk, network, GPU, and Pod according to the type of the identified application. Claim 20 delete
Citation Information
Patent Citations
Method and framework for traffic engineering in network hypervisor of SDN-based network virtualization platform
KR1020180122513A
Systems and methods for chaining control-plane virtual functions to ensure end-to-end QoS (Quality of Service) of Internet services
KR1020210024572A