SaaS software load balancing method based on improved deep double-Q network
By improving the deep dual Q network method, the Q value overestimation and local optimization problems in SaaS software load balancing scheduling are solved, and more efficient and stable load balancing and resource allocation are achieved to adapt to dynamically changing service requests .
Patent Information
- Application Number
- CN202410983784.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
The existing cloud service load balancing scheduling algorithms have problems with Q value overestimation and local optimization in SaaS software scenarios, making it difficult to achieve load balancing and efficient resource allocation, and traditional scheduling algorithms cannot adapt to dynamically changing service requests.
Using the improved deep dual Q network (RDDQN-SS) method, an agent is constructed, state space, action space and reward functions are set, and the optimized agent is obtained through iterative optimization, which is used for load balancing scheduling of SaaS software, overcoming the Q value bias. High and local optimal problems improve the stability and efficiency of load balancing.
It significantly improves the stability and efficiency of load balancing scheduling of SaaS software, achieves a more balanced virtual resource utilization rate and a higher service request success rate, and adapts to the dynamic changes in different service request density scenarios.
Smart Images

Figure CN118939422A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of load regulation, and in particular to a SaaS software load balancing method based on an improved deep dual-Q network. Background Art
[0002] Software as a Service (SaaS) has become an important industry direction in the global science and technology field. With the development of Internet technology, the SaaS model has grown rapidly around the world due to its convenience and efficiency. SaaS software provides services to users through the Internet. Its main features include relying on the Internet, multi-tenancy, and service characteristics. In order to ensure that each tenant's data and configuration are independent, as well as fast response speed and system stability, SaaS software must have an efficient load balancing mechanism. This mechanism is the key to achieving service availability and rapid response, and is also an important topic for SaaS service providers to study. Therefore, studying the load balancing of SaaS software not only helps to improve its execution efficiency, but also helps to meet dynamic service requests and ensure the high stability and rapid response capabilities of the software.
[0003] At present, relevant researchers have conducted a lot of research on the evaluation indicators and scheduling algorithms related to cloud service load balancing scheduling, and used deep reinforcement learning algorithms and hybrid algorithms combined with other strategies for cloud service scheduling. However, there is not much research on the load balancing problem based on SaaS software at this stage. The SaaS software load balancing mechanism must not only ensure the overall execution efficiency of the SaaS software, but also ensure the rationality of the load distribution of virtual resources of the SaaS software. Only in this way can the high stability and fast response characteristics of the SaaS software be ensured; therefore, are cloud service scheduling indicators applicable to SaaS software load balancing scenarios? Is the cloud service load balancing scheduling algorithm applicable to SaaS software load balancing scenarios? This requires relevant research to prove. At the same time, when using the DQN algorithm to schedule service requests, the Q network will overestimate or fall into the local optimal problem. Therefore, the challenges faced by the research on SaaS software load balancing are summarized as follows:
[0004] (1) Although SaaS software is a commercial product under cloud computing, since SaaS software must have the characteristics of high stability and fast response, is the cloud service scheduling algorithm still applicable to solving the SaaS software load balancing scenario? Can the relevant indicators in the cloud service scheduling process reflect the overall operating load status of the SaaS software cluster?
[0005] (2) When using the DQN algorithm for cloud service scheduling, the intelligent agent often encounters the problem of "overestimation" when updating the Q value. Although the DDQN algorithm cleverly solves the above problem, the algorithm can easily fall into a local optimum, and once it falls into a local optimum, a large amount of data is required to correct it, which may even lead to the emergence of a false optimal action value function.
[0006] (3) Traditional scheduling algorithms that do not use deep reinforcement learning (DRL) strategies are no longer able to solve the problem of dynamically changing service requests for SaaS software. At the same time, deep reinforcement learning algorithms are widely used in cloud service task scheduling due to their powerful self-learning capabilities. However, is the result of applying DRL strategies to SaaS software load balancing scenarios better than using traditional scheduling algorithms? Which DRL algorithm can better meet user SLA agreements in the process of reasonably distributing SaaS software running loads?
[0007] Analysis of existing methods for SaaS software load balancing research:
[0008] (1) Traditional algorithms: Traditional cloud service load balancing strategies such as min-min, max-min, round-robin, and shortest job first are mostly static load balancing techniques. The main problem with these algorithms is that they are usually unable to detect the current status of the connected servers, resulting in uneven resource distribution. In dynamically changing distributed systems, these static techniques usually do not work well unless there is low load fluctuation in the nodes.
[0009] (2) Heuristic algorithms: Heuristic load balancing algorithms are inspired by natural phenomena, such as genetic algorithms, bee-inspired algorithms, ant colony optimization (ACO), and particle swarm optimization (PSO). These algorithms can adaptively adjust parameters and combine existing mechanisms to solve load balancing problems, and usually achieve high user satisfaction and resource utilization. However, the scheduling effects of these algorithms in cloud computing environments are not always optimal, especially when processing complex service request scheduling, they may not achieve the optimal load balancing state.
[0010] (3) Deep reinforcement learning algorithms: including value function-based reinforcement learning algorithms, policy-based reinforcement learning algorithms, and hybrid algorithms of these two methods. These algorithms combine neural network models and use the powerful computing power of neural networks to calculate the value function in reinforcement learning. In specific scenarios, such as continuous action scenarios, policy-based deep reinforcement learning algorithms can solve similar scenario problems because they do not need to maintain Q values but only need to maintain action selection strategies. However, although these algorithms show strong self-learning capabilities, whether they can outperform traditional scheduling algorithms in SaaS software load balancing scenarios still requires further research and verification. Summary of the invention
[0011] In view of this, the purpose of the present invention is to provide a SaaS software load balancing method based on an improved deep dual-Q network, which is used to solve the problem that the existing algorithm cannot perform load balancing scheduling of SaaS software well.
[0012] The present invention provides a SaaS software load balancing method based on an improved deep dual-Q network, comprising the steps of:
[0013] S1: Build an intelligent agent, which includes: load balancing model, SaaS software load balancing environment model and load balancing optimization problem model;
[0014] S2: Set the agent’s state space, action space, and reward function;
[0015] S3: Iteratively optimize the state space, action space and reward function by improving the dual-depth Q network to obtain the optimized intelligent agent;
[0016] S4: Load balancing scheduling of SaaS software is performed through optimized intelligent agents.
[0017] Preferably, step S1 specifically comprises:
[0018] S11: Define the service request service of SaaS software i and virtual resources VM j , set the load balancing degree of SaaS software to build a load balancing model, i is the number of the service request, j is the number of the virtual resource;
[0019] S12: Define service request service based on load balancing model i In virtual resources VM j Execution time on Waiting time and departure time Depend on and Add up to get the total time
[0020] S13: Pass Set the success rate ij , by execution time and departure time Calculate resource utilization Utilizi_VM ij , by execution time and virtual resources VM j Calculate the processing cost ij ;
[0021] S14: Based on success ij Utilizi_VM ij and Cost ij Build a SaaS software load balancing environment model;
[0022] S15: Set the average instruction response time ratio avgVT based on the load balancing model and the SaaS software load balancing environment model IJ , overall average resource utilization avgU t , virtual resource utilization standard deviation std_VM and the final optimization target, and build a load balancing optimization problem model;
[0023] S16: The intelligent agent is composed of a load balancing model, a SaaS software load balancing environment model and a load balancing optimization problem model.
[0024] Preferred:
[0025] Resource utilization Utilizi_VM ij The expression is:
[0026]
[0027] in, Indicates that it includes service request service i The k loads included are virtualized by the VM j implement;
[0028] Processing cost ii The expression is:
[0029]
[0030] Among them, VSC j Indicates virtual resources VM j Virtual resource startup cost, VOC j Indicates virtual resources VM j The unit time running cost of virtual resources.
[0031] Preferably, step S15 is specifically as follows:
[0032] S151: Calculate and obtain service request service i Based on virtual resources VM j The command response time is shorter than VT ij , whose expression is:
[0033]
[0034] When the virtual resource VM j There are I service requests being executed, then based on VM j The average instruction response time is greater than avgVT j Defined as:
[0035]
[0036] The SaaS service cluster has a total of J virtual resources, so the average instruction response time of the entire service cluster is avgVT IJ Defined as:
[0037]
[0038] S152: Virtual resource VM of the SaaS service cluster at time t j Completion time VMF tj Defined as:
[0039]
[0040] in, Indicates that I service request is in the virtual resource VM j The latest time of completion;
[0041] Virtual resource VM in the SaaS service cluster at time t j Resource utilization VMU for executing multiple service requests tj Defined as:
[0042]
[0043] in, Indicates that I service request is in the virtual resource VM j The latest departure time, For I service requests in virtual resource VM j The total execution time on Indicates that the task is in the virtual resource VM j The start time of the execution;
[0044] The overall average resource utilization rate avgU of the virtual resources of the SaaS service cluster at time t t Defined as:
[0045]
[0046] S153: The standard deviation of virtual resource utilization std_VM is defined as:
[0047]
[0048] S154: The final optimization goal is defined as:
[0049]
[0050] Among them, π represents the load balancing algorithm;
[0051] The constraints of the final optimization objective are:
[0052]
[0053] Preferred:
[0054] State space S at time t t Current state ij It is expressed as:
[0055]
[0056] Among them, ServiceSize i Indicates service request service i Load size, DeadTime i Indicates service request service i The maximum time to complete;
[0057] Action space A at time t t Service request service i and virtual resources VM j action ij It is expressed as:
[0058]
[0059] Reward function R at time t t In virtual resources VM j The reward value in reward tj It is expressed as:
[0060]
[0061] Among them, μ is the adjustment parameter, VMU tj is the virtual resource VM in the SaaS service cluster at time t j Resource utilization for executing multiple service requests.
[0062] Preferably, step S3 specifically comprises:
[0063] S31: Initialize the evaluation network Q and the target network Q′;
[0064] S32: Update the parameters of the agent at time t and obtain the state space S at time t t , action space A t and the reward function R t ;
[0065] S33: S t , A t and R t Input the evaluation network Q and calculate the state space S at time t+1 t+1 ;
[0066] S34: S t+1 , A t and R t Input the evaluation network Q and the target network Q′ to update the parameter y t , generate loss Loss;
[0067] S35: If the Loss is less than the preset value, the optimized agent is output; otherwise, t=t+1 is set and the process returns to step S32.
[0068] Preferred:
[0069] Parameter y t The expression is:
[0070]
[0071] Among them, r t Represents the reward function Rt at time t in the virtual resource VM j The reward value in reward tj , γ is the reward decay coefficient.
[0072] A SaaS software load balancing device based on an improved deep dual-Q network includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the SaaS software load balancing method based on the improved deep dual-Q network.
[0073] The present invention has the following beneficial effects:
[0074] The intelligent agent is composed by constructing a load balancing model, a SaaS software load balancing environment model and a load balancing optimization problem model; and the state space, action space and reward function of the intelligent agent are set. The state space enables the intelligent agent to fully perceive the state of the current environment, and the action space allocates service requests to the corresponding virtual resources. The reward function not only considers the successful execution of service requests, but also focuses on the utilization balance of virtual resources. By setting the utilization range, the load balancing effect is guaranteed. By improving the dual-depth Q network to iteratively optimize the state space, action space and reward function, the high Q value and local optimal problems are overcome, and the stability and efficiency of the optimized intelligent agent in load balancing scheduling of SaaS software are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;
[0076] Figure 2 It is a schematic diagram of the workflow of the intelligent agent;
[0077] Figure 3 Schematic diagram for iterative optimization of the agent;
[0078] Figure 4 This is a schematic diagram of load balancing scheduling of SaaS software through optimized intelligent agents;
[0079] Figure 5 The average utilization of virtual resources using six algorithms under four data densities;
[0080] Figure 6 The average response time of six algorithms under four data density scenarios;
[0081] Figure 7 The execution costs of six algorithms in four data scenarios are shown;
[0082] Figure 8 The service request execution success rate of the six algorithms in four data scenarios;
[0083] Fig. 9 This is a structural diagram of the device according to an embodiment of the present invention;
[0084] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0085] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0086] Reference Figure 1 The present invention provides a SaaS software load balancing method based on an improved deep dual-Q network, comprising the steps of:
[0087] S1: Build an intelligent agent, which includes: load balancing model, SaaS software load balancing environment model and load balancing optimization problem model;
[0088] Further, the agent workflow reference Figure 2 ;
[0089] Step S1 is specifically as follows:
[0090] S11: Define the service request service of SaaS software i and virtual resources VM j , set the load balancing degree of SaaS software to build a load balancing model, i is the number of the service request, j is the number of the virtual resource;
[0091] Specifically, the load balancing model includes the following definitions:
[0092] (1) Definition of service request
[0093] Each service request is defined in the SaaS software as including the service request number, the time when the service request is connected to the SaaS software, the size of the data that needs to be processed by the service request, and the service level agreement (SLA) signed with the SaaS software. These requests are considered as immediate loads, which means that the SaaS software does not know in advance the type and number of requests that will arrive when processing the current request. In this model, it is assumed that the SaaS service provider handles the same type of service; each service request is defined as follows:
[0094] service i ={serviceID i ,arrivalT i ,serviceSize i ,DeadTime i} (1)
[0095] Among them, serviceID i ,arrivalT i ,serviceSize i ,DeadTime i They are respectively represented as request number, time when the request arrives at the SaaS software, request load size, and maximum time required for the request to be completed in the SaaS software.
[0096] (2) Definition of virtual resources
[0097] SaaS software service requests are executed by virtual resources, and each virtual resource is usually composed of one or more virtual machines (VMs). Virtual machines can simulate computer operating systems with complete hardware system functions and provide the same functions as physical machines. The definition of each virtual resource includes the number of virtual resources, processing speed per unit time, number of CPU units, startup cost, and running cost per unit time. In this model, it is assumed that the total number of fixed virtual resource nodes in the resource pool is M. Then each virtual resource can be defined as:
[0098] VM j ={VMID j ,VComp j ,VCPU j ,VSC j ,VOC j},j∈M (2)
[0099] Among them, VMID j ,VComp j ,VCPU j ,VSC j ,VOC jThey respectively represent the number of the virtual resource, the processing speed per unit time of the virtual resource, the number of CPU units of the virtual resource, the startup cost of the virtual resource, and the running cost per unit time of the virtual resource.
[0100] (3) SaaS software load balancing
[0101] The load balancing measurement of SaaS software uses the standard deviation of virtual resource utilization. This standard deviation reflects the difference in the load distributed between each virtual resource node. The smaller the standard deviation of virtual resource utilization, the more evenly the load is distributed; conversely, the larger the standard deviation, the more unbalanced the load distribution.
[0102] S12: Define service request service based on load balancing model i In virtual resources VM j Execution time on Waiting time and departure time Depend on and Add up to get the total time
[0103] Specifically, in this model, we define the response time of each SaaS software service request as the total time the request stays in the SaaS software. The actual execution time of the service request The waiting time of the service request in the queue waiting to be executed The sum of . It can be defined as:
[0104]
[0105] In this model, the pricing of virtual resources is proportional to their own processing capacity, that is, the more CPU processing units there are, the greater the capacity of the corresponding virtual resources. Therefore, the processing capacity of virtual resources is equal to the processing speed of the CPU unit and the number of CPU units contained in the virtual resource, that is, the virtual resource VM j The processing power can be defined as:
[0106] VMComp j =VComp j *VCPU j (4)
[0107] In this model, we do not distinguish between the types of service requests. By default, all virtual resources have the same execution type for task requests, that is, all virtual resources can handle the assigned service requests, so we directly define the execution time. The expression is:
[0108]
[0109] Since the performance of virtual resources is not stable in a dynamic SaaS software load scheduling environment, VMComp j are the average processing capabilities of virtual resources. Therefore, the task execution time calculated by the above formula Not necessarily the actual execution time Right now
[0110] In this model, each service request is assigned to the corresponding execution queue of each virtual resource by the SaaS software according to the load scheduling algorithm. Therefore, if the execution queue of the virtual resource is empty, the service request can directly apply for the virtual resource to complete the demand; if the corresponding virtual resource is executing other services when the service request enters the execution queue, the request continues to wait until the previous service is processed. At this time, the waiting time of the service request is the difference between the time when the corresponding virtual resource starts to execute the service request and the time when the service request enters the execution queue. i The waiting time is defined as:
[0111]
[0112] in, Represents task i In virtual resources VM j The earliest time to execute on the ij For service i The time when the virtual resource arrives at the execution queue corresponding to the virtual resource.
[0113] In this model, Indicates service i In virtual resources VM j The earliest execution time in this paper will Defined as:
[0114]
[0115] in, Indicates the service in the execution queue i Previous service request service k The earliest time to be processed; arriveT kj Respectively represent service k In virtual resources VM j The execution time on the queue and the time it arrives at the execution queue. Since the present invention assumes that the next service request is executed immediately after the previous service request is executed, the previous service request kThe time it takes to leave the virtual resource after execution is equal to the time it takes for the next service request to be made. i The time when execution starts, i.e. Indicates service request service k Leaving a virtual resource VM j If Indicates service request service k When entering the execution queue, the corresponding virtual resource VM j is executing another service; if Indicates service request service k When entering the execution queue, it is immediately assigned to the corresponding virtual resource VM j Executing.
[0116] When the service request is processed in the SaaS software, the SaaS software will feedback the execution result of the service request to the corresponding tenant; at this time, the corresponding service request service i The completion time is equal to the time when the service request starts to be executed and the time when the service service i In the corresponding virtual resource VM j The sum of running times. It can be defined as:
[0117]
[0118] S13: Pass Set the success rate ij , by execution time and departure time Calculate resource utilization Utilizi_VM ij , by execution time and virtual resources VM j Calculate the processing cost ij ;
[0119] Specifically, for each service request, the user signs an SLA agreement with the SaaS software. In this model, it is assumed that the SLA agreement for each service request represents the service request service i Try to be successfully executed in the longest stay time in the SaaS software. It means service request service i The load balancing strategy is successfully executed, otherwise it means that the SaaS data center schedules the service request service based on the load balancing strategy i Failed. Therefore, each service i The corresponding user SLA can be defined as:
[0120]
[0121] For SaaS providers, the primary goal of processing service requests is to try to meet the SLA agreement signed with users during the scheduling of service requests under limited virtual resource conditions.
[0122] In this model, SaaS software focuses on the resource utilization of each virtual resource. However, the quality of the SaaS software load balancing scheduling strategy determines the size of the overall resource utilization of each virtual and the entire service cluster. j The ratio of the total execution time of all service requests executed in the virtual resource to the running time of the virtual resource is taken as the resource utilization rate of the virtual resource. j After executing the service request service i The definition of resource utilization after is as shown in formula (10);
[0123] Resource utilization Utilizi_VM ij The expression is:
[0124]
[0125] in, Indicates that it includes service request service i The k loads included are virtualized by the VM j implement;
[0126] Finally, in this model we assume that the SaaS platform rents virtual resources from the IaaS provider and deploys application service programs on the SaaS platform. Although SaaS software provides convenient and abundant software services, its cluster virtual resources require additional maintenance costs. Therefore, while trying to meet the service request agreements signed with small and medium-sized enterprises and tenants, the SaaS software needs to consider the costs of processing the service requests. Among them, the relevant fees mainly depend on the length of time the service request occupies virtual resources and are calculated according to the unit price of the virtual resource length of time. The fee is mainly divided into two parts: virtual resource startup fee and virtual resource occupation fee; therefore, the virtual resource VM can be defined j Service request service i The processing cost is defined as formula (11)
[0127] Processing cost ij The expression is:
[0128]
[0129] Among them, VSC jIndicates virtual resources VM j Virtual resource startup cost, VOC j Indicates virtual resources VM j The unit time running cost of virtual resources.
[0130] S14: Based on success ij Utilizi_VM ij and Cost ij Build a SaaS software load balancing environment model;
[0131] S15: Set the average instruction response time ratio avgVT based on the load balancing model and the SaaS software load balancing environment model IJ , overall average resource utilization avgU t , virtual resource utilization standard deviation std_VM and the final optimization target, and build a load balancing optimization problem model;
[0132] Specifically, step S15 is as follows:
[0133] S151: This model defines response time as the time a service request stays in the entire SaaS software. That is, the time from when the service request is connected to the SaaS software to when the service request is executed by the virtual resources;
[0134] Calculate the service request service i Based on virtual resources VM j The command response time is shorter than VT ij , whose expression is:
[0135]
[0136] When the virtual resource VM j There are I service requests being executed, then based on VM j The average instruction response time is greater than avgVT j Defined as:
[0137]
[0138] The SaaS service cluster has a total of J virtual resources, so the average instruction response time of the entire service cluster is avgVT IJ Defined as:
[0139]
[0140] S152: This model defines the virtual resource VM j The resource utilization of VM at time t is jThe execution time and virtual resource VM of several service requests that have been executed j The total running time. Assume that the overall running time of the service cluster is T seconds (1, 2, 3, …, t, …, T), and at time t, there are I service requests allocated to virtual resources VM j ;
[0141] Virtual resource VM of the SaaS service cluster at time t j Completion time VMF tj Defined as:
[0142]
[0143] in, Indicates that I service request is in the virtual resource VM j The latest time of completion;
[0144] Specifically, at time t, if This means that the virtual resource VM j The service on has been completed, otherwise it means that the virtual resource VM at time t j There are still unfinished services.
[0145] This definition assumes that the last service request executed is service I , according to formula (15), the definition of resource utilization is obtained;
[0146] Virtual resource VM in the SaaS service cluster at time t j Resource utilization VMU for executing multiple service requests tj Defined as:
[0147]
[0148] in, Indicates that I service request is in the virtual resource VM j The latest departure time, For I service requests in virtual resource VM j The total execution time on Indicates that the task is in the virtual resource VM j The start time of the execution;
[0149] Specifically, at time t, if This means that the virtual resource VM j All services on the Represents the virtual resource VM at time t jConsidering that the last service I has not been completed, the actual execution time of the virtual resource at time t is the sum of the execution time of the previous I-1 service requests and the time when service request I has been completed by the virtual resource VM. j The execution time and
[0150] Formula (16) describes the relationship between each virtual resource VM in the SaaS service cluster at time t j Resource utilization for executing several service requests, assuming that the number of virtual resources in the SaaS software cluster is J;
[0151] The overall average resource utilization rate avgU of the virtual resources of the SaaS service cluster at time t t Defined as:
[0152]
[0153] S153: The load balancing degree defined in this model is defined by the standard deviation of the virtual resource utilization of the SaaS software cluster, std_VM. The calculation formula of std_VM is calculated by formula (16) and formula (17), and the specific definition formula is as follows;
[0154] The standard deviation of virtual resource utilization std_VM is defined as:
[0155]
[0156] S154: Take the average overall resource utilization of the SaaS software service cluster, the success rate of the SaaS software, and other multiple objectives as the optimization objectives of the load balancing scenario of the SaaS software at time t. t The larger the value is, the more balanced the load distribution of the overall operation of the SaaS software is. This model replaces the optimization target of the SaaS software and verifies whether the SaaS software has a higher success rate of service request execution when the SaaS software resource nodes are relatively load balanced, that is, whether it still has a lower response time. Combining formulas (9) and (14), the final optimization target of this solution is obtained;
[0157] The final optimization objective is defined as:
[0158]
[0159] Among them, π represents the load balancing algorithm;
[0160] The constraints of the final optimization objective are:
[0161]
[0162] S16: The intelligent agent is composed of a load balancing model, a SaaS software load balancing environment model and a load balancing optimization problem model.
[0163] S2: Set the agent’s state space, action space, and reward function;
[0164] Further:
[0165] 1. Define the state space:
[0166] In the cloud service load balancing scheduling algorithm based on deep Q network, the surrounding environment perceived by the agent mainly includes the virtual resources of the service cluster and the status of the service request; further expressed as, the service request task i Instruction length taskSize i , the DeadTime of the SLA agreement signed between the service request and the SaaS software i ; Service request service i Select the time column that each virtual resource will wait for Therefore, when the service request needs to be dispatched to SaaS service cluster J, the current state perceived by the agent is expressed as follows;
[0167] State space S at time t t Current state ij It is expressed as:
[0168]
[0169] Among them, ServiceSize i Indicates service request service i Load size, DeadTime i Indicates service request service i The maximum time to complete;
[0170] 2. Define the action space:
[0171] In the present invention, the virtual resources rented by the SaaS service provider are fixed and limited. And each virtual resource corresponds to a service processing queue. At the same time, there is no limit on the size of the queue, that is, as long as the service request is scheduled to a service processing queue, there is no need to consider the queue capacity. Finally, the requests in the service queue are processed according to the FCFS policy. Only one service request can be processed at a time, and the next request will be processed immediately after the previous request is processed. The interval in between is negligible. An action refers to an action taken by an intelligent agent, specifically the behavior of SaaS software allocating service requests to virtual resources; if there are J virtual resources, then the behavior of the intelligent agent is one of the J virtual sets, that is, the action space is a numbered set of these J virtual resources, and the action can be expressed as receiving a certain service request task. i The serial number of the virtual resource j;
[0172] Action space A at time t t Service request service i and virtual resources VM j action ij It is expressed as:
[0173]
[0174] 3. Define the reward function:
[0175] Considering service i Virtualized resources VM J At the same time as successful execution, the resource utilization of each virtual resource also needs to be considered; since the total number of service requests is 5,000, a virtual resource node with strong execution load capacity is allocated more load, and the virtual resource utilization is high. Naturally, virtual resource nodes with weaker execution load capacity will be allocated fewer resources, and the corresponding virtual resource utilization will be small. In terms of SaaS software load balancing scenarios, it is not about pursuing a higher virtual resource utilization, but pursuing a smaller load difference between SaaS software virtual resource nodes. Therefore, in this model, when the range of resource utilization of each virtual resource is defined as I∈[0.3,0.7], it means that the virtual resource node is load balanced. If VMU tj >0.7, it means that the load balancer has allocated too much load to the virtual resource and the virtual resource node is overloaded. tj <0.3, which means that the load balancer allocates too little load to the virtual resource, and the virtual resource is underloaded. Therefore, after the cluster load is balanced and the success rate of service requests is maximized, the reward function defined in this solution is as follows:
[0176] Reward function R at time t t In virtual resources VMj The reward value in reward ti It is expressed as:
[0177]
[0178] Among them, μ is the adjustment parameter, VMU tj is the virtual resource VM in the SaaS service cluster at time t j Resource utilization for executing multiple service requests.
[0179] Specifically, formula (22) is mainly for service request service i Assigned to virtual resources VM j When the virtual resource utilization rate is large, reward feedback is given to maximize the average utilization rate of the overall resources of the SaaS software service cluster.
[0180] S3: Iteratively optimize the state space, action space and reward function by improving the dual-depth Q network to obtain the optimized intelligent agent;
[0181] Further:
[0182] 1. Motivation for improving the dual-depth Q network:
[0183] The DQN algorithm integrates the deep double Q network and experience replay mechanism, and is a classic deep Q network algorithm in cloud service scheduling scenarios. However, it is easy to have a high Q value problem when updating the state action value function; the DDQN algorithm changes the way the action value function is updated, and cleverly weakens the "overestimation" problem that occurs when the agent calculates the action value function Q. However, when the DDQN algorithm fits the action value function Q through the neural network, the state action value function Q is easy to fall into the local optimum. Once it falls into the local optimum, a large amount of data is required to correct the agent, and even lead to the emergence of a false optimal action value function. The Double Q-learning algorithm uses a dual network to alternately update the state value function, so that the evaluation network and the target network are further decoupled, reducing the risk of the action value function Q falling into the local optimum during the calculation process. This solution combines the idea of the Double Q-learning algorithm to improve the way the DDQN algorithm updates the action value function. Based on the original algorithm training process of DDQN, the evaluation network and the target network are replaced every other time.
[0184] 2. Iterative optimization algorithm structure:
[0185] This algorithm is mainly inspired by the Double Q-learning algorithm and the DDQN algorithm. It not only introduces a dual network and experience replay mechanism, but also further changes the way to update the Q value of the evaluation network. This algorithm is similar to the DDQN structure, that is, using argmaxa Q(S j+1 ,θ) replaces the previous max a Q(S t+1 ,a,θ) update y t Unlike the DDQN algorithm, this algorithm alternately uses the evaluation network Q and the target network Q′ to update y t The Loss(θ) loss function is used to fit the Q value. This algorithm not only further reduces the correlation between the network output Q value and the target Q value, avoiding the evaluation network from falling into the local optimum, but also weakens the "overestimation" problem of the Q value during the iterative update process of the target Q value, making the evaluation network converge faster, thereby increasing the stability of the entire improved DQN load balancing algorithm.
[0186] 3. Algorithm Process
[0187] (1) Initialization. Initialize the network structure parameters θ and θ′ of the evaluation network Q and the target network Q′, and set θ = θ′; set the experience replay memory and training sample capacity.
[0188] (2) Decision control. The algorithm agent interacts with the environment and obtains the latest state S from the environment. t Use the ε-greedy greedy strategy to obtain actions. The specific strategy is to randomly select action a with a probability of ε or to select the action argmax corresponding to the maximum Q value in the current St state calculated by the evaluation network with a probability of 1-ε a Q(S t ,θ). In the early stage of agent training, due to the lack of data samples, the agent begins to explore the environment and randomly selects actions with a high probability; as the number of training times increases, the agent will reduce the probability of exploration to a certain extent and select the optimal action to execute through the neural network.
[0189] (3) Experience replay memory. After the agent selects an action, the environment executes the corresponding action a and feeds back the current state S t The reward for performing the action r t and the next state S t+1 , and at the same time form a quaternion of four elements (S t , A t , R t , S t+1 ) is stored in the experience replay memory.
[0190] Evaluation network Q training. Randomly extract batch group quadruplets from the experience replay memory according to the random extraction strategy, obtain the Q value and calculate the loss function Loss(θ) through formula (4.6), and update the network parameters θ of the evaluation network Q through the gradient descent algorithm according to the loss function.
[0191] (4) Update of target network Q′. After the evaluation network Q undergoes C steps of iteration, the parameters of the target network Q′ are directly and synchronously updated, expressed as θ′=θ.
[0192] refer to Figure 3 , step S3 is specifically as follows:
[0193] S31: Initialize the evaluation network Q and the target network Q′;
[0194] S32: Update the parameters of the agent at time t and obtain the state space S at time t t , action space A t and the reward function R t ;
[0195] S33: S t , A t and R t Input the evaluation network Q and calculate the state space S at time t+1 t+1 ;
[0196] S34: S t+1 , A t and R t Input the evaluation network Q and the target network Q′ to update the parameter y t , generate loss Loss;
[0197] Specifically, parameter y t The expression is:
[0198]
[0199] Among them, r t Represents the reward function Rt at time t in the virtual resource VM j The reward value in reward tj , γ is the reward decay coefficient.
[0200] S35: If the Loss is less than the preset value, the optimized agent is output; otherwise, t=t+1 is set and the process returns to step S32.
[0201] S4: Load balancing scheduling of SaaS software is performed through optimized intelligent agents.
[0202] Specifically, first determine the necessary parameters of the neural network, including: learning rate α, learning frequency f, start learning time τ, evaluation network parameter θ, evaluation network Q function; target network parameter θ′=θ, initialize the Q′ function in the evaluation network; and initialize the empirical regression unit parameters, including: playback memory Mini-batch sampling Δ, replay time ω; Finally, we initialize the number of iteration rounds N of the agent, the greedy strategy ∈ and the reward decay coefficient γ. Then we perform the following operations in each round:
[0203] refer to Figure 4 , record the number of iteration steps in the round, the SaaS data center obtains the service request submitted by the user and observes the current environment S t , and use the ε-greedy strategy to obtain action A t , mainly randomly selecting action A with probability ε t Or choose the current S calculated by the evaluation network with a probability of 1-ε t Action A corresponding to the maximum Q value in the state t ; Through action A t Interact with the environment to get the corresponding reward R t And the next environment S t+1 ; Determine whether the number of iteration steps is 1, if not, the quaternion (S t , A t , R t , S t+1 ) is stored in the replay memory. At this point, it is necessary to determine whether the agent has reached the time τ required for learning. If learning is required, it is necessary to continue to determine whether the parameters of the evaluation network Q need to be copied to the target network Q', that is, θ' = θ. Then randomly extract a small batch sample Δ size quadruple (S t , A t , R t , S t+1 ). And use the ∆-sized quadruple to alternately use the Q value to calculate y t =r t +γQ(S t+1 ,argmax a Q′(S t+1 ,θ′),θ) or formula y t =r t +γQ(S t+1 ,argmax a Q′(S t+1 ,θ′),θ) calculate y t Fit the evaluation network Q value and use the loss function loss = (y t -Q(S t ,a,θ)) 2 Calculate the gradient and use back propagation to update the parameters θ of the current evaluation network. One round is over. The entire algorithm process ends after N rounds of loop iteration.
[0204] The key points in the present invention are:
[0205] 1. Based on the improved deep double Q network (RDDQN-SS) algorithm: This method combines the advantages of double Q-learning and deep Q network, overcomes the problems of high Q value and local optimality, and significantly improves the stability and efficiency of the load balancing algorithm.
[0206] 2. Experience replay and dual network mechanism: Use the experience replay pool and dual network alternating update strategy to further reduce the deviation in Q value calculation and enhance the learning and generalization capabilities of the algorithm.
[0207] 3. State space design: including the instruction length of the service request, the SLA agreement time and the estimated waiting time of each virtual resource, so that the intelligent agent can fully perceive the state of the current environment.
[0208] 4. Action space design: Define the numbered set of each virtual resource as the action space, and the agent allocates service requests to the corresponding virtual resources by selecting actions.
[0209] 5. Reward function design: The reward function not only considers the successful execution of the service request, but also focuses on the utilization balance of virtual resources. By setting the utilization range, the load balancing effect is guaranteed.
[0210] 6. Dual network alternating update mechanism: The improved DDQN algorithm uses dual networks to alternately update the state value function, reducing the risk of local optimality and improving the algorithm convergence speed.
[0211] 7.ε-greedy strategy: The agent initially randomly selects actions for exploration with high probability. As training progresses, the exploration probability is gradually reduced, and the learned optimal strategy is used more.
[0212] Effect after implementation
[0213] Taking the average utilization of SaaS software virtual resources as the optimization target of SaaS software load balancing, this experiment uses the above six algorithms to conduct relevant experimental comparisons based on four service request density scenarios. Table 1 lists the average utilization data of SaaS software virtual resources of the six scheduling algorithms in four service request density scenarios.
[0214] Table 1 Average utilization of SaaS software virtual resources under six scheduling algorithms in four service request density scenarios
[0215]
[0216] Attached Figure 5The average utilization of SaaS software virtual resources of six scheduling algorithms under four service request density scenarios are compared. Overall, as the number of service requests per unit time increases, the average utilization of virtual resources of the six load balancing scheduling algorithms also increases, and the average utilization of virtual resources of the three traditional algorithms is significantly higher than that of the three deep Q network algorithms. The reason is that the three traditional algorithms always use fixed strategies to mechanically allocate loads to corresponding virtual resources, and cannot dynamically allocate virtual resources according to service requests. When using the deep Q network algorithm, while trying to maintain the load balance of cluster virtual resources, the user's SLA agreement and other related factors are considered. From the perspective of the three deep Q network algorithms, the average utilization of SaaS software virtual resources using the RDDQN-SS algorithm is lower than that of the DQN and DDQN algorithms when the data density is low, but as the data density increases, the average utilization of virtual resources using the RDDQN-SS algorithm is greater than that of the DDQN and DQN algorithms; this shows that under the premise of taking the average utilization of virtual resources as the optimization indicator, the overall load balancing degree of the SaaS cluster is best when the RDDQN-SS algorithm is used as the SaaS software service scheduling strategy as the service request density increases.
[0217] Attached Figure 6 It represents the average response time of the six algorithms in four data scenarios when the average utilization of SaaS software virtual resources is used as the optimization target of SaaS software load balancing. From the figure, it can be seen that the response time of the RDDQN-SS algorithm is basically the lowest in the four service request density scenarios, indicating that SaaS software has a strong response capability under various service request densities using this algorithm. Figure 6 and 4 In comparison, the overall response time when the average utilization of SaaS software virtual resources is used as the load balancing indicator is about 0.04s longer than the average response time when the average response time ratio is used as the load balancing indicator.
[0218] Table 2 shows the execution cost data of six algorithms in four data scenarios;
[0219] Table 2 Execution costs of six algorithms in four data scenarios
[0220]
[0221] Attached Figure 7 Indicates the execution cost of the six algorithms in four data scenarios. From the overall trend, the total execution cost of service requests using the three deep Q network algorithms under four different service request densities is lower than that of the other three traditional algorithms. From a local perspective, the total execution cost of the RDDQN-SS algorithm is higher than that of the DQN, DDQN and other algorithms, but due to Figure 6 and5 It can be seen that the experiment takes the average utilization of SaaS software virtual resources as the main optimization goal. When the RDDQN-SS algorithm is used, the average utilization of SaaS software virtual resources increases with the data density, and the load allocated to each virtual resource in the cluster is more balanced, but some execution costs are sacrificed accordingly.
[0222] Attached Figure 8 It represents the success rate of service requests executed by the six algorithms in four different service request density scenarios based on the average utilization of SaaS software virtual resources as the load balancing indicator. From an overall perspective, the success rate of service requests executed by the three deep Q network algorithms in four different data densities is higher than that of the other three traditional algorithms. From a local perspective, the success rate of service requests using the Earliest algorithm does not change much with the increase of data density, and even increases slightly. The success rate of service requests executed by the three deep Q network algorithms gradually decreases with the increase of service request density, and the success rate of service requests using the RDDQN-SS algorithm remains stable and decreases slightly when the service request data density increases. The success rates in the four data density scenarios are basically higher than those of algorithms such as DDQN and DQN.
[0223] At the same time, further analysis of the results is considered from the following aspects:
[0224] (1) Validity:
[0225] Virtual resource utilization: Experimental data show that under different service request densities, the RDDQN-SS algorithm can more effectively balance the utilization of virtual resources than other algorithms (such as Random, RR, Earliest, DQN, DDQN). Especially in high data density scenarios, the RDDQN-SS algorithm shows better load balancing performance.
[0226] Service request execution success rate: Under different data densities, the RDDQN-SS algorithm shows a higher service request execution success rate than other algorithms (especially traditional algorithms). This shows that the algorithm can manage resources more effectively to meet service requests, especially in high data density environments.
[0227] (2) Efficiency:
[0228] Average response time: The data shows that the RDDQN-SS algorithm generally has the lowest response time in various service request density scenarios, which indicates that it is highly efficient in quickly responding to service requests.
[0229] Execution Cost: Although RDDQN-SS has slightly higher execution cost than DQN and DDQN algorithms, considering its higher service request success rate and balanced resource utilization, these additional costs can be regarded as necessary investment to maintain high performance and stability.
[0230] (3) Adaptability:
[0231] Data density adaptability: In different service request density scenarios, the RDDQN-SS algorithm shows good adaptability. As the service request density increases, it can adjust the resource allocation strategy to maintain high utilization and low response time of virtual resources, showing its adaptability to different load conditions.
[0232] Dynamic environment adaptability: The RDDQN-SS algorithm can dynamically adjust resource allocation strategies according to real-time changes in service requests. This flexibility makes it suitable for dynamically changing SaaS environments.
[0233] The results show that the method proposed in the present invention can achieve better SaaS software running resource load balancing effect.
[0234] See also Fig. 9 , Fig. 9 It is a working diagram of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a SaaS software load balancing device 401 based on an improved deep dual-Q network, a processor 402 and a storage medium 403.
[0235] A SaaS software load balancing device 401 based on an improved deep dual-Q network: The SaaS software load balancing device 401 based on an improved deep dual-Q network implements the SaaS software load balancing method based on an improved deep dual-Q network.
[0236] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the SaaS software load balancing method based on the improved deep dual-Q network.
[0237] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the SaaS software load balancing method based on the improved deep dual-Q network.
[0238] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0239] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.
[0240] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A SaaS software load balancing method based on an improved deep dual-Q network, characterized in that: Includes steps: S1: Build an intelligent agent, which includes: load balancing model, SaaS software load balancing environment model and load balancing optimization problem model; S2: Set the agent’s state space, action space, and reward function; S3: Iteratively optimize the state space, action space and reward function by improving the dual-depth Q network to obtain the optimized intelligent agent; S4: Load balancing scheduling of SaaS software is performed through optimized intelligent agents.
2. The SaaS software load balancing method based on the improved deep dual-Q network according to claim 1 is characterized in that: Step S1 is specifically as follows: S11: Define the service request service of SaaS software i and virtual resources VM j , set the load balancing degree of SaaS software to build a load balancing model, i is the number of the service request, j is the number of the virtual resource; S12: Define service request service based on load balancing model i In virtual resources VM j Execution time on Waiting time and departure time Depend on and Add up to get the total time S13: Pass Set the success rate ij , by execution time and departure time Calculate resource utilization Utilizi_VM ij , by execution time and virtual resources VM j Calculate the processing cost ij ; S14: Based on success ij Utilizi_VM ij and Cost ij Build a SaaS software load balancing environment model; S15: Set the average instruction response time ratio avgVT based on the load balancing model and the SaaS software load balancing environment model IJ , overall average resource utilization avgU t , virtual resource utilization standard deviation std_VM and the final optimization target, and build a load balancing optimization problem model; S16: The intelligent agent is composed of a load balancing model, a SaaS software load balancing environment model and a load balancing optimization problem model.
3. The SaaS software load balancing method based on the improved deep dual-Q network according to claim 2 is characterized in that: Resource utilization Utilizi_VM ij The expression is: in, Indicates that it includes service request service i The k loads included are virtualized by the VM resource j implement; Processing cost ij The expression is: Among them, VSC j Indicates virtual resources VM j Virtual resource startup cost, VOC j Indicates virtual resources VM j The unit time running cost of virtual resources.
4. The SaaS software load balancing method based on the improved deep dual-Q network according to claim 2 is characterized in that: Step S15 is specifically as follows: S151: Calculate and obtain service request service i Based on virtual resources VM j The command response time is shorter than VT ij , whose expression is: When the virtual resource VM j There are I service requests being executed, then based on VM j The average instruction response time is greater than avgVT j Defined as: The SaaS service cluster has a total of J virtual resources, so the average instruction response time of the entire service cluster is avgVT IJ Defined as: S152: Virtual resource VM of the SaaS service cluster at time t j Completion time VMF tj Defined as: in, Indicates that I service request is in the virtual resource VM j The latest time of completion; Virtual resource VM in the SaaS service cluster at time t j Resource utilization VMU for executing multiple service requests tj Defined as: in, Indicates that I service request is in the virtual resource VM j The latest departure time, For I service requests in virtual resource VM j The total execution time on Indicates that the task is in the virtual resource VM j The start time of the execution; The overall average resource utilization rate avgU of the virtual resources of the SaaS service cluster at time t t Defined as: S153: The standard deviation of virtual resource utilization std_VM is defined as: S154: The final optimization goal is defined as: Among them, π represents the load balancing algorithm; The constraints of the final optimization objective are:
5. The SaaS software load balancing method based on the improved deep dual-Q network according to claim 2 is characterized in that: State space S at time t t Current state ij It is expressed as: Among them, ServiceSize i Indicates service request service i Load size, DeadTime i Indicates service request service i The maximum time to complete; Action space A at time t t Service request service i and virtual resources VM j action ij It is expressed as: Reward function R at time t t In virtual resources VM j The reward value in reward tj It is expressed as: Among them, μ is the adjustment parameter, VMU tj is the virtual resource VM in the SaaS service cluster at time t j Resource utilization for executing multiple service requests.
6. The SaaS software load balancing method based on the improved deep dual-Q network according to claim 1 is characterized in that: Step S3 is specifically as follows: S31: Initialize the evaluation network Q and the target network Q′; S32: Update the parameters of the agent at time t and obtain the state space S at time t t , action space A t and the reward function R t ; S33: S t , A t and R t Input the evaluation network Q and calculate the state space S at time t+1 t+1 ; S34: S t+1 , A t and R t Input the evaluation network Q and the target network Q′ to update the parameter y t , generate loss Loss; S35: If the Loss is less than the preset value, the optimized agent is output; otherwise, t=t+1 is set and the process returns to step S32.
7. The SaaS software load balancing method based on the improved deep dual-Q network according to claim 6 is characterized in that: Parameter y t The expression is: Among them, r t Represents the reward function Rt at time t in the virtual resource VM j The reward value in reward tj , γ is the reward decay coefficient.
8. A SaaS software load balancing device based on an improved deep dual-Q network, characterized in that: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement the SaaS software load balancing method based on the improved deep dual-Q network as described in any one of claims 1 to 7.
Citation Information
Cited By
Heterogeneous resource scheduling cloud optimization method and system based on space-time task intensity evaluation
CN120104281A
Cloud optimization method and system for heterogeneous resource scheduling based on spatiotemporal task intensity evaluation
CN120104281B
