SaaS software service request load balancing method, medium and equipment
By building a deep reinforcement learning environment and using the RDDQN algorithm, the uneven resource distribution and dynamic changes in the load balancing scenario of SaaS software are solved, and more efficient load balancing and success rate are achieved, local optimal traps are avoided, and load is increased Performance of the equalizer.
Patent Information
- Application Number
- CN202411125843.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-08
AI Technical Summary
The existing technology cannot effectively solve the service request problem of uneven resource distribution and dynamic changes in the load balancing scenario of SaaS software. Traditional algorithms cannot detect server status. The application of heuristic algorithms and deep reinforcement learning algorithms in this scenario has not been fully studied. And it is easy to fall into local optimality.
Build a deep reinforcement learning environment for service request load balancing of SaaS software, use the RDDQN algorithm to calculate the average instruction response time ratio and service request success rate, define the state space, action space and reward functions, introduce dual network structure and experience playback mechanism to avoid Q value overestimation and local optimality.
More effective load balancing of SaaS software service requests is achieved, the allocation efficiency and success rate of load balancers are improved, the agent is avoided from falling into local optimality, and the convergence speed of algorithm training is accelerated.
Smart Images

Figure CN118916177A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of software service technology, and in particular to a SaaS software service request load balancing method, medium, and device. Background Art
[0002] The intelligent service decision-making and scheduling of SaaS (Software as a Service) software providers to reasonably allocate multi-tenant requests to existing virtual resources and quickly achieve load balancing, as well as the evaluation indicators to measure the load balancing state of SaaS software server clusters, are key issues in the current SaaS software research. At present, SaaS software load balancing related technologies can be roughly divided into the following three categories:
[0003] (1) Traditional algorithms. Commonly used traditional load balancing strategies mainly include min-min, max-min, round-robin, shortest job first, two-stage and other scheduling algorithms. Traditional cloud service load balancing strategies are generally classified as static load balancing technologies, which usually cannot detect the connected servers and do not consider the current status of the system, resulting in uneven resource distribution and unable to solve the problem of dynamically changing service requests of SaaS software.
[0004] (2) Heuristic algorithms. Heuristic load balancing algorithms mainly refer to some load balancing technologies inspired by natural phenomena, such as genetic algorithms, artificial bee colony algorithms, ant colony algorithms, and particle swarm algorithms. However, there has not been much research on whether these algorithms and the related evaluation indicators of cloud service load balancing they use are suitable for SaaS software load balancing scenarios.
[0005] (3) Deep reinforcement learning algorithm. Deep reinforcement learning algorithm mainly includes reinforcement learning algorithm and deep reinforcement learning algorithm. Reinforcement learning algorithm mainly refers to reinforcement learning algorithm based on value function, based on strategy, and based on value function and strategy mixture. First of all, whether deep reinforcement learning strategy is suitable for SaaS software load balancing scenario and whether the effect is better than traditional scheduling algorithm in this scenario, and which deep reinforcement learning algorithm can better meet the user's deep reinforcement learning agreement in the process of reasonable distribution of SaaS software running load, have not been studied much at this stage. Secondly, in the process of using DQN algorithm for cloud service scheduling, the intelligent agent often has the problem of "overestimation" when updating Q value; although the use of DDQN algorithm cleverly solves the above problems, the algorithm is easy to fall into local optimum, and once it falls into local optimum, it requires a lot of data to correct, and even leads to the emergence of false optimal action value function. Summary of the invention
[0006] The purpose of the present invention is to solve the problem that the prior art cannot perform SaaS software load balancing scenarios well, and propose a SaaS software service request load balancing method, comprising the following steps:
[0007] S1. Calculate the average instruction response time ratio of the SaaS service cluster and the success rate of SaaS software execution service requests;
[0008] S2. Obtaining the optimization target of SaaS software service request load balancing based on the average instruction response time ratio of service requests and the success rate of SaaS software execution service requests;
[0009] S3. Build a deep reinforcement learning environment for load balancing of SaaS software service requests, and build the state space, action space, and reward function of the deep reinforcement learning environment;
[0010] S4. Based on the optimization goal, use the RDDQN algorithm to load balance the SaaS software service requests in the deep reinforcement learning environment.
[0011] Furthermore, the average instruction response time ratio of the SaaS service cluster is calculated as follows:
[0012] Compute the instruction response time ratio of service requests based on virtual resources:
[0013]
[0014] Among them, VT ij represents the instruction response time ratio of the i-th service request based on the j-th virtual resource, serviceSize i Indicates the data size of the i-th service request; represents the duration for the i-th service request to be responded by the j-th virtual resource;
[0015] Calculate the average instruction response time ratio of virtual resources:
[0016]
[0017] Among them, avgVT j represents the average instruction response time ratio of the jth virtual resource, and I represents the number of service requests of the jth virtual resource;
[0018] Calculate the average instruction response time ratio of the SaaS service cluster:
[0019]
[0020] Among them, avgVT represents the average instruction response time ratio of the SaaS service cluster, and J represents the number of virtual resources.
[0021] Furthermore, the success rate of SaaS software executing service requests is expressed as:
[0022]
[0023] Among them, success ij represents the success rate of the jth virtual resource of the SaaS software executing the ith service request, DeadTime represents the duration of the i-th service request being responded to by the j-th virtual resource. i represents the longest duration of the i-th service request in the SaaS software.
[0024] Furthermore, the optimization goal is:
[0025]
[0026] The constraints of the optimization objective are:
[0027]
[0028] Where π represents the load balancing algorithm, α and β represent weight coefficients, J represents the number of virtual resources, I represents the number of service requests for the jth virtual resource, and success ii It represents the success rate of the jth virtual resource of the SaaS software executing the i-th service request.
[0029] Furthermore, the state space is defined as:
[0030]
[0031] Among them, state ij represents the state of the jth virtual resource based on the i-th service request in the deep reinforcement learning environment, serviceSize i Indicates the data size of the i-th service request, DeadTime i represents the longest duration of the i-th service request in the SaaS software, They respectively represent the waiting time for the i-th service request to select the 1st, 2nd, ..., jth virtual resource.
[0032] Furthermore, the action space is defined as:
[0033]
[0034] Among them, action ij represents the action of the i-th service request in the deep reinforcement learning environment based on the j-th virtual resource, VM 1 ,VM 2 ,…,VM j,…,VM J They represent the 1st, 2nd, ..., j, ..., Jth virtual resources respectively.
[0035] Furthermore, the reward function is defined as:
[0036]
[0037] Among them, reward ij represents the reward function of the i-th service request in the deep reinforcement learning environment based on the j-th virtual resource, α represents the hyperparameter, avgVT represents the average instruction response time ratio of the SaaS service cluster, success ij It represents the success rate of the jth virtual resource of the SaaS software executing the i-th service request.
[0038] Furthermore, S4 is specifically:
[0039] S41, initializing the parameters of the RDDQN algorithm and the deep reinforcement learning environment;
[0040] S42, obtain the number of iteration steps in the agent iteration round, the current state S t , use the ε-greedy greedy strategy to obtain the current action A t ;
[0041] S43, through action A t Interact with the environment to get the corresponding reward R t and the next state S t+1 ; Determine whether the iteration step is 1, if the iteration step is 1, return to step S41, if the iteration step is not 1, the quadruple (S t ,A t ,R t ,S t+1 ) is stored in the playback memory;
[0042] S44, train and update the evaluation network and target network, determine whether the agent needs to learn, and if so, continue to determine whether the parameters θ of the evaluation network Q need to be copied to the parameters θ of the target network Q' ′ If parameter replication is required, randomly extract a small batch of quadruplets of size Δ in the replay memory (S t ,A t ,R t ,S t+1 ), and use the Δ-sized quadruple to alternately use the Q value to calculate y t To fit the Q value of the evaluation network, and calculate the gradient through the loss function, use the back propagation method to update the parameters θ of the current evaluation network. If no parameter copying or learning is required, return to step S43, and one round ends;
[0043] Calculate y t The expression is:
[0044]
[0045] Among them, y t represents the target Q value; r t represents the immediate reward obtained at time step t; γ represents the discount factor; Q(S t+1 ,argmax a Q ′ (S t+1 ,θ′),θ) represents the evaluation network Q in state S t+1 , the action is argmax a Q ′ (S t+1 ,θ′), the output when the parameter is θ; Q ′ (S t+1 ,θ′) indicates that the target network Q' is in state S t+1 , the output when the parameter is θ′;
[0046] The gradient is calculated by the loss function, where the loss function expression is:
[0047] loss=(y t -Q(S t ,a,θ)) 2
[0048] Among them, loss represents the loss function, Q(S t ,a,θ) indicates that the evaluation network Q is in state S t , the output when the action is a and the parameter is θ.
[0049] S45, the whole process ends after N rounds of loop iteration.
[0050] The present invention also proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned SaaS software service request load balancing method is implemented.
[0051] The present invention also proposes an electronic device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the above-mentioned SaaS software service request load balancing method.
[0052] The beneficial effects brought by the technical solution provided by the present invention are:
[0053] The present invention constructs a deep reinforcement learning environment for load balancing of SaaS software service requests, constructs the state space, action space, and reward function of the environment; based on the optimization goal of load balancing, the RDDQN algorithm is used to load balance SaaS software service requests in the deep reinforcement learning environment. RDDQN introduces a dual network structure, alternately uses the evaluation network Q and the target network Q' to calculate the next actual Q value, and weakens the correlation of the iterative update of the Q value, so that the change trend of the evaluation network Q and the target network Q' is asynchronous, avoiding the "overestimation" problem of the Q value in the process of interaction between the intelligent agent and the environment, and weakening the trend of the intelligent agent falling into the local optimum in a certain step. RDDQN introduces an experience replay mechanism. During the learning process of the intelligent agent, the data obtained by interacting with the environment is first stored in the replay memory unit. During offline training, the data is selected from the replay memory unit using a random sampling method, which effectively reduces the correlation between the data and accelerates the convergence speed during algorithm training. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 is a flow chart of a method for load balancing SaaS software service requests according to an embodiment of the present invention;
[0055] Figure 2 It is a training flow chart of the RDDQN algorithm according to an embodiment of the present invention;
[0056] Figure 3 is a block diagram of an electronic device in an exemplary embodiment of the present invention;
[0057] Figure 4 The execution cost of the cluster using the deep Q network algorithm in the embodiment of the present invention compared with the three traditional algorithms of Random (random algorithm), RR (Round Robin) and Earliest (earliest deadline first);
[0058] Figure 5 It is the service request execution success rate of the cluster using the deep Q network algorithm in the embodiment of the present invention compared with the three traditional algorithms of Random, RR, and Earliest;
[0059] Figure 6 is the average instruction response time ratio of the cluster using the deep Q network algorithm in the embodiment of the present invention compared with the three traditional algorithms of Random, RR and Earliest;
[0060] Figure 7 It is the average response time of the cluster using the deep Q network algorithm in the embodiment of the present invention compared with the three traditional algorithms of Random, RR, and Earliest. DETAILED DESCRIPTION
[0061] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0062] The flowchart of the SaaS software service request load balancing method according to the embodiment of the present invention is as follows: Figure 1 , specifically including the following steps:
[0063] S1. Calculate the average instruction response time ratio of the SaaS service cluster and the success rate of SaaS software execution service requests.
[0064] First, calculate the instruction response time ratio of the service request based on virtual resources:
[0065]
[0066] Among them, VT ij represents the instruction response time ratio of the i-th service request based on the j-th virtual resource, serviceSize i Indicates the data size of the i-th service request; It indicates the duration for the i-th service request to be responded by the j-th virtual resource.
[0067] Assume that the virtual resource VM j There are I service requests being executed, and the average instruction response time ratio of the virtual resources is calculated based on the instruction response time ratio of the service requests based on the virtual resources:
[0068]
[0069] Among them, avgVT j represents the average instruction response time ratio of the jth virtual resource, and I represents the number of service requests of the jth virtual resource.
[0070] Finally, the average instruction response time ratio of the SaaS service cluster is obtained:
[0071]
[0072] Among them, avgVT represents the average instruction response time ratio of the SaaS service cluster, and J represents the number of virtual resources.
[0073] If the average command response time ratio is high, it means that the overall response time of the service cluster is very small and the virtual resources are efficient, which to a certain extent represents the rationality of the load distribution in the SaaS data center. The lower the service request average command response time ratio, the more unbalanced the load allocated to each virtual resource by the load balancer.
[0074] For each service request, the user signs an SLA agreement with the SaaS software. In this model, it is assumed that the SLA agreement for each service request represents the service request service i Try to maximize the DeadTime in SaaS software i If It means service request service i The load balancing strategy is successfully executed, otherwise it means that the SaaS data center schedules the service request service based on the load balancing strategy i Failed. Therefore, each service i The corresponding user SLA can be defined as:
[0075]
[0076] Among them, success ij represents the success rate of the jth virtual resource of the SaaS software executing the ith service request, DeadTime represents the duration of the i-th service request being responded to by the j-th virtual resource. i represents the longest duration of the i-th service request in the SaaS software.
[0077] S2. According to the average instruction response time ratio of service requests and the success rate of SaaS software execution service requests, the optimization target of SaaS software service request load balancing is obtained.
[0078] Taking the average response time ratio of cluster service requests and the success rate of SaaS software execution service requests as the overall optimization goals of the SaaS software load balancing scenario, the final optimization goal is:
[0079]
[0080] The constraints of the optimization objective are:
[0081]
[0082] Where π represents the load balancing algorithm, α and β represent weight coefficients, J represents the number of virtual resources, I represents the number of service requests for the jth virtual resource, and success ii It represents the success rate of the jth virtual resource of the SaaS software executing the i-th service request.
[0083] If the average instruction response time ratio of the SaaS service cluster is high, it means that the overall response time of the service cluster is very small and the virtual resource execution efficiency is fast. The higher the success rate of SaaS software execution service requests, the more efficient the virtual resource execution is.
[0084] S3. Build a deep reinforcement learning environment for load balancing of SaaS software service requests, and build the state space, action space, and reward function of the deep reinforcement learning environment.
[0085] The state space is defined as:
[0086]
[0087] Among them, state ij represents the state of the jth virtual resource based on the i-th service request in the deep reinforcement learning environment, serviceSize i Indicates the data size of the i-th service request, DeadTime i represents the longest duration of the i-th service request in the SaaS software, They respectively represent the waiting time for the i-th service request to select the 1st, 2nd, ..., jth virtual resource.
[0088] The action space is defined as:
[0089]
[0090] Among them, action ij represents the action of the i-th service request in the deep reinforcement learning environment based on the j-th virtual resource, VM 1 ,VM 2 ,…,VM j ,…,VM J They represent the 1st, 2nd, ..., j, ..., Jth virtual resources respectively.
[0091] The reward function is defined as:
[0092]
[0093] Among them, reward ij represents the reward function of the i-th service request in the deep reinforcement learning environment based on the j-th virtual resource. α represents a hyperparameter, which is mainly used to adjust the reward. ij To a suitable size range, which is convenient for deep neural network calculation. avgVT represents the average instruction response time ratio of SaaS service cluster. success ij It represents the success rate of the jth virtual resource of the SaaS software executing the i-th service request.
[0094] S4. Based on the optimization goal, use the RDDQN (Replaced Double Deep Q Network) algorithm to load balance SaaS software service requests in a deep reinforcement learning environment.
[0095] See also Figure 2 , the training process of the RDDQN algorithm is as follows:
[0096] 1. Initialization. Initialize the evaluation network Q and the target network Q ′ The network structure parameters θ, θ′ are set, and θ=θ′ is set; the experience replay memory and the training sample capacity are set.
[0097] 2. Decision control. The RDDQN algorithm agent interacts with the environment and obtains the latest state S from the environment. t Use the ε-greedy greedy strategy to obtain actions. The specific strategy is to randomly select action a with a probability of ε or to select the action argmax corresponding to the maximum Q value in the current St state calculated by the evaluation network with a probability of 1-ε a Q(S t ,θ).
[0098] 3. Experience replay memory. When the agent selects an action, the environment executes the corresponding action a and feeds back the current state S t The reward for performing the action r t and the next state S t+1 , and at the same time, the four elements are combined into a four-tuple (St, At, Rt, St+1) and stored in the experience replay memory.
[0099] 4. Evaluate the network Q training. According to the random extraction strategy, randomly extract the batch group quadruple from the experience replay memory, and use formula a * = argmax a Q(S t+1 ,a,θ) to obtain the Q value and calculate the loss function, and update the network parameter θ of the evaluation network Q through the gradient descent algorithm as follows:
[0100]
[0101] 5. Target network Q ′ Update. After the evaluation network Q has been iterated for C steps, the target network Q is directly updated synchronously ′ The parameter, denoted as θ ′ =θ.
[0102] Step S4 is specifically as follows:
[0103] S41, initialization. Initialize the parameters of the RDDQN algorithm and the deep reinforcement learning environment, including: learning rate α, learning frequency f, start learning time τ, evaluation network parameter θ, evaluation network Q function; target network parameter θ ′ =θ, initialize Q in the evaluation network ′Function; Initialize the empirical regression unit parameters at the same time, including: playback memory Mini-batch sampling Δ, replay time ω; finally, we initialize the number of iteration rounds N of the agent, the greedy strategy ∈, and the reward decay coefficient γ.
[0104] S42, decision control. Obtain the number of iteration steps in the agent iteration round, the SaaS data center obtains the service request submitted by the user and observes the current state S t , use the ε-greedy greedy strategy to obtain the current action A t , mainly randomly selecting action At with probability ε or selecting action At corresponding to the maximum Q value in the current St state calculated by the evaluation network with probability 1-ε.
[0105] S43, experience playback memory. Through action A t Interact with the environment to get the corresponding reward R t and the next state S t+1 ; Determine whether the iteration step is 1, if the iteration step is 1, return to step S41, if the iteration step is not 1, the quadruple (S t ,A t ,R t ,S t+1 ) into the playback memory.
[0106] S44, train and update the evaluation network and target network, determine whether the agent has reached the time τ required for learning, and if learning is required, continue to determine whether the parameters θ of the evaluation network Q need to be copied to the parameters θ of the target network Q' ′ If parameter replication is required, randomly extract a small batch of quadruplets of size Δ in the replay memory (S t ,A t ,R t ,S t+1 ), and use the ∆-sized quadruple to alternately use the Q value to calculate y t To fit the evaluation network Q value, and calculate the gradient through the loss function, use the back propagation method to update the parameters θ of the current evaluation network. If no parameter copying or learning is required, return to step S43 and one round ends.
[0107] Calculate y t The expression is:
[0108]
[0109] or t =r t +γQ ′ (S t+1 ,argmax a Q(St+1 ,θ),θ′)
[0110] Among them, y t represents the target Q value; r t represents the immediate reward obtained at time step t; γ represents the discount factor; Q(S t+1 ,argmax a Q ′ (S t+1 ,θ′),θ) represents the evaluation network Q in state S t+1 , the action is argmax a Q ′ (S t+1 ,θ′), the output when the parameter is θ; Q ′ (S t+1 ,θ′) indicates that the target network Q' is in state S t+1 , the output when the parameter is θ′.
[0111] The gradient is calculated by the loss function, where the loss function expression is:
[0112] loss=(y t -Q(S t ,a,θ)) 2
[0113] Among them, loss represents the loss function, Q(S t ,a,θ) indicates that the evaluation network Q is in state S t , the output when the action is a and the parameter is θ.
[0114] S45, the whole process ends after N rounds of loop iteration.
[0115] During the training process, the network gradually learned how to distribute requests under different load conditions to minimize response time and maximize success rate, achieving the optimization goal of SaaS software load balancing.
[0116] In an exemplary embodiment, a computer-readable storage medium is included, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned SaaS software service request load balancing method is implemented.
[0117] See also Figure 3 In an exemplary embodiment, an electronic device is also included, including at least one processor, at least one memory, and at least one communication bus.
[0118] Among them, a computer program is stored in the memory, and the computer program includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through the communication bus to execute the above-mentioned SaaS software service request load balancing method.
[0119] The present invention is based on the CloudSim Plus cloud computing simulation platform as the SaaS software testing environment, and uses TensorFlow2.2 as a related component for building a deep neural network. The service request parameters and cluster virtual resource data included are set as follows:
[0120] Table 1 Experimental parameter settings
[0121]
[0122]
[0123] Comparison algorithms: Random scheduling algorithm (Random), Earliest scheduling algorithm (Earliest), Round Robin scheduling algorithm (RR), DQN scheduling algorithm, DDQN scheduling algorithm, RDDQN-SS scheduling algorithm.
[0124] Evaluation indicators: cluster average instruction response time ratio, average response time, total execution cost, and service request execution success rate.
[0125] Results Analysis
[0126] (1) Validity:
[0127] See also Figure 4 , Figure 5 , the horizontal axis is divided into four groups of data, 10, 20, 30, and 40, according to the number of average arrival rates of service requests per unit time, and different algorithms are compared. The algorithms of the data bar graphs from left to right in each of the four groups of data are: Random (random algorithm), RR (Round Robin), Earliest (earliest deadline priority), DQN (Deep Q-Network, deep reinforcement learning), DDQN (Replaced Double Deep Q Network, improved deep double Q network), RDDQN-SS (method of the present invention). Compared with the three traditional algorithms of Random, RR, and Earliest, the cluster using the deep Q network algorithm has a lower total execution cost and a higher service request execution success rate. Compared with the use of DDQN and DQN algorithms, the use of the RDDQN-SS algorithm (method of the present invention) in the deep Q network algorithm has the highest or similar service request execution success rate in different scenarios, and the lowest or similar total execution cost. It can be seen that the use of the RDDQN-SS algorithm can improve the success rate of SaaS software service request execution.
[0128] (2) Efficiency:
[0129] See also Figure 6 , Figure 7 Compared with the three traditional algorithms Random, RR, and Earliest, the cluster using the deep Q network algorithm has a larger average instruction response time ratio and a smaller average response time. Compared with the DDQN and DQN algorithms, the RDDQN-SS algorithm in the deep Q network algorithm has a larger average instruction response time ratio and a smaller average response time. It can be seen that the RDDQN-SS algorithm can improve the overall execution efficiency of the SaaS software cluster.
[0130] (3) Adaptability:
[0131] See also Figure 4 , Figure 5 , Figure 7 When the average instruction response time ratio of the cluster is used as the load balancing optimization target, the average instruction response time ratio, total cost of service request execution, and service request success rate of the deep Q network algorithm are better than those of the three traditional algorithms of Random, RR, and Earliest in the four service request density scenarios. At the same time, the relevant indicators of the RDDQN-SS algorithm in the four service request density scenarios are better than those of the DDQN and DQN algorithms. The Earliest algorithm performs relatively stably under the four service request densities, which is better than the other two traditional algorithms. It is a better method in the static load balancing strategy, but there is still a certain gap compared with the three deep Q network algorithms in the SaaS software load balancing dynamic scheduling load scenario.
[0132] The results show that the method proposed in the present invention can achieve better SaaS software service request load balancing effect.
[0133] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for load balancing SaaS software service requests, characterized in that: The following steps are involved: S1. Calculate the average instruction response time ratio of the SaaS service cluster and the success rate of SaaS software execution service requests; S2. Obtaining the optimization target of SaaS software service request load balancing based on the average instruction response time ratio of service requests and the success rate of SaaS software execution service requests; S3, constructing a reinforcement learning network for load balancing of SaaS software service requests, and constructing a state space, action space, and reward function of the reinforcement learning network; S4. Based on the optimization goal, use the RDDQN algorithm to load balance the SaaS software service requests in the reinforcement learning network.
2. A SaaS software service request load balancing method according to claim 1, characterized in that: The specific calculation of the average instruction response time ratio of the SaaS service cluster is: Compute the instruction response time ratio of service requests based on virtual resources: Among them, VT ij represents the instruction response time ratio of the i-th service request based on the j-th virtual resource, serviceSize i Indicates the data size of the i-th service request; represents the duration for the i-th service request to be responded by the j-th virtual resource; Calculate the average instruction response time ratio of virtual resources: Among them, avgVT j represents the average instruction response time ratio of the jth virtual resource, and I represents the number of service requests of the jth virtual resource; Calculate the average instruction response time ratio of the SaaS service cluster: Among them, avgVT represents the average instruction response time ratio of the SaaS service cluster, and J represents the number of virtual resources.
3. A SaaS software service request load balancing method according to claim 1, characterized in that: The success rate of SaaS software execution service request is expressed as: Among them, success ij represents the success rate of the jth virtual resource of the SaaS software executing the ith service request, DeadTime represents the duration of the i-th service request being responded to by the j-th virtual resource. i represents the longest duration of the i-th service request in the SaaS software.
4. A SaaS software service request load balancing method according to claim 1, characterized in that: The optimization goal is: The constraints of the optimization objective are: Where π represents the load balancing algorithm, α and β represent weight coefficients, J represents the number of virtual resources, I represents the number of service requests for the jth virtual resource, and success ij It represents the success rate of the jth virtual resource of the SaaS software executing the i-th service request.
5. A SaaS software service request load balancing method according to claim 1, characterized in that: The state space is defined as: Among them, state ij represents the state of the jth virtual resource based on the i-th service request in the deep reinforcement learning environment, serviceSize i Indicates the data size of the i-th service request, DeadTime i represents the longest duration of the i-th service request in the SaaS software, They respectively represent the waiting time for the i-th service request to select the 1st, 2nd, ..., jth virtual resource.
6. A SaaS software service request load balancing method according to claim 1, characterized in that: The action space is defined as: Among them, action ij represents the action of the i-th service request in the deep reinforcement learning environment based on the j-th virtual resource, VM1, VM2, …, VM j ,…,VM J They represent the 1st, 2nd, ..., j, ..., Jth virtual resources respectively.
7. A SaaS software service request load balancing method according to claim 1, characterized in that: The reward function is defined as: Among them, reward ij represents the reward function of the i-th service request in the deep reinforcement learning environment based on the j-th virtual resource, α represents the hyperparameter, avgVT represents the average instruction response time ratio of the SaaS service cluster, success ij It represents the success rate of the jth virtual resource of the SaaS software executing the i-th service request.
8. A SaaS software service request load balancing method according to claim 1, characterized in that: S4 is specifically: S41, initializing the parameters of the RDDQN algorithm and the deep reinforcement learning environment; S42, obtain the number of iteration steps in the agent iteration round, the current state S t , use the ε-greedy greedy strategy to obtain the current action A t ; S43, through action A t Interact with the environment to get the corresponding reward R t and the next state S t+1 ; Determine whether the iteration step is 1, if the iteration step is 1, return to step S41, if the iteration step is not 1, the quadruple (S t ,A t ,R t ,S t+1 ) is stored in the playback memory; S44, train and update the evaluation network and target network, determine whether the agent needs to learn, and if so, continue to determine whether the parameters θ of the evaluation network Q need to be copied to the parameters θ of the target network Q' ′ If parameter replication is required, randomly extract a small batch of quadruplets of size Δ in the replay memory (S t ,A t ,R t ,S t+1 ), and use the Δ-sized quadruple to alternately use the Q value to calculate y t To fit the Q value of the evaluation network, and calculate the gradient through the loss function, use the back propagation method to update the parameters θ of the current evaluation network. If no parameter copying or learning is required, return to step S43, and one round ends; Calculate y t The expression is: Among them, y t represents the target Q value; r t represents the instant reward obtained at the time step; γ represents the discount factor; Q(S t+1 , argmax a Q′(S t+1 ,θ′),θ) represents the evaluation network Q in state S t+1 , the action is argmax a Q′(S t+1 ,θ′), the output when the parameter is θ; Q′(S t+1 , θ′) indicates that the target network Q' is in state S t+1 , the output when the parameter is θ′; The gradient is calculated by the loss function, where the loss function expression is: loss=(y t -Q(S t ,a,θ)) 2 Among them, loss represents the loss function, Q(S t , a,θ) represents the evaluation network Q in state S t , the output when the action is a and the parameter is θ; S45, the whole process ends after N rounds of loop iteration.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the method according to any one of claims 1 to 8.
