Resource optimization method based on reinforcement learning in cloud game multi-opening scene

Abstract

The application discloses a resource optimization method based on reinforcement learning in a cloud game multi-opening scene, comprising the following steps: collecting game starting data, game running data and cloud monitoring data, constructing a game feature sequence set based on all the data, and serializing the game running data to obtain a bandwidth sample sequence; constructing a bandwidth prediction model by using an LSTM and the bandwidth sample sequence, and obtaining bandwidth prediction data based on the bandwidth prediction model; constructing a reinforcement learning network based on a DQN algorithm, training the reinforcement learning network by using the game feature sequence set and the bandwidth sample sequence to obtain a game decision network model, wherein the reinforcement learning network comprises train_networks and target_networks with the same structure; collecting corresponding data online and inputting the data into the game decision network model, and the game ID with the maximum reward in the reward sequence output by the game decision network model is the current optimal starting game decision. The application can reduce the failure rate in the game multi-opening scene.

Description

Technical Field

[0001] This invention belongs to the field of cloud gaming technology, specifically relating to a resource optimization method based on reinforcement learning in multi-instance cloud gaming scenarios. It can be applied to common cloud gaming scenarios such as edge cloud and cloud-to-edge cloud, and can be reused in multiple scenarios such as central cloud, IDC, CDN, and edge routing. Background Technology

[0002] Cloud gaming refers to games running in the cloud, where rendered graphics are compressed before being transmitted to the user, allowing them to experience the full game content without needing high-end hardware. Compared to traditional gaming, cloud gaming offers the following advantages: First, it reduces costs for players, as they don't need high-end hardware to enjoy the full game. Second, by running in the cloud, it benefits from advanced graphics rendering and processing capabilities. Finally, it offers advantages such as no installation required, instant play, and a vast library of game combinations.

[0003] Cloud gaming, though small in scale within the overall gaming market, is a rapidly growing sector. As people increasingly seek to lower the barriers to entry for gaming, it will become increasingly popular. The cloud gaming model involves games running in the cloud, often with multiple games running simultaneously on a single cloud server. This scenario makes the rational and optimized utilization of cloud-based CPU and GPU computing resources particularly crucial.

[0004] Existing cloud gaming multi-instance scenarios often employ hybrid deployment or dedicated GPU resources, which has the following drawbacks: 1. Relatively significant waste of GPU and other algorithmic resources. Direct-connect hybrid deployments often result in GPU and other resource utilization rates below 30%, leading to relatively high cloud costs. 2. Prone to computational failures. This is because different games have different peak runtime periods and resource consumption. Improper management can easily cause congestion and ultimately, failures. 3. Inability to effectively address the elastic concurrency issues of a large number of cloud game players. Peak periods often see a surge in player computational demands, and conventional technologies often lack reasonable solutions for handling such peak loads. Furthermore, conventional game multi-instance technologies primarily rely on greedy algorithms or dynamic programming for real-time resource allocation. However, games often exhibit significant resource heterogeneity and business complexity, making it difficult to achieve reasonable game multi-instance through statistical or machine learning methods. Summary of the Invention

[0005] To address the above problems, this invention proposes a resource optimization method based on reinforcement learning for multi-instance cloud gaming scenarios. The technical solution adopted by this invention to solve the above technical problems is as follows:

[0006] A resource optimization method based on reinforcement learning for multi-instance cloud gaming scenarios includes the following steps:

[0007] S1 collects game startup data, game running data, and cloud monitoring data, constructs a game feature sequence set based on all data, and serializes the game running data to obtain a bandwidth sample sequence;

[0008] S2, a bandwidth prediction model is constructed using LSTM and bandwidth sample sequences, and bandwidth prediction data is obtained based on the bandwidth prediction model;

[0009] S3, a reinforcement learning network is constructed based on the DQN algorithm. The game decision network model is obtained by training the reinforcement learning network using the game feature sequence set and bandwidth sample sequence. The reinforcement learning network includes a train network and a target network with the same structure.

[0010] S4. Collect the corresponding data online and input it into the game decision network model. The game ID with the highest reward in the reward sequence output by the game decision network model is the current optimal decision to start the game.

[0011] In step S1, the game launch data includes the game ID, single-client game revenue, and game merchant ID; the cloud monitoring data includes CPU usage, GPU usage, and network bandwidth usage; the historical single-game running data includes the CPU usage, GPU usage, and network bandwidth usage corresponding to a single game ID; and the historical multi-client game running data includes the game IDs of each device ID that were newly started or stopped at different times.

[0012] Step S3 includes the following steps:

[0013] S3.1 Define the game pre-entry state sample sequence, game post-entry state sample sequence, and action sample sequence based on the game feature sequence set and bandwidth sample sequence, and simultaneously define the reward function to obtain the reward sample sequence corresponding to the game post-entry state sample sequence based on the reward function.

[0014] S3.2, Set the iteration rounds and parameter update cycle and initialize the training rounds. ;

[0015] S3.3, Randomly select from all the sample sequences obtained in step S3.1 The data points are labeled as the pre-game state sampling sequence, the post-game state sampling sequence, the action sampling sequence, and the reward sampling sequence, respectively.

[0016] S3.4 Input the pre-game state sampling sequence into the target network to obtain the first predicted reward sequence;

[0017] S3.5, randomly sample a random number from [0,1]. ,like Then, update each reward in the reward sampling sequence;

[0018] S3.6, input the updated reward sampling sequence, action sampling sequence and pre-game state sampling sequence into the train_network to train the game decision network model, and update the loss function of the model at the same time;

[0019] S3.7, Judgment If yes, output the game decision network model and execute step S4; otherwise, execute step S3.8.

[0020] S3.8, Judgment If so, copy the parameters from the train_network to the target_network and execute. And return S3.3; otherwise, execute. And return to S3.3.

[0021] In step S3.1, the expression for the reward function is:

[0022] ;

[0023] In the formula, This refers to the reward given when, within 24 hours of switching to the game ID, the device's peak CPU utilization, peak GPU utilization, or peak network bandwidth utilization exceeds 90%. This indicates the reward received when the device's CPU usage reaches 100% or the device's GPU usage reaches 100% within 24 hours of switching to the game ID. This is a reward given when the device's CPU, GPU, and network card are at 100% load and a hardware failure occurs. This refers to the reward given when the cloud bandwidth utilization rate drops to less than 40%. This refers to rewards in normal game scenarios.

[0024] In step S3.5, the expression for updating each reward in the reward sampling sequence is:

[0025] ;

[0026] In the formula, Indicates the first in the reward sampling sequence One element, This indicates the first predicted sequence of rewards. One element, This represents the time difference factor.

[0027] In step S3.6, the formula for updating the model's loss function is as follows:

[0028] ;

[0029] In the formula, Indicates the first in the reward sampling sequence One element, This represents the second predicted sequence of rewards obtained by inputting the pre-game state sampling sequence into the basic model of the decision network. One element, This represents the value of the loss function.

[0030] The beneficial effects of this invention are:

[0031] 1. By using reinforcement learning algorithms to learn the optimal multi-instance solution for games, and by rationally allocating and scheduling resources through algorithms, precise resource control can be achieved. While making full use of computing resources such as GPUs and CPUs, it can not only improve the efficiency of computing resources such as GPUs and CPUs in the cloud, but also reduce the failure rate in scenarios such as multi-instance games, and at the same time solve the dynamic expansion needs of a large number of cloud game players.

[0032] 2. By rationally scheduling and allocating resources, the problems of network congestion and resource overload caused by multiple instances of the game are minimized.

[0033] 3. Introducing reinforcement learning for profit modeling in multi-client game scenarios transforms the problem into a cost and revenue issue, enabling better commercial applications.

[0034] 4. Introduce time-series prediction to mine periodic game data, thereby providing a more accurate description of the environment and status of cloud gaming.

[0035] 5. By using reinforcement learning algorithms, the problem of large-scale game concurrency with an increasing number of game players can be solved relatively reasonably. By making reasonable decisions based on the current status data of various cloud resources and the demand for increased volume, the game can be multi-instanced and scaled up. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the process of the present invention.

[0038] Figure 2 This is a comparative diagram of this application and NO-LSTM, i.e., without using LSTM. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Long Short-Term Memory (LSTM) networks are a type of recurrent neural network specifically designed to address the long-term dependency problem inherent in general recurrent neural networks (RNNs). All RNNs exhibit a chain-like structure of repeating neural network modules. In a standard RNN, this repeating structural module has only a very simple structure, such as a single tanh layer.

[0041] Reinforcement learning (RL), also known as reward learning, evaluation learning, or reinforcement learning, is a paradigm and methodology of machine learning used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment.

[0042] A resource optimization method based on reinforcement learning in multi-instance cloud gaming scenarios, such as... Figure 1 As shown, it includes the following steps:

[0043] S1. Collect game startup data, game running data, and cloud monitoring data. Construct a game feature sequence set based on all data, and serialize the game running data to obtain a bandwidth sample sequence, including the following steps:

[0044] S1.1 collects game launch data, cloud monitoring data, and historical single and multi-instance game running data over a certain period of time;

[0045] The game launch data includes game ID, single-instance game revenue, game merchant ID, etc. Cloud monitoring data includes hardware utilization data such as CPU, GPU, and network bandwidth, as well as cloud CPU core count, cloud GPU model, and cloud device bandwidth. Historical single-game operation data includes CPU, GPU, and network bandwidth consumption data corresponding to a single game ID. Historical multi-instance game operation data includes game IDs that were newly started or stopped at different times on various devices. All of the above data can be collected using tools such as databases and log4j. Specifically, sampling periods can be set to collect relevant data within those periods, such as 31 days, 21 days, etc.

[0046] S1.2, serialize the game running data to obtain a bandwidth sample sequence including several historical samples and one real sample;

[0047] Both the historical samples and the real samples are 95% bandwidth data within the sample period. The 95% bandwidth data is obtained based on network bandwidth loss, which is existing technology and will not be elaborated further in this embodiment. The sampling period is evenly divided to set the sample period, which can be a day or an hour, etc. For example, the 95% bandwidth loss data of each game ID at different times can be organized into serialized data such as {x=[0.23,0.26,0.78,0.90...0.10],y=[0.81]}, where x is the historical 95% bandwidth data of a certain game ID at a certain node over the past 30 days, and y is the real 95% bandwidth data on the 31st day.

[0048] S1.3, Construct a game feature sequence set based on all the data obtained in step S1.1;

[0049] A game feature sequence set is established based on the time period division points of historical and real samples in the bandwidth sample sequence. Each element in the game feature sequence set includes {device ID, current list of multi-instance game IDs, initiating game ID, current CPU utilization, current GPU utilization, current network bandwidth utilization, historical 7-day average CPU utilization of the initiating game ID, historical 7-day average GPU utilization of the initiating game ID, historical 7-day average bandwidth utilization of the initiating game ID, highest device CPU utilization in the 24 hours after initiating the initiating game ID, highest device GPU utilization in the 24 hours after initiating the initiating game ID, highest device bandwidth utilization in the 24 hours after initiating the initiating game ID, and revenue corresponding to the game ID}. It may also include the number of cloud CPU cores, cloud GPU model, and cloud device bandwidth. The average usage is obtained by accumulating the daily usage and averaging it.

[0050] S2, using LSTM and bandwidth sample sequences to construct a bandwidth prediction model, and obtaining bandwidth prediction data based on the bandwidth prediction model, including the following steps:

[0051] S2.1, Construct a basic bandwidth prediction model based on LSTM (Long Short-Term Memory) network, and train the basic bandwidth prediction model using the bandwidth sample sequence obtained in step S1 to obtain the bandwidth prediction model.

[0052] The network structure of the basic bandwidth prediction model is as follows:

[0053]

[0054] The optimizer for the model is defined as adam, and the loss function used is mse (mean square error).

[0055] S2.2 Input historical samples from the bandwidth sample sequence into the bandwidth prediction model to obtain bandwidth prediction data.

[0056] In reality, games exhibit diverse operational characteristics, some with gradual resource usage, others with bursts, and still others with periodic resource usage. By predicting the bandwidth corresponding to different game IDs (e.g., 95 GHz), we can essentially assess and predict the future traffic of each game ID, thus providing more accurate support for decision-making. Compared to directly modeling game traffic based on its current state without time-series prediction, this method ultimately improves the MAPE (Mean Absolute Percentage Error) metric by 1.56% during evaluation.

[0057] S3. Construct a reinforcement learning network based on the DQN algorithm, and train the reinforcement learning network using the game feature sequence set and bandwidth sample sequence to obtain a game decision network model. The reinforcement learning network includes a train network and a target network with the same structure, and includes the following steps:

[0058] S3.1 Define the game pre-entry state sample sequence, game post-entry state sample sequence, and action sample sequence based on the game feature sequence set and bandwidth sample sequence, and simultaneously define the reward function to obtain the reward sample sequence corresponding to the game post-entry state sample sequence based on the reward function.

[0059] The expression for the action sample sequence is: ;

[0060] In the formula, Indicates the number of cloud games. Indicates whether to cut into the first... The operation of the game, when When, it indicates no cut-in, when At that time, it indicates that the game ID is Cloud gaming.

[0061] Preferably, the expression for the action sample sequence can also be set as: ,in, An identifier used to determine whether to enter the game, when When, it means not to enter any game. This indicates that at least one game has been switched in. Setting an identifier helps the system quickly identify whether a game has been switched in.

[0062] For example, there are 20 different types of games, with a game ID list of [1,2,3,...,20]. If a cutoff operation is performed on game ID 1, the final output will be... ={0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}, where an action value of 1 represents switching to the game with game ID 1, and other action values ​​of 0 represent not performing the action of switching to the corresponding game ID. If ={0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20} indicates that the switching operations for game ID 1 and game ID 20 were performed.

[0063] The game entry state sample sequence The bandwidth prediction data obtained in step S2 is added to the game feature sequence set as the 95% bandwidth data for the day the game ID is switched in. That is, {device ID, current list of multi-client game IDs, game ID switched in, current CPU usage, current GPU usage, current network bandwidth usage, historical 7-day average CPU usage of game ID switched in, historical 7-day average GPU usage of game ID switched in, historical 7-day average bandwidth usage of game ID switched in, 95% bandwidth data for the day the game ID is switched in, highest device CPU usage 24 hours after switching in, highest device GPU usage 24 hours after switching in, highest device bandwidth usage 24 hours after switching in, revenue corresponding to game ID switched in}. It can also include the number of device CPU cores, device GPU model, and device bandwidth.

[0064] The game pre-entry state sample sequence After the game is entered, the state sample sequence is obtained. The difference is that each element in the sequence does not contain the following elements: the highest CPU utilization of the device 24 hours after switching to the game ID, the highest GPU utilization of the device 24 hours after switching to the game ID, the highest bandwidth utilization of the device 24 hours after switching to the game ID, and the revenue corresponding to the game ID.

[0065] The expression for the reward function is:

[0066] ;

[0067] In the formula, This refers to the reward given when, within 24 hours of switching to the game ID, the device's peak CPU utilization, peak GPU utilization, or peak network bandwidth utilization exceeds 90%. This indicates the reward received when the device's CPU usage reaches 100% or the device's GPU usage reaches 100% within 24 hours of switching to the game ID. This is a reward given when the device's CPU, GPU, and network card are at 100% load and a hardware failure occurs. This refers to the reward given when the cloud bandwidth utilization rate drops to less than 40%. This refers to the normal game scenario, excluding rewards. ,award ,award and rewards For rewards in scenarios other than the corresponding scenario, in this embodiment, when the corresponding conditions are met, the rewards are set as shown in the table below; when the corresponding conditions are not met, the rewards are all zero.

[0068]

[0069] For example, if we approach the game with game ID 1, which is a normal game scenario with no risk, the revenue from game ID 1 would be 0.8, and the final output would be... =0.8.

[0070] The `train` network uses data generated by the `target` network to train and update its parameters, learning the final performance improvement from each action. The `target` network outputs the final performance improvement from each action and receives all parameters from the `train` network. Both networks have a three-layer structure, as shown in the table below:

[0071]

[0072] S3.2, Set the iteration rounds and parameter update cycle and initialize the training rounds. ;

[0073] S3.3, Randomly select from all the sample sequences obtained in step S3.1 The data points are labeled as the pre-game state sampling sequence, the post-game state sampling sequence, the action sampling sequence, and the reward sampling sequence, respectively.

[0074] In this embodiment, The value is set to be less than the number of samples in the collected game feature sequence set.

[0075] S3.4 Input the pre-game state sampling sequence into the target network to obtain the first predicted reward sequence;

[0076] S3.5, randomly sample a random number from [0,1]. ,like Then, update each reward in the reward sampling sequence;

[0077] The formula for updating each reward in the reward sampling sequence is as follows:

[0078] ;

[0079] In the formula, Indicates the first in the reward sampling sequence One element, This indicates the first predicted sequence of rewards. One element, Indicates the time difference factor. .

[0080] S3.6, input the updated reward sampling sequence, action sampling sequence and pre-game state sampling sequence into the train_network to train the game decision network model, and update the loss function of the model at the same time;

[0081] The formula for updating the model's loss function is as follows:

[0082] ;

[0083] In the formula, Indicates the first in the reward sampling sequence One element, This represents the second predicted sequence of rewards obtained by inputting the pre-game state sampling sequence into the basic model of the decision network. Each element.

[0084] S3.7, Judgment If yes, output the game decision network model and execute step S4; otherwise, execute step S3.8.

[0085] S3.8, Judgment If so, copy the parameters from the train_network to the target_network and execute. And return S3.3; otherwise, execute. And return to S3.3.

[0086] S4. Collect the corresponding data online and input it into the game decision network model. The game ID with the highest reward in the reward sequence of the game decision network model is the current optimal decision to start the game.

[0087] state = {device ID, current list of multi-instance game IDs, game ID entered, current CPU usage, current GPU usage, current network bandwidth usage, historical 7-day average CPU usage of game ID entered, historical 7-day average GPU usage of game ID entered, historical 7-day average bandwidth usage of game ID entered, and 95% bandwidth data of game ID entered on the current day}, where the 95% bandwidth data of game ID entered on the current day is the bandwidth prediction data calculated using the method in step S2, and the other data comes from real-time data.

[0088] For example, If the reward for game ID 1 in the reward sequence is the maximum, then entering the game with that ID is the optimal entry point. Conversely, if the game reward is -101, it carries a very high risk of failure compared to other game IDs. Figure 2 As shown, NO-LSTM represents the reinforcement learning performance without introducing an LSTM model, while ADD-LSTM represents the performance with temporal prediction, i.e., the reinforcement learning performance in this application. Through actual testing and grid hyperparameter training, the performance is improved by an average of 0.4% compared to NO-LSTM.

[0089] Reinforcement learning can learn which game ID to open a new instance under the current machine resource conditions, leading to optimal resource utilization in multi-instance game scenarios and reducing systemic risks associated with multi-instance gaming. Combining time series prediction with reinforcement learning not only solves the problem of optimizing resource utilization in cloud gaming multi-instance scenarios, but also addresses multiple business issues such as how to effectively reduce failures, operational risks, and the utilization rate of nodes.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A resource optimization method based on reinforcement learning in cloud gaming multi-instance scenarios, characterized in that, Includes the following steps: S1 collects game startup data, game running data, and cloud monitoring data, constructs a game feature sequence set based on all data, and serializes the game running data to obtain a bandwidth sample sequence; S2, a bandwidth prediction model is constructed using LSTM and bandwidth sample sequences, and bandwidth prediction data is obtained based on the bandwidth prediction model; S3, a reinforcement learning network is constructed based on the DQN algorithm. The game decision network model is obtained by training the reinforcement learning network using the game feature sequence set and bandwidth sample sequence. The reinforcement learning network includes a train network and a target network with the same structure. S4. Collect the corresponding data online and input it into the game decision network model. The game ID with the highest reward in the reward sequence output by the game decision network model is the current optimal decision to start the game. The network structure of the bandwidth prediction model includes a first LSTM, a second LSTM, a third LSTM, a Dropout layer, and a fully connected layer connected in sequence. The structure of the train network and the target network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. Step S3 includes the following steps: S3.1 Define the game pre-entry state sample sequence, game post-entry state sample sequence, and action sample sequence based on the game feature sequence set and bandwidth sample sequence, and simultaneously define the reward function to obtain the reward sample sequence corresponding to the game post-entry state sample sequence based on the reward function. S3.2, Set the iteration rounds and parameter update cycle and initialize the training rounds. ; S3.3, Randomly select from all the sample sequences obtained in step S3.1 The data points are labeled as the pre-game state sampling sequence, the post-game state sampling sequence, the action sampling sequence, and the reward sampling sequence, respectively. S3.4 Input the pre-game state sampling sequence into the target network to obtain the first predicted reward sequence; S3.5, randomly sample a random number from [0,1]. ,like Then, update each reward in the reward sampling sequence; S3.6, input the updated reward sampling sequence, action sampling sequence and pre-game state sampling sequence into the train_network to train the game decision network model, and update the loss function of the model at the same time; S3.7, Judgment If yes, output the game decision network model and execute step S4; otherwise, execute step S3.

8. S3.8, Judgment If so, copy the parameters from the train_network to the target_network and execute. And return S3.3; otherwise, execute. And return to S3.3; In step S3.1, the expression for the reward function is: ； In the formula, This refers to the reward given when, within 24 hours of switching to the game ID, the device's peak CPU utilization, peak GPU utilization, or peak network bandwidth utilization exceeds 90%. This indicates the reward received when the device's CPU usage reaches 100% or the device's GPU usage reaches 100% within 24 hours of switching to the game ID. This is a reward given when the device's CPU, GPU, and network card are at 100% load and a hardware failure occurs. This refers to the reward given when the cloud bandwidth utilization rate drops to less than 40%. This represents the reward in a normal game scenario; In step S3.5, the expression for updating each reward in the reward sampling sequence is: ； In the formula, Indicates the first element in the reward sampling sequence. One element, This indicates the first predicted sequence of rewards. One element, This represents the time difference factor.

2. The resource optimization method based on reinforcement learning in cloud gaming multi-instance scenarios according to claim 1, characterized in that, In step S1, the game launch data includes the game ID, single-client game revenue, and game merchant ID; the cloud monitoring data includes CPU usage, GPU usage, and network bandwidth usage; the historical single-game running data includes the CPU usage, GPU usage, and network bandwidth usage corresponding to a single game ID; and the historical multi-client game running data includes the game IDs of each device ID that were newly started or stopped at different times.

3. The resource optimization method based on reinforcement learning in multi-instance cloud gaming scenarios according to claim 1, characterized in that, In step S3.6, the formula for updating the model's loss function is as follows: ； In the formula, Indicates the first element in the reward sampling sequence. One element, This represents the second predicted sequence of rewards obtained by inputting the pre-game state sampling sequence into the basic model of the decision network. One element, This represents the value of the loss function.

Images