Game resource scheduling method based on reinforcement learning

Abstract

The application discloses a game resource scheduling method based on reinforcement learning and belongs to the field of game server resource scheduling and online service arrangement. In order to solve the problem that the local congestion of server computing power, bandwidth and instance resources is caused by the load fluctuation in the multi-player online game, and then the time delay is increased, the state is out of synchronization and the line is dropped, the application constructs a game resource scheduling state diagram, performs time sequence coding, uses a multi-agent reinforcement learning to output a candidate action, and generates a resource scheduling instruction after the correction of a congestion risk action shielding layer and a constraint checking layer, so that the technical effects of dynamically matching resources among the replica, the battle and the social scene and reducing the sudden increase of scheduling time delay and the synchronization risk are realized.

Description

Technical Field

[0001] This invention relates to the field of game server resource scheduling and online service orchestration, and in particular to a game resource scheduling method based on reinforcement learning. Background Technology

[0002] As massively multiplayer online games (MMORPGs) continue to evolve towards larger maps, stronger interactivity, and higher concurrency, game servers typically need to simultaneously handle multiple workloads, including dungeon battles, open-world exploration, social message synchronization, and cross-regional player access. Existing resource scheduling solutions mostly employ static threshold scaling, fixed route allocation, or manually reserved redundant instances to maintain operation, which can meet basic service needs during normal periods.

[0003] However, when the number of players, the frequency of combat events, and the frequency of social messages change rapidly within a short period of time, fixed scheduling methods struggle to identify load differences between different scenarios in a timely manner. This can easily lead to a concentrated increase in the computing power and bandwidth usage of some instances, resulting in increased network latency, delayed state synchronization, and instance migration mismatch. Furthermore, traditional scheduling strategies lack joint constraints on synchronization consistency, migration costs, and action cooldown times when performing cross-instance migrations or temporarily starting and stopping hot standby instances. This can easily introduce new risks of state rollback and disconnection while alleviating local congestion.

[0004] Therefore, a game resource scheduling method that can address the shortcomings of existing technologies is needed. Summary of the Invention

[0005] One objective of this invention is to propose a game resource scheduling method based on reinforcement learning. Addressing the problem that existing technologies struggle to timely match the load differences between dungeons, battles, and social scenarios, leading to localized server congestion, increased latency, and state desynchronization, this invention proposes a technical solution based on game resource scheduling state graph temporal coding, multi-agent reinforcement learning collaborative decision-making, and congestion risk action masking and correction. This invention achieves the technical effect of dynamically allocating instances, bandwidth, and routing resources while reducing synchronization risks and disconnection rates.

[0006] This invention provides a game resource scheduling method based on reinforcement learning, comprising: S1, collecting operational indicators representing player distribution, scene load, resource consumption, network quality, and synchronization status, and constructing a game resource scheduling state graph including region nodes, scene nodes, instance nodes, and player group nodes; S2, performing time-series encoding on the game resource scheduling state graph within a continuous time window to obtain the short-term load trend, congestion risk value, and synchronization risk value for each scene; S3, inputting the short-term load trend, the congestion risk value, and the synchronization risk value into a multi-agent reinforcement learning scheduler, and outputting candidate resource scheduling actions; S4, ... The resource scheduling candidate actions are input into the congestion risk action shielding layer. Combining predicted load, resource availability, network latency, state version difference, cross-instance synchronization time, and migration cost, the congestion risk value and consistency risk value of each candidate action are calculated. Candidate actions that exceed the execution conditions are shielded or projected for correction. The processed candidate actions are then input into the constraint verification layer for constraint verification, and the final resource scheduling instruction is output. S5. According to the final resource scheduling instruction, the resource orchestrator is controlled to perform instance scaling, player route switching, bandwidth quota adjustment, hot standby resource start / stop, and instance migration, and the capacity of the hot standby resource pool is adjusted according to the congestion risk value.

[0007] Optionally, S1 includes:

[0008] Generate region nodes based on the player's geographical region, scene nodes based on scene type and instance room, instance nodes based on server instances hosting the same scene, and player group nodes based on the co-occurrence relationship of players in the room and the frequency of interaction events.

[0009] The relationships between nodes are constructed by regional affiliation, scene carrying, instance allocation, and player group interaction, and server computing power consumption, bandwidth consumption, network latency, state synchronization latency, and disconnection rate are written into the corresponding node attributes.

[0010] Furthermore, players are grouped according to their geographical region, room co-occurrence relationship, and interaction event frequency to generate player group nodes. The message interaction frequency, combat coordination event frequency, and cross-region migration frequency are used as the edge weights of the connection edges between the player group nodes and the scene nodes and the instance nodes.

[0011] Optionally, S2 includes:

[0012] Temporal encoding is performed on the attribute sequences of each node and the weight change sequences of each connecting edge within a continuous time window to generate a scene-level temporal representation;

[0013] Based on the scenario-level time-series representation, the short-term load trend, congestion risk value, and synchronization risk value of each scenario in the subsequent scheduling window are calculated, and the scenario-level time-series representation, the short-term load trend, the congestion risk value, and the synchronization risk value are used as the joint state input for subsequent scheduling decisions.

[0014] Optionally, S3 includes:

[0015] The scenario-level time-series representation, the short-term load trend, the congestion risk value, and the synchronization risk value are input into the scenario scheduling agent, the routing scheduling agent, and the resource pool scheduling agent.

[0016] The scenario scheduling agent outputs candidate actions for scaling up / down and candidate actions for hot standby start / stop; the routing scheduling agent outputs candidate actions for player routing and candidate actions for migration; and the resource pool scheduling agent outputs candidate actions for bandwidth quotas. These actions are then aggregated to form the resource scheduling candidate actions.

[0017] Furthermore, the multi-agent reinforcement learning scheduler is a scheduler obtained through offline training. During training, a reward function is constructed based on network latency, state synchronization latency, disconnection rate, rollback count, resource utilization, and migration cost. Penalty terms are set for exceeding the limit of state version difference and state synchronization latency exceeding the synchronization consistency threshold to obtain action strategies corresponding to different scenarios.

[0018] Optionally, S4 includes:

[0019] The resource scheduling candidate actions are input into the congestion risk action shielding layer. For each candidate action, the congestion risk value and consistency risk value are calculated by combining the predicted load, resource margin, network latency, state version difference, cross-instance synchronization time and migration cost.

[0020] Candidate actions that do not meet the execution conditions are masked, candidate actions that meet the correction conditions are projected and corrected, and the corrected candidate actions are output together with the candidate actions that meet the execution conditions and do not need correction to the constraint verification layer.

[0021] The constraint verification layer performs executability verification on the input candidate actions based on the latency threshold, synchronization consistency threshold, resource margin, and action cooldown time, and outputs the final resource scheduling instruction.

[0022] Furthermore, the latency threshold and the synchronization consistency threshold are determined based on the scenario type, the upper limit of instance capacity, and the network latency quantile and state synchronization latency quantile within a continuous time window. The action cooldown time is determined based on the completion time and state stabilization time of the previous round of similar scheduling actions.

[0023] Optionally, S5 includes:

[0024] The instance resources are divided into a running resource pool, a hot standby resource pool, and a cold start resource pool.

[0025] The resource orchestrator performs instance scaling from the running resource pool and starts / stops standby instances from the hot standby resource pool according to the final resource scheduling instruction. When the congestion risk value exceeds the preset hot standby threshold, instance capacity is transferred from the cold start resource pool to the hot standby resource pool.

[0026] Furthermore, when the congestion risk value exceeds the preset hot standby threshold for two consecutive time windows, the resource orchestrator transfers instance capacity from the cold start resource pool to the hot standby resource pool according to the predicted load increment, and performs scenario image preloading on the instances to be enabled in the hot standby resource pool.

[0027] The beneficial effects of this invention are:

[0028] 1. This invention constructs a game resource scheduling state diagram that includes regional nodes, scene nodes, instance nodes, and player group nodes, and generates short-term load trends, congestion risk values, and synchronization risk values ​​by combining the time-series coding results of continuous time windows. This enables the identification of load change directions in different game scenarios before resource scheduling, thereby improving the scheduling decision's responsiveness to sudden load changes.

[0029] 2. This invention outputs candidate actions for scaling up / down, player routing, bandwidth quotas, hot standby start / stop, and instance migration through a multi-agent reinforcement learning scheduler. The candidate actions are then masked or projected and corrected by a congestion risk action shielding layer and a constraint verification layer. This can take into account latency control, synchronization consistency, and migration costs when performing resource scheduling, and reduce the risk of state asynchrony and disconnection caused by online scheduling.

[0030] 3. This invention divides instance resources into a running resource pool, a hot standby resource pool, and a cold start resource pool, and dynamically adjusts the capacity of the hot standby resource pool based on the congestion risk value. This allows for the reserve of available instance capacity before local congestion occurs, reducing service jitter caused by cold start waiting during peak periods. Attached Figure Description

[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0032] Figure 1 This is a flowchart of a game resource scheduling method based on reinforcement learning.

[0033] Figure 2 This is a flowchart of step S4 of the present invention. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0035] refer to Figures 1-2 A reinforcement learning-based game resource scheduling method includes: S1, collecting operational metrics representing player distribution, scene load, resource consumption, network quality, and synchronization status, and constructing a game resource scheduling state graph containing region nodes, scene nodes, instance nodes, and player group nodes; S2, performing time-series encoding on the game resource scheduling state graph within a continuous time window to obtain short-term load trends, congestion risk values, and synchronization risk values ​​for each scene; S3, inputting the short-term load trends, congestion risk values, and synchronization risk values ​​into a multi-agent reinforcement learning scheduler, and outputting candidate resource scheduling actions; S4, processing the resource... The source scheduling candidate action input congestion risk action shielding layer calculates the congestion risk value and consistency risk value of each candidate action based on predicted load, resource margin, network latency, state version difference, cross-instance synchronization time and migration cost. Candidate actions that exceed the execution conditions are shielded or projected for correction. The processed candidate actions are then input into the constraint verification layer for constraint verification, and the final resource scheduling instruction is output. S5. According to the final resource scheduling instruction, the resource orchestrator is controlled to perform instance scaling, player route switching, bandwidth quota adjustment, hot standby resource start / stop and instance migration, and the hot standby resource pool capacity is adjusted according to the congestion risk value.

[0036] In this specific embodiment, S1 includes:

[0037] Resource monitoring agent based on sampling period Operational metrics are collected from the game gateway, various scene service processes, synchronization service processes, and resource orchestrator, and presented in a sliding window. right The sampled values ​​within are aggregated to form the time. Game resource scheduling status diagram;

[0038] The game resource scheduling state graph is constructed as a heterogeneous graph structure with types. The node set consists of regional nodes, scene nodes, instance nodes, and player group nodes. Regional nodes are generated according to the regional code obtained by mapping the public IP of the player client through a geographic database. The regional code is taken as the provincial administrative region code, and only one regional node is generated within the window for the same regional code. Scene nodes are generated according to the tuple <scene_type, room_id> formed by scene type and room identifier, and only one scene node is generated within the window for the same tuple. Instance nodes are generated according to the instance identifier of the server that carries the scene, instance_id, and only one instance node is generated within the window for the same instance identifier. Player group nodes are generated according to the deterministic rule of "consistent player geographic region, strong room co-occurrence relationship, and high frequency of interaction events".

[0039] Specifically, within the set of players corresponding to each room identifier (room_id), the selection is first performed in the window. Active players with an online time percentage of at least 0.8% are further divided into regional subsets based on their regional codes. A player relationship graph is constructed within each regional subset, and groups are formed using a union-find set approach. A candidate relationship is established between any two players in the player relationship graph if and only if their cumulative co-occurrence time in the same room within the same window is at least [amount missing]. And the total number of interactive events that occur within the window divided by the window duration. The frequency of the obtained interaction events is no less than The interactive events include chat messages, team-up operations, trading operations, and combat collaboration operations. Each event carries the player identifiers of both parties and the event timestamp. After the disjoint-set data collection is completed and converged, each connected component generates a player group node and records the set of player identifiers contained in the group.

[0040] The connection edges between nodes are constructed according to four types of relationships: region affiliation relationship edges are used to connect region nodes and player group nodes. An edge is established when the region code of a player in the group is consistent with the region node code, and the number of online players in the group is recorded in the edge attribute; scene hosting relationship edges are used to connect scene nodes and instance nodes. An edge is established when the instance process registry shows that the instance hosts the <scene_type, room_id>, and the number of concurrent sessions of the scene on the instance is recorded in the edge attribute; instance allocation relationship edges are used to connect instance nodes and player group nodes. An edge is established when the routing table shows that the player session in the group is allocated to the instance, and the number of allocated sessions is recorded in the edge attribute; player group interaction relationship edges are used to connect player group nodes and scene nodes, as well as player group nodes and instance nodes, and the edge weight is written to represent the interaction strength and migration pressure.

[0041] The edge weights are calculated using a unified frequency-weighted model as follows:

[0042] ;

[0043] in Indicates the player group node identifier and Indicates scene node identifier or instance node identifier. Represents a node With nodes The edge weight connecting the edges, Display window inner group At the node The number of message interaction events generated within the corresponding business scope and The ratio of the frequency of message interactions obtained is Display window inner group At the node The number of cooperative events generated within the corresponding combat range and The frequency of combat coordination events obtained by the ratio, Display window inner group The number of sessions that occurred during cross-region access switching or cross-instance migration and The cross-regional migration frequency obtained by the ratio of , This represents the weighting coefficient for message interaction frequency. This represents the frequency weighting coefficient of combat coordination events. This represents the cross-regional migration frequency weighting coefficient;

[0044] Simultaneously, server computing power consumption, bandwidth consumption, network latency, state synchronization latency, and disconnection rate are written to the corresponding node attributes. Computing power consumption is written as the average CPU utilization and average memory utilization of instance processes within the window; bandwidth consumption is written as the average and peak throughput of instance inbound and outbound directions within the window; and network latency is written as the RTT within the window for the link from the region node to the instance node. The quantile value is written to the associated instance node, and the status synchronization delay is written to the P95 quantile value of "server status version number change timestamp to client confirmation receipt timestamp" in the window and written to the associated scene node and instance node. The disconnection rate is written to the ratio of the number of disconnected sessions in the window to the cumulative number of active sessions in the window and written to the associated player group node and instance node, thus obtaining a game resource scheduling status diagram with node attributes and edge weights for subsequent scheduling.

[0045] In this specific embodiment, S2 includes:

[0046] The timing encoder takes the game resource scheduling state graph as input and uses a sampling period. and sliding window At the current scheduling time Looking back at the interval A continuous graph sequence is formed within the graph, which is composed of node attributes and edge weights fixed at each sampling time.

[0047] For each scene node, construct the scene-level input sequence within the window for that scene. subscript Indicates scene node identifier and superscript Indicates the first in the window The snapshot number corresponding to the next sampling time, the It is a fixed-dimensional numerical vector, obtained by concatenating two parts. The first part is the node's own node attributes at the [first] rank. The value vector at the next sampling time includes the number of concurrent sessions in the room corresponding to the scene, the P95 quantile of the state synchronization latency, and the player disconnection rate in the scene obtained from the routing table. The second part is the value vector of all instance nodes that have scene-bearing relationships with the scene node at the next sampling time. The node attribute aggregation vector and edge weight change aggregation vector at each sampling time are used to obtain the instance node attribute aggregation vector by reading the CPU utilization, memory utilization, average throughput in the inbound and outbound directions, and network latency of all instance nodes carrying this scenario. Percentile value, state synchronization delay The edge weight change aggregation vector is obtained by taking the quantile value and the arithmetic mean. The edge weight change is obtained by reading the edge weight of the player group interaction relationship edge connected to the scene node. And calculate the difference between two adjacent samples. The arithmetic mean was then taken, where Indicates the player group node identifier and Indicates scene node identifier;

[0048] The scene-level input sequence is input into a gated recurrent unit (GRU) temporal encoder with a defined structure. The GRU temporal encoder has a two-layer stacked structure with a hidden state dimension of 64 for each layer, and outputs the hidden state vector at the final moment for each scene node. As a scene-level temporal representation of this scenario, the It remains deterministic within a single scheduling cycle and is obtained through forward inference of a GRU with fixed parameters;

[0049] In obtaining Then, the short-term load trend of this scenario within the subsequent scheduling window is generated simultaneously through the linear output header. Congestion risk value With synchronous risk value Its calculation satisfies:

[0050] ;

[0051] in This represents the weight matrix of the linear output head. This represents the bias vector of the linear output head. This represents a mapping in which activation functions are applied to components, and the first component is... Make and with Indicates a short-term increase in load and This indicates a short-term load decrease, and the second and third components are treated as follows: Make and ,in Represents the Sigmoid function;

[0052] The congestion risk value With window The probability of any instance in this scenario having CPU utilization or bandwidth utilization exceeding 0.90 is defined, and the synchronization risk value is defined as follows. With window State synchronization delay occurs in this scenario. The probability meaning of quantile values ​​exceeding 200ms or state version differences exceeding 1 is defined to thereby... , and The combined state inputs are then used for subsequent scheduling decisions and output to step S3.

[0053] In this specific embodiment, S3 includes:

[0054] The multi-agent reinforcement learning scheduler receives the joint state input from the output of step S2 and schedules it at the designated time. For each scene node Generate resource scheduling candidate actions;

[0055] Joint state input pairs for scene nodes Defined as a vector:

[0056] ;

[0057] in This represents the scene-level timing representation and is output by the GRU timing encoder in step S2. Indicates short-term load trends. Indicates the congestion risk value. Indicates the synchronous risk value. The scene type embedding vector is obtained from the scene type scene_type through a trainable lookup table and is updated together with the policy network during the offline training phase.

[0058] The multi-agent reinforcement learning scheduler consists of a scene scheduling agent, a route scheduling agent, and a resource pool scheduling agent, and adopts distributed output during execution and merges the candidate action set in the aggregation module;

[0059] The scene scheduling agent schedules each scene node. Output candidate actions for capacity expansion / shrinkage and candidate actions for hot standby start / stop, where the candidate actions for capacity expansion / shrinkage are defined as follows: And respectively, the corresponding scenarios will be assigned to the next scheduling window. The change in the number of running instances is defined as the number of instances decreasing by 2, decreasing by 1, remaining unchanged, increasing by 1, or increasing by 2. Hot standby start / stop candidate actions are defined as follows: And respectively, to deal with the hot standby resources, one standby instance is disabled, left unchanged, or one standby instance is enabled;

[0060] The routing and scheduling agent handles each scenario node. Output the player's route candidate actions and migration candidate actions. The action codes for both routing and migration are defined as tuples. and Indicates the player group node identifier, Indicates the instance node identifier; the player routing candidate action is used to group... New session routing to instance Migration candidate actions are used to trigger groups. Existing sessions are migrated from the current instance to the current instance. It also requires that the scene state version sequence be preserved during the migration process;

[0061] To ensure that the action space is computable within a defined range, at time... For each scene node Construct a candidate group set and a candidate instance set, and evaluate the action only on their Cartesian product, where the candidate group set is taken from the scene node. The edge weights of player group nodes with player group interaction relationships The largest front Group nodes and For player group node identification, The scene node is identified and its edge weight is defined in step S1. The candidate instance set is taken from the scene node. Network latency in instance nodes of scene-bearing relationship edges The smallest quantile value Each instance node and its network latency attribute is written in step S1;

[0062] The resource pool scheduling agent schedules each instance node. Output bandwidth quota candidate actions, bandwidth quota candidate actions are defined as follows: And corresponding to the instances The bandwidth quota relative to the current quota is calculated as follows: , The proportion is adjusted and the adjusted bandwidth quota is clipped to the range [ ]. This is used to form executable candidate instruction parameters;

[0063] All three agents were trained offline and had their network parameters fixed during online execution. Their policy network structures were identical, consisting of two fully connected layers, with the input being... or and The hidden layer has a joint state vector corresponding to the instance node, and includes the aggregation of the node attributes of that instance and the risk aggregation of the set of scenarios it carries. The hidden layer widths are 128 and 64 respectively, and the activation function is... The output layer outputs scores for discrete actions based on action type, and the aggregation module then selects the highest-scoring actions for each category. A set of candidate actions is formed. The target scene node, target instance node, and target player group node identifier associated with the action are also output to step S4;

[0064] During the offline training phase, training data sequences are constructed using historical running logs and a playback simulator, and a discount factor is applied. Learning rate A batch size of 256 is used to iteratively update each agent, and the immediate reward for each step is calculated using the following formula to drive the agent to learn action strategies in different scenarios:

[0065] ;

[0066] in Indicates an immediate reward. This represents resource utilization, calculated as the arithmetic mean of CPU utilization and bandwidth utilization of the running instance, and normalized to [value missing]. Represent the network latency cost and take the network latency. The quantile value is divided by 200ms and then cropped to... Represent the state synchronization delay cost and take the state synchronization delay. The quantile value is divided by 200ms and then cropped to... This represents the disconnection rate and is calculated as the ratio of the number of disconnected sessions within the window to the cumulative number of active sessions. The cost of each rollback is represented by dividing the number of rollbacks within the window by 10 and then cropping to the specified value. The migration cost is represented by a weighted sum of the number of migration sessions and the amount of migration data within the window, then normalized to a value. This represents the version difference and takes the maximum difference between the server-side version number and the client-confirmed version number within the same scenario. This represents the version difference threshold corresponding to the synchronization consistency threshold. This represents the synchronization delay threshold corresponding to the normalized synchronization consistency threshold. express The positive part function, , These represent the resource utilization reward weight, network latency penalty weight, state synchronization latency penalty weight, disconnection rate penalty weight, rollback count penalty weight, migration cost penalty weight, version difference exceeding penalty weight, and synchronization latency exceeding penalty weight, respectively. This allows the scheduler to minimize latency, synchronization risk, and migration cost during the offline training phase and output a set of resource scheduling candidate actions that can be filtered and corrected in step S4 during the online inference phase.

[0067] In this specific embodiment, S4 includes:

[0068] Congestion risk action shielding layer receives resource scheduling candidate action set And for each candidate action Construct it in a subsequent scheduling window The predicted quantities after execution include predicted load, resource availability, network latency, state version difference, cross-instance synchronization time, and migration cost.

[0069] The predicted load is derived from the short-term load trend output in step S2. Drive and perform scene node operations The number of concurrent sessions within the current window is extrapolated from the CPU utilization and bandwidth utilization of the instance node. During extrapolation, the following parameters are used: Crop to And mapped to load factor ,when Press at time Calculate and when Press at time Calculate and take the larger value between this and the lower bound of 0.5 to obtain the scene within the window. Predicted CPU and bandwidth requirements within the system;

[0070] Resource reserves based on actions The corresponding changes in capacity expansion / contraction, hot standby start / stop, or bandwidth quota are calculated after updating the number of running instances and instance quotas. The resource reserve judgment criteria for meeting the execution conditions are that the predicted CPU utilization after execution does not exceed 0.90 and the predicted bandwidth utilization after execution does not exceed 0.90.

[0071] Network latency is based on the link RTT written in step S1. Percentage values ​​are combined with the target instance nodes of candidate routes or migrations to perform replacement predictions. Replacement predictions are then grouped. Switch to target instance The network latency is then taken as the RTT from the region node to the instance node. The quantile value is determined;

[0072] The state version difference is calculated by taking the difference between the current scene state version number held by the target instance node and the current major version number of the scene node, and then using it as the basis for calculation. As the version difference threshold, Indicates the maximum allowed state version difference;

[0073] The cross-instance synchronization time is calculated using the state synchronization latency written in step S1. The quantile value is added to the cross-instance state transmission time when a migration action occurs. The cross-instance state transmission time is obtained by dividing the migration data volume by the available bandwidth of the migration channel, and the available bandwidth of the migration channel is the remaining bandwidth of the target instance in the inbound direction.

[0074] The migration cost is calculated by weighting the number of migration sessions, the amount of data migrated, and the number of repeated synchronizations during the migration period, and then normalizing it. ;

[0075] After obtaining the above predicted values, the congestion risk action shielding layer calculates the congestion risk value for each candidate action. Consistency risk value And both are used for masking or projection correction, the calculations satisfying:

[0076] ;

[0077] in This indicates the candidate action identifier, and the candidate actions include five categories: scaling up / down, hot standby start / stop, player routing, instance migration, and bandwidth quota adjustment. Indicates action The resulting congestion risk value and its range is: Indicates action The resulting consistency risk value and its range are: Indicates the execution of an action The normalized value of the predicted CPU utilization was then used, with 0.90 as the upper bound for normalization and cropped to... Indicates the execution of an action The normalized value of the predicted bandwidth utilization is then used, with 0.90 as the upper bound for normalization and clipped to... Indicates the execution of an action The normalized value of the post-prediction network latency, and the latency threshold for this scenario. To normalize the upper bound and clip to Indicates the execution of an action The normalized value of the post-predicted state synchronization delay, and the synchronization consistency threshold for this scenario. To normalize the upper bound and clip to Indicates the execution of an action The normalized value of the version difference of the later predicted state and with To normalize the upper bound and clip to Indicates the execution of an action The normalized value of migration cost and directly derived from the value of migration cost. This represents the weighting coefficient for the CPU utilization item. This represents the weighting coefficient for the bandwidth utilization term. This represents the weighting coefficient of the network latency term. This represents the weighting coefficient of the state synchronization delay term. This represents the weight coefficient of the state version difference term. This represents the weighting coefficient for migration cost items. This indicates that the input will be cropped by components. Interval clipping function;

[0078] The delay threshold With synchronization consistency threshold By scene type, instance capacity limit, and window The quantile values ​​within the scene node are determined. First, calculate the RTT of the associated link for this scenario from the monitoring data in step S1. The quantile value is multiplied by 1.2 to obtain the statistical threshold. Then, a type upper limit is determined based on the scene type, and tightened according to the instance capacity limit of the scene. The type upper limit is 150ms for combat scenes, 180ms for dungeon scenes, and 250ms for social scenes. When the instance capacity limit of the scene is not less than 2000 concurrent sessions, the type upper limit is multiplied by 0.9. The final result... Take the smaller value between the statistical threshold and the tightened upper limit of the type;

[0079] Synchronization Consistency Threshold State synchronization delay The quantile value is multiplied by 1.2 to obtain the statistical threshold, and 160ms for combat scenarios, 200ms for dungeon scenarios, and 300ms for social scenarios are used as the upper limit for each type. The smaller value is then taken after applying the same instance capacity upper limit tightening rule.

[0080] Congestion risk action shielding layer based on the above and The action is processed according to the execution conditions. Meets the criteria of predicted CPU utilization exceeding 0.90, predicted bandwidth utilization exceeding 0.90, or predicted network latency exceeding [a certain threshold]. Or the predicted state synchronization delay exceeds Or the predicted state version difference exceeds When the action is a candidate action for scaling up or down or a candidate action for bandwidth quota and only violates the "resource margin" or "quota upper and lower bounds" and can be restored to executability by adjusting the action range, projection correction is performed.

[0081] The projection correction projects the instance number change onto a discrete set for the scaling candidate action. The bandwidth quota candidate action is the one closest to the original change that satisfies the condition that the predicted CPU utilization and predicted bandwidth utilization after execution do not exceed 0.90, and simultaneously ensures that the number of running instances supporting this scenario is not less than 1 and not more than 32. The adjusted bandwidth quota is then pruned to [ ]. And recalculate after cutting. and For routing candidate actions and migration candidate actions, if the target instance does not satisfy the condition... or The target instance is replaced with an instance within the same set of candidate instances that satisfies the constraints and has the minimum predicted network latency, while keeping the target player group unchanged.

[0082] After completing the masking and projection correction, we obtain the processed candidate action set. And input constraint verification layer, the constraint verification layer for Perform a global executability check and output the final set of resource scheduling instructions. The global executability verification includes three parts: resource reserve verification, action conflict resolution, and action cooldown time verification. Action conflict resolution is constrained by the following: only one route switching instruction or one migration instruction is allowed for the same player group at the same scheduling time; if both a route and a migration exist simultaneously, the migration is retained and the route is deleted. Similarly, only one bandwidth quota adjustment instruction is allowed for the same instance at the same scheduling time; if multiple instructions exist, they are retained. For smaller instructions, only one scaling up / down instruction and one hot standby start / stop instruction are allowed in the same scenario at the same scheduling time. When multiple instructions exist, the one with the highest score is retained, and the following conditions are met: and The instructions;

[0083] The action cooldown time verification maintains the completion timestamp and state stability timestamp of the most recent action of the same type for each type of action. The completion timestamp is determined by the action completion receipt reported by the resource orchestrator, and the state stability timestamp is determined by the fact that the network latency of the scene nodes associated with the action does not exceed a certain threshold within two consecutive windows after the action ends. And the state synchronization delay does not exceed Furthermore, the disconnection rate is determined by a criterion of not exceeding 0.01, and the constraint verification layer only allows disconnections at the current time. If the timestamp is greater than or equal to the larger of the two types mentioned above, output the same type of action instruction;

[0084] The constraint verification layer generates the final resource scheduling instruction set based on the execution order of the actions that pass the verification. The execution order is fixed as follows: first, output scaling up / down and hot standby start / stop commands to ensure capacity is available first; then, output bandwidth quota adjustment commands to ensure link resources are available first; finally, output player route switching and instance migration commands to reduce online session jitter and... Output to S5.

[0085] In this specific embodiment, S5 includes:

[0086] The resource orchestrator receives the final set of resource scheduling instructions. All instance resources are divided into three categories according to instance status tags: running resource pool, hot standby resource pool, and cold start resource pool. The running resource pool is used to carry online player sessions, and each running instance is in an accessible state and has completed the registration of the scene service process. The hot standby resource pool is used to store standby instances that have been started but do not carry player sessions, and each hot standby instance has completed the loading of basic dependencies and maintains a heartbeat connection with the configuration center and state storage. The cold start resource pool is used to store capacity quotas that are reserved for computing and network quotas but have not started instance processes, and these quotas can be used to create corresponding instances by the infrastructure interface after receiving the call-in instruction.

[0087] The resource orchestrator executes instance scaling up / down, hot standby resource start / stop, bandwidth quota adjustment, player route switching, and instance migration sequentially according to instruction type, and performs idempotent control on each instruction. When the instance scaling up instruction is executed, it first selects a standby instance that meets the target scenario image version from the hot standby resource pool, switches it to a running instance, and completes load balancing registration. When there are not enough available standby instances in the hot standby resource pool, it first transfers capacity from the cold start resource pool, creates an instance, adds it to the hot standby resource pool, and then switches it to a running instance. When the instance scaling down instruction is executed, it first performs session emptying on the target running instance and stops receiving new sessions. After the session count reaches zero, it transfers the instance to the hot standby resource pool or releases it as a cold start resource pool quota when the capacity of the hot standby resource pool reaches its limit.

[0088] Player route switching commands are implemented by updating the gateway routing table. The routing table update is written with the player group node identifier and the target instance node identifier as keys and synchronized across the entire network within 1 second. Instance migration commands are executed at the player group level and include five stages: freezing the source instance session, establishing the target instance shadow session, cross-instance status transmission, status version confirmation, and traffic switching. After the switch is completed, the source instance session is released and a migration completion timestamp is reported for the action cooldown time of step S4. Bandwidth quota adjustment commands apply the inbound and outbound quotas of the target instance simultaneously through the instance network card QoS interface and are read back for confirmation within 3 seconds.

[0089] The resource orchestrator also considers the congestion risk value for each scenario. The capacity of the hot standby resource pool is dynamically adjusted, and the preset hot standby threshold is denoted as [missing information]. And with two consecutive scheduling windows All satisfy the same scenario As a trigger condition, when the trigger condition is met, the resource orchestrator calculates the instance capacity that needs to be transferred from the cold start resource pool to the hot standby resource pool for this scenario and performs the transfer and preloading. The instance capacity to be transferred is calculated according to... Confirmed, among which Represents scene-oriented nodes The number of instances that need to be moved from the cold start resource pool to the hot standby resource pool. This indicates the short-term load trend from step S2. The mapped predicted load factor for the scenario in subsequent windows, with a value range of [value missing]. And consistent with the multiplier definition in S4. Represents the scene nodes within the current window. The number of concurrent sessions is recorded in the scene node attributes by step S1. This represents the maximum concurrent session capacity that a single instance can stably support and serves as a fixed capacity parameter for the resource orchestrator. This represents the floor function;

[0090] The resource orchestrator deducts no less than [amount] from the cold start resource pool. The capacity quota is allocated and a corresponding number of instances are created and added to the hot standby resource pool and marked as instances to be enabled. Scene image preloading is performed on each instance to be enabled to shorten the preparation time for subsequent switching to running instances. The scene image preloading includes pulling a container image version consistent with the target scene service, loading the map and script configuration of the scene, establishing a read-only connection to the state storage and preheating the state snapshot index of the most recent 10 minutes, loading the combat and social message processing modules and completing health checks but not opening the player access port to the outside world, so that the instances to be enabled can complete registration and enter the running resource pool within 10 seconds after receiving the enable command.

[0091] When all scenarios within three consecutive scheduling windows meet the requirements Furthermore, when the number of standby instances in the hot standby resource pool exceeds 20, the resource orchestrator will gradually release the excess standby instances into the cold start resource pool quota and maintain the hot standby resource pool with no less than 20 standby instances to avoid jitter caused by frequent start-stop.

[0092] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0093] This invention maps multi-source load indicators during game operation into a computable state diagram representation, and introduces a verification and correction mechanism for congestion and consistency risks before the execution of candidate scheduling actions, so that the resource orchestration process keeps pace with changes in online business load, thereby more effectively alleviating the problem of local resource congestion in multiplayer online games.

[0094] This invention adds a congestion risk action shielding layer between the reinforcement learning scheduler and the resource orchestrator, and combines a hierarchical management structure of running resource pool, hot standby resource pool and cold start resource pool to structurally improve the traditional single scheduling logic, thereby making it more conducive to achieving online game service scheduling effects with low latency, low disconnection and high synchronization consistency.

Claims

1. A game resource scheduling method based on reinforcement learning, characterized in that, include: S1. Collect operational metrics representing player distribution, scene load, resource consumption, network quality, and synchronization status, and construct a game resource scheduling state diagram including regional nodes, scene nodes, instance nodes, and player group nodes; S2. Perform time-series encoding on the game resource scheduling state diagram within a continuous time window to obtain the short-term load trend, congestion risk value, and synchronization risk value for each scene. S3. Input the short-term load trend, congestion risk value and synchronization risk value into the multi-agent reinforcement learning scheduler, and output resource scheduling candidate actions; S4. Input the resource scheduling candidate actions into the congestion risk action shielding layer. Combine the predicted load, resource availability, network latency, state version difference, cross-instance synchronization time, and migration cost to calculate the congestion risk value and consistency risk value of each candidate action. Shield or project and correct candidate actions that exceed the execution conditions. Input the processed candidate actions into the constraint verification layer for constraint verification and output the final resource scheduling instruction. S5. Control the resource orchestrator to execute instance scaling, player routing switching, bandwidth quota adjustment, hot standby resource start / stop, and instance migration according to the final resource scheduling instruction. Adjust the capacity of the hot standby resource pool according to the congestion risk value.

2. The game resource scheduling method based on reinforcement learning according to claim 1, characterized in that, S1 includes: Region nodes are generated based on the player's geographical region, scene nodes are generated based on scene type and instance room, instance nodes are generated based on server instances that host the same scene, and player group nodes are generated based on the co-occurrence relationship of players in the room and the frequency of interaction events. The regional affiliation relationship, scene hosting relationship, instance allocation relationship and player group interaction relationship are constructed into connection edges between nodes, and server computing power consumption, bandwidth consumption, network latency, state synchronization latency and disconnection rate are written into the corresponding node attributes.

3. The game resource scheduling method based on reinforcement learning according to claim 1, characterized in that, S2 includes: Temporal encoding is performed on the attribute sequences of each node and the weight change sequences of each connecting edge within a continuous time window to generate a scene-level temporal representation; Based on the scenario-level time-series representation, the short-term load trend, congestion risk value, and synchronization risk value of each scenario in the subsequent scheduling window are calculated, and the scenario-level time-series representation, the short-term load trend, the congestion risk value, and the synchronization risk value are used as the joint state input for subsequent scheduling decisions.

4. The game resource scheduling method based on reinforcement learning according to claim 3, characterized in that, S3 includes: inputting the scenario-level time-series representation, the short-term load trend, the congestion risk value, and the synchronization risk value into the scenario scheduling agent, the routing scheduling agent, and the resource pool scheduling agent; the scenario scheduling agent outputs scaling-up / scaling candidate actions and hot standby start / stop candidate actions; the routing scheduling agent outputs player routing candidate actions and migration candidate actions; the resource pool scheduling agent outputs bandwidth quota candidate actions; and these are aggregated to form the resource scheduling candidate actions.

5. The game resource scheduling method based on reinforcement learning according to claim 1, characterized in that, S4 includes: inputting the resource scheduling candidate actions into the congestion risk action shielding layer; calculating the congestion risk value and consistency risk value for each candidate action by combining predicted load, resource availability, network latency, state version difference, cross-instance synchronization time, and migration cost; performing shielding processing on candidate actions that do not meet the execution conditions; performing projection correction on candidate actions that meet the correction conditions; and outputting the corrected candidate actions along with candidate actions that meet the execution conditions and do not require correction to the constraint verification layer; the constraint verification layer performs executability verification on the input candidate actions based on latency threshold, synchronization consistency threshold, resource availability, and action cooldown time, and outputs the final resource scheduling instruction.

6. The game resource scheduling method based on reinforcement learning according to claim 1, characterized in that, S5 includes: dividing instance resources into a running resource pool, a hot standby resource pool, and a cold start resource pool; the resource orchestrator performs instance scaling from the running resource pool and starts / stops standby instances from the hot standby resource pool according to the final resource scheduling instruction, and when the congestion risk value exceeds the preset hot standby threshold, it transfers instance capacity from the cold start resource pool to the hot standby resource pool.

7. The game resource scheduling method based on reinforcement learning according to claim 4, characterized in that, The multi-agent reinforcement learning scheduler in S3 is a scheduler trained offline. During training, a reward function is constructed based on network latency, state synchronization latency, disconnection rate, rollback count, resource utilization, and migration cost. Penalty terms are set for exceeding the limit of state version difference and state synchronization latency exceeding the synchronization consistency threshold to obtain action policies corresponding to different scenarios.

8. The game resource scheduling method based on reinforcement learning according to claim 5, characterized in that, The latency threshold and the synchronization consistency threshold mentioned in S4 are determined based on the scenario type, the upper limit of instance capacity, and the network latency quantile and state synchronization latency quantile within a continuous time window. The action cooldown time is determined based on the completion time of the previous round of similar scheduling actions and the state stabilization time.

9. The game resource scheduling method based on reinforcement learning according to claim 2, characterized in that, In S1, players are grouped according to their geographical region, room co-occurrence relationship, and interaction event frequency, and the player group nodes are generated. The message interaction frequency, combat coordination event frequency, and cross-region migration frequency are used as the edge weights of the connection edges between the player group nodes and the scene nodes and the instance nodes.

10. The game resource scheduling method based on reinforcement learning according to claim 6, characterized in that, In S5, when the congestion risk value exceeds the preset hot standby threshold for two consecutive time windows, the resource orchestrator transfers instance capacity from the cold start resource pool to the hot standby resource pool according to the predicted load increment, and performs scene image preloading on the instances to be enabled in the hot standby resource pool.

Images