User abnormal game behavior detection method and system based on reinforcement learning

Abstract

This invention discloses a method and system for detecting abnormal user gaming behavior based on reinforcement learning. The method employs a proximal policy optimization algorithm to construct a dual-Critic value head policy network model. The first value head outputs the final event result representation value, and the second value head outputs the global final event aggregation magnitude representation value. The two value heads share a pre-concept value feature layer and output separately at the end. The first value head is used for final event result value estimation, while the second value head is used for final event magnitude auxiliary supervision and abnormal scoring. During the inference phase, a two-dimensional action deviation is calculated based on the output of the dual value heads, generating three levels of scores: single-step, game, and user history. Abnormal behavior is determined through a three-level threshold system of single game, related games, and game currency circulation. The corresponding system includes modules for data acquisition and preprocessing, a dual-Critic value head model, action scoring, threshold detection, and early warning handling. This invention can accurately identify various abnormal game behaviors and effectively maintain the virtual economic ecosystem of the game.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and online game security technology, specifically a method and system for detecting abnormal user gaming behavior based on reinforcement learning. Background Technology

[0002] Against the backdrop of the rapid development of the online gaming industry, the security issues of in-game virtual economic systems have become increasingly prominent. Some unscrupulous users deliberately transfer game currency to specific accounts through passive gameplay and collusion, enabling the illegal resale of game currency and the transfer of virtual resources, severely damaging the economic ecosystem of gaming platforms.

[0003] Existing abnormal behavior detection technologies mainly include: (1) a database retrieval method based on rule matching, which retrieves and statistically analyzes background data by pre-setting association rules; (2) a deep learning classification method based on the visualization of behavioral data, which converts user behavior time series data into a two-dimensional image matrix, uses convolutional neural networks for feature extraction and positive and negative sample classification, and then combines unsupervised clustering to discover abnormal patterns.

[0004] The above methods have the following shortcomings: (1) Rule matching methods can only discover known obvious abnormal associations, and lack effective means of analysis for hidden negative game behaviors (such as users deliberately choosing low-value actions at key decision points); (2) Behavioral visualization methods mainly extract macroscopic features of time series data, and cannot perform fine-grained analysis of the rationality of each step of the operation decision; (3) Existing methods lack the ability to quantify decision quality based on policy models, and cannot determine whether user operations deviate from reasonable strategies; (4) Traditional single-value head reinforcement learning methods only predict returns from a single dimension, and cannot simultaneously take into account both the end-game event result and the end-game magnitude, resulting in an incomplete evaluation. Therefore, there is an urgent need for an abnormal behavior detection method and system based on a multi-dimensional value assessment model. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the present invention aims to provide a method and system for detecting abnormal user gaming behavior based on reinforcement learning. This addresses the problems of insufficient ability to identify covert negative gaming behavior, inability to quantify the rationality of operational decisions with fine granularity, lack of decision quality assessment methods based on policy models, and the limited evaluation dimension of single-value head reinforcement learning, which makes it difficult to effectively detect abnormal behaviors such as negative gaming, collusion to give away points, and reselling game currency.

[0006] To achieve the above objectives, embodiments of the present invention disclose a method for detecting abnormal user gaming behavior based on reinforcement learning, the method comprising the following steps: S1: Collect user game data in real time through the game server interface, perform feature encoding and vectorization processing on the game data, and generate state feature vectors and historical action sequences; S2: A policy network model comprising a shared feature backbone network, an Actor policy network, and a dual-value-head Critic network is constructed using a proximal policy optimization algorithm. The state feature vector and historical action sequence features are input into the shared feature backbone network to obtain a state representation. The state representation is then input into the Actor policy network to output a legal action probability distribution and into the dual-value-head Critic network to output a value representation. The value representation includes the final event result representation value output by the first value head and the global final aggregation magnitude representation value output by the second value head. During the training phase, policy updates are performed based on the legal action probability distribution and the first value head output. The value loss and auxiliary supervision loss are calculated based on the first and second value heads, and joint parameter updates are performed. After training, the Actor policy network is discarded during the inference phase. S3: Based on the trained dual-value-head Critic network, calculate the final result deviation and final magnitude deviation of the user's single-step action to generate a comprehensive score for the single-step action, and further calculate the comprehensive score for the game and the user's historical behavior score based on the comprehensive score for the single-step action. S4: Combining the single-step action comprehensive score, game comprehensive score, and user historical behavior score, execute single-game anomaly detection, associated game anomaly detection, and game currency flow anomaly detection in sequence to determine abnormal user game behavior.

[0007] Furthermore, the Actor policy network takes the state representation output by the shared feature backbone network as input, and obtains the log probability of the action through three fully connected layers and ReLU activation function layer by layer nonlinear transformation. Then, it is normalized by the Softmax function to output the probability distribution of each legal action.

[0008] Furthermore, the dual-value-head Critic network includes a shared value feature layer, a first value-head output layer, and a second value-head output layer. The shared value feature layer consists of five fully connected layers and is jointly invoked by the two value heads. The first value-head output layer outputs a scalar-form representation of the final event result based on the shared value feature layer and is subject to independent supervision by the final event result label. The second value-head output layer outputs a scalar-form representation of the global final aggregation magnitude based on the same shared value feature layer, regressing the global final aggregation magnitude rather than incrementally increasing local values, and is subject to independent auxiliary supervision by the global final aggregation magnitude label.

[0009] Furthermore, the formula for calculating the final result deviation of the single-step action is as follows:

[0010] in, This is the final amplitude deviation; Current state The output value of the second value head; To perform the action Next state The output value of the second value head; Current state Next action The instant reward obtained in the final scale dimension;

[0011] in, This is the final amplitude deviation; Current state The output value of the second value head; To perform the action Next state The output value of the second value head; Current state Next action The instant reward obtained in the final scale dimension; The formula for calculating the comprehensive score of a single-step action is as follows:

[0012] in, The overall score is based on the single-step action. As the weight for the result deviation, As the magnitude deviation weight, and .

[0013] The overall game score is based on a game containing M state-action pairs. :

[0014] in, This is a comprehensive score for the game. For the first Each state has an importance weight, with a weight of 2 for critical decision states and 1 for ordinary states. For the first Overall score for footwork.

[0015] The calculation method for the user's historical behavior score is as follows: statistically analyze the comprehensive score of all games played within a preset period, calculate its mean and standard deviation to form a user behavior profile; at the same time, statistically analyze the distribution characteristics of the user's two-dimensional deviation to construct a user behavior feature vector.

[0016] Furthermore, the single-game anomaly detection specifically refers to: when the overall score of a single game... Below the single-game abnormal threshold At that time, it was marked as a suspected negative game; among them, ; This represents the average overall score for a normal user's match. The standard deviation of the overall game score for normal users. The significance coefficient is used when the cumulative amplitude deviation at the end of a single game exceeds the amplitude anomaly threshold. At that time, the game is additionally marked as having abnormal amplitude.

[0017] Furthermore, the aforementioned abnormal game detection specifically involves: statistically analyzing game data between pairs of users within a preset period and calculating the severity of collusion. When the severity of collusion Exceeding the associated anomaly threshold At that time, it was determined that there was serious suspicion of collusion; the severity of the collusion. The calculation formula is:

[0018] Among them, the severity of collusion is represented; Average score for passive games; The amplitude anomaly coefficient represents the proportion of amplitude anomalies in negative games. Furthermore, the anomaly detection of game currency flow specifically involves: combining user anomaly rating data and game currency flow data to construct a user-game currency flow graph G, calculated using the following formula:

[0019] in, This is an indicator of abnormal circulation. For concentration; if The game currency resale was deemed abnormal, among which... This is the threshold for abnormal circulation.

[0020] Furthermore, step S5 is included after step S4: generating structured early warning information based on the type and severity of the user's abnormal gaming behavior, and performing graded handling operations.

[0021] Furthermore, reinforcement learning-based systems for detecting abnormal user gaming behavior include: The data acquisition and preprocessing module is used to acquire user game data in real time through the game server interface, perform feature encoding and vectorization processing on the game data, and generate state feature vectors. The dual-Critic value head model module stores and deploys the trained dual-Critic value head near-end policy optimization algorithm policy model. The policy model includes an Actor policy network and a dual-value head Critic network consisting of a shared value feature layer and a terminal dual-head output layer. It is used to output the final event result representation value and the final magnitude representation value, and to form a two-dimensional value representation in the anomaly scoring stage. The action scoring module is communicatively connected to the dual-critic value head model module. It is used to calculate the final result deviation and final amplitude deviation of the user's single-step action based on the output of the dual value head, and generate a single-step action comprehensive score, a game comprehensive score, and a user historical behavior score. The threshold detection module is communicatively connected to the action scoring module. It has a built-in threshold system and is used to combine various scores to perform single-game anomaly detection, associated game anomaly detection, and game currency flow anomaly detection in sequence to determine abnormal user game behavior. The early warning and response module is communicatively connected to the threshold detection module. It is used to generate structured early warning information based on the judgment result of abnormal behavior and to perform graded response operations.

[0022] Compared with the prior art, the beneficial effects of the present invention are: (1) The present invention adopts a dual-value head joint evaluation mechanism to simultaneously quantify the rationality of user operation from two dimensions: final result and global magnitude. Compared with the traditional single-dimensional detection method, it significantly improves the accuracy of identifying abnormal behaviors such as concealed negative behavior and malicious point-giving, and solves the problems of one-sided identification and high false negative rate of existing technologies.

[0023] (2) This invention forms a full-link anomaly identification system by using three levels of threshold progressive detection: single game, related game, and game currency circulation. It can accurately identify multi-level violations such as individual negligence, collusion among multiple people, and resale of virtual assets. The detection coverage is comprehensive and the logic is rigorous and reliable.

[0024] (3) This invention relies on a reinforcement learning model to achieve end-to-end automatic learning and dynamic adaptation, eliminating the need for manual configuration of complex rules, reducing operation and maintenance costs, and supporting real-time online detection, which can quickly respond to and handle abnormal behavior, effectively ensuring the stability of the game virtual economy ecosystem. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the strategy network model structure of the method of the present invention; Figure 2 This is a flowchart of the action scoring system of the present invention; Figure 3 This is a flowchart of the multi-level threshold detection method of the present invention; Figure 4 This is a diagram of the overall system architecture of the present invention. Detailed Implementation

[0026] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please see Figure 1 , Figure 2 , Figure 3 and Figure 4 This embodiment uses card games such as Dou Dizhu as an application scenario to describe in detail the technical solutions provided by each embodiment of this application.

[0028] This invention provides a method for detecting abnormal user gaming behavior based on reinforcement learning, the method comprising the following steps: S1: Collect user game data in real time through the game server interface, perform feature encoding and vectorization processing on the game data, and generate state feature vectors; The game information of a single game of Dou Dizhu is collected from the game server or self-playing environment at the current decision moment. The situation information includes the current player's position, the current player's hand, the unknown hand set or observable summary information of the other players, the relative position of the player who made the last valid move, the last valid move, the set of cards played by each non-current player, the number of remaining cards in each non-current player's hand, the number of bombs that have appeared, the current set of legal moves, and the recent historical move sequence. The training samples also record the final win / loss label and the final score range label after the game ends.

[0029] The card set and action set are both encoded using a card count matrix. The 13 cards from 3 to A and 2 are expanded column by column. Each card is encoded into a 4-dimensional count vector according to the number of times it appears from 0 to 4. The Little Joker and Big Joker are encoded into 2 independent dimensions, forming a 54-dimensional card vector. Empty actions are encoded into a 54-dimensional zero vector.

[0030] For the landlord position, the current player's hand, the sum of other players' hands, the relative position of the player who last played a valid card, the last valid card-playing action, the set of cards played by the next player, the set of cards played by the previous player, the number of remaining cards in the next player's hand, the number of remaining cards in the previous player's hand, and the number of bombs that have appeared are concatenated in a fixed order to form a 321-dimensional state feature vector that does not include the current candidate action; then it is concatenated with the 54-dimensional candidate action code to form a 375-dimensional action condition state feature.

[0031] For the peasant's position, the current player's hand, the sum of other players' hands, the relative position of the player who last played a valid card, the last valid card-playing action, the set of cards played by the landlord, the set of cards played by the teammate, the number of cards remaining in the landlord's hand, the number of cards remaining in the teammate's hand, and the number of bombs that have appeared are concatenated in a fixed order to form a 324-dimensional state feature vector that does not include the current candidate action; then it is concatenated with the 54-dimensional candidate action code to form a 378-dimensional action condition state feature.

[0032] The most recent 18 steps of historical card play sequence are encoded as 54-dimensional card vectors, and rearranged into 6×162 historical action sequence features in groups of 3 steps. These features are then input into an LSTM to extract temporal state representations. When the number of historical card play steps is less than 18, empty action vectors are used to pad the beginning of the sequence.

[0033] The set of legal actions is generated by the environment based on the current hand and the last valid action played. During the training phase, candidate actions are selected from the set of legal actions and action probabilities are generated. During the inference or scoring phase, a dual-value head result is output based on the above state feature vector and historical action sequence features.

[0034] Therefore, the state feature vector in this embodiment is not composed of MOBA-type features such as health, map coordinates, and equipment resources, nor is it a uniform 512-dimensional vector. Instead, it forms 321-dimensional or 324-dimensional state features based on the different roles in Dou Dizhu, and combines 54-dimensional action encoding and 6×162 historical action sequences to form the model input.

[0035] S2: A policy network model comprising a shared feature backbone network, an Actor policy network, and a dual-value-head Critic network is constructed using a proximal policy optimization algorithm. The state feature vector and historical action sequence features are input into the shared feature backbone network to obtain a state representation. The state representation is then input into the Actor policy network to output a legal action probability distribution and into the dual-value-head Critic network to output a value representation. The value representation includes the final event result representation value output by the first value head and the global final aggregation magnitude representation value output by the second value head. During the training phase, policy updates are performed based on the legal action probability distribution and the first value head output. The value loss and auxiliary supervision loss are calculated based on the first and second value heads, and joint parameter updates are performed. After training, the Actor policy network is discarded during the inference phase. like Figure 1 As shown, a policy network model is constructed using a proximal policy optimization algorithm, comprising a shared feature backbone network, an Actor policy network, and a dual-value-head Critic network. The shared feature backbone network is used to extract deep situation features; the Actor policy network is used to learn reasonable action distributions during the training phase; and the dual-value-head Critic network is used to output result value and magnitude value during the inference phase.

[0036] The state feature vector Features of historical action sequences The input is fed into the shared feature backbone network, which consists of three fully connected layers and a ReLU activation function, as expressed in the formula:

[0037]

[0038]

[0039]

[0040]

[0041] in, , and This is the network weight matrix; and These are the feature vectors of the intermediate layer; , and This is the network bias vector; This is the output state representation.

[0042] Will Input the Actor policy network and output the probability distribution of legal actions. The calculation formula for the Actor policy network is as follows:

[0043]

[0044] and These are the weights and biases of the Actor policy network, respectively. The logarithmic probability of the action; The probability distribution of legal actions is 6-dimensional, with the sum of each dimension being 1, representing the probability of choosing each action in the current state.

[0045] Actor policy network output Used to interact with the simulated game environment, sampling to generate actions to be performed. .

[0046] Next, the state will be represented. Input a dual-value-head Critic network; the dual-value-head Critic network comprises two value heads; The first value head and the second value head share the parameters of the first five fully connected layers, and only output the final result value and the final magnitude value at the final output layer, respectively. The formulas are as follows:

[0047]

[0048] in, and As the value head weight; and Value head bias; ,when This indicates a high probability of winning. This indicates a high probability of losing; This indicates an advantage; the higher the value, the greater the advantage. Indicates a disadvantage; the smaller the value, the greater the disadvantage. By continuously interacting with the simulated game environment through the Actor policy network, a complete game timeline is generated. :

[0049] in, This is the state feature vector of the game from the initial time to the final time. To correspond to the execution action sampled by the Actor policy network at each time step; The immediate reward value returned by the environment after each action is performed; This represents the total number of steps in a single game. Label the final result; This is the range label for the final result.

[0050] To address the core challenge of traditional reinforcement learning's single value head, which cannot simultaneously evaluate the dual impact of game actions on the final outcome and the magnitude of the win / loss, this invention utilizes the final outcome value output by a dual value head. and final value A two-dimensional fusion temporal advantage function specifically adapted for game anomaly detection is constructed, as shown in the following formula:

[0051] in, This is the dual-value fusion balance coefficient; and These are the final outcome value and the final magnitude value at the next moment, respectively.

[0052] Action probability distribution through the Actor policy network and dominance function This represents the relative value of the current action compared to the average state. This invention employs a pruning strategy loss function adapted to the game scenario to guide the model in learning the general behavioral patterns of ordinary players, rather than extreme optimal actions. The calculation formula is expressed as:

[0053] in, The policy loss is used to optimize the Actor policy network; Let $\mathbf{x}$ be the expected value, and $\mathbf{x}$ be the mean of For the current action; The updated policy network outputs the action probabilities; The probability of the action output by the policy network before the update; The pruning function limits the range of the probability ratio between the old and new strategies. The clipping factor is a preset hyperparameter.

[0054] Simultaneously, the value regression loss is calculated based on the output of the dual-value-head Critic network and the true labels: The first-value head regression loss is expressed as:

[0055] The second-value head regression loss is expressed as:

[0056] in, The loss is due to the reversion of the final outcome value. For the final value reversion loss; This refers to the number of training batch samples. For the first The first value head output for each sample at the current moment; For the first The second value head output of each sample at the current moment; For the first Labels for the actual endgame results of individual sample games; For the first True endgame amplitude labels for each sample game.

[0057] To achieve the goal of simultaneously detecting three core abnormal behaviors in games—negative gameplay, intentional feeding, and collusion to resell—this invention constructs a multi-objective total loss function that includes policy loss and independent loss for both value heads. A weighted additive approach is used for joint optimization, as shown in the following formula:

[0058] in, This represents the total model loss, used for backpropagation to update network parameters. The first value head loss weight; The second value head loss weight.

[0059] Minimize the total loss using gradient descent. Simultaneously update all trainable parameters of the shared feature backbone network, Actor policy network, and dual-value head Critic network.

[0060] After training, the Actor policy network no longer participates in the inference phase calculations; only the two-value-head Critic network is retained for subsequent action value evaluation and scoring calculations.

[0061] S3: Based on the trained dual-value-head Critic network, calculate the final result deviation and final magnitude deviation of the user's single-step action to generate a comprehensive score for the single-step action, and further calculate the comprehensive score for the game and the user's historical behavior score based on the comprehensive score for the single-step action. like Figure 2 As shown, based on the trained dual-value-head Critic network, the user's single-step action bias score, overall game score, and user historical behavior score are calculated.

[0062] To address the core pain point that traditional methods cannot provide interpretable and quantifiable assessments of the rationality of single-step actions by game users, this invention designs two complementary action bias calculation formulas—final outcome bias and final magnitude bias—based on a trained dual-value-head Critic network. This transforms the optimal policy value difference from reinforcement learning into a quantitative indicator of abnormal behavior in game scenarios.

[0063] Let the current state be s. After performing action a, the state transitions to the next state. ; The first value head bias (final outcome bias) is specifically used to quantify the impact of user actions on the final probability of winning or losing a game. The calculation formula is as follows:

[0064] The second value head bias (endgame magnitude bias) is specifically used to quantify the impact of user actions on the final win or loss of a game. The calculation formula is as follows:

[0065] in, For the final result dimension of action deviation; This refers to the deviation in the final amplitude dimension; Current state The first value head output; For the next state Down First value head output; Current state The second value head output; For the next state Down Second value head output; Immediate rewards for results (such as positive rewards for improving win rate by playing key card combinations and negative rewards for reducing win rate by ineffective card splitting). The immediate reward is the magnitude dimension (such as the positive reward for reducing one's remaining hand or widening the score difference, and the negative reward for giving the opponent a high-multiplier settlement opportunity).

[0066] Calculated and This is used to calculate the user's single-step action deviation score, overall game score, and user historical behavior score.

[0067] To comprehensively quantify the degree of anomaly in a user's single-step operation and address the issues of missed detection and inability to fully reflect behavioral abnormalities due to single-dimensional deviations, this invention weights and fuses final result deviation and final magnitude deviation to construct a unified comprehensive scoring formula for single-step actions. This formula enables the assessment of the rationality of user operations. The calculation formula for the user's single-step action deviation score is as follows:

[0068] in, The overall score is based on the single-step action. As the weight for the result deviation, As the magnitude deviation weight, and .

[0069] This invention, by aggregating anomalies in single-step operations to obtain the anomaly level of the entire game, designs a comprehensive game scoring formula based on state importance weighting. This formula highlights the impact of anomalies at key decision nodes while filtering out noise interference from normal operational errors. The comprehensive game score is calculated based on M state-action pairs in the game. :

[0070] in, This is a comprehensive score for the game. For the first Each state has an importance weight, with a weight of 2 for critical decision states and 1 for ordinary states. For the first Overall score for footwork.

[0071] Next, we will calculate the user's overall score for their most recent K games. :

[0072]

[0073] in, Rate users based on their historical behavior; The standard deviation of user historical behavior rating; like Figure 3 As shown, the output is the user's single-step action deviation score. Overall game score User historical behavior score Standard deviation of user historical behavior rating The data is then sent to step S4 for multi-level threshold anomaly detection and behavior determination.

[0074] S4: Based on the preset thresholds for single-game anomalies, related-game anomalies, and game currency circulation anomalies, and combined with the comprehensive score of single-step actions, the comprehensive score of games, and the user's historical behavior score, perform single-game anomaly detection, related-game anomaly detection, and game currency circulation anomaly detection in sequence to determine abnormal user gaming behavior.

[0075] The single-site anomaly detection is achieved by setting a single-site anomaly threshold. Its representation is as follows:

[0076] in, This represents the average match rating for normal users. The standard deviation of the game rating for normal users; The significance coefficient is denoted as .

[0077] The judgment rules are as follows: (1) If Marked as potentially passive play; (2) Additional threshold for abnormal amplitude If the bureau accumulates Negative deviation exceeds The game was marked as having an abnormal amplitude.

[0078] The aforementioned associated game anomaly detection is based on statistical analysis of user interactions... Total number of games played within period H and negative game number :

[0079] in, Severity of collusion; Average score for passive games; The amplitude anomaly coefficient represents the proportion of amplitude anomalies in negative games. like The system determined that a collusion anomaly existed, in which This is the threshold for associated anomalies.

[0080] The game currency flow anomaly detection is achieved by constructing a user-game currency flow graph. Its mathematical representation is:

[0081] in, For a set of user nodes, Each node Each node corresponds to a unique game user account ID, and its attributes include basic information such as the user's historical abnormal ratings, account registration duration, and login IP address. Let be a set of directed edges.

[0082] The direction of a directed edge strictly indicates the flow direction of game currency (from the sender to the receiver); This is the edge weight matrix. Indicates users within the preset statistical period H To users Record the total amount of game currency transferred, along with additional information such as the timestamp and transfer scenario for each transfer.

[0083] Construct the user-game currency flow graph Perform the following steps: (1) Extract all game currency transaction records of all users on the game platform within the preset period H, including game settlement rewards, peer-to-peer transfers between users, item trading and exchange, mall recharge and withdrawal, and guild welfare distribution, etc. (2) Filter out transaction records that are strongly associated with the “suspected negative game” or “abnormal game” marked in the previous steps. The specific filtering rules are: the transaction occurred within 24 hours after the abnormal game ended, and the transferor was a user who participated in the abnormal game. (3) Based on the filtered related transaction records, construct an abnormal game association subgraph. ,in, This refers to the set of all user nodes involved in abnormal related transactions. This is the set of directed edges corresponding to abnormal related transactions; This is the weight matrix for the corresponding edges; (4) Based on Calculate graph structure features such as node in-degree, out-degree, weighted in-degree, weighted out-degree, and node centrality for subsequent anomaly index calculation.

[0084] The abnormal circulation indicators The calculation formula is:

[0085] in, This is an indicator of abnormal circulation. To show concentration, the top 3 accounts account for the percentage of coins received; like The game currency resale was deemed abnormal, among which... This is the threshold for abnormal circulation.

[0086] Based on the combined results of the three levels of detection, the system outputs the abnormal account, abnormal type, abnormal level, and corresponding evidence chain.

[0087] like Figure 4 As shown, the present invention also provides a user abnormal game behavior detection system based on reinforcement learning, comprising: The data acquisition and preprocessing module is used to acquire user game data in real time through the game server interface, perform feature encoding and vectorization processing on the game data, and generate state feature vectors. The dual-Critic value head model module stores and deploys the trained dual-Critic value head PPO policy model. The policy model includes an Actor policy network and a dual-value head Critic network consisting of a shared value feature layer and a terminal dual-head output layer. It is used to output the final event result representation value and the final magnitude representation value, and to form a two-dimensional value representation in the anomaly scoring stage. The action scoring module is communicatively connected to the dual-critic value head model module. It is used to calculate the final result deviation and final amplitude deviation of the user's single-step action based on the output of the dual value head, and generate a single-step action comprehensive score, a game comprehensive score, and a user historical behavior score. The threshold detection module is communicatively connected to the action scoring module. It has a built-in threshold system and is used to combine various scores to perform single-game anomaly detection, associated game anomaly detection, and game currency flow anomaly detection in sequence to determine abnormal user game behavior. The early warning and response module is communicatively connected to the threshold detection module. It is used to generate structured early warning information based on the judgment result of abnormal behavior and to perform graded response operations.

[0088] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting user abnormal game behavior based on reinforcement learning, characterized in that, The method includes the following steps: S1: Collect user game data in real time through the game server interface, perform feature encoding and vectorization processing on the game data, and generate state feature vectors and historical action sequences; S2: A policy network model comprising a shared feature backbone network, an Actor policy network, and a dual-value-head Critic network is constructed using a proximal policy optimization algorithm. The state feature vector and historical action sequence features are input into the shared feature backbone network to obtain a state representation. The state representation is then input into the Actor policy network to output a legal action probability distribution and into the dual-value-head Critic network to output a value representation. The value representation includes the final event result representation value output by the first value head and the global final aggregation magnitude representation value output by the second value head. During the training phase, policy updates are performed based on the legal action probability distribution and the first value head output. The value loss and auxiliary supervision loss are calculated based on the first and second value heads, and joint parameter updates are performed. After training, the Actor policy network is discarded during the inference phase. S3: Based on the trained dual-value-head Critic network, calculate the final result deviation and final magnitude deviation of the user's single-step action to generate a comprehensive score for the single-step action, and further calculate the comprehensive score for the game and the user's historical behavior score based on the comprehensive score for the single-step action. S4: Combining the single-step action comprehensive score, game comprehensive score, and user historical behavior score, execute single-game anomaly detection, associated game anomaly detection, and game currency flow anomaly detection in sequence to determine abnormal user game behavior. 2.The method of claim 1, wherein, The Actor policy network takes the state representation output by the shared feature backbone network as input, and obtains the log probability of the action through three fully connected layers and ReLU activation function layer by layer nonlinear transformation. Then, it is normalized by the Softmax function to output the probability distribution of each legal action. 3.The method of claim 1, wherein, The dual-value-head Critic network comprises a shared value feature layer, a first value-head output layer, and a second value-head output layer. The shared value feature layer consists of five fully connected layers and is jointly invoked by the two value heads. The first value-head output layer outputs a scalar representation of the final event result based on the shared value feature layer and is subject to independent supervision by the final event result label. The second value-head output layer outputs a scalar representation of the global final aggregation magnitude based on the same shared value feature layer, regressing the global final aggregation magnitude rather than incrementally increasing local values, and is subject to independent auxiliary supervision by the global final aggregation magnitude label. 4.The method of claim 1, wherein, The formula for calculating the final result deviation of the single-step action is as follows: wherein, is the terminal outcome bias; is the current state is the output value of the first value head; is the performed action is the next state is the output value of the first value head; is the current state is the performed action is the obtained terminal outcome dimension immediate reward; wherein, is the terminal magnitude bias; is the current state is the output value of the second value head; is the executed action is the next state is the output value of the second value head; is the current state is the executed action is the obtained terminal magnitude dimension instant reward; The formula for calculating the comprehensive score of a single-step action is as follows: in, The overall score is based on the single-step action. As the weight for the result deviation, As the magnitude deviation weight, and ; The overall game score is based on a game containing M state-action pairs. in, This is a comprehensive score for the game. For the first Each state has an importance weight, with a weight of 2 for critical decision states and 1 for ordinary states. For the first Overall score for footwork; The calculation method for the user's historical behavior score is as follows: statistically analyze the comprehensive score of all games played within a preset period, calculate its mean and standard deviation to form a user behavior profile; at the same time, statistically analyze the distribution characteristics of the user's two-dimensional deviation to construct a user behavior feature vector.

5. The method for detecting abnormal user gaming behavior based on reinforcement learning according to claim 1, characterized in that, The specific method for detecting anomalies in a single game is as follows: when the overall score of a single game... Below the single-game abnormal threshold At that time, it was marked as a suspected negative game; among them, ; This represents the average overall score for a normal user's match. The standard deviation of the overall game score for normal users. The significance coefficient is used when the cumulative amplitude deviation at the end of a single game exceeds the amplitude anomaly threshold. At that time, the game is additionally marked as having abnormal amplitude.

6. The method for detecting abnormal user gaming behavior based on reinforcement learning according to claim 1, characterized in that, The aforementioned abnormal game detection specifically involves: statistically analyzing game data between pairs of users within a preset period and calculating the severity of collusion. When the severity of collusion Exceeding the associated anomaly threshold At that time, it was determined that there was serious suspicion of collusion; the severity of the collusion. The calculation formula is: in, Severity of collusion; Average score for passive games; The amplitude abnormality coefficient represents the proportion of amplitude abnormalities in negative games.

7. The method for detecting abnormal user gaming behavior based on reinforcement learning according to claim 1, characterized in that, The specific method for detecting abnormal game currency flow is as follows: By combining user anomaly rating data and game currency flow data, a user-game currency flow graph G is constructed, and the calculation formula is: in, This is an indicator of abnormal circulation. For concentration; if The game currency resale was deemed abnormal, among which... This is the threshold for abnormal circulation.

8. The method for detecting abnormal user gaming behavior based on reinforcement learning according to claim 1, characterized in that, Step S4 is followed by step S5: generating structured early warning information based on the type and severity of the user's abnormal gaming behavior, and performing tiered handling operations.

9. A user abnormal game behavior detection system based on reinforcement learning, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire user game data in real time through the game server interface, perform feature encoding and vectorization processing on the game data, and generate state feature vectors. The dual-Critic value head model module stores and deploys the trained dual-Critic value head near-end policy optimization algorithm policy model. The policy model includes an Actor policy network and a dual-value head Critic network consisting of a shared value feature layer and a terminal dual-head output layer. It is used to output the final event result representation value and the final magnitude representation value, and to form a two-dimensional value representation in the anomaly scoring stage. The action scoring module is communicatively connected to the dual-critic value head model module. It is used to calculate the final result deviation and final amplitude deviation of the user's single-step action based on the output of the dual value head, and generate a single-step action comprehensive score, a game comprehensive score, and a user historical behavior score. The threshold detection module is communicatively connected to the action scoring module. It has a built-in threshold system and is used to combine various scores to perform single-game anomaly detection, associated game anomaly detection, and game currency flow anomaly detection in sequence to determine abnormal user game behavior. The early warning and response module is communicatively connected to the threshold detection module. It is used to generate structured early warning information based on the judgment result of abnormal behavior and to perform graded response operations.

Images