Mahjong matching game level generation method based on reinforcement learning, medium and equipment

Abstract

The invention discloses a mahjong matching game level generation method based on reinforcement learning, a medium and equipment, and belongs to the technical field of machine learning, game levels are randomly generated, then game problem solving is executed based on a reinforcement learning network model, the level passing rate is obtained, and if the level passing rate is larger than or equal to a set threshold value, the game levels are output. In the reinforcement learning network model, a neural network is used as a strategy network and a value network to extract level features, and the value network cooperates and simulates the time required for playing of a real player and the level passing rate. The game level can be generated according to the preset difficulty and layout of the game, so that the game has the level design of diversified card surface types, quantity and arrangement modes under each difficulty level, and the playability and challenge of the game are remarkably improved. According to the invention, level challenge experience with invariable difficulty can be provided, and the problem of inconsistent user experience caused by inconsistent difficulty due to random generation of levels is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a method, medium and device for generating levels of a mahjong matching game based on reinforcement learning. Background Art

[0002] With the development of electronic devices and the game industry, electronic games have been widely welcomed in the market. In order to maintain the interest of players, multiple levels are usually designed in the game, and different levels have different difficulties. The different matching rules, level layouts, the number of cards and the arrangement of cards in the game will affect the difficulty and strategy of the game. However, the level design of most existing games is either completely randomly generated or manually pre-designed, which has high labor costs and cannot flexibly generate fixed levels according to specific difficulty requirements, which limits the scalability of the game and the personalized needs of the player experience. For example, the rules of mahjong matching games are usually different from traditional mahjong, mainly by pairing the same mahjong tiles to eliminate them until all the cards are cleared. The layout of the game level is artificially created and limited, but the number of cards and the arrangement of the cards in the same layout can be infinitely combined, and the layout can be reused to generate levels of different difficulties. Therefore, in the case of a limited number of layouts, there is an urgent need for a method that can effectively and efficiently generate levels with different difficulties to improve the playability and challenge of the game. Summary of the invention

[0003] The purpose of the present invention is to provide a mahjong matching game level generation method, medium and device based on reinforcement learning, aiming to solve the problem of single or fixed level difficulty setting in the game. The present invention can generate game levels according to the preset difficulty and layout of the game, and efficiently reuse the level layout, so that the game has a level design with diversified card types, quantities and arrangements at each difficulty level, which significantly improves the playability and challenge of the game. The present invention can provide a level challenge experience with constant difficulty, avoiding the problem of inconsistent user experience caused by the inconsistent difficulty caused by randomly generated levels.

[0004] The present invention is mainly achieved through the following technical solutions:

[0005] The mahjong matching game level generation method based on reinforcement learning includes the following steps:

[0006] Step S1: randomly generating game levels according to set game difficulty parameters;

[0007] Step S2: Execute the game problem solving based on the reinforcement learning network model and obtain the level passing rate. If the level passing rate is greater than or equal to the set threshold, output the game level;

[0008] Step S21: Building a reinforcement learning network model, and using a neural network as a strategy network and a value network to extract level features and coordinate and simulate the time required for real players to play and the level passing rate through the value network;

[0009] Step S22: Use the strategy network to simulate the player to perform multiple action outputs; identify the simulated player's game clearance results associated with the control object, as well as the conditions encountered by the control object, identify the possible actions taken by the control object, and calculate the behavior-based probability value of each possible action based on the success rate of the action in historical scenarios, the similarity between the current state and the historical success state, and the expected immediate reward value; then, combine the value function V(s) to evaluate the long-term cumulative reward that the action may bring in the future; the decision to select the action is based on the strategy π, and the strategy π selects the action according to the behavior-based probability value and the evaluation value of the value function V(s), and commands the control object to perform the action in the game;

[0010] Step S23: accepting input of the strategy network in the background of the game interface; discretizing the game screen space into M×N grid points, each grid point represents an interactive position, the output layer of the neural network corresponds to the discrete position point, and each output node represents the probability of performing a click / touch operation at the position;

[0011] Step S24: setting a reward function for the reinforcement learning network model; the reward function of the reinforcement learning network model includes an extrinsic reward and an intrinsic reward; the intrinsic reward is generated by a self-supervised curiosity model in the Unity ML-agents framework to motivate the artificial intelligence neural network to explore unknown game states and behavior patterns;

[0012] Step S25: configuring the training program environment and training the policy network;

[0013] First, it interacts with the environment to collect experience data until the buffer is filled, and then randomly samples batches of 512 from the buffer for learning; each time the policy is updated, the model calculates the ratio of the new and old policies and constrains the update amplitude based on the clipping range, while combining the policy entropy term to promote exploration;

[0014] Step S26: training the value network, and using the value network to output features for the level;

[0015] The value network is trained based on minimizing the loss function between the predicted value and the actual value. The training data includes the feature information of the existing levels and the corresponding player feedback or expert ratings. The trained value network evaluates the new level design and predicts the potential player response and game experience quality.

[0016] The value network performs dimension mapping and preliminary feature extraction on the input level features through a fully connected layer; uses a one-dimensional convolutional layer to capture local patterns and spatial correlations in the feature sequence; then introduces a long short-term memory network layer to model the temporal dependencies of the feature sequence and capture the long-term structural features in the level design; finally, all the extracted feature information is integrated through a fully connected layer to output the level value score, i.e., the level pass rate.

[0017] Specifically, in step S22, the simulated player play results associated with the control object and the conditions encountered by the control object are identified; based on the conditions and the model parameter state, the possible actions of the control object are identified, and multiple behavior-based probability values ​​are generated based on the stored action inputs. The behavior-based probability value of each possible action is calculated based on the three factors of the success rate of the action in historical scenarios (40% weight), the similarity between the current state and the historical success state (30% weight), and the expected instant reward value (30% weight), and then the behavior-based probability values ​​of all actions are divided by the total for normalization to obtain the final action probability distribution.

[0018] The model parameter state refers to all weights and bias values ​​stored in the current network. These parameters are continuously trained and updated by simulating the player's game data. Specifically, it contains the game state information received by the input layer, the feature representation learned in the hidden layer (such as behavior patterns under different conditions, state transition rules, etc.), and the action value predicted by the output layer. When the controlled object encounters a certain condition, these trained parameters can quickly calculate the optimal action selection based on the current input state. The parameter state of the network actually records the empirical knowledge learned by the model from historical game data to guide new decision-making.

[0019] The decision to select an action is based on a strategy π, which guides the selection of several optimal actions in a given state, with the goal of maximizing the total reward; each state is associated with a value function V(s), which predicts the expected amount of several future rewards obtained by executing the corresponding strategy in a given state, and the future reward is the sum of a series of forward discounted rewards. Behavior-based probability value selection is a specific implementation of strategy π. Specifically, when making a decision, strategy π first calculates the behavior-based probability value of each possible action (based on historical success rate, state similarity, and immediate reward expectations), and then combines the value function V(s) to evaluate the long-term cumulative rewards that these actions may bring in the future. The final action selection probability is a weighted integration of the two parts. The behavior-based probability value reflects the short-term value of the action, while the value function takes into account the long-term benefits, which not only ensures the immediate rationality of the action selection, but also ensures the maximization of the long-term benefits. In this way, the system can select the optimal action strategy in the current state.

[0020] In order to better implement the present invention, further, in step S21, the neural network includes a 4-layer structure, and each layer is configured with 512 hidden units; during the forward propagation process of the neural network, the input data passes through the hidden units in turn for feature transformation and extraction, and each layer of the neural network enhances the expression ability of the network through a nonlinear activation function.

[0021] In order to better implement the present invention, further, in step S23, the neural network calculates the operation probability distribution of each grid point according to the current game state, and then selects the optimal interaction position based on the operation probability distribution, and performs corresponding click and touch operations at the selected grid point position, thereby simulating the touch screen operation mode of a real player; dynamically adjusts the grid density according to the game type, and adds additional action attributes of click duration and multi-touch to each grid point.

[0022] In order to better implement the present invention, further, in step S24, the extrinsic reward includes a positive reward or a negative penalty, a linear reward given according to the proportion of completed goals, a phased reward calculated based on the percentage of unlocking progress, a fixed negative reward for punishing the model for using more than the required number of steps, and a click reward, and the click reward is calculated by a distance-based decay function.

[0023] In order to better implement the present invention, further, the calculation formula of the click reward is:

[0024]

[0025] Where: r distance is the normalized distance between the click position and the nearest target;

[0026] d is an adjustable weight parameter used to control the impact of distance on rewards;

[0027] c0 is the base reward value that determines the initial strength or magnitude of the reward;

[0028] c1 controls the speed at which the reward decays with distance. The larger c1 is, the slower the decay and the more gradually the reward decreases with distance.

[0029] In order to better implement the present invention, further, in step S25, a maximum training limit of 5 billion steps is set during the training process, and every 1000 steps is regarded as a complete time horizon cycle, during which experience data of interaction with the environment is continuously collected, and reward calculation and strategy update are performed at the end of the cycle; a model checkpoint is automatically saved for every 500,000 cumulative training steps, and the five most recent checkpoint files are always kept; when the model performance reaches the expected level or reaches the maximum number of training steps, the training is ended and the final model is saved.

[0030] In order to better implement the present invention, further, in the step S25, when the model initially performs exploratory learning and does not know where to click, the click reward provides additional guidance for clicking only valid targets; a batch size of 512 is used to update the neural network, and an experience replay buffer of 20480 is maintained to store interaction data; a learning rate of 0.0002 is used during training to ensure stable convergence, and an entropy coefficient of 0.5 is used to balance exploration and utilization; a clipping range parameter of 0.1 is used to limit the amplitude of policy changes, and a generalized advantage estimation parameter of 0.95 is used to optimize the accuracy of advantage estimation; after each complete batch of data is collected, 3 rounds of policy updates are performed to make full use of the sampled data.

[0031] In order to better implement the present invention, further, in step S25, the policy network and the value network both calculate the action prediction of the current state and generate a time difference error signal at each time step; the input of the policy network is the current state, and the policy network outputs a value representing an action derived from the action space; the input of the value network is the level state, the return feedback of the policy network or an approximation of the predicted action. The policy network outputs the probability distribution of various actions and the expected return from the current position; the value network outputs several values ​​for each state-action pair.

[0032] In order to better implement the present invention, further, step S1 includes the following steps:

[0033] Step S11: Obtaining game difficulty parameters, the game difficulty parameters include level layout data, the number of mahjong tiles and level generation input data, the level generation input data includes difficulty level, special card layout structure type, card type total coefficient and pass rate;

[0034] Step S12: randomly selecting a specified number of card types from the 36 card types, where the number of card types = (total number of mahjongs in the level layout / 4)*total number of card types coefficient; randomly allocating the number of cards of each type according to the number of mahjongs, and satisfying that the number of each card type is an integer multiple of 2;

[0035] Step S13: generating corresponding level layout data, mahjong quantity and card quantity structure in the level data according to the special card arrangement structure type;

[0036] Step S14: Mark the mahjong corresponding to the structure generated in step S13 as cleared, and the remaining mahjong is randomly matched with the remaining cards.

[0037] In order to better implement the present invention, further, in the step S13, if the special arrangement structure type of the card faces is two identical card faces overlapping or horizontally blocking, the following steps are included:

[0038] a. Randomly select a type of card A with 4 cards, randomly select a Mahjong M1 in the level layout, obtain all the Mahjong sets MS1 that overlap with Mahjong M1, and randomly select a Mahjong M2 from the set MS1;

[0039] b. Randomly obtain all mahjongs in a layer below M1, and exclude all mahjongs overlapping with mahjong M1 and mahjong M2, calculate the distance between the remaining mahjongs and mahjong M1 on the layout, and obtain mahjong M3 with the largest distance;

[0040] c. Randomly obtain all mahjongs in a layer below M1, and exclude all mahjongs overlapping with mahjong M1 or mahjong M3, as well as mahjong M2 and mahjong M3, calculate the distances between the remaining mahjongs and mahjong M1 and mahjong M3 respectively, and add them up to obtain mahjong M4 with the maximum sum of distances;

[0041] d. Mark Mahjong M1 to Mahjong M4 as the selected card A.

[0042] In order to better implement the present invention, further, in the step S13, if the special arrangement structure type of the card faces is two identical card faces overlapping and horizontally blocking, the following steps are included:

[0043] a. Traverse the mahjong of each layer of the current layout, find the mahjong with no obstruction on the left and obstruction on the right, and use this as the starting point to recursively search for all blocking mahjong sequences, and satisfy the sequence length greater than or equal to 4, to generate the set sequence SL1;

[0044] b. Randomly select a sequence L1 from the set sequence SL1, randomly select 4 mahjongs from the sequence L1, and sort them from left to right according to the horizontal coordinate of the layout to generate a mahjong sequence L2;

[0045] c. Randomly select 2 cards with a number of 4 from the card number set, and randomly use AABB or ABAB arrangement order to correspond one-to-one with the mahjong sequence L2;

[0046] d. Obtain all mahjongs from the upper layer to the top layer of mahjong sequence L2, search for mahjongs that satisfy all the overlapping mahjongs below it, and do not include all mahjongs that have been selected before, and obtain 4 mahjong sequences L3;

[0047] e. Mahjong sequence L3 randomly distributes two A and two B cards.

[0048] In order to better implement the present invention, further, in the step S13, if the special card arrangement structure type is chain overlapping blocking, the following steps are included:

[0049] a. Set the chain structure to a random length of 4 to 7, or to two lengths of 4;

[0050] b. Obtain a set MS2 of two-by-two overlapping mahjong combinations of a specified chain length, wherein the two-by-two overlapping mahjong combinations cannot overlap each other;

[0051] c. Randomly obtain 4 cards of A, and 2 cards of the chain length minus one. The mahjong combination with overlapping cards of A is the solution of the chain. Card A should choose the combination with the lowest level in set MS2. The remaining cards correspond to mahjong one by one according to the rules of chain overlapping and blocking.

[0052] A computer-readable storage medium stores a computer program, which implements the above method when executed by a processor.

[0053] An electronic device comprises a memory and a processor; the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the above method.

[0054] The beneficial effects of the present invention are as follows:

[0055] The present invention efficiently reuses the level layout, so that the game has a level design with diversified card types, quantities and arrangements at each difficulty level, which improves the playability and challenge of the game. It improves the efficiency of level generation and reduces manual operations. It provides players with an unchanged level challenge experience at the same difficulty level, avoiding the problem of inconsistent difficulty and incoherent experience caused by fully randomly generated levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flowchart of the method for generating game levels based on reinforcement learning of the present invention;

[0057] Figure 2 This is a graph showing the change in the reward and punishment scores of the model in the first 30 million training iterations;

[0058] Figure 3This is a curve chart showing the change in the duration of a single iteration of the model in the first 30 million training iterations;

[0059] Figure 4 Schematic diagram of the model structure of the strategy network;

[0060] Figure 5 A statistical graph showing the passing rate of the online test level of the present invention;

[0061] Figure 6 This is a flow chart of Example 3. DETAILED DESCRIPTION

[0062] Embodiment 1:

[0063] A mahjong matching game level generation method based on reinforcement learning generates level data according to game setting parameters, obtains the pass rate based on the reinforcement learning network model; if the pass rate meets the setting, the formatted data is saved to the storage space for subsequent output. The game setting parameters include difficulty level, special card arrangement structure, card ratio, and pass rate.

[0064] The main purpose of the present invention is to learn a behavior strategy that can maximize the cumulative reward, extract the level features, and simulate the time required for real players to play and the level pass rate through the value network. The present invention learns the optimal strategy through direct interaction between the agent and the game environment, using neural networks as the strategy network and value network. The agent continuously explores new actions during the game and stores the experience in the playback buffer. Through policy gradient and temporal difference learning, the agent can gradually improve its decision-making ability and strike a balance between exploration and exploitation.

[0065] The invention includes core components such as game environment interface, neural network processor and experience replay memory. The system accumulates experience through continuous self-play and uses advanced algorithms such as double Q learning to optimize model performance. The training controller is responsible for dynamically adjusting the exploration rate and regularly evaluating the model to ensure that the game strategy is mastered through trial and error and self-improvement in the absence of or with little human data.

[0066] Preferably, if Figure 1 As shown, the present invention specifically comprises the following steps:

[0067] a. Build a reinforcement learning network model: During the network construction phase, normalization is first enabled for the input features to standardize all input data to the same numerical range to improve training stability.

[0068] The neural network adopts a 4-layer structure, with 512 hidden units in each layer, forming a feature extractor with sufficient depth. During the forward propagation of the network, the input data passes through these hidden layers in turn for feature transformation and extraction, and each layer enhances the network's expression ability through nonlinear activation functions. A layer of the network contains several neurons. Each neuron has several inputs and produces a single output that can be sent to multiple other neurons (signals).

[0069] The inputs are feature values ​​of the game data, or they can be the outputs of other neurons. The output of the final output neuron implements the desired / configured task. Neurons and edges may also have weights that adjust as learning proceeds. Weights increase or decrease the strength of the signal on the connection. Neurons can have a threshold so that they only send a signal when the aggregate signal exceeds this threshold.

[0070] Neurons can be aggregated or grouped into several layers L, and different layers L can perform different transformations on their inputs. Signals propagate from the first layer (input layer L1) to the last layer (output layer Ly), possibly arriving after passing through layers La, Lb, and Lc multiple times. The input layer La receives data from input variables xi (where i=1,...,p, where p is a number). Hidden layers La, Lb, and Lc process the input xi, and finally the output layer Ly provides output variables yj (where j=1,...,p', where p' is equal to p or a different number). The output variable yj can exist in the form of a decision, inference, prediction, and / or evaluation. The input variable xi can be specified as a vector containing relevant data (e.g., observations, ML features, etc.). In addition or in addition, the output variable yj can be specified as a vector containing relevant data (e.g., decisions, inferences, predictions, scores, etc.). Neural networks have fixed input and output dimensions. In a multi-access network, the number of traffic flows with several transmissions and / or different requirements varies over time. If a neural network is implemented whose input and output dimensions depend on the number of active flows, then a neural network designed for a certain number of active flows cannot be used when the number of flows changes.

[0071] b. Use a reinforcement learning network model to simulate a player's multiple action outputs: This method is associated with a model player and receives multiple action inputs that are related to the game played by the virtual model player on the device. Receive a simulated model play indication, indicating that the virtual player is controlling. Identify the simulated player's play results associated with the control object previously controlled by the virtual player, and identify the conditions encountered by the control object. Based on the condition and the model parameter state, identify multiple actions that the control object may take, and generate multiple behavior-based probability values ​​based on the stored action inputs, each of which is used to identify how likely it is that each of the multiple possible actions will be executed under the condition and model parameter state.

[0072] The method also includes selecting an action based on multiple behavior-based probability values ​​and commanding the controlled object to perform the action in the game. The decision to select action a is based on a policy π, which provides guidance on taking several optimal actions in a given state, with the goal of maximizing the overall reward. Each state is associated with a value function V(s) that predicts the expected amount of several future rewards that can be obtained in the given state by executing the corresponding policy. The "reward" for an action is the future reward that the model can obtain after performing the action. Future rewards or future benefits can be calculated as the sum of a series of forward discounted rewards. An optimal value function produces a maximum benefit, and an optimal policy produces an optimal value function.

[0073] In some implementations, a policy π is a mapping from a state s to a number of probabilities for choosing each possible action a in that state s. A reward r is a numerical value that the model gets from the environment directly as a response to the model's action a. The model attempts to maximize the total reward it gets during an epoch or episode (where an episode refers to all states that occur between an initial state and a final state), so the reward r is the incentive the model needs to act in a desired manner. All actions produce rewards r, which can be broadly divided into three types, including positive rewards that emphasize desired actions, negative rewards that discount specific actions, and zero rewards that represent neutral actions.

[0074] c. Accept the input of the strategy network in the background of the game interface: discretize the game screen space into M×N grid points to simplify the action space. Each grid point represents a possible interaction position. The output layer of the neural network corresponds to these discrete position points, and each output node represents the probability of performing a click / touch operation at that position.

[0075] During actual operation, the neural network will calculate the operation probability distribution of each grid point according to the current game state, and then select the optimal interaction position based on this probability distribution, and can perform corresponding click, touch and other operations at the selected grid point position, thereby simulating the touch screen operation mode of real players. The system can also dynamically adjust the grid density according to the game type, and add additional action attributes such as click duration and multi-touch to each grid point, in this way, it can accurately simulate the touch behavior of human players.

[0076] d. Set the reward function for the model and prepare the training strategy: Figure 2 and Figure 3 As shown in the figure, the reward function for the model combines the extrinsic reward and the intrinsic reward generated by the self-supervised curiosity model implemented in Unity ML-agents, and adopts a composite reward mechanism that combines extrinsic rewards and intrinsic rewards to guide model learning. The extrinsic reward is calculated as the sum of multiple reward components: victory reward or failure penalty, the proportion of cleared targets, the percentage of unlocked progress, a small constant negative reward used to punish the model for using more than the required number of steps, and a click reward. Specifically, the click reward is calculated using a distance-based decay function, where the click reward calculation formula is:

[0077]

[0078] where r distance represents the normalized distance between the click position and the nearest target, and d is an adjustable weight parameter used to control the impact of distance on the reward.

[0079] Intrinsic rewards are generated through a self-supervised curiosity model in the Unity ML-agents framework, which incentivizes AI neural networks to explore unknown game states and behavior patterns.

[0080] Preferably, the extrinsic reward consists of multiple components: a positive reward (e.g., +100) is given when the artificial intelligence neural network wins, and a negative penalty (e.g., -50) is imposed when it fails; a linear reward is given based on the proportion of completed goals (e.g., 50 points for 50% completion); a phased reward is calculated based on the percentage of unlocking progress; and a smaller fixed negative reward (e.g., -0.01) is introduced to encourage the model to find the optimal path and avoid unnecessary operation steps.

[0081] Through this carefully designed reward function, the model can learn efficient and game-strategic operational behaviors while maintaining exploratory nature. In terms of reward mechanism, the system uses a discount factor of 0.99 to balance short-term and long-term benefits, while setting the external reward signal strength to 1.0 to maintain the importance of the original reward. This structural design enables the model to effectively learn the deep feature representation of the game state and optimize the long-term strategy through a reasonable reward discount mechanism.

[0082] e. Configure the training program environment: A maximum training limit of 5 billion steps is set during the training process to prevent waste of resources caused by overtraining. Every 1,000 steps is regarded as a complete time horizon cycle, during which the system continuously collects experience data of the interaction between the agent and the environment, and performs reward calculation and strategy update at the end of the cycle. To ensure the traceability and fault tolerance of the training process, the system automatically saves a model checkpoint for every 500,000 cumulative training steps, while always keeping the five most recent checkpoint files, so that it can not only fall back to the previous stable state when training anomalies occur, but also select models at different stages for comparison and verification when necessary. Through this strict training management mechanism, the model can minimize the risk of training interruption caused by unexpected factors while ensuring training continuity, making the entire training process more robust and controllable. When the model performance reaches the expected level or reaches the maximum number of training steps, the system will automatically end the training and save the final model.

[0083] f. Start training the model: When the model initially explores and learns and does not know where to click, the click reward provides additional guidance to click only valid targets. Use a batch size of 512 for neural network updates, and maintain an experience replay buffer of 20480 to store interaction data. During training, a learning rate of 0.0002 is used to ensure stable convergence, and an entropy coefficient of 0.5 is used to balance exploration and exploitation. To prevent the policy update from being too aggressive, a clipping range parameter of 0.1 is used to limit the amplitude of policy changes, and a generalized advantage estimation parameter of 0.95 is used to optimize the accuracy of advantage estimation.

[0084] After each complete batch of data is collected, the model performs three rounds of policy updates to make full use of the sampled data. The specific execution process is as follows: the agent first interacts with the environment to collect experience data until the buffer is filled, and then randomly samples batches of 512 from the buffer for learning. Each time the policy is updated, the model calculates the ratio of the new and old policies and constrains the update amplitude based on the clipping range, while combining the policy entropy term to promote exploration. Through multiple rounds of policy iteration updates, the model gradually optimizes its behavior strategy until the expected performance level is achieved.

[0085] g. The value network is used to intelligently evaluate and predict game levels. The value network is trained by minimizing the loss function between the predicted value and the actual value. The training data includes the feature information of existing levels and the corresponding player feedback or expert ratings. After training, the network can automatically evaluate new level designs and predict potential player reactions and game experience quality. This evaluation result can directly guide the optimization process of level design, help designers quickly identify and improve potential problems in the level, and thus improve the overall game quality.

[0086] In this way, the value network forms an efficient automated quality assessment and feedback mechanism during game development. Both the policy network and the value network compute action predictions for the current state and generate a time-difference error signal at each time step. The input to the policy network is the current state, and the policy network outputs a value representing an action derived from the action space. The input to the value network is the level state, and the policy network returns feedback or an approximation of the predicted action. Feedback is a score, ranking, rating, evaluation, quality, indicating the appropriateness or superiority of the predicted action relative to other potential actions. Feedback can be a state value from a state-value function (such as V(s)), or a quality value ("Q-value" or "action value") from a Q-value function (such as Q(s, a)). The policy network outputs a probability distribution over various actions and the expected reward from the current position. The value network outputs several values ​​for each state-action pair. In either case, the policy network attempts to improve its policy based on the approximation provided by the value network. In other words, the policy network determines action a, and the value network evaluates all actions a and determines adjustments to the action determination / learning process.

[0087] h. The value network performs dimension mapping and preliminary feature extraction on the input level features through the fully connected layer; uses a one-dimensional convolutional layer to capture local patterns and spatial correlations in the feature sequence; then introduces a long short-term memory network layer to model the temporal dependencies of the feature sequence, effectively capturing the long-term structural features in the level design; finally, all the extracted feature information is integrated through the fully connected layer to output a comprehensive level value score, i.e., the level pass rate.

[0088] Preferably, if Figure 4 As shown, the present invention defines the structure of the policy network, which is composed of a model including the output of the model in step b and the input of the model in step c through multi-layer general matrix multiplication and logic functions, and is particularly suitable for reinforcement learning or policy generation scenarios. From input action masks (action_masks) to the final output discrete actions (discrete_actions), the following steps are specifically included:

[0089] 1. Input: action mask action_masks

[0090] Function:

[0091] The input action mask is used to mark which actions are enabled and which are disabled.

[0092] The action mask is a constraint on the action selection process, ensuring that the network does not output illegal actions.

[0093] 2. Left branch: mask adjustment and transformation

[0094] This part processes the input action mask and performs a series of numerical adjustments on it in order to provide a more explicit calculation result for the action constraint.

[0095] 2.1 Matrix Transformation Operations

[0096] Mul(B=-1): Multiply the matrix value by -1 to get a negative value.

[0097] Add(B=1): Add 1 to the result of the previous step to adjust the numerical range.

[0098] Mul(B=100000000): further amplifies the result value, possibly to strengthen the constraint so that the value of the invalid action has a significant advantage in subsequent calculations.

[0099] 2.2 Output

[0100] After these transformations, a revised action mask matrix is ​​generated, which will be combined with the result of the right branch in the subsequent Sub operation.

[0101] 3. Right branch: network reasoning calculation

[0102] This part is the main neural network inference path, which is used to calculate the probability distribution of actions.

[0103] 3.1Gemm (General Matrix Multiplication)

[0104] First layer:

[0105] Input matrix B (512×512), output channel C=512.

[0106] Represents a fully connected layer for extracting features.

[0107] The Sigmoid activation function maps the output to the (0,1) range and adds a nonlinear transformation.

[0108] Second layer:

[0109] The matrix multiplication operation is performed again, the input dimension remains unchanged, and the output channel is still C=512.

[0110] The Sigmoid activation function is repeatedly activated to further enhance the nonlinear expression ability.

[0111] Third layer:

[0112] Finally, after a layer of matrix multiplication, the output is reduced to B(30×512), and the dimension of the action space is 30.

[0113] Each line of the output corresponds to the weight value of an action.

[0114] 3.2 Output

[0115] The output of this branch is the weight of each action, which will be combined with the action mask results from the left branch to determine the probability distribution of legal actions.

[0116] 4.Sub (matrix subtraction)

[0117] Subtract the results of the left and right branches:

[0118] Left branch: The adjusted action mask indicates which actions are invalid.

[0119] Right branch: The inferred action weights indicate the priority of the actions.

[0120] Function:

[0121] The action mask constraint is imposed on the action weights output by the network so that the weights of invalid actions are suppressed or zeroed.

[0122] 5. Action Selection Strategy

[0123] Process the output of Sub, generate the final probability distribution of actions, and select specific actions based on different strategies.

[0124] 5.1 Softmax

[0125] Convert the result of Sub into a probability distribution so that all weight values ​​are normalized to 0 to 1 and the sum is 1.

[0126] Each action corresponds to a probability value, indicating the likelihood of it being selected.

[0127] 5.2 Two Action Selection Methods

[0128] Random selection: Log+Multinomial

[0129] Log: Takes the logarithm of a probability distribution, which may be used to compute values ​​that are more consistent with sampling from a multinomial distribution.

[0130] Multinomial: Random sampling based on the Softmax probability distribution to obtain an action.

[0131] Randomness: This approach introduces an element of exploration, allowing the model to choose non-optimal actions to achieve exploration.

[0132] Deterministic selection: ArgMax;

[0133] Directly select the action with the highest probability in the Softmax probability distribution.

[0134] Determinism: This method completely relies on the current network output, has no randomness, and is a greedy strategy.

[0135] 5.3Concat

[0136] Combine randomly selected actions and deterministically selected actions together to form a more complete decision information.

[0137] 6. Output: discrete_actions

[0138] content:

[0139] discrete_actions consists of two parts:

[0140] Deterministic action: The optimal action obtained by ArgMax.

[0141] Random actions: Exploratory actions obtained by Multinomial sampling.

[0142] Function:

[0143] Support different decision-making needs:

[0144] Deterministic actions are suitable for either the testing phase or the execution phase.

[0145] Random actions are applied during the training phase for policy exploration.

[0146] Specifically, the main process is:

[0147] Action mask restriction: filter illegal actions to ensure valid output.

[0148] Network reasoning: Use a three-layer fully connected network to extract features and generate action weight distribution.

[0149] Probability distribution calculation: Combine the action mask and inference results to calculate the Softmax probability distribution.

[0150] Action selection: Output actions through two methods: random sampling and greedy strategy.

[0151] Applicable scenarios:

[0152] Policy networks in reinforcement learning tasks, such as robot control, game AI, etc.

[0153] You can use random actions (exploration) during training and deterministic actions (exploitation) during inference.

[0154] This process emphasizes the importance of the input action mask in the action selection process.

[0155] The present invention solves the problem of single or fixed level difficulty settings in existing mahjong matching games, and provides a method for generating elementary, intermediate, advanced, and expert levels according to preset difficulty and layout, so that the game has diversified card types, quantities, and arrangements at each difficulty level, and the level experience at the same difficulty level fluctuates less, thereby improving the playability and challenge of the game. Improve the efficiency of level generation and reduce manual operations. Figure 5 As shown, the present invention has been tested by nearly 100,000 online players, and 100 elementary, intermediate, advanced and expert levels each are generated by the present invention. The pass rate of each difficulty level is obviously layered and meets the level requirements.

[0156] Embodiment 2:

[0157] The mahjong matching game level generation method based on reinforcement learning includes the following steps:

[0158] 1. Definition of terms:

[0159] Card types: There are 36 types of patterns, including 1-9 Wan, 1-9 Tong, 1-9 Tiao, East, West, South, North, Center, White, Spring, Summer, Autumn, Winter (grouped into one category), Plum, Orchid, Bamboo, Chrysanthemum (grouped into one category);

[0160] Card face: a certain type of card face, that is, a specific pattern;

[0161] Blockage: When a piece of mahjong is touched by other mahjong on the left and right, or when the mahjong is (partially) covered (the covered mahjong can be any layer above the mahjong in the layout), the mahjong is blocked;

[0162] Blocker: A blocker is a tile that blocks another tile, either directly or indirectly by blocking other blocking tiles;

[0163] Free: A piece is free when it is not blocked.

[0164] 2. Determine the difficulty level and divide the level difficulty into 0, 1, 2, and 3, namely beginner, intermediate, advanced, and expert levels;

[0165] 3. Each level corresponds to different difficulty parameters and layout designs. The specific difficulty can be set according to the following dimensions:

[0166] a. Total number of mahjongs in level layout: the total number of mahjongs used in the level layout. Primary level layouts use fewer mahjongs, while advanced levels use more mahjongs.

[0167] b. Complexity of deck layout: Primary levels use a relatively simple matrix arrangement or single-layer layout, while advanced levels use multi-layer, irregular stacking, which increases the difficulty of unlocking and matching;

[0168] c. Total number of card types in a level: the total number of card types used in a level, which in the present invention is represented by the total number of mahjong tiles in a level layout / 4*the coefficient of the total number of card types in a level (in the range of 1 to 2);

[0169] d. Pass rate (reference value): The mahjong matching elimination problem can be regarded as a type of problem called "path elimination problem". For such a problem, the most natural way to think about it is to try to find the number of steps to solve it, or to determine whether the game has a solution. In computer science, similar problems can often be classified as NP (Nondeterministic Polynomial time) problems. Especially in its most complex form, the solution to this type of problem cannot be efficiently solved by a known polynomial time algorithm. The pass rate of the present invention refers to the percentage of the number of times all mahjongs are successfully eliminated after executing a specific solution 1,000 times. Therefore, this value is not the actual level pass rate, but only a reference value (technical personnel in this technical field should be clear). In the present invention, the higher the level of the level, the lower the pass rate, and the reverse is not true.

[0170] e. Special arrangement structure of cards: Because the total number of mahjong tiles, the complexity of the deck layout, the types of cards, and the pass rate are related to the level difficulty but not in a linear relationship (technical personnel in this technical field should be aware of this), and the level difficulty is an experiential issue, the special arrangement structure of cards defined in the present invention determines the basic classification of the level difficulty.

[0171] 4. The special layout structure of the cards includes:

[0172] a. Two identical cards (A1, A2) overlap, and the other two do not overlap (A3, A4), and A1, A3, and A4 are the farthest from each other in the layout. Overlapping situations include two identical cards completely overlapping, partially overlapping (half or quarter overlapping), or indirectly overlapping (there are multiple blockers between the two identical cards and they are all overlapping);

[0173] b. Horizontal blocking structure: There is a horizontal blocking structure of AABB or ABAB, and other cards except A and B can be inserted in the middle;

[0174] c. Chain overlapping blocking structure: The entire chain card layout structure can be eliminated only when specific blocks are eliminated, such as A1B1B2C1 C2D1 D2A2. The maximum length of the chain is related to the layout. Due to layout restrictions, a chain of 5 to 7 or two chains of 4 are selected. For example, there are 4 A cards, 2 B, C, and D cards in the level, A1 overlaps on B1, B2 overlaps on C1, C2 overlaps on D1, and D2 overlaps on A2. Only when A1 is matched with A3 or A4 and eliminated first, the entire chain can be eliminated.

[0175] 5. Define the range of each difficulty parameter as shown in Table 1.

[0176] Table 1

[0177] Difficulty level Special layout structure type Total card type coefficient Pass Rate 0 none 1.2-1.5 0.9-1 1 2 identical cards overlap or block horizontally 1.3-1.5 0.7-0.8 2 2 identical cards overlap and block horizontally 1.3-1.5 0.3-0.6 3 Chain Overlap Blocking 1.3-1.5 0.1-0.3

[0178] 6. Get the level layout data and the number of mahjong tiles.

[0179] 7. Obtain the input data difficulty level, special card layout structure, card ratio, and pass rate for level generation;

[0180] 8. Randomly select a specified number of tile types from the 36 tile types (total number of mahjong tiles / 4*ratio of tile types);

[0181] 9. Randomly allocate the number of cards of each type according to the number of mahjong tiles, so that the number of cards of each type is an integer multiple of 2;

[0182] 10. According to the input card layout structure type, level layout data, number of mahjong tiles, and number of cards, the corresponding structure is generated in the level data:

[0183] a. 2 identical cards stacked:

[0184] Randomly select type A with 4 cards, randomly select a Mahjong in the level layout (excluding the top layer) as M1, obtain all the Mahjong sets MS1 (including all layers below it) that overlap with M1, and randomly select a Mahjong M2 from MS1;

[0185] Randomly obtain all mahjongs in a layer below M1, exclude all mahjongs overlapping with M1, exclude M2, calculate the distance between the remaining mahjongs and M1 on the layout, and obtain the mahjong M3 with the maximum distance;

[0186] Randomly obtain all mahjongs in a layer below M1, and exclude all mahjongs overlapping with M1, exclude all mahjongs overlapping with M3, exclude M2 ​​and M3, calculate the sum of the distances between the remaining mahjongs and M1 and M3, and obtain the mahjong M4 with the maximum distance;

[0187] Mark M1, M2, M3, and M4 as the selected card A.

[0188] b. Horizontal blocking:

[0189] Traverse each layer of mahjong in the current layout (excluding the top layer), find the mahjong that is not blocked on the left and blocked on the right, and use this as a starting point to recursively search for all blocking mahjong sequences that satisfy the sequence length greater than or equal to 4, generate a set sequence SL1, randomly select a sequence L1 from SL1, randomly select 4 mahjongs in the selected sequence, generate L2 according to the horizontal coordinate of the layout and sort from left to right, randomly select two card faces (the number of card faces is equal to 4) and correspond them one by one to the mahjong in the card face order of AABB or ABAB.

[0190] Get all mahjongs between the upper layer and the top layer of L2, and find mahjongs that satisfy all overlapping mahjongs below and do not contain all mahjongs that have been selected before. According to this condition, 4 mahjongs are obtained, and two of the 4 mahjongs are randomly matched to the cards A and B.

[0191] c. Chain overlapping blocking:

[0192] A set S1 of two overlapping mahjong combinations of a specified chain length is obtained. Two overlapping mahjong combinations cannot overlap each other. Four card faces A are obtained randomly. Two card faces are the chain length minus one. The mahjong combinations with overlapping card faces A are the solutions to the chain. Card face A should choose the combination with the lowest level in S1. The remaining card faces correspond to mahjong one by one according to the rules of chain overlapping blocking.

[0193] 11. Mark the mahjong of a specific structure as cleared, and the remaining mahjong will be randomly matched with the remaining cards. The matching methods include:

[0194] a. Obtain all currently free mahjong pieces AL1 according to the layout, and calculate the selection weight of each mahjong piece, which is calculated by dividing the total number of mahjong pieces blocked by the mahjong piece in the horizontal direction by 2 and the total number of mahjong pieces blocked by the mahjong piece in the vertical direction;

[0195] b. Weighted random selection of 1 to 5 pairs of mahjong (try to ensure that the number of mahjong pairs that can be matched during the elimination process is between 1 and 5) and marking each pair of mahjong with a randomly selected card and marking it as cleared;

[0196] c. Repeat steps a and b until all mahjong tiles are marked.

[0197] 12. According to the generated level data, execute the solution 1000 times and obtain the pass rate;

[0198] 13. If the pass rate meets the requirement, format the data and save it to the storage space. If not, repeat steps 8-12.

[0199] Embodiment 3:

[0200] Mahjong matching game level generation method based on reinforcement learning, such as Figure 6 As shown, the following steps are included:

[0201] Step T1: Obtaining level layout data, the number of mahjong tiles, and level generation input data, wherein the level generation input data includes the level difficulty level, the type of special card layout structure, the total number coefficient of card types, and the pass rate.

[0202] Step T2: Randomly select a specified number of card types from the 36 card types = (total number of level layout mahjong / 4)*total number of level card types coefficient;

[0203] Step T3: Randomly allocate the number of cards of each type according to the number of mahjong tiles, so that the number of each card type is an integer multiple of 2. First, set the number of mahjong tiles of all card types to 2, and then randomly select one type from the card types. The number of mahjong tiles of this type increases by 2, and remove this type from the card types. Repeat this process. The number of executions is the number of mahjong tiles laid out divided by 2 minus the number of card types to complete the allocation of all card numbers.

[0204] Step T4: Generate a corresponding structure in the level data according to the input card special arrangement structure type, level layout data, number of mahjong tiles, and number of cards; and execute any one of steps T5 to T9 accordingly;

[0205] Step T5: Generate an arrangement structure in which two identical cards (A1, A2) overlap, and the other two do not overlap (A3, A4), and A1, A3, and A4 are the farthest from each other in the layout. The overlapping situations include two identical cards completely overlapping, partially overlapping (half overlapping, quarter overlapping), or indirectly overlapping (there are multiple blockers between the two identical cards and they are all overlapping):

[0206] a. Randomly select type A with 4 cards, randomly select a Mahjong in the level layout (excluding the top layer, the top layer is less difficult, if you want to further reduce the difficulty but keep this structure, you can do it without constraints) as M1, obtain all the Mahjong sets MS1 (including all the layers below it) that overlap with M1, and randomly select a Mahjong M2 from MS1;

[0207] b. Randomly obtain all mahjongs in a layer below M1, exclude all mahjongs overlapping with M1, exclude M2, calculate the distance between the remaining mahjongs and M1 on the layout, and obtain the mahjong M3 with the maximum distance;

[0208] c. Randomly obtain all mahjongs in a layer below M1, and exclude all mahjongs overlapping with M1, exclude all mahjongs overlapping with M3, exclude M2 ​​and M3, calculate the sum of the distances between the remaining mahjongs and M1 and M3, and obtain the mahjong M4 with the maximum distance;

[0209] d. Mark M1, M2, M3, and M4 as the selected card A.

[0210] Step T6: Generate a horizontal blocking card layout structure with AABB or ABAB blocking in the horizontal direction

[0211] Structure, other cards besides A and B can also be inserted in the middle:

[0212] a. Traverse each layer of mahjong in the current layout (excluding the top layer) and find mahjong that is not blocked on the left but blocked on the right. Using this as a starting point, recursively search for all blocking mahjong sequences that satisfy the sequence length greater than or equal to 4, and generate a set sequence SL1;

[0213] b. Randomly select a sequence L1 from SL1, randomly select 4 mahjongs from L1, and sort them from left to right according to the horizontal coordinate of the layout to generate a mahjong sequence L2;

[0214] c. Randomly select 2 cards with a number of 4 from the card number set, and randomly use AABB or ABAB arrangement order to correspond one-to-one with the mahjong sequence L2;

[0215] d. Get all mahjongs between the upper layer and the top layer of L2, and find mahjongs that satisfy all the overlapping mahjongs below them and do not contain all the mahjongs that have been selected before. According to this condition, obtain 4 mahjong sequences L3. If there are no 4 mahjongs that meet the condition, re-execute from step a;

[0216] e. L3 randomly distributes two A and two B cards.

[0217] 13. Step T7: Generate a chain-like overlapping blocking card layout structure:

[0218] a. To increase diversity, the chain structure can choose a random length of 4 to 7, or two lengths of 4;

[0219] b. Obtain a set MS2 of two-by-two overlapping mahjong combinations of a specified chain length, where two overlapping mahjong combinations cannot overlap each other;

[0220] c. Randomly obtain 4 cards of A, and 2 cards of the chain length minus one. The mahjong combination with overlapping cards of A is the solution of the chain. Card A should choose the combination with the lowest level in MS2. The remaining cards correspond to mahjong one by one according to the rules of chain overlapping and blocking.

[0221] Step T8: Execute step T5 first and then step T6;

[0222] Step T9: Skip if there is no special arrangement structure;

[0223] Step T10: Mark the mahjong of a specific structure as cleared, and the remaining mahjong will correspond to the remaining cards randomly:

[0224] a. Obtain all currently free mahjong pieces AL1 according to the layout, and calculate the selection weight of each mahjong piece in AL1. The selection weight is calculated by dividing the total number of mahjong pieces blocked by the mahjong piece in the horizontal direction by 2 and the total number of mahjong pieces blocked in the vertical direction;

[0225] b. Randomly select 1 to 5 pairs of mahjong according to the weight (try to ensure that the number of mahjong pairs that can be matched during the elimination process is between 1 and 5) and mark each pair of mahjong with the randomly selected cards and mark them as cleared;

[0226] c. Repeat steps a and b until all mahjong tiles are marked.

[0227] Step T11: Verify the pass rate of the generated level:

[0228] a. Access the trained model through Unity ML-agents;

[0229] b. Use the model to evaluate the level pass rate, execute 1000 times, and calculate the pass rate.

[0230] c. If the pass rate does not meet the level generation configuration conditions, repeat steps T2-S11.

[0231] Step T12: If the pass rate meets the level generation configuration conditions, the data is formatted and the level data is saved to the storage space (disk).

[0232] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A mahjong matching game level generation method based on reinforcement learning, characterized in that: The following steps are involved: Step S1: randomly generating game levels according to set game difficulty parameters; Step S2: Execute the game problem solving based on the reinforcement learning network model and obtain the level passing rate. If the level passing rate is greater than or equal to the set threshold, output the game level; Step S21: Building a reinforcement learning network model, and using a neural network as a strategy network and a value network to extract level features and coordinate and simulate the time required for real players to play and the level passing rate through the value network; Step S22: using the strategy network to simulate the player to perform various action outputs; Identify the simulated player play results associated with the control object and the conditions encountered by the control object, identify the possible actions taken by the control object, and calculate the behavior-based probability value of each possible action based on the success rate of the action in historical scenarios, the similarity between the current state and the historical successful state, and the expected immediate reward value; Then, the long-term cumulative rewards that the action may bring in the future are evaluated in combination with the value function V(s); the decision to select an action is made based on the strategy π, which selects an action based on the probability value based on the behavior and the evaluation value of the value function V(s), and commands the controlled object to perform the action in the game; Step S23: accepting input of the strategy network in the background of the game interface; discretizing the game screen space into M×N grid points, each grid point represents an interactive position, the output layer of the neural network corresponds to the discrete position point, and each output node represents the probability of performing a click / touch operation at the position; Step S24: setting a reward function for the reinforcement learning network model; The reward function of the reinforcement learning network model includes extrinsic rewards and intrinsic rewards; the intrinsic rewards are generated through the self-supervised curiosity model in the Unity ML-agents framework to motivate the artificial intelligence neural network to explore unknown game states and behavior patterns; Step S25: Configure the training program environment and train the policy network; first, interact with the environment to collect experience data until the buffer is filled, and then randomly sample batches of 512 from the buffer for learning; each time the policy is updated, the model calculates the ratio of the new and old policies and constrains the update amplitude based on the clipping range, while combining the policy entropy term to promote exploration; Step S26: training the value network, and using the value network to output features of the level; training the value network based on minimizing the loss function between the predicted value and the actual value, and the training data includes feature information of the existing level and the corresponding player feedback or expert ratings; the value network performs dimension mapping and preliminary feature extraction on the input level features through a fully connected layer; a one-dimensional convolutional layer is used to capture local patterns and spatial correlations in the feature sequence; then a long short-term memory network layer is introduced to model the temporal dependency of the feature sequence and capture the long-term structural features in the level design; finally, all the extracted feature information is integrated through a fully connected layer to output the level value score, i.e., the level pass rate.

2. The method for generating mahjong matching game levels based on reinforcement learning according to claim 1, characterized in that: In step S23, the neural network calculates the operation probability distribution of each grid point according to the current game state, then selects the optimal interaction position based on the operation probability distribution, and performs corresponding click and touch operations at the selected grid point position, thereby simulating the touch screen operation mode of a real player; dynamically adjusts the grid density according to the game type, and adds additional action attributes of click duration and multi-touch to each grid point.

3. The method for generating mahjong matching game levels based on reinforcement learning according to claim 1, characterized in that: In step S24, the extrinsic reward includes a positive reward or a negative penalty, a linear reward given according to the proportion of completed goals, a phased reward calculated based on the percentage of unlocking progress, a fixed negative reward for punishing the model for using more than the required number of steps, and a click reward, which is calculated by a distance-based decay function.

4. The method for generating mahjong matching game levels based on reinforcement learning according to claim 1, characterized in that: In the step S25, when the model initially performs exploratory learning and does not know where to click, the click reward provides additional guidance for clicking only valid targets; a batch size of 512 is used to update the neural network, and an experience replay buffer of 20480 is maintained to store interaction data; a learning rate of 0.0002 is used during training to ensure stable convergence, and an entropy coefficient of 0.5 is used to balance exploration and utilization; a clipping range parameter of 0.1 is used to limit the amplitude of policy changes, and a generalized advantage estimation parameter of 0.95 is used to optimize the accuracy of advantage estimation; after each complete batch of data is collected, three rounds of policy updates are performed to make full use of the sampled data.

5. The method for generating mahjong matching game levels based on reinforcement learning according to any one of claims 1 to 4, characterized in that: The step S1 comprises the following steps: Step S11: Obtaining game difficulty parameters, the game difficulty parameters include level layout data, the number of mahjong tiles and level generation input data, the level generation input data includes difficulty level, special card layout structure type, card type total coefficient and pass rate; Step S12: randomly selecting a specified number of card types from the 36 card types, where the number of card types = (total number of mahjongs in the level layout / 4)*total number of card types coefficient; randomly allocating the number of cards of each type according to the number of mahjongs, and satisfying that the number of each card type is an integer multiple of 2; Step S13: generating corresponding level layout data, mahjong quantity and card quantity structure in the level data according to the special card arrangement structure type; Step S14: Mark the mahjong corresponding to the structure generated in step S13 as cleared, and the remaining mahjong is randomly matched with the remaining cards.

6. The method for generating mahjong matching game levels based on reinforcement learning according to claim 5, characterized in that: In the step S13, if the special card arrangement structure type is two identical cards overlapping or horizontally blocking, the following steps are included: a. Randomly select a type of card A with 4 cards, randomly select a Mahjong M1 in the level layout, obtain all the Mahjong sets MS1 that overlap with Mahjong M1, and randomly select a Mahjong M2 from the set MS1; b. Randomly obtain all mahjongs in a layer below M1, and exclude all mahjongs overlapping with mahjong M1 and mahjong M2, calculate the distance between the remaining mahjongs and mahjong M1 on the layout, and obtain mahjong M3 with the largest distance; c. Randomly obtain all mahjongs in a layer below M1, and exclude all mahjongs overlapping with mahjong M1 or mahjong M3, as well as mahjong M2 and mahjong M3, calculate the distances between the remaining mahjongs and mahjong M1 and mahjong M3 respectively, and add them up to obtain mahjong M4 with the maximum sum of distances; d. Mark Mahjong M1 to Mahjong M4 as the selected card A.

7. The method for generating mahjong matching game levels based on reinforcement learning according to claim 5, characterized in that: In the step S13, if the special card arrangement structure type is two identical cards overlapping and horizontally blocking, the following steps are included: a. Traverse the mahjong of each layer of the current layout, find the mahjong with no obstruction on the left and obstruction on the right, and use this as the starting point to recursively search for all blocking mahjong sequences, and satisfy the sequence length greater than or equal to 4, to generate the set sequence SL1; b. Randomly select a sequence L1 from the set sequence SL1, randomly select 4 mahjongs from the sequence L1, and sort them from left to right according to the horizontal coordinate of the layout to generate a mahjong sequence L2; c. Randomly select 2 cards with a number of 4 from the card number set, and randomly use AABB or ABAB arrangement order to correspond one-to-one with the mahjong sequence L2; d. Obtain all mahjongs from the upper layer to the top layer of mahjong sequence L2, search for mahjongs that satisfy all the overlapping mahjongs below it, and do not include all mahjongs that have been selected before, and obtain 4 mahjong sequences L3; e. Mahjong sequence L3 randomly distributes two A and two B cards.

8. The method for generating mahjong matching game levels based on reinforcement learning according to claim 5, characterized in that: In the step S13, if the special card arrangement structure type is chain overlapping blocking, the following steps are included: a. Set the chain structure to a random length of 4 to 7, or to two lengths of 4; b. Obtain a set MS2 of two-by-two overlapping mahjong combinations of a specified chain length, wherein the two-by-two overlapping mahjong combinations cannot overlap each other; c. Randomly obtain 4 cards of A, and 2 cards of the chain length minus one. The mahjong combination with overlapping cards of A is the solution of the chain. Card A should choose the combination with the lowest level in set MS2. The remaining cards correspond to mahjong one by one according to the rules of chain overlapping and blocking.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the game level generation method based on reinforcement learning described in any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the game level generation method based on reinforcement learning as described in any one of claims 1-8.

Images