A method, apparatus, and device for recommending block placement based on target region guidance

Abstract

The application discloses a kind of based on target area guide's block placement recommendation method, device and equipment, belong to game artificial intelligence technical field.The method includes: obtaining the state of board and generating candidate target area ROI;Joint state input including board occupation and ROI mask is constructed;The shape of block and drop point are uniformly encoded into discrete action space and action mask is executed;Based on ROI completion degree and board structure change (such as connected block, boundary length) compound reward is structured.The application solves the problem of sparse reward signal and long-range credit assignment difficulty in reinforcement learning through ROI guide mechanism, and the situation fragmentation is constrained by structured reward, which significantly improves the training convergence efficiency and inference recommendation quality.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and game algorithm technology. Specifically, it relates to a method, apparatus, and device for recommending block placement based on target region guidance for reinforcement learning training and inference. This method utilizes reinforcement learning technology, combined with board state perception and target region selection mechanisms, to solve the strategy planning problem of block placement in elimination puzzle games. Background Technology

[0002] Elimination-type block games are classic combinatorial optimization problems. Their core objective is to find the optimal placement of the current block within an uncertain future block sequence to maximize score and maintain the sustainability of the game. Existing block placement decision-making schemes mainly fall into two categories:

[0003] 1. Rule-based and experience-based decision-making solutions

[0004] These solutions typically rely on existing rules of elimination board games and high-score strategies summarized by human experience. For example, they consider factors such as board size, row and column elimination rules, the number of candidate blocks in the current round, non-rotatable block constraints, reserved space for large blocks, corner area management, and multi-step advance planning to make decisions regarding the generation, sorting, or placement of candidate blocks. Some solutions also control the types and combinations of candidate blocks for the next round based on the current situation and empirical rules, such as avoiding consecutive repetitions, reserving space for large blocks, or ensuring the board has basic placeability. These solutions are simple to implement and easy to deploy, but they heavily rely on human experience, and the rules are usually statically designed, making dynamic adjustments difficult to make based on complex situations. Especially in the mid-to-late game, they are prone to problems such as severe board fragmentation, insufficient compatibility with large blocks, and poor overall sustainability.

[0005] 2. Situation prediction or strategy prediction schemes directly based on neural networks

[0006] These approaches typically input the entire game board state directly into the model (e.g., DQN, PPO), and the model outputs recommendations from the candidate action space. Existing research includes solutions for using deep reinforcement learning in puzzle games like Tetris, such as Park et al.'s "Tetris Bot using Deep Reinforcement Learning," and Bairaktaris et al.'s comparison of reinforcement learning and heuristic methods in Atari Tetris. In broader game decision-making scenarios, Mnih et al.'s research on deep reinforcement learning in Atari games can also serve as a representative of neural network policy prediction methods. Current technical limitations include: Numerous invalid searches: When the action space is large and lacks effective constraints, many invalid action searches are likely to occur during training. Lack of phased objectives: When the model does not introduce explicit phased objectives, it is difficult to learn which regions should be prioritized for filling in the current situation. Short-sighted behavior: When using reinforcement learning training, if the reward is mainly constructed based on immediate scores, the model tends to favor short-term gains, paying insufficient attention to structural information affecting subsequent placeability, such as the number of holes, the degree of region fragmentation, and boundary complexity.

[0007] The core bottleneck of existing technology:

[0008] The sparsity of reward signals and the difficulty of temporal credit assignment: In elimination-type block games, meaningful reward signals (eliminating rows / columns) are low-frequency events, with zero reward values ​​in a large number of rounds, resulting in extremely scarce effective gradient signals. More importantly, a successful elimination often depends on layout decisions made several or even dozens of steps ago, while existing methods based on temporal difference error (TD error) (such as DQN and PPO) rely on the temporal proximity of actions and rewards to attribute contributions, failing to effectively solve the long-range credit assignment problem. The gradient contributions of key layout steps are diluted by a large number of zero-reward steps in the backpropagation chain, causing slow model convergence, unstable training, or falling into greedy and short-sighted suboptimal strategies.

[0009] Lack of awareness of board structure: Existing solutions often use "eliminating the number of rows / columns" as the sole optimization objective, failing to incorporate structural indicators reflecting the long-term sustainability of the board, such as the number of holes, isolated connected regions, degree of boundary fragmentation, and reachability through neighbor elimination, into the state representation or reward design. Research shows that introducing structural features (such as hole penalties and the number of row / column transitions) can significantly improve performance, a point often overlooked by existing deep reinforcement learning methods.

[0010] Lack of target region recognition capability: In scenarios where multiple block shapes are given in each round (such as Block Blast), existing methods lack the ability to recognize target regions, and cannot guide the blocks to prioritize filling the high-value regions closest to elimination, resulting in placement strategies that are only locally optimal rather than globally optimal.

[0011] Therefore, a block placement training and reasoning method that can combine local target guidance, legal action selection, and overall situation structure optimization is needed. Summary of the Invention

[0012] The purpose of this invention is to provide a method, apparatus, and device for recommending block placement based on target region guidance, in order to solve the problems of numerous invalid actions, unclear stage objectives, short-sighted recommendation results, and easy deterioration of the situation structure in the prior art, thereby improving the exploration efficiency, stability, and recommendation quality in the inference stage of reinforcement learning training.

[0013] To achieve the above objectives, the present invention provides the following technical solution:

[0014] A method for recommending block placement based on target region guidance, the method comprising the following steps:

[0015] Obtain the current chessboard state, which includes the occupancy information of each square on the chessboard;

[0016] Based on the chessboard state, candidate target regions are constructed and the current target region (ROI) is determined.

[0017] Based on the chessboard state and the target region, a joint state input is constructed;

[0018] Based on the joint state input, a target action is output using a reinforcement learning policy model. The target action is used to indicate the shape and landing position of the block to be placed.

[0019] In a preferred embodiment, the step of constructing candidate target regions and determining the current target region based on the chessboard state specifically includes:

[0020] Construct a two-dimensional prefix sum matrix to accelerate regional statistical calculations;

[0021] Enumerate candidate regions: Generate a candidate set containing horizontal full-width rectangles (width equal to the full width of the chessboard, height of 2 to 4 squares) and vertical full-width rectangles (height equal to the full height of the chessboard, width of 2 to 4 squares);

[0022] Scoring and Selection: A scoring function is used to calculate the score of each candidate region, and the region with the highest score is selected as the current ROI. If the highest score is lower than a preset threshold, a fallback strategy is activated, selecting the row or column with the highest fill density as the ROI. The scoring function comprehensively considers internal fill density, penalty for occupied areas outside the region, and edge offset, with the preferred weighting coefficient being... , .

[0023] In a preferred embodiment, constructing a joint state input based on the chessboard state and the target region specifically includes: concatenating the chessboard occupancy state, the target region mask, and the extended feature vector to generate a joint input vector with a dimension of 139. : in:

[0024] (64 dimensions): will The chessboard is arranged according to "first". back "The occupancy vector is expanded sequentially;"

[0025] (64-dimensional): The mask vector of the current ROI, with 1 inside the ROI and 0 outside the ROI;

[0026] (11-dimensional): Reserved extended feature vectors.

[0027] In a preferred embodiment, the step of outputting the target action using a reinforcement learning policy model specifically includes:

[0028] Unified action coding: 41 types of block shape metadata are pre-defined, and a mapping from "block shape + landing position" to discrete indexes is established to construct a unified discrete action space of size 1767.

[0029] Action Mask: Generates a hard constraint mask to shield actions that are physically conflicting or on cooldown; if a player fails to place a piece within an ROI for more than a certain number of consecutive steps, a penalty or ROI switching mechanism is triggered.

[0030] In a preferred embodiment, the method further includes calculating a reward signal based on changes in the chessboard state, and calculating the total reward for a single step using the following composite reward function.

[0031] The specific implementation details are as follows:

[0032] Local fill item: coefficient Encourage filling within the ROI;

[0033] Milestone Reward: Awarded when ROI is first achieved. High rewards ( );

[0034] Overflow penalty: The penalty is for squares that fall outside the ROI;

[0035] Structured rewards:

[0036] Blank connected component reward (coefficient) );

[0037] Filled area number bonus (coefficient) );

[0038] Boundary length reward (coefficient) ).

[0039] Dynamic decay factor Defined as When the structural change is positive, the reward is multiplied by [a certain factor]. When the structural change is negative, the penalty is applied in full, achieving asymmetric modulation.

[0040] The present invention also provides a block placement recommendation device based on target area guidance, including a state acquisition module, a target area generation module, a state encoding module, an action processing module, a strategy execution module, a reward calculation module and a parameter update module;

[0041] An electronic device includes a processor, a memory, and a computer program stored in the memory, wherein the processor executes the program to implement any of the above methods;

[0042] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0043] Compared with the prior art, the technical solution provided by the embodiments of the present invention has the following significant beneficial effects:

[0044] By introducing a unified discrete action space (1767 dimensions) and an action masking mechanism, this invention transforms the complex combinatorial optimization problem into a standard discrete action selection problem. This avoids the brute-force search required by traditional algorithms at every step, which involves traversing all block shapes and their rotation states. It reduces the computational cost of forward inference for the policy model by several orders of magnitude, making real-time decision-making possible on resource-constrained terminal devices.

[0045] Traditional reinforcement learning faces the sparsity problem in elimination games, where rewards are only given at the moment of elimination. This invention addresses this issue by using a Region of Interest (ROI) guidance mechanism to decompose the global elimination objective into local filling sub-objectives. This "divide and conquer" strategy transforms sparse global rewards into dense local rewards, significantly accelerating the model's convergence speed and effectively solving the long-range credit allocation problem, enabling AI to learn macro-level strategies spanning multiple steps.

[0046] This invention innovatively designs a composite reward function and a dynamic decay factor. This not only encourages AI to quickly achieve local ROI (short-term gains), but also forces AI to focus on the global structure (long-term gains) through an asymmetric modulation mechanism. This mechanism effectively prevents AI from creating a large number of difficult-to-eliminate "dead holes" or broken boundaries in pursuit of short-term filling, ensuring that the health of the board position is maintained while pursuing high scores. This is a technical effect that existing single-objective optimization algorithms cannot achieve.

[0047] This design employs 139-dimensional vectors to directly represent the chessboard state and objective, eliminating the need for high-dimensional raw image input. This standardized vector input not only eliminates the complexity of image preprocessing but also avoids training instability caused by image noise or resolution variations. This design significantly reduces the number of model parameters, makes the training process more stable, and facilitates easy transfer and deployment across different platforms. Attached Figure Description

[0048] Figure 1 A schematic diagram of a block placement recommendation method based on target region guidance provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating the generation process of the target region ROI provided in this application embodiment;

[0050] Figure 3 A schematic diagram illustrating the construction of joint state inputs provided in this application embodiment;

[0051] Figure 4 A schematic diagram illustrating the unified motion coding of the block shape and landing point provided in an embodiment of this application;

[0052] Figure 5 A schematic diagram illustrating the structure of the composite reward function provided in the embodiments of this application;

[0053] Figure 6 A flowchart illustrating the dynamic update process of the target area provided in this application embodiment;

[0054] Figure 7 This is a schematic diagram of another recommended method for placing blocks provided in an embodiment of this application;

[0055] Figure 8 This is a schematic diagram of the structure of a block placement recommendation system based on target area guidance provided in an embodiment of this application. Detailed Implementation

[0056] To better understand the present invention, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Various modifications can be made to the embodiments as long as the effects of the present invention are achieved.

[0057] The present invention will now be described in detail with reference to specific embodiments.

[0058] like Figure 1 The diagram shows a flowchart of the block placement recommendation method based on target region guidance provided by this invention. This embodiment elaborates on a block placement recommendation method based on target region guidance for reinforcement learning training and inference. This method aims to solve the problems of sparse rewards, excessive search space, and lack of long-term strategy planning in existing elimination game AI. The core of this method lies in constructing a closed-loop reinforcement learning system of "perception-decision-execution-feedback".

[0059] Step 10: Obtain the current chessboard state

[0060] First, the system obtains the current game environment state. In a preferred implementation scenario, the game board is defined as a... The chessboard is a two-dimensional grid space. To facilitate computer processing and neural network input, this invention binarizes the chessboard state.

[0061] Specifically, construct a matrix For each element in the matrix (in ):

[0062] If the cell is occupied by a block, its state value is recorded as 1;

[0063] If the cell is in an empty state, its state value is recorded as 0.

[0064] This step transforms the complex graphical chessboard into a mathematical matrix that a computer can accurately calculate, laying the foundation for subsequent feature extraction.

[0065] Step 20: Construct candidate target regions and determine the current target region (ROI)

[0066] This is the key step that distinguishes this invention from existing technologies. See the appendix for details. Figure 2 To avoid the model blindly searching on a large chessboard, this invention introduces a "target region guidance" mechanism.

[0067] Step 201: Construct a two-dimensional prefix sum array

[0068] To expedite the subsequent statistical analysis of the square filling within any rectangular region, this invention first bases the analysis on the current chessboard state. Construct a two-dimensional prefix sum array By using dynamic programming, any submatrix can be made... arrive The total number of blocks inside can be The calculation is performed within time complexity, which greatly improves the real-time performance of the algorithm.

[0069] Step 202: Enumerate candidate target regions

[0070] Based on the prefix sum array, the system enumerates all possible candidate target regions. In this embodiment, the candidate regions are limited to two types of rectangular structures with high elimination potential:

[0071] Horizontal full-width rectangle: width is fixed to the full width of the chessboard (i.e., 8 squares), height... The value can be 2, 3, or 4 cells.

[0072] Vertical full-width rectangle: height is fixed at the full height of the chessboard (i.e., 8 squares), width... The value can be 2, 3, or 4 cells.

[0073] This enumeration strategy is based on the rules of match-3 games, prioritizing potential areas that can eliminate multiple rows or columns at once.

[0074] Step 203: Calculate candidate region scores and select ROIs

[0075] For each candidate region This invention employs a specific scoring function for evaluation. This scoring function aims to identify areas with both "high filling potential" and "advantageous location".

[0076] The scoring function is specifically defined as follows:

[0077] The physical meaning and calculation method of each parameter are as follows:

[0078] (Intra-region fill density): defined as .in For the region The number of cells already filled. For the region The total area. This encourages models to focus on areas that already have a certain foundation and are easy to fill.

[0079] (Penalty for occupied squares outside the designated area): This penalty is applied to areas where the surrounding environment is too harsh (the outer squares are too full, making it difficult to place a block). This represents the total number of squares on the chessboard (64 squares).

[0080] (Edge-to-Boundary Ratio): Calculates the degree of overlap between the target area and the board boundary. Areas closer to the boundary are generally easier to eliminate, so an offset bonus is given.

[0081] Weighting coefficient: In this embodiment, the preferred parameter is... Used to control the severity of punishment in the external environment; It is used to control the strength of the boundary offset.

[0082] The system traverses all candidate regions and calculates their... Select the candidate region with the highest score as the current target region. If there are ties in scores, the smaller area is prioritized to focus on short-term goals; if all candidate areas score below a preset threshold (e.g., 0.1), a fallback strategy is activated, forcibly selecting the row or column with the highest current fill density as the candidate. This ensures that the decision-making process is never interrupted.

[0083] Step 300: Construct the joint state input

[0084] To enable reinforcement learning policy models to simultaneously perceive both the "global situation" and the "local objective," this invention designs a joint state encoding mechanism, as detailed in the appendix. Figure 3 .

[0085] This step specifically includes:

[0086] Chessboard Occupation Vector :Will The chessboard state matrix is ​​scanned in a preset order (e.g., first by pressing...). Increment the coordinates, then press Expanding the vector (in ascending row order) results in a 64-dimensional binary vector. If the corresponding cell in the original matrix is ​​1, the corresponding bit in the vector is 1; otherwise, it is 0.

[0087] Target region mask vector Construct a mask matrix of the same size as the chessboard, and then... The covered cells are marked with 1, and the remaining cells are marked with 0. This is then expanded into a 64-dimensional vector in the same order.

[0088] Reserved extended feature vectors To accommodate future game rule expansions or the addition of extra statistical features (such as the number of current connected components, the current block type ID, etc.), this invention reserves an 11-dimensional feature vector space.

[0089] Finally, the three vectors are concatenated along the channel dimension to form a vector with a total dimension of [missing value]. dimensional joint state input vector This fixed-dimensional encoding method avoids the model instability problem caused by variable-length inputs and significantly reduces the number of parameters required by the neural network.

[0090] Step 400: Construct a unified discrete action space

[0091] To improve the training efficiency of reinforcement learning, this invention maps the complex process of "selecting a block + selecting a position" into a discrete set of integers, as detailed in the appendix. Figure 4 .

[0092] The shapes of the blocks and the placement points on the chessboard are uniformly encoded into a discrete action space, which serves as the action output space for the reinforcement learning policy model.

[0093] Specifically, for each type of square, all its theoretically possible landing points on the chessboard are traversed, and the combination of "square shape + landing point position" is mapped to a discrete action index.

[0094] The system pre-sets metadata for multiple block shapes. Each block shape includes at least a set of relative coordinates, width, height, number of valid cells, and number of theoretically possible landing points.

[0095] In this embodiment, the game pre-sets 41 candidate blocks of different shapes (including different geometric forms and rotation states). For each type of block... The system traverses its... All theoretically possible positions on the chessboard.

[0096] The size of the discrete action space satisfies

[0097]

[0098] in, Let N represent the theoretical number of possible placement points for the i-th type of square on the chessboard, and let N represent the total number of square shapes.

[0099] The mapping logic for establishing action indexes is as follows:

[0100] Base address calculation: First, calculate the first block according to the preset order of the block shapes. Action index of the block starting offset .

[0101]

[0102] PositionCount(shapek) represents the theoretical number of possible placement points for the k-th type of square on the chessboard.

[0103] Position Numbering: Within the action index range corresponding to each block, each theoretically possible landing point is numbered according to a preset position traversal order (e.g., from left to right, from bottom to top). .

[0104] Uniform Index: Planting blocks in position The unified action index is:

[0105]

[0106] Through the above encoding, the entire action space is mapped to a sequence of actions. arrive The encoding uses a continuous integer space (total number of actions is 1767). This encoding method allows the policy network to directly output an integer representing "which block to use and where to land", greatly simplifying the design of the output layer.

[0107] Step 500: Execute the action mask

[0108] Illegal actions must be eliminated before the action space is input into the policy model. This invention applies a binary mask to the output of the neural network.

[0109] Specifically, for each action index in the discrete action space, if its corresponding landing position satisfies any of the following conditions, it is marked as invalid (Mask=0):

[0110] Physical collision: When a block is placed, it overlaps with an already occupied square on the board.

[0111] Cooldown constraint: This block type is within the cooldown period set by the game rules (if such a rule exists).

[0112] By using action masks, the model will only select from the remaining legal actions (Mask=1) during sampling. This not only avoids the model outputting invalid instructions, but also effectively injects the game's basic physics rules into the early stages of training, significantly reducing invalid exploration and improving training efficiency.

[0113] Step 600: Output the target action and execute block placement.

[0114] The 139-dimensional joint state input vector generated in step 300 is fed into a reinforcement learning policy model (such as a PPO or DQN network). Under the constraint of the action mask, the model outputs a probability distribution and selects the action index with the highest probability from it.

[0115] Based on the mapping table established in step 400, the system deciphers the action index into the specific target block shape and target landing point coordinates. Then, the system controls the game engine to place the block on the board. After the block lands, the system checks if an entire row or column is filled and performs a resolution, thereby changing the board state.

[0116] Step 700: Construct rewards based on target area completion and changes in situation structure

[0117] This is the core of the problem of "short-sighted strategy" solved by this invention. This invention designs a composite reward function that focuses not only on immediate filling but also on the long-term board structure; see appendix for details. Figure 5 .

[0118] Total reward per step The calculation formula is:

[0119]

[0120] The components of this formula and their technical meanings are as follows:

[0121] Add a fill bonus to the target area:

[0122] When the ROI is not yet complete, the model is encouraged to fill as many squares as possible into the target area. This represents the number of new fill cells added within the ROI for this action.

[0123] Preferred parameters: .

[0124] Milestone Rewards:

[0125] When the current action causes the ROI to be fully filled for the first time (i.e.) A high reward will be given when this is achieved.

[0126] To encourage models to prioritize completing large areas, the reward includes the area of ​​the ROI. Related linear terms.

[0127] Preferred parameters: (Basic Rewards) (Area coefficient).

[0128] Spillover constraint penalties:

[0129] Penalize actions that place blocks outside the ROI area to ensure the strategy remains focused.

[0130] Preferred parameters: This is the penalty coefficient.

[0131] Structural reward based on spillover decay factor:

[0132] This is a key dynamic adjustment mechanism. Spillover decay factor. The overflow ratio of the current placement action is dynamically calculated:

[0133]

[0134] in The number of squares that fall outside the target area. This represents the total number of squares in the current area; when all squares fall within the ROI. = 1.0 (no decay), when all squares fall outside the ROI = 0.0 (completely shields positive structure rewards).

[0135] Asymmetric processing logic: This invention employs an asymmetric processing strategy for structural rewards and penalties.

[0136] When the game structure improves (e.g., fewer connected components, smoother boundaries), the reward is multiplied by $\alpha_t$. If most of the blocks fall outside the ROI ($\alpha_t$ is close to 0), the reward for structural improvement is significantly reduced. This forces the model to be within the ROI to achieve a high score, thus binding local objectives to the global structure.

[0137] When the board structure deteriorates (e.g., holes appear, fragmentation worsens), the penalty remains fully applied, unaffected by \alpha_t. This means that regardless of where the squares are placed, the cost of disrupting the board structure must be borne by the model.

[0138] Specific calculation of structural change terms:

[0139] : This represents the number of connected components in the empty areas of the chessboard. Weight (preferred) If the action causes the empty region to split (the number of connected components increases), then The model is penalized if the action fills the gaps and the connected components merge, then it is rewarded.

[0140] : This represents the number of areas filled on the chessboard. Weight (preferred) The model is encouraged to connect blocks together, reducing fragmentation.

[0141] : Represents the total length of the boundary segments of the chessboard's filled area (outline complexity). Weight (preferred) Models are encouraged to construct smooth boundaries to avoid jagged terrain.

[0142] Finally, to ensure training stability and prevent gradient explosion, the system calculates the initial reward... Normalization and truncation are performed:

[0143]

[0144] Step 800: Reassess the target area and proceed to the next round of recommendations. See the appendix for details. Figure 6 .

[0145] The system calculates the reward signal The parameters of the policy model (such as the weights and biases of the neural network) are updated using reinforcement learning algorithms (such as backpropagation).

[0146] Subsequently, the system determines the current target area. Has the completion condition been met (e.g., the ROI fill rate reaches 100%)?

[0147] If already completed: The system discards the current... Returning to step S2, new candidate target regions are regenerated based on the latest chessboard state to achieve "rolling optimization".

[0148] If not completed: Keep current. No change, proceed to the next time step. Repeat steps 100 to 800 above.

[0149] This dynamic update mechanism means that the model does not perform a one-time static plan for the entire game situation, but continuously adjusts its focus as the game progresses, making it highly adaptable.

[0150] like Figure 7The diagram illustrates the overall system flow disclosed in another embodiment of this application. The data flow is as follows: taking the current chessboard binary grid (8×8) as input, it sequentially passes through: ROI generation module—outputting the target region mask; joint state encoding module—concatenating the chessboard vector and the ROI mask into a 139-dimensional observation vector; policy model—outputting the target action index under the action mask constraint; block placement execution module—decoding the action and updating the chessboard state; reward calculation module—outputting a composite reward based on the ROI completion degree and changes in the board structure; parameter update module—updating the policy model parameters according to the reward signal; when the ROI is completed, the target region is re-evaluated cyclically, forming a closed-loop training process.

[0151] like Figure 8 As shown, this embodiment provides a hardware / software architecture device corresponding to the above embodiment. This device is used to execute the above-described block placement recommendation method based on target region guidance. The device includes the following modules:

[0152] Status Acquisition Module 10: Used to capture chessboard data in game memory in real time and convert it into... A binary matrix.

[0153] Target Region Generation Module 20: Includes a two-dimensional prefix sum building unit, a candidate region enumeration unit (horizontal / vertical full-width rectangle), and a scoring selection unit, used to output the coordinates of the current target region ROI.

[0154] State encoding module 30: Responsible for executing the logic of step 300. This module expands the chessboard occupancy state into a 64-dimensional vector. Expand the ROI into a 64-dimensional mask vector. and with 11-dimensional extended vector By splicing, a 139-dimensional joint state input is generated. .

[0155] Action processing module 40: Contains action space definition unit and action mask unit. Maintains metadata for 41 types of blocks, constructs a lookup table containing 1767 actions, and generates an action mask matrix by combining legality constraints.

[0156] Policy execution module 50: The core is a reinforcement learning policy network (such as a multilayer perceptron MLP). It is used to input the joint state into the reinforcement learning policy model (such as a neural network), sample the target action index under the action mask constraint, and decompose it into a specific block shape and landing point coordinates to control the execution mechanism for placement.

[0157] Reward Calculation Module 60: Includes an ROI completion detection unit and a structural feature extraction unit. It monitors chessboard state changes in real time and calculates composite rewards, including local filling, milestone rewards, overflow penalties, and structural changes (connected components, boundary lengths). and apply Attenuation factor.

[0158] Parameter update module 70: It is used to update the network parameters of the policy model based on the calculated reward signal using reinforcement learning algorithms (such as PPO, DQN), and trigger the target region generation module to re-evaluate the new ROI after the ROI is completed.

[0159] Each module communicates and connects via a bus or API interface, working together to achieve intelligent block recommendation.

[0160] This invention can be implemented on an electronic device. The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the recommended block placement method as described in the above embodiments.

[0161] In summary, the embodiments of the present invention achieve the following technical effects through the above-described methods, apparatus, and devices:

[0162] Significantly reduced search space: By using the action masking mechanism, the theoretical space of 1767 actions is reduced to a subset of legal actions, which greatly reduces the computational cost of forward inference of the policy model and improves decision-making efficiency.

[0163] Addressing the reward sparsity problem: By introducing a target region (ROI) guidance mechanism, the originally sparse global elimination reward is transformed into a dense local filling reward, which solves the temporal credit assignment problem and accelerates model convergence.

[0164] Global and local synergy: This invention is not a simple "divide and conquer" approach. It utilizes a composite reward function... The asymmetric design of the decay factor achieves a coupled and synergistic effect between ROI guidance and structured rewards. Without structural constraints, guidance alone would create numerous gaps in the model; without guidance, structure alone would result in extremely low search efficiency. The combination of both forces the model to consider both short-term ROI gains and long-term global smoothness.

[0165] Standardized input and output: The chessboard and the target are uniformly encoded as 139-dimensional fixed vectors, avoiding the parameter expansion caused by variable-length features or image inputs, so that the model has extremely high stability and consistency in both training and inference stages.

[0166] The solution provided in this embodiment is not only applicable to the specific board game described in the document, but can also be extended to AI decision-making scenarios for various elimination and puzzle games, and has extremely high engineering application value.

[0167] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0169] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0172] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0173] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0174] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0175] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0176] The above are merely embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0177] The accompanying drawings illustrate several block diagrams and / or flowcharts. It should be understood that some blocks, or combinations thereof, in the block diagrams and / or flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when executed by the processor, these instructions can create means for implementing the functions / operations described in these block diagrams and / or flowcharts. The technology of this application can be implemented in hardware and / or software (including firmware, microcode, etc.). Alternatively, the technology of this application can take the form of a computer program product stored on a computer-readable storage medium, which can be used by or in conjunction with an instruction execution system.

Claims

1. A method for recommending block placement based on target region guidance, characterized in that, Includes the following steps: Obtain the current chessboard state and represent the chessboard state as a two-dimensional grid of fixed size; Based on the current chessboard state, construct multiple candidate target regions and select the current target region ROI from them; A joint state input is constructed based on the current chessboard state and the current target region ROI, which serves as the observation input for the reinforcement learning policy model; The shapes of the blocks and the chessboard placement points are uniformly encoded into a discrete action space, and combined with the legality constraint execution action mask, the action output space of the reinforcement learning policy model is obtained. The joint state is input into the policy model, the target action is output under the action mask constraint, and the block placement is performed. After placing the block, a composite reward is constructed based on the completion of the current target area and the overall chessboard structure change, and the composite reward is used as a reward signal in reinforcement learning training. The policy model parameters are updated based on environmental feedback and reward signals, and the above steps are repeated during the training phase. Once the current target region meets the completion conditions, the candidate target regions are reconstructed based on the updated chessboard state, and a new target region ROI is selected.

2. The method according to claim 1, characterized in that, The process of constructing multiple candidate target regions based on the current chessboard state and selecting the current target region ROI from them specifically includes: Construct a two-dimensional prefix sum for the chessboard; Enumerate horizontal and vertical full-width rectangles as candidate target areas. The width of the horizontal full-width rectangle is the full width of the chessboard and the height is 2 to 4 squares. The height of the vertical full-width rectangle is the full height of the chessboard and the width is 2 to 4 squares. A score is calculated for each candidate target region, and the score includes at least the region's internal fill density, the penalty for occupied squares outside the region, and the bias term for proximity to the chessboard boundary; Based on the scoring results, the candidate target region with the highest score is selected as the current target region ROI.

3. The method according to claim 2, characterized in that, The scoring function for the candidate target region R is: in, Fill density within the target area. This refers to the number of cells already occupied outside the target area. The edge ratio between the target area and the chessboard boundary. , These represent the width and height of the chessboard, respectively. , These are the weighting coefficients.

4. The method according to claim 1, characterized in that, The construction of the joint state input based on the current chessboard state and the current target region ROI specifically includes: Unfolding the chessboard occupancy state yields a 64-dimensional chessboard occupancy vector. The value is 1 when the corresponding cell is occupied, and 0 otherwise. Expanding the current target region ROI yields a 64-dimensional target region mask vector. The value is 1 when the corresponding cell is located within the current target region (ROI), and 0 otherwise. Constructing an 11-dimensional reserved extended feature vector ; Concatenating the above vectors yields a joint state input with a total dimension of 139. .

5. The method according to claim 1, characterized in that, The process of uniformly encoding the shape of the blocks and the chessboard placement points into a discrete action space specifically includes: 41 pre-defined block shape metadata; For each type of square, iterate through all its theoretically possible placement points on the chessboard. According to the preset order of the block shapes, each block is assigned a continuous action index interval, and within the action index interval corresponding to each block, each theoretical landing point is numbered according to the preset position traversal order, establishing a one-to-one mapping relationship between discrete action indexes and the combination of "block shape + landing point position". The total number of discrete action spaces is 1767.

6. The method according to claim 1, characterized in that, The composite reward constructed based on the current target area completion rate and changes in the overall chessboard structure specifically includes: Calculate the bonus for newly added fill within the target area; Calculate the milestone reward when the current target area is completed for the first time; Calculate the overflow constraint penalty for newly added cells outside the target region; Calculate the reward or penalty corresponding to the change in the number of connected components in the blank areas of the chessboard; Calculate the reward or penalty corresponding to the change in the number of filled areas on the chessboard; Calculate the reward or penalty corresponding to the change in the total length of the boundary segments of the chessboard's filled area.

7. The method according to claim 6, characterized in that, The formula for calculating the total reward for a single step is as follows: in, This represents the amount of fill added to the target area in the current step. This indicates whether the current target area has been completed for the first time. Add the number of new grids outside the target area. For the first The number of connected components in the empty areas of the chessboard. For the first Number of areas to fill on the chessboard in each move. For the first The total length of the boundary segment of the chessboard's filled area. The structural reward decay factor is related to the degree of spillover. These are the weighting coefficients.

8. The method according to claim 7, characterized in that, The spillover attenuation factor The overflow ratio of the current placement action is dynamically calculated: in, This represents the number of squares where the piece is placed outside the target area. This represents the total number of squares in the current block; and when the structural change is positive, the corresponding reward is multiplied by [value missing]. The penalty is applied attenuated; when the structural change is negative, the corresponding penalty remains fully applied.

9. A block placement recommendation device based on target area guidance, characterized in that, include: The status acquisition module is used to obtain the current chessboard state; The target region generation module is used to construct candidate target regions and determine the current target region ROI; The state coding module is used to construct the joint state input; The motion processing module is used to construct a unified discrete motion space and execute motion masks. The strategy execution module is used to output the target action and execute the block placement. The reward calculation module is used to construct rewards based on the completion rate of the target area and changes in the situation structure. The parameter update module is used to update the strategy model parameters based on the reward signal.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, characterized in that the processor executes the computer program to implement the method as described in any one of claims 1 to 8.

Images