A game interaction method and device based on a policy network, equipment and a medium

Abstract

The application discloses a game interaction method and device based on a policy network, equipment and a medium, and relates to the technical field of computers, which comprises the following steps: generating a to-be-trained sample through a sub-process, a local network copy and a deep counterfactual regret minimization algorithm in a preset parallel training architecture; the local network copy comprises a copy of a policy network and a copy of an advantage network of each player; writing the to-be-trained sample into an experience buffer in batches through a main process in the architecture; training the policy network and the advantage network by using a preset multi-dimensional hierarchical sample sampling strategy and the sample in the experience buffer through the main process, obtaining to-be-updated network parameters; updating the network according to the to-be-updated network parameters through the sub-process, generating the to-be-trained sample again based on the updated local network copy, and stopping the training until the training is terminated; and generating a current action decision for the player based on the trained target policy network, so that the player can interact with other players. The application effectively realizes parallel training for DeepCFR.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a game interaction method, apparatus, device, and medium based on policy networks. Background Technology

[0002] In traditional strategy network training schemes for game theory games: 1) When training with DeepCFR (Deep Counterfactual Regret Minimization), the game tree traversal (CPU-intensive) and network training (GPU-intensive) are performed sequentially, preventing the CPU / GPU from being fully utilized simultaneously. Furthermore, general distributed learning methods, which focus on environment interaction rather than the game tree, increase complexity. 2) Direct writing after local traversal in a single process is prone to blocking, while direct writing to the shared buffer by child processes in a multi-process environment can lead to deadlock. 3) Purely random sampling of training samples may result in a low proportion of recent or highly important samples, causing distribution drift and training jitter. Consequently, the interaction effect is poor when players use the trained strategy network. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a game interaction method, apparatus, device, and medium based on policy networks, which can effectively realize parallel training for DeepCFR, thereby improving training efficiency. Furthermore, under preset queue capacity / concurrency control conditions, it reduces the risk of blocking and deadlock, mitigates distribution drift and training jitter, and improves sample utilization efficiency. The specific solution is as follows: In a first aspect, this application provides a game interaction method based on a policy network, applied to a computer device, comprising: The game tree is traversed using a pre-defined parallel training architecture, a local network copy of the current subprocess, and a deep counterfactual regret minimization algorithm to obtain training samples. The local network copy includes a copy of the policy network and a copy of the advantage network corresponding to each player. The advantage network is a neural network used to learn the counterfactual regret of each game interaction action relative to the baseline policy. The main process in the preset parallel training architecture writes the training samples that have been stored in the sample queue into the experience buffer in batches. Through the main process, and by using the preset multi-dimensional hierarchical sample sampling strategy and the samples in the experience buffer, the strategy network and the advantage network are trained in the current iteration training round to obtain the network parameters to be updated. The subprocess updates the network based on the network parameters to be updated that have been stored in the parameter synchronization queue, and then triggers the game tree traversal operation again based on the updated local network copy until the preset training termination condition is met. After the preset training termination condition is met, a current action decision is generated for the target player based on the trained target policy network, so that the target player can use the current action decision to interact with other players in the game.

[0004] Optionally, the step of performing game tree traversal using subprocesses in a preset parallel training architecture, a local network copy of the current subprocess, and a deep counterfactual regret minimization algorithm to obtain training samples includes: The latest network parameters are obtained from the parameter synchronization queue through subprocesses in the pre-defined parallel training architecture. The local network copy of the current subprocess is obtained by performing network updates based on the network parameters through the subprocess. Through the subprocess and based on the local network copy, the game tree traversal operation is completed by searching or sampling along the game tree using the Monte Carlo sampling method, and the traversal result is obtained. Through the subprocess, and using the traversal results and the deep counterfactual regret minimization algorithm, several training samples are determined; wherein, the training samples include advantage samples and policy samples; the advantage samples are samples used to train the advantage network; the policy samples are samples used to train the policy network; the policy network is a neural network used to learn the probability distribution of the selection of each game interaction action under a given information state. The subprocess determines whether the current sample queue is full in order to determine the queue determination result. If the queue determination result is negative, then the subprocess stores the plurality of training samples into the sample queue. If the queue determination result is yes, then the historical samples in the sample queue are processed through the subprocess and using a preset hybrid replacement strategy to determine the processed sample queue. The subprocess stores the plurality of training samples into the processed sample queue.

[0005] Optionally, the step of processing historical samples in the sample queue through the subprocess and using a preset hybrid replacement strategy includes: The subprocess discards several historical samples from the sample queue using the first-in-first-out principle to determine the first sample queue. The subprocess randomly selects and discards historical samples from the first sample queue to determine the processed sample queue.

[0006] Optionally, after determining a number of training samples, the method further includes: The subprocess performs normalized baseline parsing on the target fields in the game configuration string to obtain the parsing results. The prior features are obtained through the subprocess; wherein, the prior features are features of information states designed and spliced ​​based on game rules and domain knowledge, including starting hand strength or current hand strength, hand type, waiting hand, community card features, and game round. The splicing result is determined by splicing the data through the subprocess based on the prior features and the original information state corresponding to the training sample. Through the subprocess, and using the splicing result and the parsing result, the monetary features are normalized to determine the first normalization result; The second normalization result is determined by normalizing and pruning the regret values ​​in the advantageous samples based on the analysis results through the subprocess. The optimized training samples are obtained through the subprocess and based on the first normalization result and the second normalization result.

[0007] Optionally, the step of writing the training samples, which have been stored in the sample queue, into the experience buffer in batches through the main process in the preset parallel training architecture includes: The main process in the preset parallel training architecture reads the training samples in batches from the sample queue until the sample queue is empty or the maximum sample collection amount is reached. The main process reads the training samples and writes them into the experience buffer. The main process marks each sample in the experience buffer to obtain the sample iteration count field and sample importance metric field corresponding to each sample.

[0008] Optionally, the step of training the policy network and the advantage network for the current iteration training round through the main process and using a preset multi-dimensional hierarchical sample sampling strategy and samples in the experience buffer includes: The main process determines whether the current experience buffer is full in order to determine the buffer judgment result. If the buffer determination result is negative, then the main process performs multi-dimensional hierarchical sampling on the samples in the experience buffer to determine the current training batch. If the buffer determination result is yes, then the main process processes the data in the experience buffer using a preset hybrid elimination strategy, and performs multi-dimensional hierarchical sampling based on the corresponding processed buffer to determine the current training batch. The main process trains the advantage network corresponding to each player using the current training batch, and combines the action advantage value output by the advantage network to guide the training of the policy network, so as to obtain the network parameters to be updated; the action advantage value is the advantage value corresponding to the game interaction action. The main process serializes the network parameters to be updated to determine the serialization result. The main process stores the parameter serialization result into the parameter synchronization queue.

[0009] Optionally, the step of performing multi-dimensional hierarchical sampling of samples in the experience buffer through the main process includes: The main process, based on the sample iteration count field and window, divides the samples in the experience buffer into time-dimensional segments to determine the recent sample index set and the historical sample index set. Through the main process, and based on the sample importance metric field, the recent sample index set and the historical sample index set are divided according to importance dimensions to determine the high-importance sample index set and the low-importance sample index set. The first sampling result is determined by sampling through the main process, using a preset sampling ratio, the high-importance sample index set, and the low-importance sample index set; The main process is used to perform sampling, utilizing a preset importance-weighted sampling strategy, the high-importance sample index set, and the low-importance sample index set, to determine the second sampling result. The first sampling result and the second sampling result are merged through the main process to obtain the sampling merge result; The main process shuffles the sample indices in the sampling and merging results to determine the shuffled sampling and merging results. The main process reads samples from the experience buffer based on the sample index in the sampling merging result to determine the current training batch.

[0010] Secondly, this application provides a game interaction device based on a policy network, applied to a computer device, comprising: The sample generation module is used to perform game tree traversal through subprocesses in a preset parallel training architecture, local network copies of the current subprocesses, and a deep counterfactual regret minimization algorithm to obtain training samples; wherein, the local network copy includes a copy of the policy network and a copy of the advantage network corresponding to each player; the advantage network is a neural network used to learn the counterfactual regret of each game interaction action relative to the baseline policy; The sample writing module is used to write the training samples that have been stored in the sample queue into the experience buffer in batches through the main process in the preset parallel training architecture. The network training module is used to train the strategy network and the advantage network for the current iteration training round through the main process and by using a preset multi-dimensional hierarchical sample sampling strategy and samples in the experience buffer, so as to obtain the network parameters to be updated. The network update module is used to perform network updates through the subprocess and according to the network parameters to be updated that have been stored in the parameter synchronization queue, and to trigger the game tree traversal operation again based on the updated local network copy until the preset training termination condition is met. The decision generation module is used to generate a current action decision for the target player based on the trained target policy network after the preset training termination condition is met, so that the target player can use the current action decision to interact with other players in the game.

[0011] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the aforementioned policy network-based game interaction method.

[0012] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the steps of the aforementioned policy network-based game interaction method.

[0013] As can be seen, in this application, the method applied to a computer device includes: performing game tree traversal through a subprocess in a preset parallel training architecture, a local network copy of the current subprocess, and a deep counterfactual regret minimization algorithm to obtain training samples; wherein, the local network copy includes a copy of the policy network and copies of the advantage network corresponding to each player; the advantage network is a neural network used to learn the counterfactual regret of each game interaction action relative to the baseline policy; through the main process in the preset parallel training architecture, the training samples that have been stored in the sample queue are written in batches into the experience buffer; through the main process, and using a preset multi-... The dimensional hierarchical sample sampling strategy and the samples in the experience buffer are used to train the policy network and the advantage network for the current iteration training round to obtain the network parameters to be updated. The network is updated through the subprocess according to the network parameters to be updated that have been stored in the parameter synchronization queue. The game tree traversal operation is triggered again based on the updated local network copy until the preset training termination condition is met. After the preset training termination condition is met, the current action decision is generated for the target player based on the trained target policy network so that the target player can use the current action decision to interact with other players in the game. In other words, this application, applied to a computer device, first generates training samples using a subprocess in a preset parallel training architecture, a local network copy of the subprocess, and a deep counterfactual regret minimization algorithm. Then, the main process in the preset parallel training architecture batches the training samples from the sample queue into an experience buffer, and uses a preset multi-dimensional hierarchical sample sampling strategy and samples in the experience buffer to train the policy network and the dominance network to obtain the network parameters to be updated. Next, the subprocess updates the network copy according to the network parameters to be updated, and triggers the game tree traversal operation again based on the updated local network copy until a preset training termination condition is met. Finally, after the preset training termination condition is met, a current action decision is generated for the target player based on the trained target policy network, so that the target player can use the current action decision to interact with other players. This effectively achieves parallel training for DeepCFR, thereby improving training efficiency. Under preset queue capacity / concurrency control conditions, it reduces the risk of blocking and deadlock, mitigates distribution drift and training jitter, and improves sample utilization efficiency. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0015] Figure 1 A flowchart of a game interaction method based on a policy network is provided in this application; Figure 2 A schematic diagram of a pre-defined parallel training architecture provided in this application; Figure 3 This application provides a specific flowchart illustrating the subprocess traversal and sample enqueue process. Figure 4 A schematic diagram of a specific prior feature and normalization process provided for this application; Figure 5 A flowchart illustrating a specific main process training network provided in this application; Figure 6 A schematic diagram illustrating a specific multi-dimensional sample stratified sampling process provided in this application; Figure 7 A schematic diagram of a game interaction device based on a policy network is provided for this application; Figure 8 This application provides a structural diagram of an electronic device. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] In traditional strategy network training schemes for game theory games: 1) During DeepCFR network training, the game tree traversal (CPU-intensive) and network training (GPU-intensive) are performed sequentially, preventing the CPU / GPU from being fully utilized simultaneously. Furthermore, general distributed learning methods, which focus on environment interaction rather than the game tree, increase complexity. 2) Direct writing after local traversal in a single process is prone to blocking, while direct writing to the shared buffer by child processes in a multi-process environment can lead to deadlock. 3) Purely random sampling of training samples may result in a low proportion of recent or highly important samples, causing distribution drift and training jitter. Consequently, the interaction effect is poor when players use the trained strategy network.

[0018] To this end, this application provides a game interaction scheme based on policy networks, which can effectively realize parallel training for DeepCFR, thereby improving training efficiency. Under the conditions of preset queue capacity / concurrency control, it reduces the risk of blocking and deadlock, alleviates distribution drift and training jitter, and improves sample utilization efficiency.

[0019] See Figure 1 As shown, this embodiment of the invention discloses a game interaction method based on a policy network, applied to a computer device, comprising: Step S11: The game tree is traversed through the subprocesses in the preset parallel training architecture, the local network copy of the current subprocess, and the deep counterfactual regret minimization algorithm to obtain the training samples; wherein, the local network copy includes a copy of the policy network and a copy of the advantage network corresponding to each player; the advantage network is a neural network used to learn the counterfactual regret of each game interaction action relative to the baseline policy.

[0020] Combination Figure 2 As shown, this embodiment designs a parallel training architecture for DeepCFR. The main process (GPU (Graphics Processing Unit)) is responsible for sampling from the experience buffer, training the policy network and the advantage network, distributing parameters to child processes via the parameter synchronization queue, and managing the experience buffer (retrieving samples from the sample queue and writing them, and sampling from the buffer). Multiple child processes (CPU (Central Processing Unit)) retrieve parameters from the parameter synchronization queue, perform local game tree traversal (Monte Carlo sampling), generate advantage / policy samples, and add them to the sample queue. The sample queue and the parameter synchronization queue are connected to the main / child processes, and the experience buffer is managed by the main process. The data flow is 'child process traversal → enqueue → main process retrieves samples and writes them to the buffer → hierarchical sampling → GPU training → parameter distribution → child process update', realizing parallel traversal and training pipelines.

[0021] It should be noted that the experience buffer is a storage structure for storing advantage samples and strategy samples. In this embodiment, it is supplied by the child process via a queue and written by the main process. The child process queue is a multi-process safe queue between the child process and the main process, used to transfer samples. Advantage samples are used to train the advantage network, representing the advantage / regret value of each action in a certain information state, generated by calculating counterfactual values ​​after traversing the game tree. Strategy samples are used to train the strategy network, representing the target probability distribution of each game interaction action in a certain information state, generated by updating the strategy (such as regret value matching) after advantage / regret information. The advantage network is a neural network used to approximately learn the counterfactual regret (advantage) of each game interaction action relative to the baseline strategy. The input is the information state (which may contain prior features), and the output is the advantage value (or regret-related value) of each legal action, used to guide subsequent strategy updates and action selection tendencies. The strategy network is a neural network used to approximately learn the selection probability distribution of each legal action in a given information state. The input is the information state (which may contain prior features), and the output is the action probability (usually normalized), used for sampling strategies during actual game decisions or training. Understandably, the number of policy networks is one, while the number of advantage networks is the same as the number of players participating in a game. In other words, the type of game being targeted determines the number of advantage networks to be trained, based on the number of players participating in games of that type. Furthermore, the specific game interactions involved also depend on the game type.

[0022] Specifically, in this embodiment, a subprocess queue is used to replace the experience buffer samples: the samples in the experience buffer are changed from being generated locally to being supplied by subprocesses via a queue. Before placing the generated samples into the sample queue, the subprocess determines whether the sample queue is full to determine whether to use a hybrid replacement strategy to process the samples in the queue. That is: the subprocess in the preset parallel training architecture obtains the latest network parameters from the parameter synchronization queue; the subprocess updates the network based on the network parameters to obtain a local network copy of the current subprocess; the subprocess searches or samples along the game tree using Monte Carlo sampling based on the local network copy to complete the game tree traversal operation and obtain the traversal result; the subprocess, using the traversal result and the depth-counterfactual regret minimization algorithm, determines several... The training samples include advantage samples and policy samples. The advantage samples are used to train the advantage network. The policy samples are used to train the policy network. The policy network is a neural network used to learn the probability distribution of the selection of each game interaction action under a given information state. The subprocess determines whether the current sample queue is full to determine the queue determination result. If the queue determination result is negative, the subprocess stores the training samples into the sample queue. If the queue determination result is positive, the subprocess processes the historical samples in the sample queue using a preset hybrid replacement strategy to determine the processed sample queue. The subprocess stores the training samples into the processed sample queue.

[0023] Understandably, in combination Figure 3 As shown, regarding the sample generation and enqueueing of the subprocess, in this embodiment, after starting the subprocess, the latest network parameters are first retrieved from the parameter synchronization queue to update the local network copy. Then, the local network copy is used to perform game tree traversal (Monte Carlo sampling method to search / sample along the game tree), and training samples containing advantage samples and policy samples are generated according to the DeepCFR algorithm. Afterwards, local feature preprocessing or normalization can be performed on the samples before checking if the sample queue is full; otherwise, if no selection is made, the sample queue can be directly checked for fullness. If not full, the generated current batch of training samples is put into the sample queue; otherwise, if full, a hybrid replacement strategy is adopted: a portion of the samples are discarded using the first-in-first-out principle, and another portion is randomly selected from any position in the queue and discarded. After discarding, the current batch of training samples is then added to the queue. Furthermore, after the samples are placed, it is also possible to check if the main process is alive.

[0024] Furthermore, regarding the processing of a full sample queue, in this embodiment, the subprocess adopts a hybrid strategy of first-in-first-out (FIFO) and random replacement (e.g., discarding the oldest sample in some cases and randomly selecting a position to discard and then put it in in others), taking into account both timeliness and diversity, and reducing the risk of blocking. That is, through the subprocess, several historical samples in the sample queue are discarded according to the first-in-first-out principle to determine the first sample queue; through the subprocess, the historical samples in the first sample queue are randomly selected and discarded to determine the processed sample queue.

[0025] Furthermore, in combination Figure 4 As shown, regarding feature preprocessing or normalization, i.e., input feature prior and normalization, the prior features are features designed based on game rules and domain knowledge and concatenated with the information state (card strength, stage, etc.). In this embodiment: the target field in the game configuration string is parsed using the subprocess to obtain the normalization benchmark, and the parsing result is obtained; the prior features are obtained through the subprocess; wherein, the prior features are features of the information state designed based on game rules and domain knowledge and concatenated, including starting hand strength or current hand strength, hand type, waiting hand, community card features, etc. The process involves: 1) Separating the game rounds; 2) Using the subprocess and based on the prior features and the original information state corresponding to the training sample, concatenating the features to determine the concatenation result; 3) Using the subprocess and the concatenation result and the parsing result, normalizing the monetary features to determine the first normalization result; 4) Using the subprocess and based on the parsing result, normalizing and pruning the regret values ​​in the advantageous samples to determine the second normalization result; 5) Using the subprocess and based on the first and second normalization results, obtaining the optimized training sample.

[0026] In other words, in this embodiment, normalization benchmarks such as `max_stack` (the maximum scale at the start of the game) are parsed from the game configuration string; prior features (starting hand / current hand strength, stage, hand type, draw, community card characteristics, game round, action-related factors, etc.) are concatenated onto the original information state; regret values ​​are normalized and pruned using `max_stack` (e.g., to a bounded interval [-C, C]); and monetary features are normalized to a bounded interval using `max_stack` or the pot. This improves stability and convergence.

[0027] Step S12: The main process in the preset parallel training architecture writes the training samples that have been stored in the sample queue into the experience buffer in batches.

[0028] In this embodiment, combined with Figure 2As shown, after multiple subprocesses complete the generation and enqueueing of samples, the main process will retrieve the samples from the sample queue and write them into the experience buffer. That is, the main process in the preset parallel training architecture reads the training samples in batches from the sample queue until the sample queue is empty or the maximum sample collection amount is reached; the main process writes the read training samples into the experience buffer; the main process marks each sample in the experience buffer to obtain the sample iteration number field and sample importance measurement field corresponding to each sample.

[0029] Combination Figure 5 As shown, the main process uses `get_nowait` to batch sample from the sample pair queue until the maximum sample collection size is reached or the sample queue is empty. Afterward, when writing the sample to the experience buffer, the current iteration number is marked on the sample. Optionally, the importance of the sample can also be marked. The sample importance is a metric used for stratified or weighted sampling, such as the regret value magnitude or the contribution estimate to policy / dominance network updates.

[0030] Step S13: Through the main process, and using the preset multi-dimensional hierarchical sample sampling strategy and the samples in the experience buffer, perform policy network training and advantage network training for the current iteration training round to obtain the network parameters to be updated.

[0031] In this embodiment, combined with Figure 2 As shown, after the main process writes and manages the samples into the experience buffer, it can sample the training network from the buffer. That is: the main process determines whether the current experience buffer is full to determine the buffer judgment result; if the buffer judgment result is not full, the main process performs multi-dimensional hierarchical sampling on the samples in the experience buffer to determine the current training batch; if the buffer judgment result is yes, the main process processes the data in the experience buffer using a preset hybrid elimination strategy, and performs multi-dimensional hierarchical sampling based on the corresponding processed buffer to determine the current training batch; the main process trains the advantage network corresponding to each player using the current training batch, and combines the action advantage value output by the advantage network to guide the training of the strategy network to obtain the network parameters to be updated; the action advantage value is the advantage value corresponding to the game interaction action; the main process serializes the network parameters to be updated to determine the parameter serialization result; the main process stores the parameter serialization result in the parameter synchronization queue.

[0032] It is important to understand that, regarding the network training of the main process, combined with... Figure 5As shown, before sampling the buffer, it is necessary to check whether the experience buffer is full. If it is not full, the sample buffer is sampled directly. Otherwise, if it is full, some samples in the elimination buffer need to be mixed first: one part uses the first-in-first-out principle to eliminate the oldest sample, and the other part randomly selects a slot in the buffer and overwrites the selected slot to eliminate the original sample in that slot. The sample buffer is sampled after the mixed elimination.

[0033] Furthermore, in combination Figure 6 As shown, in this embodiment, multi-dimensional stratified sampling is adopted for samples in the experience buffer. In addition to dividing recent and historical samples by time / iteration number, high / low importance layers can also be divided by importance (regret value magnitude, contribution to strategy update, etc.). Sampling is performed proportionally from each layer or using importance-weighted stratified sampling to ensure the proportion of recent and high-importance samples, reduce distribution drift, and improve sample utilization efficiency. That is: through the main process, and based on the sample iteration number field and window, the samples in the experience buffer are divided by time dimension to determine the recent sample index set and the historical sample index set; through the main process, and based on the sample importance metric field, the recent sample index set and the historical sample index set are divided by importance dimension to determine high importance. The system comprises: a sample index set and a low-importance sample index set; sampling using the main process and a preset sampling ratio, the high-importance sample index set and the low-importance sample index set to determine a first sampling result; sampling using the main process and a preset importance-weighted sampling strategy, the high-importance sample index set and the low-importance sample index set to determine a second sampling result; merging the first sampling result and the second sampling result using the main process to obtain a merged sampling result; shuffling the sample indices in the merged sampling result using the main process to determine the shuffled merged sampling result; and reading samples from the experience buffer using the sample indices in the merged sampling result to determine the current training batch.

[0034] In addition, after the main process completes this round of network training, it can choose to check the survival status of each child process and save the checkpoint. Then it can proceed to the next round of network training.

[0035] Step S14: Through the subprocess, update the network according to the network parameters to be updated that have been stored in the parameter synchronization queue, and trigger the game tree traversal operation again based on the updated local network copy until the preset training termination condition is met.

[0036] In this embodiment, combined with Figure 2As shown, the child process retrieves the network parameters to be updated obtained by the main process through the parameter synchronization queue, and then updates its local network copy using these parameters. Furthermore, if further training is needed after the update, the updated local network copy will be used to generate training samples for a new training epoch. This logic is repeated until a preset training termination condition is met. This condition includes, but is not limited to, the training epoch threshold and network convergence status.

[0037] Step S15: After the preset training termination condition is met, generate the current action decision for the target player based on the trained target policy network, so that the target player can use the current action decision to interact with other players in the game.

[0038] In this embodiment, after training is complete, the trained target policy network can be used to generate current action decisions for the player when needed. These decisions include the selection probabilities of various game interaction actions. The player can then use these current action decisions to interact with other players in the game.

[0039] In summary, this embodiment proposes a parallel training scheme for the deep counterfactual regret minimization algorithm, which is particularly suitable for Texas Hold'em and similar incomplete information games. The beneficial effects of this scheme are as follows: (1) Parallel training: Traversal and training are carried out in parallel, CPU and GPU utilization are improved at the same time, and training throughput is significantly improved; queues and buffers are decoupled, and no Parameter Server is required.

[0040] (2) Queue supply buffer: The sample source is clear (queue → main process → buffer), which is easy to implement and debug; the mixed strategy of part first-in-first-out (FIFO) and part random replacement takes into account the timing and diversity, and reduces the risk of blocking and deadlock; the main process uniformly marks the importance of iteration and optionality, which facilitates multi-dimensional hierarchical sampling.

[0041] (3) Prior features and normalization: Prior information such as card strength and stage is explicitly introduced to reduce the learning difficulty; configuration analysis normalization benchmark makes multiple configurations reusable; regret value normalization and pruning reduce gradient explosion and oscillation.

[0042] (4) Multi-dimensional hierarchical sampling: The proportion of recent samples and high-importance samples is guaranteed, reducing distribution drift and training jitter, improving the utilization efficiency of key samples, and making convergence more stable.

[0043] In addition, regarding the above aspects of this plan, in order to ensure the effectiveness of implementation, the following modifications may be made as necessary: (1) Parallelism and communication: Simulate concurrency with multiple threads / coroutines within a single process; transmit large blocks of samples via shared memory and transmit only parameters via queues; when deploying on multiple machines, the queue is changed to a distributed message queue.

[0044] (2) Queues and buffers: When the queue is full, a mixed strategy of first-in-first-out (FIFO) and random replacement is adopted; the experience buffer can adopt mixed elimination, fixed first-in-first-out (FIFO), priority experience replay, etc.

[0045] (3) Prior and normalization: The prior dimension and formula can be adjusted according to the game; the normalization benchmark can be added to blind betting and bottom pot; other bounded functions can be used for pruning; some priors can be calculated on the main process side and written into the buffer.

[0046] (4) Stratified sampling: In addition to time (iteration / time stamp), the stratified dimensions can be importance (regret value magnitude, contribution, priority), player, game stage, etc.; it can be extracted from each layer proportionally or weighted by importance; the proportion and window can be dynamically adjusted; stratified sampling can be used only for strategy or advantage networks.

[0047] Therefore, in this embodiment of the application, applied to a computer device, training samples are first generated using a subprocess in a preset parallel training architecture, a local network copy of the subprocess, and a deep counterfactual regret minimization algorithm. Then, the main process in the preset parallel training architecture batches the training samples from the sample queue into an experience buffer, and uses a preset multi-dimensional hierarchical sample sampling strategy and samples in the experience buffer to train the policy network and the dominance network to obtain the network parameters to be updated. Next, the subprocess updates the network copy according to the network parameters to be updated, and triggers the game tree traversal operation again based on the updated local network copy until a preset training termination condition is met. Finally, after the preset training termination condition is met, a current action decision is generated for the target player based on the trained target policy network, so that the target player can use the current action decision to interact with other players. This effectively achieves parallel training for DeepCFR, thereby improving training efficiency. Under preset queue capacity / concurrency control conditions, it reduces the risk of blocking and deadlock, mitigates distribution drift and training jitter, and improves sample utilization efficiency.

[0048] See Figure 7 As shown in the illustration, this application also discloses a game interaction device based on a policy network, applied to a computer device, comprising: The sample generation module 11 is used to perform game tree traversal through subprocesses in a preset parallel training architecture, local network copies of the current subprocesses, and a deep counterfactual regret minimization algorithm to obtain training samples; wherein, the local network copy includes a copy of the policy network and a copy of the advantage network corresponding to each player; the advantage network is a neural network used to learn the counterfactual regret of each game interaction action relative to the baseline policy. The sample writing module 12 is used to write the training samples that have been stored in the sample queue into the experience buffer in batches through the main process in the preset parallel training architecture. The network training module 13 is used to train the strategy network and the advantage network in the current iteration training round through the main process and by using a preset multi-dimensional hierarchical sample sampling strategy and samples in the experience buffer, so as to obtain the network parameters to be updated. The network update module 14 is used to perform network updates through the subprocess and according to the network parameters to be updated that have been stored in the parameter synchronization queue, and to trigger the game tree traversal operation again based on the updated local network copy until the preset training termination condition is met. The decision generation module 15 is used to generate a current action decision for the target player based on the trained target policy network after the preset training termination condition is met, so that the target player can use the current action decision to interact with other players in the game.

[0049] In some specific embodiments, the sample generation module 11 may specifically include: The parameter acquisition unit is used to obtain the latest network parameters from the parameter synchronization queue through a subprocess in the preset parallel training architecture. A network copy determination unit is used to obtain a local network copy of the current subprocess by performing network updates based on the network parameters through the subprocess. The game tree traversal unit is used to search or sample along the game tree using the Monte Carlo sampling method through the subprocess and based on the local network copy, so as to complete the game tree traversal operation and obtain the traversal result; A sample determination unit is used to determine a number of training samples through the subprocess and by using the traversal results and the deep counterfactual regret minimization algorithm; wherein the training samples include dominant samples and policy samples; the dominant samples are samples used to train the dominant network; the policy samples are samples used to train the policy network; the policy network is a neural network used to learn the probability distribution of the selection of each game interaction action under a given information state. The queue judgment unit is used to determine whether the current sample queue is full through the subprocess, so as to determine the queue judgment result; The first sample storage unit is used to store the plurality of training samples into the sample queue through the subprocess if the queue judgment result is negative. A queue processing unit is used to process historical samples in the sample queue through the subprocess and using a preset hybrid replacement strategy if the queue determination result is yes, so as to determine the processed sample queue. The second sample storage unit is used to store the plurality of training samples into the processed sample queue through the subprocess.

[0050] In some specific embodiments, the queue processing unit may specifically include: The first-in-first-out (FIFO) processing subunit is used to discard several historical samples in the sample queue using the FIFO principle through the subprocess, so as to determine the first sample queue. The random discard subunit is used to randomly select and discard historical samples in the first sample queue through the subprocess, so as to determine the processed sample queue.

[0051] In some specific embodiments, the policy network-based game interaction device may further include: The parsing unit is used to parse the target field in the game configuration string using the subprocess, based on a normalized benchmark, to obtain the parsing result; The feature acquisition unit is used to acquire prior features through the subprocess; wherein, the prior features are features of information states designed and spliced ​​based on game rules and domain knowledge, including starting hand strength or current hand strength, hand type, waiting hand, community card features, and game round; The feature information splicing unit is used to splice the features based on the prior features and the original information state corresponding to the training sample through the subprocess to determine the splicing result. The first normalization unit is used to normalize the monetary features through the subprocess and by using the splicing result and the parsing result to determine the first normalization result; The second normalization unit is used to normalize and prune the regret values ​​in the advantageous samples through the subprocess and based on the parsing results to determine the second normalization result; The sample optimization completion unit is used to obtain the optimized training sample through the subprocess and based on the first normalization result and the second normalization result.

[0052] In some specific embodiments, the sample writing module 12 may specifically include: The sample reading unit is used to read the training samples in batches from the sample queue through the main process in the preset parallel training architecture until the sample queue is empty or the maximum sample collection amount is reached. The buffer writing unit is used to write the read training samples into the experience buffer through the main process; The sample labeling unit is used to label each sample in the experience buffer through the main process to obtain the sample iteration number field and the sample importance metric field corresponding to each sample.

[0053] In some specific embodiments, the network training module 13 may specifically include: The buffer judgment unit is used to determine, through the main process, whether the current experience buffer is full, so as to determine the buffer judgment result; The first sample sampling unit is used to perform multi-dimensional hierarchical sampling of the samples in the experience buffer through the main process if the buffer judgment result is negative, so as to determine the current training batch. The second sample sampling unit is used to process the data in the experience buffer through the main process and using a preset hybrid elimination strategy if the buffer judgment result is yes, and to perform multi-dimensional hierarchical sampling based on the corresponding processed buffer to determine the current training batch. The network training unit is used to train the advantage network corresponding to each player through the main process and using the current training batch, and to guide the training of the strategy network by combining the action advantage value output by the advantage network to obtain the network parameters to be updated; the action advantage value is the advantage value corresponding to the game interaction action. The parameter serialization unit is used to serialize the network parameters to be updated through the main process to determine the parameter serialization result; The queue storage unit is used to store the parameter serialization result into the parameter synchronization queue through the main process.

[0054] In some specific embodiments, the first sample sampling unit may specifically include: The time-dimension partitioning subunit is used to partition the samples in the experience buffer according to the time dimension through the main process and based on the sample iteration number field and window, so as to determine the recent sample index set and the historical sample index set; The importance dimension partitioning subunit is used to partition the recent sample index set and the historical sample index set according to the importance dimension through the main process and based on the sample importance metric field, so as to determine the high importance sample index set and the low importance sample index set. The ratio adopts a sub-unit, which is used to sample through the main process and using a preset sampling ratio, the high importance sample index set and the low importance sample index set to determine the first sampling result; An importance-weighted sampling subunit is used to perform sampling through the main process, using a preset importance-weighted sampling strategy, the high-importance sample index set, and the low-importance sample index set, to determine a second sampling result; A sampling merging unit is used to merge the first sampling result and the second sampling result through the main process to obtain a sampling merging result; The sequence shuffling unit is used to shuffle the sample indices in the sampling and merging result through the main process, so as to determine the shuffled sampling and merging result; The training batch determination unit is used to read samples from the experience buffer through the main process and according to the sample index in the sampling merging result to determine the current training batch.

[0055] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0056] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the policy network-based game interaction method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0057] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0058] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0059] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the policy-network-based game interaction method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0060] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned policy network-based game interaction method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0061] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0062] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0063] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0064] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0065] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A game interaction method based on policy networks, characterized in that, Applied to computer devices, including: The game tree is traversed using a pre-defined parallel training architecture, a local network copy of the current subprocess, and a deep counterfactual regret minimization algorithm to obtain training samples. The local network copy includes a copy of the policy network and a copy of the advantage network corresponding to each player. The advantage network is a neural network used to learn the counterfactual regret of each game interaction action relative to the baseline policy. The main process in the preset parallel training architecture writes the training samples that have been stored in the sample queue into the experience buffer in batches. Through the main process, and by using the preset multi-dimensional hierarchical sample sampling strategy and the samples in the experience buffer, the strategy network and the advantage network are trained in the current iteration training round to obtain the network parameters to be updated. The subprocess updates the network based on the network parameters to be updated that have been stored in the parameter synchronization queue, and then triggers the game tree traversal operation again based on the updated local network copy until the preset training termination condition is met. After the preset training termination condition is met, a current action decision is generated for the target player based on the trained target policy network, so that the target player can use the current action decision to interact with other players in the game.

2. The game interaction method based on policy networks according to claim 1, characterized in that, The process of traversing the game tree to obtain training samples by using subprocesses in a preset parallel training architecture, local network copies of the current subprocesses, and a deep counterfactual regret minimization algorithm includes: The latest network parameters are obtained from the parameter synchronization queue through subprocesses in the pre-defined parallel training architecture. The local network copy of the current subprocess is obtained by performing network updates based on the network parameters through the subprocess. Through the subprocess and based on the local network copy, the game tree traversal operation is completed by searching or sampling along the game tree using the Monte Carlo sampling method, and the traversal result is obtained. Through the subprocess, and using the traversal results and the deep counterfactual regret minimization algorithm, several training samples are determined; wherein, the training samples include advantage samples and policy samples; the advantage samples are samples used to train the advantage network; the policy samples are samples used to train the policy network; the policy network is a neural network used to learn the probability distribution of the selection of each game interaction action under a given information state. The subprocess determines whether the current sample queue is full in order to determine the queue determination result. If the queue determination result is negative, then the subprocess stores the plurality of training samples into the sample queue. If the queue determination result is yes, then the historical samples in the sample queue are processed through the subprocess and using a preset hybrid replacement strategy to determine the processed sample queue. The subprocess stores the plurality of training samples into the processed sample queue.

3. The game interaction method based on policy networks according to claim 2, characterized in that, The process of processing historical samples in the sample queue through the subprocess and using a preset hybrid replacement strategy includes: The subprocess discards several historical samples from the sample queue using the first-in-first-out principle to determine the first sample queue. The subprocess randomly selects and discards historical samples from the first sample queue to determine the processed sample queue.

4. The game interaction method based on policy networks according to claim 2, characterized in that, After determining a number of training samples, the process also includes: The subprocess performs normalized baseline parsing on the target fields in the game configuration string to obtain the parsing results. The prior features are obtained through the subprocess; wherein, the prior features are features of information states designed and spliced ​​based on game rules and domain knowledge, including starting hand strength or current hand strength, hand type, waiting hand, community card features and game round; The splicing result is determined by splicing the data through the subprocess based on the prior features and the original information state corresponding to the training sample. Through the subprocess, and using the splicing result and the parsing result, the monetary features are normalized to determine the first normalization result; The second normalization result is determined by normalizing and pruning the regret values ​​in the advantageous samples based on the analysis results through the subprocess. The optimized training samples are obtained through the subprocess and based on the first normalization result and the second normalization result.

5. The game interaction method based on policy networks according to any one of claims 1 to 4, characterized in that, The step of writing the training samples, which have been stored in the sample queue, into the experience buffer in batches through the main process in the preset parallel training architecture includes: The main process in the preset parallel training architecture reads the training samples in batches from the sample queue until the sample queue is empty or the maximum sample collection amount is reached. The main process reads the training samples and writes them into the experience buffer. The main process marks each sample in the experience buffer to obtain the sample iteration count field and sample importance metric field corresponding to each sample.

6. The game interaction method based on policy networks according to claim 5, characterized in that, The process of training the policy network and the advantage network for the current iteration through the main process and using a preset multi-dimensional hierarchical sample sampling strategy and samples in the experience buffer includes: The main process determines whether the current experience buffer is full in order to determine the buffer judgment result. If the buffer determination result is negative, then the main process performs multi-dimensional hierarchical sampling on the samples in the experience buffer to determine the current training batch. If the buffer determination result is yes, then the main process processes the data in the experience buffer using a preset hybrid elimination strategy, and performs multi-dimensional hierarchical sampling based on the corresponding processed buffer to determine the current training batch. The main process trains the advantage network corresponding to each player using the current training batch, and combines the action advantage value output by the advantage network to guide the training of the policy network, so as to obtain the network parameters to be updated; the action advantage value is the advantage value corresponding to the game interaction action. The main process serializes the network parameters to be updated to determine the serialization result. The main process stores the parameter serialization result into the parameter synchronization queue.

7. The game interaction method based on policy networks according to claim 6, characterized in that, The step of performing multi-dimensional, hierarchical sampling of samples in the experience buffer through the main process includes: The main process, based on the sample iteration count field and window, divides the samples in the experience buffer into time-dimensional segments to determine the recent sample index set and the historical sample index set. Through the main process, and based on the sample importance metric field, the recent sample index set and the historical sample index set are divided according to importance dimensions to determine the high-importance sample index set and the low-importance sample index set. The first sampling result is determined by sampling through the main process, using a preset sampling ratio, the high-importance sample index set, and the low-importance sample index set; The main process is used to perform sampling, utilizing a preset importance-weighted sampling strategy, the high-importance sample index set, and the low-importance sample index set, to determine the second sampling result. The main process merges the first sampling result and the second sampling result to obtain a sampling merge result. The main process shuffles the sample indices in the sampling and merging results to determine the shuffled sampling and merging results. The main process reads samples from the experience buffer based on the sample index in the sampling merging result to determine the current training batch.

8. A game interaction device based on a policy network, characterized in that, Applied to computer devices, including: The sample generation module is used to perform game tree traversal through subprocesses in a preset parallel training architecture, local network copies of the current subprocesses, and a deep counterfactual regret minimization algorithm to obtain training samples; wherein, the local network copy includes a copy of the policy network and a copy of the advantage network corresponding to each player; the advantage network is a neural network used to learn the counterfactual regret of each game interaction action relative to the baseline policy; The sample writing module is used to write the training samples that have been stored in the sample queue into the experience buffer in batches through the main process in the preset parallel training architecture. The network training module is used to train the strategy network and the advantage network for the current iteration training round through the main process and by using a preset multi-dimensional hierarchical sample sampling strategy and samples in the experience buffer, so as to obtain the network parameters to be updated. The network update module is used to perform network updates through the subprocess and according to the network parameters to be updated that have been stored in the parameter synchronization queue, and to trigger the game tree traversal operation again based on the updated local network copy until the preset training termination condition is met. The decision generation module is used to generate a current action decision for the target player based on the trained target policy network after the preset training termination condition is met, so that the target player can use the current action decision to interact with other players in the game.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the policy network-based game interaction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the game interaction method based on a policy network as described in any one of claims 1 to 7.

Images