Multi-agent deep reinforcement learning method, device, equipment and medium

Abstract

The invention relates to a multi-agent deep reinforcement learning method and device, equipment and a medium, and relates to the technical field of reinforcement learning. The method comprises the steps of constructing a multi-agent reinforcement learning model for a multi-agent collaborative decision problem in a severe game scene; training the model by a self-adaptive course learning mechanism based on multi-type prior strategy guidance and strategy entropy; in the training interaction process, staged synchronous storage of a multi-agent joint trajectory is carried out through a double-experience playback area, mixed sampling is carried out by utilizing a curriculum type sampling proportion and a priority experience playback mechanism, and a learning strategy is trained by combining loss calculation of mixed sampling and strategy updating frequency with attenuation; along with course learning promotion, the prior guidance length is gradually reduced until training is finished, and an optimal collaborative decision of multiple agents is output. According to the method, the training efficiency of multi-agent reinforcement learning can be improved, and the overall performance of collaborative decision-making can be improved.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning technology, and in particular to a multi-agent deep reinforcement learning method, apparatus, device, and medium. Background Technology

[0002] With the widespread application of serious games in professional fields such as emergency rescue training and medical collaborative operations, the complexity and realism requirements of game scenarios have significantly increased. Especially in serious scenarios such as team collaboration training requiring multi-person collaborative decision-making, extremely high demands are placed on the collaborative decision-making capabilities and dynamic environment adaptability of agents. Multi-agent reinforcement learning, as an important research direction in machine learning, breaks through the limitations of traditional single-agent reinforcement learning. By constructing interaction and collaborative learning mechanisms among multiple agents in a shared environment, it can effectively address multi-agent sequence decision-making problems in complex task environments. Therefore, this technology is widely used in key tasks such as collaborative behavior modeling, decision inference, and adaptive adjustment of training difficulty for non-player characters in serious games, providing important technical support for improving the training effect, immersion, and intelligence level of serious games, and possessing broad application prospects.

[0003] However, in serious game environments, multi-agent control tasks face significant bottlenecks due to the large number of agents involved in decision-making. On the one hand, the system state space, observation space, and joint action space expand exponentially with the number of agents, leading to the "curse of dimensionality." On the other hand, the number of effective policies in multi-agent decision-making tasks does not increase synchronously with the expansion of the solution space, resulting in a sharp decline in the proportion of "high-quality experience" in the samples, creating a "sample sparsity" problem. These two factors together pose challenges to multi-agent reinforcement learning algorithms in terms of sample utilization, computational overhead, and training efficiency. In particular, it is difficult to obtain effective policy feedback signals in the early stages of training, which can easily lead to slow convergence or even training stagnation, ultimately affecting the accuracy of agent decisions and the rationality of the task.

[0004] Among existing improvement schemes, some studies have proposed combining prior policies with offline pre-training. This involves using the pre-trained value function on an offline dataset as the initial point for online training, and constructing a temporal difference learning objective within a value decomposition framework based on this function to guide the training process through the prior policy. However, this method requires the prior policy to be a network with a specific structure, and it must be consistent with the network structure of the agent being trained online, limiting its applicability. Furthermore, this scheme determines whether to guide the policy by comparing the state-action values ​​of the prior policy and the learned policy; this guidance logic is susceptible to biases in the value function evaluation. Simultaneously, the value function obtained through offline training may have insufficient estimation accuracy when facing new states, potentially further impacting training efficiency and final decision performance.

[0005] In summary, current multi-agent reinforcement learning techniques applied to decision-making in serious games struggle to effectively balance training efficiency and decision-making performance, necessitating the development of superior technical solutions to overcome existing bottlenecks. Summary of the Invention

[0006] Therefore, it is necessary to provide a multi-agent deep reinforcement learning method, apparatus, device, and medium to address the aforementioned technical problems.

[0007] A multi-agent deep reinforcement learning method, the method comprising: S1 models the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and constructs a multi-agent reinforcement learning model. S2 employs an adaptive course learning mechanism based on multi-type prior policy guidance and policy entropy to train the multi-agent reinforcement learning model. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps; S3, during the interaction between the multi-agent and the game environment, synchronously stores the joint trajectory of the multi-agent generated in the prior policy stage and the learning policy stage in the dual experience replay area, and uses the course-based sampling ratio and the priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agent in the dual experience replay area. Based on the mixed sampling samples, the reinforcement learning loss is calculated and the policy update frequency with decay is combined to train and update the learning policy. S4. At the end of each training round, determine whether the policy entropy of the learning policy meets the preset threshold. If it does, the course learning progresses to the next stage, and after reducing the value of h, S2 and S3 are repeated. Otherwise, the course learning does not progress, and S2 and S3 are repeated directly. Training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0008] In one embodiment, the multi-agent cooperative decision-making problem in serious game scenarios is modeled as a decentralized partially observable Markov decision process, represented as: ; In the formula, For the number of agents, For the state space, the agent at time... The state is ; For the action space, the intelligent agent at any time The joint action is : The transition function describes the state evolution of the game environment as a result of joint actions; The reward function outputs the reward obtained by the agent. and These are the observation function and the observation space, respectively, since the agent only has partial observability. Map states and actions to moments. Observations ; This is a discount factor used to discount future rewards; each agent learns a stochastic policy. This strategy is based on action-observation history. Generate actions; rewards Defined as discounted cumulative reward: Under the decentralized partially observable Markov decision process setting, the core objective of multi-agent reinforcement learning is to optimize the policy to maximize the expected discounted reward, formally represented as... ;in, For real numbers, It indicates a desire for the expected value.

[0009] In one embodiment, the multiple types of prior strategies include: knowledge engineering-based strategies, planning and decision theory-based strategies, and machine learning-based strategies; knowledge engineering-based strategies include game rule scripts and finite state machines that encode multi-agent decision-making logic; planning and decision theory-based strategies include planning oriented towards multi-agent collaborative decision-making objectives using standard language in the planning domain; and machine learning-based strategies include neural networks that guide multi-agent collaborative decision-making.

[0010] In one embodiment, policy entropy is defined as: ;in, For a moment The hidden state estimation of the observation, It is the entropy function. The parameter is The action probability distribution of the strategy under the hidden state estimation; During the course learning process, the prior guidance length h is adaptively adjusted by calculating the policy entropy of the learning strategy. If the policy entropy meets the preset threshold, the course learning progresses to the next stage, according to the formula h=h The value of h is gradually decreased by / m; if this is not satisfied, the current value of h is maintained for training until the policy entropy meets the preset threshold; where h is the initial value. Set H to the total time span of the task, where m is the number of preset course learning stages.

[0011] In one embodiment, a course-based sampling ratio and a priority experience replay mechanism are used to perform mixed sampling of the multi-agent joint trajectory in the dual experience replay region, including: The dual experience replay area includes the prior experience replay area. and online experience replay area , Used to store the prior multi-agent joint trajectory generated in the prior policy phase of the first h steps of each training round. Used to store H after each training round The online multi-agent joint trajectory generated during the h-step learning strategy phase; the multi-agent joint trajectory is defined as the entire state-action sequence of a multi-agent in a complete game task attempt, from the initial game state to the final game state. In each training round, the time-dependent proportions of the two experience replay zones are used. Perform hybrid sampling of multi-agent joint trajectories, where, from The proportion of sampling in the middle is ;from The proportion of sampling in the middle is As training progresses, The value is determined according to predefined scheduling or performance feedback, from bias. The sampling transitions smoothly to almost complete dependence. Sampling; During mixed sampling, and Internally, priority indicators related to multi-agent TD-error or joint action advantage are maintained respectively. Based on the priority indicators, key collaborative behavior segments within the experience replay area are assigned higher sampling probabilities, thereby realizing priority experience replay of prior multi-agent joint trajectories and online multi-agent joint trajectories respectively.

[0012] In one embodiment, the attenuated policy update frequency uses a cosine decay method, expressed as: ; in, For a moment The frequency of strategy updates This indicates the initial policy update frequency. This indicates the policy update frequency at the end of the decay period. This indicates the maximum number of decay steps.

[0013] A multi-agent deep reinforcement learning device, the device comprising: The first module is used to model the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and to construct a multi-agent reinforcement learning model. The second module is used to train a multi-agent reinforcement learning model using an adaptive course learning mechanism based on multi-type prior policies and policy entropy. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps; The third module is used to synchronously store the joint trajectory of the multi-agents generated in the prior policy stage and the learning policy stage in the dual experience replay area during the interaction between the multi-agents and the game environment. It uses the course-based sampling ratio and the priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agents in the dual experience replay area. Based on the mixed sampling samples, it calculates the reinforcement learning loss and combines it with the policy update frequency with decay to train and update the learning policy. The fourth module is used to determine whether the policy entropy of the learning policy meets the preset threshold at the end of each training round. If it does, the course learning progresses to the next stage, and after reducing the value of h, the second and third modules are called again. Otherwise, the course learning does not progress, and the second and third modules are called again directly. The training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0014] In one embodiment, the apparatus further includes: a priori policy knowledge base for storing multiple types of prior policies, including knowledge engineering-based policies, planning and decision theory-based policies, and machine learning-based policies; the knowledge engineering-based policies include game rule scripts and finite state machines that encode multi-agent decision-making logic; the planning and decision theory-based policies include planning oriented towards multi-agent collaborative decision-making goals using standard language in the planning domain; and the machine learning-based policies include neural networks that guide multi-agent collaborative decision-making.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: S1 models the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and constructs a multi-agent reinforcement learning model. S2 employs an adaptive course learning mechanism based on multi-type prior policy guidance and policy entropy to train the multi-agent reinforcement learning model. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps; S3, during the interaction between the multi-agent and the game environment, synchronously stores the joint trajectory of the multi-agent generated in the prior policy stage and the learning policy stage in the dual experience replay area, and uses the course-based sampling ratio and the priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agent in the dual experience replay area. Based on the mixed sampling samples, the reinforcement learning loss is calculated and the policy update frequency with decay is combined to train and update the learning policy. S4. At the end of each training round, determine whether the policy entropy of the learning policy meets the preset threshold. If it does, the course learning progresses to the next stage, and after reducing the value of h, S2 and S3 are repeated. Otherwise, the course learning does not progress, and S2 and S3 are repeated directly. Training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0016] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: S1 models the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and constructs a multi-agent reinforcement learning model. S2 employs an adaptive course learning mechanism based on multi-type prior policy guidance and policy entropy to train the multi-agent reinforcement learning model. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps; S3, during the interaction between the multi-agent and the game environment, synchronously stores the joint trajectory of the multi-agent generated in the prior policy stage and the learning policy stage in the dual experience replay area, and uses the course-based sampling ratio and the priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agent in the dual experience replay area. Based on the mixed sampling samples, the reinforcement learning loss is calculated and the policy update frequency with decay is combined to train and update the learning policy. S4. At the end of each training round, determine whether the policy entropy of the learning policy meets the preset threshold. If it does, the course learning progresses to the next stage, and after reducing the value of h, S2 and S3 are repeated. Otherwise, the course learning does not progress, and S2 and S3 are repeated directly. Training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0017] The aforementioned multi-agent deep reinforcement learning method, apparatus, device, and medium have the following advantages compared to existing technologies: 1. To address the multi-agent collaborative decision-making problem in serious game scenarios, a multi-agent reinforcement learning model is constructed. An adaptive learning mechanism based on multi-type prior policies and policy entropy is employed for model training. The multi-type prior policy guidance makes the decision-making process of the multi-agents more accurate and reasonable, thereby improving their adaptability and flexibility in complex game environments. Policy entropy enables adaptive learning from strong priors to self-organized collaboration, gradually eliminating reliance on prior policies while avoiding ineffective exploration. This significantly shortens the model training convergence cycle, reduces hardware resource consumption such as computing power and storage, and improves the consistency and training efficiency of multi-agent collaborative decision-making.

[0018] 2. By using dual experience replay zones, the data integrity and temporal consistency of the joint trajectory of multiple agents are ensured, laying a reliable data foundation for collaborative decision-making. The hybrid sampling strategy, which utilizes a course-based sampling ratio and a priority experience replay mechanism, dynamically adapts to the learning progress while focusing on high-value collaborative experiences. This improves sample utilization efficiency and helps to accurately capture key interaction patterns. Furthermore, by combining loss calculation of hybrid sampling samples with a decaying policy update frequency, dynamic adaptation and robust optimization of policy iteration are achieved, ultimately significantly improving the accuracy, stability, and overall performance of multi-agent collaborative decision-making. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a multi-agent deep reinforcement learning method in one embodiment; Figure 2 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] In one embodiment, such as Figure 1 As shown, taking a serious game scenario as an example, a multi-agent deep reinforcement learning method is provided, including the following steps: S1 models the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and constructs a multi-agent reinforcement learning model.

[0022] S2 employs an adaptive course learning mechanism based on multi-type prior policy guidance and policy entropy to train the multi-agent reinforcement learning model. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps.

[0023] Among them, multiple types of prior strategies rely on prior knowledge constructed during real-world gameplay. Using these as teachers in the early stages of multi-agent learning facilitates the collection of action guidance with non-zero reward samples, supports the use of arbitrary forms of prior strategies, and makes the multi-agent's decision-making process more accurate and rational, significantly improving the robustness of multi-agent decision-making. This, in turn, helps the agent exhibit higher adaptability and flexibility when facing complex game environments. The learning strategy refers to reinforcement learning strategies where multi-agents learn through trial and error, primarily used for exploration and online decision improvement. In S2, since the learning strategy only needs to determine the remaining H steps starting from step h... The action of step h greatly reduces the problem space, thus simplifying the task. The number of steps h gradually increases as the course progresses. h enables training tasks to transition naturally from simple to complex.

[0024] Policy entropy is an indicator that measures the uncertainty of policy action selection and is used in reinforcement learning to quantify the model's exploration ability. High entropy indicates strong policy randomness, while low entropy indicates strong policy determinism. During model training, the prior guidance length h can be adaptively adjusted based on policy entropy. In the initial stage, h is relatively large, and the system strongly relies on prior collaborative behavior, which can reduce the effective search range of the joint action space, reduce computational complexity, and improve training efficiency. As the performance and stability of the multi-agent policy improve, h is gradually shortened (course learning changes from "strong prior" to "self-organizing collaboration") by monitoring uncertainty indicators such as policy entropy, until h→0, completing the de-dependence on the prior (de-priorification), significantly shortening the model training convergence cycle, reducing the consumption of hardware resources such as computing power and storage, and improving the consistency and training efficiency of multi-agent collaborative decision-making.

[0025] S3, during the interaction between the multi-agent and the game environment, synchronously stores the joint trajectory of the multi-agent generated in the prior policy stage and the learning policy stage in the dual experience replay area. It also uses a course-based sampling ratio and a priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agent in the dual experience replay area. Based on the mixed sampling samples, it calculates the reinforcement learning loss and combines it with a policy update frequency with decay to train and update the learning policy.

[0026] The dual experience replay area stores the multi-agent joint trajectory in stages, balancing the utilization of prior collaborative behaviors with the full learning of online exploration behaviors. This ensures the temporal consistency and information integrity of the experience data, providing accurate data support for multi-agent collaborative decision-making. A hybrid sampling strategy combining a course-based sampling ratio and a priority experience replay mechanism dynamically adapts to the learning progress, ensuring the scientific rigor and robustness of strategy iteration. Focusing on high-value interaction experiences improves sample utilization and reduces computational overhead, helping agents accurately capture key collaborative patterns in complex scenarios and significantly improving the accuracy and overall performance of multi-agent collaborative decision-making. Furthermore, by coordinating loss calculations from hybrid sampling samples with a decaying strategy update frequency, the learning strategy updates dynamically adjust to the course learning process, ensuring the robustness and rationality of strategy iteration, thereby improving the accuracy and reliability of multi-agent joint decision-making.

[0027] S4. At the end of each training round, determine whether the policy entropy of the learning policy meets the preset threshold. If it does, the course learning progresses to the next stage, and after reducing the value of h, S2 and S3 are repeated. Otherwise, the course learning does not progress, and S2 and S3 are repeated directly. Training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0028] It should be noted that the serious game scenario is a specific illustration of one application scenario of this method. The method proposed in this application can actually be applied to various scenarios involving multi-agent decision-making, such as robot control and traffic driving.

[0029] In one embodiment, the multi-agent cooperative decision-making problem in serious game scenarios is modeled as a decentralized partially observable Markov decision process, represented as: ; In the formula, For the number of agents, For the state space, the agent at time... The state is ; For the action space, the intelligent agent at any time The joint action is : The transition function describes the state evolution of the game environment as a result of joint actions; The reward function outputs the reward obtained by the agent. and These are the observation function and the observation space, respectively, since the agent only has partial observability. Map states and actions to moments. Observations ; This is a discount factor used to discount future rewards; each agent learns a stochastic policy. This strategy is based on action-observation history. Generate actions; rewards Defined as discounted cumulative reward: Under the decentralized partially observable Markov decision process setting, the core objective of multi-agent reinforcement learning is to optimize the policy to maximize the expected discounted reward, formally represented as... ;in, For real numbers, It indicates a desire for the expected value.

[0030] Specifically, in serious game scenarios, the game state of multiple agents encompasses the agent's own state (such as location, resources, and identifiers), the environment's state (scene terrain, task progress, etc.), and the states associated with other agents (teammate / opponent positions, action intentions, etc.). Actions include discrete or continuous behaviors such as movement, attack, and cooperation, adapting to the core gameplay and multi-agent interaction needs of the scenario. Rewards are designed around task objectives (victory, task completion) and collaborative / competitive rules, including individual contribution rewards (completing sub-tasks) and team-wide rewards (team victory, goal achievement), guiding agents to optimize decision-making.

[0031] In one embodiment, the multiple types of prior strategies include: knowledge engineering-based strategies, planning and decision theory-based strategies, and machine learning-based strategies; knowledge engineering-based strategies include game rule scripts and finite state machines that encode multi-agent decision-making logic; planning and decision theory-based strategies include planning oriented towards multi-agent collaborative decision-making goals using the Planning Domain Definition Language (PDDL); and machine learning-based strategies include neural networks that guide multi-agent collaborative decision-making.

[0032] In one embodiment, policy entropy is defined as: ;in, For a moment The hidden state estimation of the observation, It is the entropy function. The parameter is The strategy's action probability distribution under hidden state estimation. During the course learning process, the prior guidance length h is adaptively adjusted by calculating the policy entropy of the learned strategy. If the policy entropy meets a preset threshold, the course learning progresses to the next stage, according to the formula h = h... The value of h is gradually decreased by / m; if this is not satisfied, the current value of h is maintained for training until the policy entropy meets the preset threshold; where h is the initial value. Set H to the total time span of the task, where m is the number of preset course learning stages.

[0033] In one embodiment, a course-based sampling ratio and a priority experience replay mechanism are used to perform mixed sampling of the multi-agent joint trajectory in the dual experience replay region, including: The dual experience replay area includes the prior experience replay area. and online experience replay area , Used to store the prior multi-agent joint trajectory generated in the prior policy phase of the first h steps of each training round, which can preserve and mine prior collaborative decision-making patterns and their contribution to long-term rewards. Used to store H after each training round The online multi-agent joint trajectory generated in the h-step learning strategy phase reflects the behavioral distribution and dynamics of the current multi-agent strategy under real online collaborative exploration; the multi-agent joint trajectory is defined as the entire state-action sequence of a multi-agent in a complete game task attempt, from the initial game state to the final game state.

[0034] In each training round, the time-dependent proportions of the two experience replay zones are used. Perform hybrid sampling of multi-agent joint trajectories, where, from The proportion of sampling in the middle is ;from The proportion of sampling in the middle is As training progresses, The value is determined according to predefined scheduling or performance feedback, from bias. The sampling transitions smoothly to almost complete dependence. The sampling allows for a transition in the curriculum from imitating prior collaborative behaviors to learning autonomous collaborative strategies.

[0035] During mixed sampling, and Internally, priority indices related to multi-agent TD-error (temporal difference error) or joint action advantages are maintained. Based on these priority indices, key cooperative behavior segments within the experience replay zone are assigned higher sampling probabilities, enabling prioritized experience replay of prior multi-agent joint trajectories and online multi-agent joint trajectories, respectively. Specifically, the TD-error expression is: It should be understood that by prioritizing the replay of prior and online trajectories, key online collaborative experiences can be fully extracted to guide multi-agent decision-making without being overwhelmed by a large amount of prior data.

[0036] In one embodiment, the attenuated policy update frequency uses a cosine decay method, expressed as: ; in, For a moment The frequency of strategy updates This indicates the initial policy update frequency. This indicates the policy update frequency at the end of the decay period. This represents the maximum number of decay steps. It should be understood that a decaying policy update frequency allows for a high update frequency in the early stages of training, enabling multiple conservative updates to utilize heuristic samples generated by the prior policy, thus significantly improving training efficiency. As training progresses, the update frequency gradually normalizes.

[0037] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0038] In one embodiment, a multi-agent deep reinforcement learning apparatus is provided, comprising: The first module is used to model the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and to construct a multi-agent reinforcement learning model. The second module is used to train a multi-agent reinforcement learning model using an adaptive course learning mechanism based on multi-type prior policies and policy entropy. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps; The third module is used to synchronously store the joint trajectory of the multi-agents generated in the prior policy stage and the learning policy stage in the dual experience replay area during the interaction between the multi-agents and the game environment. It uses the course-based sampling ratio and the priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agents in the dual experience replay area. Based on the mixed sampling samples, it calculates the reinforcement learning loss and combines it with the policy update frequency with decay to train and update the learning policy. The fourth module is used to determine whether the policy entropy of the learning policy meets the preset threshold at the end of each training round. If it does, the course learning progresses to the next stage, and after reducing the value of h, the second and third modules are called again. Otherwise, the course learning does not progress, and the second and third modules are called again directly. The training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0039] In one embodiment, the apparatus further includes: a priori policy knowledge base for storing multiple types of prior policies, including knowledge engineering-based policies, planning and decision theory-based policies, and machine learning-based policies; the knowledge engineering-based policies include game rule scripts and finite state machines that encode multi-agent decision-making logic; the planning and decision theory-based policies include planning oriented towards multi-agent collaborative decision-making goals using standard language in the planning domain; and the machine learning-based policies include neural networks that guide multi-agent collaborative decision-making.

[0040] Specific limitations regarding a multi-agent deep reinforcement learning device can be found in the limitations of a multi-agent deep reinforcement learning method described above, and will not be repeated here. Each module in the aforementioned multi-agent deep reinforcement learning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0041] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 2As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-agent deep reinforcement learning method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0042] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0043] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: S1 models the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and constructs a multi-agent reinforcement learning model. S2 employs an adaptive course learning mechanism based on multi-type prior policy guidance and policy entropy to train the multi-agent reinforcement learning model. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps; S3, during the interaction between the multi-agent and the game environment, synchronously stores the joint trajectory of the multi-agent generated in the prior policy stage and the learning policy stage in the dual experience replay area, and uses the course-based sampling ratio and the priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agent in the dual experience replay area. Based on the mixed sampling samples, the reinforcement learning loss is calculated and the policy update frequency with decay is combined to train and update the learning policy. S4. At the end of each training round, determine whether the policy entropy of the learning policy meets the preset threshold. If it does, the course learning progresses to the next stage, and after reducing the value of h, S2 and S3 are repeated. Otherwise, the course learning does not progress, and S2 and S3 are repeated directly. Training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0044] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: S1 models the multi-agent collaborative decision-making problem in serious game scenarios as a decentralized partially observable Markov decision process and constructs a multi-agent reinforcement learning model. S2 employs an adaptive course learning mechanism based on multi-type prior policy guidance and policy entropy to train the multi-agent reinforcement learning model. Before training, H is preset as the number of steps in the time span, and h is the prior guidance length. In each training round, the multi-agent first interacts with the game environment for h steps based on the prior policy, and then interacts with the game environment for the remaining H steps based on the learned policy. h steps; S3, during the interaction between the multi-agent and the game environment, synchronously stores the joint trajectory of the multi-agent generated in the prior policy stage and the learning policy stage in the dual experience replay area, and uses the course-based sampling ratio and the priority experience replay mechanism to perform mixed sampling on the joint trajectory of the multi-agent in the dual experience replay area. Based on the mixed sampling samples, the reinforcement learning loss is calculated and the policy update frequency with decay is combined to train and update the learning policy. S4. At the end of each training round, determine whether the policy entropy of the learning policy meets the preset threshold. If it does, the course learning progresses to the next stage, and after reducing the value of h, S2 and S3 are repeated. Otherwise, the course learning does not progress, and S2 and S3 are repeated directly. Training ends when the number of training steps reaches the preset maximum value. Based on the trained multi-agent reinforcement learning model, the optimal collaborative decision of the multi-agent is output, and the optimal collaborative decision guides the joint actions and environmental interactions of the multi-agent in serious games.

[0045] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0046] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0047] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A multi-agent deep reinforcement learning method, characterized in that, The method comprises: S1, modeling a multi-agent cooperative decision-making problem in a serious game scene as a decentralized partially observable Markov decision process, and constructing a multi-agent reinforcement learning model; S2, an adaptive curriculum learning mechanism based on multi-type prior strategy guidance and strategy entropy is used to train the multi-agent reinforcement learning model, before training, presetting H as the time span step number and h as the prior guidance length; in each training round, the multi-agent first interacts with the game environment for h steps based on the prior strategy, and then interacts with the game environment for the remaining H steps based on the learning strategy S3, in the interaction process of the multi-agent and the game environment, synchronously storing the multi-agent joint trajectories generated in the priori strategy stage and the learning strategy stage in a double experience replay area, and using a curriculum sampling ratio and a priority experience replay mechanism to perform mixed sampling on the multi-agent joint trajectories in the double experience replay area, calculating a reinforcement learning loss according to the mixed sampling samples and jointly using a strategy update frequency with decay to train and update the learning strategy; S4, at the end of each training round, judging whether the strategy entropy of the learning strategy meets a preset threshold, if yes, the curriculum learning is promoted to the next stage, and after the value of h is reduced, S2 and S3 are repeatedly executed; otherwise, the curriculum learning is not promoted, and S2 and S3 are directly repeatedly executed; until the training step number reaches a preset maximum value, the training is ended, the optimal cooperative decision-making of the multi-agent is output based on the trained multi-agent reinforcement learning model, and the optimal cooperative decision-making is used to guide the joint action and environment interaction of the multi-agent in the serious game. 2.The multi-agent deep reinforcement learning method of claim 1, wherein, The multi-agent cooperative decision-making problem in the serious game scene is modeled as a decentralized partially observable Markov decision process, and is represented as: ； In the formula, For the number of agents, For the state space, the agent at time... The state is ; For the action space, the intelligent agent at any time The joint action is : The transition function describes the state evolution of the game environment as a result of joint actions; The reward function outputs the reward obtained by the agent. and These are the observation function and the observation space, respectively, since the agent only has partial observability. Map states and actions to moments. Observations ; This is a discount factor used to discount future rewards; each agent learns a stochastic policy. This strategy is based on action-observation history. Generate actions; rewards Defined as a discounted cumulative reward; under the decentralized partially observable Markov decision process setting, the core objective of multi-agent reinforcement learning is to optimize the policy. To maximize the expected discounted return, formally expressed as: ;in, For real numbers, It indicates a desire for the expected value.

3. The multi-agent deep reinforcement learning method of claim 1, wherein, The multi-type priori strategy includes a knowledge engineering-based strategy, a planning decision theory-based strategy and a machine learning-based strategy; the knowledge engineering-based strategy includes a game rule script and a finite state machine coding multi-agent decision logic, the planning decision theory-based strategy includes a multi-agent cooperative decision-making goal-oriented planning using a planning domain standard language, and the machine learning-based strategy includes a neural network guiding the multi-agent cooperative decision-making.

4. The multi-agent deep reinforcement learning method of claim 1, wherein, The policy entropy is defined as: ; where, is the observation at time is the estimated hidden state, is the entropy function, denotes the action probability distribution of the policy with parameters under the estimated hidden state. During the course learning process, the prior guidance length h is adaptively adjusted by calculating the policy entropy of the learning strategy. If the policy entropy meets the preset threshold, the course learning progresses to the next stage, according to the formula h=h The value of h is gradually decreased by / m; if this is not satisfied, the current value of h is maintained for training until the policy entropy meets the preset threshold; where h is the initial value. Set H to the total time span of the task, where m is the number of preset course learning stages.

5. The multi-agent deep reinforcement learning method of claim 1, wherein, The mixed sampling on the multi-agent joint trajectories in the double experience replay area using the curriculum sampling ratio and the priority experience replay mechanism comprises: The double experience replay area includes a prior experience replay area and an online experience replay area , for storing the prior multi-agent joint trajectories generated by the prior policy phase h steps before each training round, for storing the online multi-agent joint trajectories generated by the learning policy phase H h steps after each training round; the multi-agent joint trajectory is defined as the entire state-action sequence of the multi-agent from the initial game state to the final game state in one complete game task attempt; In each training round, the data is extracted from the dual-experience replay region in a time-related proportion. Perform hybrid sampling of multi-agent joint trajectories, where, from The proportion of sampling in the middle is ;from The proportion of sampling in the middle is As training progresses, The value is determined according to predefined scheduling or performance feedback, from bias. The sampling transitions smoothly to almost complete dependence. Sampling; When sampling is mixed, in and Internally, priority indicators related to multi-agent TD-error or joint-action advantage are maintained respectively, and according to the priority indicators, key cooperative behavior segments within the experience replay area are given higher sampling probability, and priority experience replay of prior multi-agent joint trajectory and online multi-agent joint trajectory is realized respectively.

6. The multi-agent deep reinforcement learning method of claim 1, wherein, The strategy update frequency with decay adopts a cosine decay mode, and is represented as: ； wherein is the policy update frequency at time represents the initial policy update frequency, represents the policy update frequency at the end of the decay, represents the maximum number of decay steps.​ 7. A multi-agent deep reinforcement learning apparatus, comprising: The device comprises: A first module is configured to model a multi-agent cooperative decision-making problem in a serious game scene as a decentralized partially observable Markov decision process, and construct a multi-agent reinforcement learning model; The second module is used for training the multi-agent reinforcement learning model by using an adaptive curriculum learning mechanism based on a multi-type prior strategy guidance and strategy entropy, wherein, before the training, H is preset as a time span step number, and h is preset as a prior guidance length; in each training round, the multi-agent first interacts with a game environment for h steps based on a prior strategy, and then interacts with the game environment for the remaining H steps based on a learning strategy. A third module is configured to synchronously store multi-agent joint trajectories generated in a priori strategy stage and a learning strategy stage in a double experience replay area in the interaction process of the multi-agent and the game environment, and use a curriculum sampling ratio and a priority experience replay mechanism to perform mixed sampling on the multi-agent joint trajectories in the double experience replay area, calculate a reinforcement learning loss according to the mixed sampling samples, and jointly use a strategy update frequency with decay to train and update the learning strategy. A fourth module is configured to determine whether the policy entropy of the learning strategy meets a preset threshold at the end of each training round, and if yes, the curriculum learning proceeds to a next stage, and the second module and the third module are repeatedly invoked after the value of h is reduced; otherwise, the curriculum learning does not proceed, and the second module and the third module are directly repeatedly invoked; until the training step reaches a preset maximum value, the training is ended, and an optimal collaborative decision of the multi-agent is output based on the multi-agent reinforcement learning model trained, and the optimal collaborative decision is used to guide the joint action of the multi-agent and the environment interaction in the serious game.

8. The multi-agent deep reinforcement learning apparatus according to claim 7, wherein, The device further comprises a priori strategy type knowledge base configured to store multiple types of priori strategies, including a strategy based on knowledge engineering, a strategy based on planning decision theory, and a strategy based on machine learning; the strategy based on knowledge engineering includes game rule scripts and finite state machines that encode multi-agent decision logic, the strategy based on planning decision theory includes planning for multi-agent collaborative decision objectives using a planning domain standard language, and the strategy based on machine learning includes a neural network for guiding multi-agent collaborative decision. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images