Scene-adaptive heterogeneous multi-agent game confrontation method and device
Through the scenario-adaptive heterogeneous multi-agent game adversarial method, the reinforcement learning module and environment perception module are used to perceive and adjust the strategy network in real time, and the problem of poor adaptability of multi-agent game adversarial in dynamic environments is solved, achieving more efficient generalization learning and parameter sharing.
Patent Information
- Application Number
- CN202311433260.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-31
- Publication Date
- 2025-05-02
AI Technical Summary
The existing multiagent game adversarial algorithm cannot effectively adapt to dynamic changes in the external environment, and the parameter sharing and training complexity among heterogeneous agents are high.
The heterogeneous multi-agent game adversarial method is adopted to realize the environmental state and behavior in real time by strengthening the actor module and critic module in the learning module, calculate the sub-value function and action value function, adjust the policy network to adapt to environmental changes, and quantify and switch network parameters through the environment perception module to realize parameter sharing between agents.
The generalized learning ability of multi-agent game confrontation in dynamic environments has been improved, the difficulty of heterogeneous multi-agent game confrontation reinforcement learning has been reduced, and better strategy adaptability and parameter sharing efficiency have been achieved.
Smart Images
Figure CN119918565A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-agent confrontation game, and in particular to a scenario-adaptive heterogeneous multi-agent game confrontation method and device. Background Art
[0002] With the development of artificial intelligence technology, the problem of multi-agent game confrontation in complex environments has become a frontier hot issue that needs to be solved urgently. Multi-agent game confrontation scenarios have the characteristics of high dynamics, strong confrontation, and real-time. In complex confrontation environments, there are problems such as multi-agent heterogeneity, changes in the number of agents, and changes in system external environment parameters. The game confrontation strategy trained using traditional reinforcement learning algorithms for steady-state environments cannot effectively adapt to changes in the external environment, and thus cannot achieve a good generalization effect, and there is a problem of high complexity in training large-scale heterogeneous agents. Therefore, the self-adaptation of multi-agents to the external dynamically changing environment and the parameter sharing mechanism between heterogeneous agents can significantly improve the level of heterogeneous multi-agent game confrontation.
[0003] Existing multi-agent game confrontation algorithms are mainly developed for fixed, steady-state external environments. Using traditional multi-agent reinforcement learning algorithms for training in a steady-state environment, once the external environment (such as changes in the position of obstacles) or the number of agents changes, the original learning strategy will become less effective or even ineffective, and will not be able to adapt to the dynamic changes of the environment. In addition, traditional multi-agent reinforcement learning is mainly developed for homogeneous agents, that is, the types, state spaces, action spaces, and reward functions of the agents are consistent, while the complexity of heterogeneous multi-agent reinforcement learning in terms of joint states, actions, and strategy spaces is much higher than that of homogeneous agents. Therefore, the use of policy gradient optimization methods is very difficult in terms of calculation and has certain limitations. Summary of the invention
[0004] The present invention provides a scenario-adaptive heterogeneous multi-agent game confrontation method and device, which can solve the technical problem that the existing multi-agent game confrontation method cannot adapt to the dynamic changes of the environment.
[0005] According to one aspect of the present invention, a scenario-adaptive heterogeneous multi-agent game confrontation method is provided, the method comprising:
[0006] S10, the N agents in the reinforcement learning module perceive their respective states and behaviors from the current external environment in real time;
[0007] S20, the evaluation network of the actor module in the reinforcement learning module obtains N sub-value functions at the current moment based on the N states at the current moment, and obtains the value function at the current moment based on the N sub-value functions at the current moment; obtains N sub-value functions at the next moment based on the N states at the next moment, and obtains the value function at the next moment based on the N sub-value functions at the next moment; obtains the actor partial training error based on the difference between the value function at the current moment and the value function at the next moment;
[0008] S30, the evaluation network of the critic module in the reinforcement learning module obtains N sub-action value functions at the current moment based on the N states and behaviors at the current moment, and obtains the action value function at the current moment based on the N sub-action value functions at the current moment; obtains the N sub-action value functions at the next moment based on the N states and behaviors at the next moment, and obtains the action value function at the next moment based on the N sub-action value functions at the next moment; obtains the critic partial training error based on the difference between the action value function at the current moment and the action value function at the next moment;
[0009] S40, judging whether the actor's training error and the critic's training error have converged, if so, obtaining a strategy network adapted to the current external environment and going to S50, otherwise, going to S10;
[0010] S50, judging whether the traversal of the M external environments is completed, if so, obtaining the strategy network corresponding to the M external environments one by one, and turning to S60, otherwise, turning to the next external environment, and turning to S10;
[0011] S60, the environment perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results to complete the multi-agent game confrontation.
[0012] Preferably, after S30 of the present invention, the method further comprises: the N agents share the evaluation network parameters of the actor module and the evaluation network parameters of the critic module in a global or local network sharing manner.
[0013] Preferably, the environmental perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results, including: the environmental perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results using a network switching algorithm.
[0014] According to another aspect of the present invention, a scenario-adaptive heterogeneous multi-agent game confrontation device is provided, the device comprising a reinforcement learning module and an environment perception module; the reinforcement learning module comprises N agents, an actor module and a critic module;
[0015] The N intelligent agents are used to perceive their respective states and behaviors from the current external environment in real time;
[0016] The evaluation network of the actor module is used to obtain N sub-value functions at the current moment based on the N states at the current moment, and to obtain the value function at the current moment based on the N sub-value functions at the current moment; to obtain N sub-value functions at the next moment based on the N states at the next moment, and to obtain the value function at the next moment based on the N sub-value functions at the next moment; and to obtain the actor partial training error based on the difference between the value function at the current moment and the value function at the next moment;
[0017] The evaluation network of the critic module is used to obtain N sub-action value functions at the current moment based on the N states and behaviors at the current moment, and to obtain the action value function at the current moment based on the N sub-action value functions at the current moment; to obtain the N sub-action value functions at the next moment based on the N states and behaviors at the next moment, and to obtain the action value function at the next moment based on the N sub-action value functions at the next moment; and to obtain the critic partial training error based on the difference between the action value function at the current moment and the action value function at the next moment;
[0018] The reinforcement learning module is used to obtain a policy network adapted to the current external environment when both the actor part training error and the critic part training error converge; and is also used to complete the traversal of M external environments to obtain a policy network corresponding to the M external environments one by one;
[0019] The environmental perception module is used to perceive and classify the actual use environment, and select the corresponding strategy network according to the perception and classification results to complete the multi-agent game confrontation.
[0020] Preferably, the N agents are also used to share the evaluation network parameters of the actor module and the evaluation network parameters of the critic module by adopting a global or local network sharing method.
[0021] Preferably, the environmental perception module is used to select a corresponding strategy network using a network switching algorithm based on the perception and classification results to complete multi-agent game confrontation.
[0022] According to another aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.
[0023] By applying the technical solution of the present invention, the network parameters of the external environment of the multi-agent system are quantified using the environment perception module, and the environment is perceived and classified. The generalization learning ability of the multi-agent in a dynamic environment is guaranteed by the network parameter switching algorithm, and the game confrontation of heterogeneous multi-agents can be realized through the parameter sharing and expansion mechanism between the agents, thus making up for the shortcomings of the prior art. The present invention can achieve a good generalization effect on the external dynamic environment and effectively reduce the difficulty of reinforcement learning of heterogeneous multi-agent game confrontation. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1 A flow chart of a scenario-adaptive heterogeneous multi-agent game confrontation method provided according to an embodiment of the present invention is shown;
[0026] Figure 2 A schematic diagram of the structure of a scenario-adaptive heterogeneous multi-agent game confrontation device provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0027] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0029] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.
[0030] Considering the challenges that the non-static nature of the system's external environment brings to the existing multi-agent game confrontation, the present invention can be applied to game scenarios where the system's external environment parameters change dynamically. Multi-agents have a certain perception of the external environment, but the network characteristics of reinforcement learning limit the system's adaptability in a changing environment. In order to better perceive the uncertain external environment of the system, the environmental perception module of the present invention performs network parameterization on the external environment in which the agent system is located, identifies and classifies the environment through the existing random forest classification algorithm, and uses a heterogeneous multi-agent reinforcement learning module based on improved AC (actor-critic) and a network parameter switching algorithm to train and learn multi-agent game confrontation. The heterogeneous multi-agent reinforcement learning module based on improved AC has a strong random strategy update capability, which can achieve efficient cooperation between multi-agents; the network parameter switching algorithm can effectively make up for the lack of the reinforcement learning module's ability to adapt to the external environment of the dynamic system. The combination of the two ensures that the multi-agent game confrontation has a strong learning ability, while also improving the generalization learning ability of the multi-agent system in a changing environment.
[0031] Considering the large-scale existence of heterogeneous agents in the game confrontation environment, as the number of agents increases, the state space and action space of the system gradually increase, which greatly increases the difficulty of reinforcement learning training. The network parameter sharing mechanism for heterogeneous multi-agents avoids the training problem caused by the oversized model by dynamically expanding the neural network architecture and corresponding weights. According to the type of agents and the relevance of their tasks, it can be divided into three categories: "global network sharing retraining", "local network sharing extension retraining" and "local network sharing extension selection retraining".
[0032] Among them, "global network sharing retraining" refers to building a super network architecture containing multiple tasks when the structure or task of the intelligent agent is highly correlated, and sharing the global network weights by retraining the weight model overload; "local network sharing extension retraining" refers to retaining the shared network weights between heterogeneous intelligent agents, and introducing new network weights for retraining on this basis; "local network sharing extension selection retraining" refers to selecting weights that are more sensitive to the current task through the neural network architecture search algorithm for sharing, and retraining according to heterogeneous intelligent agents and new tasks. The above three methods can reduce the burden of reinforcement learning training for large-scale heterogeneous multi-agents.
[0033] like Figure 1 As shown, the present invention provides a scenario-adaptive heterogeneous multi-agent game confrontation method, the method comprising:
[0034] S10, the N agents in the reinforcement learning module perceive their respective states and behaviors from the current external environment in real time;
[0035] S20, the evaluation network of the actor module in the reinforcement learning module obtains N sub-value functions at the current moment based on the N states at the current moment, and obtains the value function at the current moment based on the N sub-value functions at the current moment; obtains N sub-value functions at the next moment based on the N states at the next moment, and obtains the value function at the next moment based on the N sub-value functions at the next moment; obtains the actor partial training error based on the difference between the value function at the current moment and the value function at the next moment;
[0036] S30, the evaluation network of the critic module in the reinforcement learning module obtains N sub-action value functions at the current moment based on the N states and behaviors at the current moment, and obtains the action value function at the current moment based on the N sub-action value functions at the current moment; obtains the N sub-action value functions at the next moment based on the N states and behaviors at the next moment, and obtains the action value function at the next moment based on the N sub-action value functions at the next moment; obtains the critic partial training error based on the difference between the action value function at the current moment and the action value function at the next moment;
[0037] S40, judging whether the actor's training error and the critic's training error have converged, if so, obtaining a strategy network adapted to the current external environment and going to S50, otherwise, going to S10;
[0038] S50, judging whether the traversal of the M external environments is completed, if so, obtaining the strategy network corresponding to the M external environments one by one, and turning to S60, otherwise, turning to the next external environment, and turning to S10;
[0039] S60, the environment perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results to complete the multi-agent game confrontation.
[0040] The present invention uses an environmental perception module to quantify network parameters of the external environment in which the multi-agent system is located, and perceives and classifies the environment. The network parameter switching algorithm ensures the generalization learning ability of the multi-agent in a dynamic environment, and can realize the game confrontation of heterogeneous multi-agents through parameter sharing and expansion mechanisms between agents, making up for the shortcomings of the prior art. The present invention can achieve a good generalization effect on the external dynamic environment and effectively reduce the difficulty of heterogeneous multi-agent game confrontation reinforcement learning.
[0041] In the present invention, the reinforcement learning module is a heterogeneous multi-agent reinforcement learning module based on improved AC (actor module and critic module).
[0042] According to an embodiment of the present invention, after S30 of the present invention, the method further includes: the N agents share the evaluation network parameters of the actor module and the evaluation network parameters of the critic module in a global or local network sharing manner.
[0043] Among them, after sharing, in the subsequent cycle process, if there are two or more intelligent agents with the same state and behavior, the sub-value function and sub-action value function of the two or more intelligent agents are consistent. Therefore, only the sub-value function and sub-action value function of any one intelligent agent need to be calculated.
[0044] According to an embodiment of the present invention, the environment perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results, including: the environment perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results using a network switching algorithm.
[0045] like Figure 2 As shown, the present invention provides a scenario-adaptive heterogeneous multi-agent game confrontation device, the device comprising a reinforcement learning module and an environment perception module; the reinforcement learning module comprises N agents, an actor module and a critic module;
[0046] The N intelligent agents are used to perceive their respective states and behaviors from the current external environment in real time;
[0047] The evaluation network of the actor module is used to obtain N sub-value functions at the current moment based on the N states at the current moment, and to obtain the value function at the current moment based on the N sub-value functions at the current moment; to obtain N sub-value functions at the next moment based on the N states at the next moment, and to obtain the value function at the next moment based on the N sub-value functions at the next moment; and to obtain the actor partial training error based on the difference between the value function at the current moment and the value function at the next moment;
[0048] The evaluation network of the critic module is used to obtain N sub-action value functions at the current moment based on the N states and behaviors at the current moment, and to obtain the action value function at the current moment based on the N sub-action value functions at the current moment; to obtain the N sub-action value functions at the next moment based on the N states and behaviors at the next moment, and to obtain the action value function at the next moment based on the N sub-action value functions at the next moment; and to obtain the critic partial training error based on the difference between the action value function at the current moment and the action value function at the next moment;
[0049] The reinforcement learning module is used to obtain a policy network adapted to the current external environment when both the actor part training error and the critic part training error converge; and is also used to complete the traversal of M external environments to obtain a policy network corresponding to the M external environments one by one;
[0050] The environmental perception module is used to perceive and classify the actual use environment, and select the corresponding strategy network according to the perception and classification results to complete the multi-agent game confrontation.
[0051] According to one embodiment of the present invention, the N agents are also used to share the evaluation network parameters of the actor module and the evaluation network parameters of the critic module in a global or local network sharing manner.
[0052] According to one embodiment of the present invention, the environment perception module is used to select a corresponding strategy network using a network switching algorithm based on perception and classification results to complete multi-agent game confrontation.
[0053] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.
[0054] In summary, the present invention provides a scenario-adaptive heterogeneous multi-agent game confrontation method and device, which uses an environment perception module to quantify network parameters of the external environment in which the multi-agent system is located, and perceives and classifies the environment. The network parameter switching algorithm ensures the generalization learning ability of the multi-agent in a dynamic environment, and can realize heterogeneous multi-agent game confrontation through parameter sharing and expansion mechanisms between agents, making up for the shortcomings of the prior art. The present invention can achieve a good generalization effect on the external dynamic environment and effectively reduce the difficulty of heterogeneous multi-agent game confrontation reinforcement learning.
[0055] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.
[0056] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.
[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A scenario-adaptive heterogeneous multi-agent game confrontation method, characterized in that: The method comprises: S10, the N agents in the reinforcement learning module perceive their respective states and behaviors from the current external environment in real time; S20, the evaluation network of the actor module in the reinforcement learning module obtains N sub-value functions at the current moment based on the N states at the current moment, and obtains the value function at the current moment based on the N sub-value functions at the current moment; obtains N sub-value functions at the next moment based on the N states at the next moment, and obtains the value function at the next moment based on the N sub-value functions at the next moment; obtains the actor partial training error based on the difference between the value function at the current moment and the value function at the next moment; S30, the evaluation network of the critic module in the reinforcement learning module obtains N sub-action value functions at the current moment based on the N states and behaviors at the current moment, and obtains the action value function at the current moment based on the N sub-action value functions at the current moment; obtains the N sub-action value functions at the next moment based on the N states and behaviors at the next moment, and obtains the action value function at the next moment based on the N sub-action value functions at the next moment; obtains the critic partial training error based on the difference between the action value function at the current moment and the action value function at the next moment; S40, judging whether the actor's training error and the critic's training error have converged, if so, obtaining a strategy network adapted to the current external environment and going to S50, otherwise, going to S10; S50, judging whether the traversal of the M external environments is completed, if so, obtaining the strategy network corresponding to the M external environments one by one, and turning to S60, otherwise, turning to the next external environment, and turning to S10; S60, the environment perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results to complete the multi-agent game confrontation.
2. The method according to claim 1, characterized in that After S30 of the present invention, the method further includes: the N agents share the evaluation network parameters of the actor module and the evaluation network parameters of the critic module in a global or local network sharing manner.
3. The method according to claims 1 and 2, characterized in that The environmental perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results. The environmental perception module perceives and classifies the actual use environment, and selects the corresponding strategy network according to the perception and classification results using a network switching algorithm.
4. A scenario-adaptive heterogeneous multi-agent game confrontation device, characterized in that: The device includes a reinforcement learning module and an environment perception module; the reinforcement learning module includes N agents, an actor module and a critic module; The N intelligent agents are used to perceive their respective states and behaviors from the current external environment in real time; The evaluation network of the actor module is used to obtain N sub-value functions at the current moment based on the N states at the current moment, and to obtain the value function at the current moment based on the N sub-value functions at the current moment; Used to obtain N sub-value functions at the next moment based on the N states at the next moment, and to obtain the value function at the next moment based on the N sub-value functions at the next moment; and also used to obtain the actor's partial training error based on the difference between the value function at the current moment and the value function at the next moment; The evaluation network of the critic module is used to obtain N sub-action value functions at the current moment based on the N states and behaviors at the current moment, and to obtain the action value function at the current moment based on the N sub-action value functions at the current moment; Used to obtain N sub-action value functions at the next moment based on the N states and behaviors at the next moment, and to obtain the action value function at the next moment based on the N sub-action value functions at the next moment; also used to obtain the critic partial training error based on the difference between the action value function at the current moment and the action value function at the next moment; The reinforcement learning module is used to obtain a strategy network adapted to the current external environment when both the actor part training error and the critic part training error converge; and is also used to complete the traversal of M external environments to obtain a strategy network corresponding to the M external environments one by one; The environmental perception module is used to perceive and classify the actual use environment, and select the corresponding strategy network according to the perception and classification results to complete the multi-agent game confrontation.
5. The device according to claim 1, characterized in that The N agents are also used to share the evaluation network parameters of the actor module and the evaluation network parameters of the critic module in a global or local network sharing manner.
6. The device according to claim 1, characterized in that The environment perception module is used to select the corresponding strategy network using a network switching algorithm according to the perception and classification results to complete the multi-agent game confrontation.
7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.