Training method and system for adversarial cluster, adversarial method and system for adversarial cluster

Abstract

The present disclosure relates to the field of artificial intelligence and cluster confrontation technology, and provides a training method and system for confrontation clusters, and a confrontation method and system for confrontation clusters. The training method includes: storing confrontation data in an online strategy sample memory in real time; whenever the amount of data in the online strategy sample memory exceeds a preset first threshold, transferring the data to an offline strategy sample memory; filtering out multiple data with the best confrontation indicators of an intelligent agent from the offline strategy sample memory to obtain filtered data; determining a policy gradient based on the current data and the filtered data in the online strategy sample memory; and updating the neural network parameters of the intelligent agent in the confrontation cluster based on the policy gradient to train the confrontation cluster. The present disclosure can solve the problems of weak confrontation ability of clusters, solidification of confrontation strategies and poor adaptability, can enhance the confrontation ability of clusters, improve the continuous learning and evolution of clusters, and improve the flexibility and adaptability of confrontation strategies.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and cluster confrontation technology, and in particular to a training method and system for a confrontation cluster, and a confrontation method and system for a confrontation cluster. Background Art

[0002] Recently, swarm systems consisting of drones, unmanned vehicles, cruise missiles, etc. have received great attention and have broad application prospects in scenarios such as red-blue confrontation, coordinated reconnaissance or cruise. In order to improve the swarm confrontation capability, researchers have invested a lot of time and energy, and have achieved a lot of results based on theories and methods such as game theory, operations optimization and machine learning.

[0003] With the development of artificial intelligence, microelectronics, quantum mechanics and other technologies, models with self-learning capabilities such as deep reinforcement learning will gradually become the key technical route for general artificial intelligence. They are currently a hot research topic with great application potential, but also present great challenges. In particular, for the dynamic confrontation problem of multi-team multi-agent clusters, the agents must not only cooperate with neighboring agents, but also compete and confront with opponents. However, in related technologies, there are still problems such as weak confrontation ability, rigid confrontation strategy and poor adaptability. Therefore, how to improve the cluster's adaptability to opponents is an urgent problem that needs to be solved. Summary of the invention

[0004] The present disclosure provides a training method and system for adversarial clusters, and a method and system for adversarial clusters, so as to at least solve the problems of weak adversarial ability, rigid adversarial strategy and poor adaptability of clusters in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of the present disclosure, a training method for an adversarial cluster is provided, wherein the adversarial cluster is applied to unmanned equipment, the adversarial cluster includes multiple agents, each agent corresponds to an unmanned equipment, and the agent can make adversarial decisions in the process of confrontation with the opponent, wherein the training method includes: in the process of confrontation between the adversarial cluster and the opponent, the adversarial data of the adversarial cluster is stored in an online strategy sample memory in real time, wherein the adversarial data includes the observed state, winning rate and reward and punishment value of each agent in the adversarial cluster; whenever the amount of data in the online strategy sample memory exceeds a preset first threshold, the data in the online strategy sample memory is transferred to an offline strategy sample memory; for the adversarial index preset for the agent, multiple data with the best adversarial index of the agent are screened out from the offline strategy sample memory to obtain filtered data; based on the current data in the online strategy sample memory and the filtered data, the policy gradient for training the adversarial cluster is determined; based on the policy gradient, the neural network parameters of the agents in the adversarial cluster are updated to train the adversarial cluster.

[0006] Optionally, the confrontation indicator is the round life cycle of the agent, and the round life cycle refers to the number of rounds that the agent survives during the confrontation process, wherein the filtered data is obtained in the following manner: according to a predetermined filtering ratio, the data with the longest round life cycle of the agent is filtered out from the offline strategy sample storage to obtain the filtered data.

[0007] Optionally, the training method further includes: during determining the policy gradient, in response to the amount of data in the offline policy sample memory exceeding a preset second threshold, deleting a preset amount of data stored earliest in the offline policy sample memory, wherein the second threshold is greater than the first threshold.

[0008] Optionally, the training method also includes: performing statistics on the data in the online strategy sample memory to obtain adversarial statistical data of the adversarial cluster; determining an updated learning rate based on the adversarial statistical data, a preset maximum learning rate and a minimum learning rate, wherein the updated learning rate is less than or equal to the maximum learning rate, and greater than or equal to the minimum learning rate.

[0009] Optionally, the training method also includes: performing statistics on the data in the online strategy sample storage to obtain adversarial statistical data of the adversarial cluster; determining updated training rounds based on the adversarial statistical data, a preset maximum training round and a minimum training round, wherein the updated training round is less than or equal to the maximum training round, and greater than or equal to the minimum training round.

[0010] Optionally, the neural network parameters of the agents in the adversarial cluster are updated in the following manner: based on the policy gradient and the updated learning rate, updated neural network parameters are determined; in response to the current number of training rounds exceeding the updated number of training rounds, the current neural network parameters of the agents in the adversarial cluster are updated using the updated neural network parameters.

[0011] According to a second aspect of the present disclosure, a confrontation method for an adversarial cluster is provided, the confrontation method comprising: obtaining an adversarial cluster, wherein the adversarial cluster is applied to unmanned equipment, the adversarial cluster comprises a plurality of intelligent agents, each intelligent agent corresponds to an unmanned equipment, the intelligent agent can make adversarial decisions in the process of confrontation with the opponent, the adversarial cluster is trained according to the training method for the adversarial cluster described in an embodiment of the present disclosure; and using the adversarial cluster to confront the target opponent.

[0012] According to a third aspect of the present disclosure, a training system for an adversarial cluster is provided, wherein the adversarial cluster is applied to an unmanned device, the adversarial cluster includes a plurality of intelligent agents, each of which corresponds to an unmanned device, and the intelligent agent can make adversarial decisions in the process of confronting an opponent, wherein the training system includes: a storage unit, configured to store the adversarial data of the adversarial cluster in real time in an online strategy sample memory during the process of the adversarial cluster confronting an opponent, wherein the adversarial data includes the observed state, winning rate and reward and punishment value of each intelligent agent in the adversarial cluster; a data transfer unit, configured to transfer the data in the online strategy sample memory to an offline strategy sample memory whenever the amount of data in the online strategy sample memory exceeds a preset first threshold; a filtering unit, configured to filter out a plurality of data with the optimal adversarial index of the intelligent agent from the offline strategy sample memory for the adversarial index preset for the intelligent agent, and obtain filtered data; a policy gradient determination unit, configured to determine the policy gradient for training the adversarial cluster based on the current data in the online strategy sample memory and the filtered data; a training unit, configured to update the neural network parameters of the intelligent agents in the adversarial cluster based on the policy gradient to train the adversarial cluster.

[0013] According to a fourth aspect of the present disclosure, a confrontation system for an adversarial cluster is provided, the confrontation system comprising: a cluster acquisition unit, configured to acquire an adversarial cluster, wherein the adversarial cluster is applied to unmanned equipment, the adversarial cluster comprises a plurality of intelligent agents, each intelligent agent corresponds to an unmanned equipment, the intelligent agent can make adversarial decisions in the process of confrontation with the opponent, the adversarial cluster is trained according to the training method of the adversarial cluster described in an embodiment of the present disclosure; and a confrontation unit, configured to use the adversarial cluster to confront the target opponent.

[0014] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the instructions executable by the processor, when executed by the processor, cause the processor to execute the training method for an adversarial cluster according to an embodiment of the present disclosure or the adversarial method for an adversarial cluster according to an embodiment of the present disclosure.

[0015] According to the sixth aspect of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the training method of the adversarial cluster according to the embodiment of the present disclosure or the adversarial method of the adversarial cluster according to the embodiment of the present disclosure.

[0016] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which, when executed by at least one processor, implement the training method for an adversarial cluster according to an embodiment of the present disclosure or the adversarial method for an adversarial cluster according to an embodiment of the present disclosure.

[0017] The technical solution provided by the present disclosure brings at least the following beneficial effects:

[0018] According to the present disclosure, during the training process of an adversarial cluster, the adversarial data of the adversarial cluster can be stored in an online policy sample memory in real time, and each time the amount of data in the online policy sample memory exceeds a preset first threshold, the data in the online policy sample memory is transferred to the offline policy sample memory, and the filtered data with the best adversarial index of the intelligent agent can be screened out from the offline policy sample memory, so that the policy gradient of the adversarial cluster can be determined based on both the data in the online policy sample memory and the filtered data in the offline policy sample memory. The policy gradient determined in this way not only takes into account the advantageous historical data in the offline policy sample memory, but also can obtain the real-time generated online data from the online policy sample memory, thereby realizing the enhancement of the policy gradient. The neural network parameters of the intelligent agent in the adversarial cluster are updated based on such an enhanced policy gradient, which can enhance the adversarial ability of the cluster, improve the continuous learning and evolution of the cluster, and improve the flexibility and adaptability of the adversarial strategy.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0021] Figure 1 is a schematic flowchart of a training method for adversarial clustering according to an exemplary embodiment of the present disclosure.

[0022] Figure 2 It is a schematic architecture diagram of cluster adversarial training in the adversarial cluster training method according to an exemplary embodiment of the present disclosure.

[0023] Figure 3 is a schematic flowchart of determining an updated learning rate in a training method for adversarial clustering according to an exemplary embodiment of the present disclosure.

[0024] Figure 4 is a schematic flowchart of determining updated training rounds in a training method for adversarial clustering according to an exemplary embodiment of the present disclosure.

[0025] Figure 5 It is a schematic diagram of an adversarial scenario in a training method for adversarial clustering according to an exemplary embodiment of the present disclosure.

[0026] Figure 6 It is a simulation effect diagram of the training method of adversarial clustering according to an exemplary embodiment of the present disclosure.

[0027] Figure 7 is a schematic flowchart of a confrontation method of a confrontation cluster according to an exemplary embodiment of the present disclosure.

[0028] Figure 8 is a schematic block diagram of a training system for adversarial clustering according to an exemplary embodiment of the present disclosure.

[0029] Fig. 9 is a schematic block diagram of a confrontation system for a confrontation cluster according to an exemplary embodiment of the present disclosure.

[0030] Fig.10 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the present application will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0033] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0034] As mentioned above, in the related technologies, there are problems such as weak cluster confrontation capability, rigid confrontation strategy and poor adaptability.

[0035] Specifically, existing research on cluster confrontation can be divided into two categories: knowledge-driven and data-driven. Knowledge-driven methods include expert systems, Bayesian networks, game theory, decision trees, etc., which use human prior knowledge to design cluster confrontation decision models. They have simple structures and strong interpretability, but they have problems such as incomplete coverage of confrontation strategies, difficulty in modeling complex problems, and lack of autonomous learning and iterative update capabilities.

[0036] The data-driven method mainly focuses on deep reinforcement learning. It uses the construction of training elements and scenarios to let the intelligent agent cluster compete with the opponent, generate massive data through trial and error and exploration, learn to optimize the cluster confrontation strategy, and has strong exploration capabilities for the solution space and simple mechanism modeling. However, in the training methods of related technologies, there are problems such as weak dynamic confrontation capabilities, rigid confrontation strategies, and poor adaptability. In particular, when facing opponents with self-learning capabilities, the intelligent agent lacks the ability to continuously evolve training. Therefore, how to improve the adaptability of the cluster is an urgent problem that needs to be solved, and further in-depth research is urgently needed to propose feasible solutions.

[0037] In view of the above, exemplary embodiments of the present disclosure propose a training method for adversarial clusters, an adversarial method for adversarial clusters, a training system for adversarial clusters, an adversarial system for adversarial clusters, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0038] In a first aspect of an exemplary embodiment of the present disclosure, a training method for adversarial clustering is provided.

[0039] The training method for adversarial clusters according to the exemplary embodiments of the present disclosure can be applied to scenarios where users interact with software. For example, the software can be loaded on a user terminal, and the user can input adversarial cluster training instructions on the user terminal. The user terminal can obtain the final adversarial cluster by executing the training method for adversarial clusters according to the exemplary embodiments of the present disclosure.

[0040] Specifically, the confrontation cluster is applied to unmanned equipment. The confrontation cluster may include multiple agents, each of which corresponds to an unmanned equipment. The agent can make confrontation decisions in the process of confrontation with the opponent. The user terminal can store the confrontation data of the confrontation cluster in real time in the online strategy sample memory during the confrontation between the confrontation cluster and the opponent, wherein the confrontation data includes the observation state, winning rate and reward and punishment value of each agent in the confrontation cluster. The user terminal can transfer the data in the online strategy sample memory to the offline strategy sample memory whenever the amount of data in the online strategy sample memory exceeds the preset first threshold. The user terminal can also filter out multiple data with the best confrontation index of the agent from the offline strategy sample memory for the confrontation index preset for the agent, and obtain filtered data. The user terminal can also determine the policy gradient for training the confrontation cluster based on the current data in the online strategy sample memory and the filtered data, and can update the neural network parameters of the agent in the confrontation cluster based on the policy gradient to train the confrontation cluster.

[0041] The above-mentioned user terminal can be, for example, a tablet computer, a laptop computer, a digital assistant, a wearable device, etc. However, the implementation scenario of the above method is only an example scenario. The training method of the adversarial cluster according to the exemplary embodiment of the present disclosure can also be applied to other application scenarios. For example, the user can request a training model from the server through the network on the user terminal (for example, a mobile phone, a desktop computer, a tablet computer, etc.). The server can complete the request by executing the training method of the adversarial cluster according to the exemplary embodiment of the present disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform or a virtualization center.

[0042] According to the training method of the adversarial cluster of the exemplary embodiment of the present disclosure, the advantageous historical data in the offline strategy sample storage and the real-time online data obtained in the online strategy sample storage can be utilized to enhance the policy gradient. The neural network parameters of the intelligent agents in the adversarial cluster can be updated based on such enhanced policy gradient. This can enhance the adversarial ability of the cluster, improve the continuous learning and evolution of the cluster, and improve the flexibility and adaptability of the adversarial strategy.

[0043] The following will refer to Figures 1 to 6 An example of a training method for adversarial clustering according to an embodiment of the present disclosure is described.

[0044] In the embodiments of the present disclosure, the confrontation cluster is applied to unmanned equipment. The confrontation cluster may include multiple agents, each of which corresponds to an unmanned equipment, and the agent can make confrontation decisions in the process of confrontation with the opponent. As an example, the confrontation cluster may refer to a system composed of unmanned equipment such as drones, unmanned vehicles, unmanned ships, cruise missiles, automated security robots, confrontation competitions or game robots. The confrontation cluster can complete complex tasks through autonomous and intelligent collaborative strategies, and has broad application prospects in multiple fields such as industry, agriculture, and transportation.

[0045] For example, in security inspection scenarios such as power grids, factory areas, and geographic areas, drone swarms can work together to fight against foreign invaders and improve inspection efficiency and accuracy; in forest protection scenarios, drone swarms can monitor fire hazards and promptly discover and deal with (which can be understood as fighting against) fire points; in competitive competitions or game business scenarios, intelligent agent clusters can be characters participating in competitive competitions or games; in military exercises or simulation scenarios, clusters of drones, unmanned vehicles, and unmanned ships can work together to fight the enemy.

[0046] Here, it should be noted that although the above-mentioned scenario is described as an example in this article, the application scenario of the exemplary embodiment according to the present disclosure can be arbitrary, for example, it can also be applied to other decision-making scenarios, and the types of virtual characters and environmental data represented by the agent can be determined according to business needs, and the present disclosure does not impose any special restrictions on this.

[0047] like Figure 1 As shown, the training method of the adversarial cluster may include the following steps:

[0048] In step S110, during the confrontation between the confrontation cluster and the opponent, the confrontation data of the confrontation cluster may be stored in the online strategy sample storage in real time.

[0049] Here, the adversarial data may include the observed state, winning rate, and reward and punishment value of each agent in the adversarial cluster. As an example, the observed state may include the position, speed, posture, etc. of other opponents or agents observed by the agent. As an example, the agents in the adversarial cluster may be the same, specifically, they may have the same neural network structure and parameters, and during the training process, the neural network parameters of all agents may be updated synchronously.

[0050] In the embodiments of the present disclosure, Figure 2As shown, a confrontation environment can be built, which includes each agent of the confrontation cluster and an opponent to be confronted. The agents in the confrontation cluster can conduct a round-based confrontation with the opponent in the confrontation environment, and a winning and losing rule can be preset to determine the winning and losing of the agent in each round and give corresponding rewards and punishments. Here, the opponent can be, for example, a preset strategy mechanism, or an opponent with self-learning ability, for example, another cluster including multiple agents that has been trained.

[0051] In each round, each agent can make decisions based on environmental data that characterizes the global state of the environment (for example, including but not limited to the state of the opponent, the state of other agents in the confrontation cluster, etc.), and perform corresponding actions. As an example, the action space of the agent can be preset, and the action space can include, for example, 4 acceleration actions, 2 rotation shooting directions, and static actions. The agent can perform one or more actions in the action space according to the current decision, such as moving to a certain position or attacking a certain target. The reward and punishment value can represent the judgment of the action results of the agent, for example, it can be a score for the action results of the agent, and the higher the score, the more likely it is that the current action result of the agent has won or is in line with expectations.

[0052] In this step, the confrontation cluster can be made to confront the opponent, and the confrontation data can be stored in the online strategy sample memory in real time. The online strategy sample memory (in this article, the memory can also be called a buffer or a buffer area) can be, for example, a preset storage location, which can receive the confrontation data generated in the above confrontation environment.

[0053] In step S120, whenever the amount of data in the online policy sample storage exceeds a preset first threshold, the data in the online policy sample storage may be transferred to the offline policy sample storage.

[0054] Specifically, during the confrontation process, the confrontation data is continuously stored in the online strategy sample memory. The amount of data in exceeds the preset first threshold , the online strategy sample memory can be All data in the offline strategy sample storage , and clear the online strategy sample storage Here, the first threshold may be preset according to actual needs.

[0055] After the online strategy sample memory is cleared, since the confrontation continues, the confrontation data generated in real time will still be stored in the online strategy sample memory until the amount of data in the online strategy sample memory exceeds the first threshold again, and then the data transfer is performed again. The above process can be repeated until the training is completed.

[0056] As an example, the storage capacity of the offline policy sample memory may be greater than the storage capacity of the online policy sample memory, for example, it may be an integer multiple of the first threshold set for the online policy sample memory. Storage capacity ,in, Indicates adjustment of offline strategy sample storage The size hyperparameter can be set according to actual needs, for example, it can be 4.

[0057] In step S130, with respect to the confrontation index preset for the agent, multiple data with the best confrontation index of the agent can be screened out from the offline strategy sample storage to obtain filtered data.

[0058] Here, the confrontation indicator can be set according to actual needs, for example, it can be the round life cycle of the agent, and the round life cycle can refer to the number of rounds that the agent survives during the confrontation process.

[0059] Specifically, in unmanned equipment confrontation, confrontation game simulation or chess games, an agent can represent an individual that can be eliminated. Here, elimination can refer to, for example, the unmanned equipment being shot down or detected, the player being sent off or knocked down, the chess piece being captured in the chess game, etc. The round life cycle of an agent can refer to the number of rounds experienced from the start of the confrontation to the elimination of the agent.

[0060] As an example, the data with the longest round life cycle of the agent can be filtered out from the offline strategy sample storage according to a predetermined filtering ratio to obtain filtered data.

[0061] The predetermined screening ratio can be set according to actual needs. As an example, the data in the offline strategy sample memory can be sorted according to the round life cycle of each group of data. The longer the life cycle, the higher the ranking. The predetermined screening ratio can be, for example, Here, the data in the offline strategy sample memory may include multiple groups of data, each group of data corresponds to an agent, and each group of data may be sorted according to the round life cycle of each agent.

[0062] In an embodiment of the present disclosure, the data in the online policy sample storage and the filtered data can be stored in the extended sample storage. , for subsequent determination of the policy gradient.

[0063] Since the longer the round life cycle of the agent, the longer the agent survives in the confrontation, it can be considered that the agent has a greater chance of winning and the agent's strategy may be more optimal. Therefore, by determining the policy gradient based on such filtered data, more data of the dominant agent can be considered, thereby further enhancing the calculation of the policy gradient, which is conducive to further enhancing the cluster's confrontation ability.

[0064] In step S140, a policy gradient for training the adversarial cluster may be determined based on the current data in the online policy sample memory and the filtered data.

[0065] In the embodiment of the present disclosure, the data in the online strategy sample storage and the filtered data can be stored in the preset extended sample storage. Here, the online strategy sample storage , Offline strategy sample storage and extended sample memory It can refer to different storage locations, such as different databases.

[0066] In step S130, if Figure 2 As shown, the policy gradient can be calculated using real-time data from the online policy sample storage and historical data in the offline policy sample storage according to an existing policy gradient algorithm. Here, the algorithm for determining the policy gradient can be any algorithm such as a proximal policy optimization method, and the present disclosure does not impose any particular limitation on this.

[0067] In addition, as an example, the training method further includes: during determining the policy gradient, in response to the amount of data in the offline policy sample memory exceeding a preset second threshold, deleting the earliest preset amount of data stored in the offline policy sample memory.

[0068] Here, the second threshold value can be greater than the first threshold value, and the second threshold value and the preset number can be set according to actual needs. For example, before transferring data in the online strategy sample storage to the offline strategy sample storage each time, it can be determined whether the offline strategy sample storage Whether the amount of data in exceeds the second threshold If the second threshold is exceeded, the offline strategy sample storage can be deleted. The oldest data is stored in the online policy sample storage, and after deletion, the data in the online policy sample storage is transferred to the offline policy sample storage; if it does not exceed the second threshold, the data in the online policy sample storage can be directly transferred to the offline policy sample storage.

[0069] Through the above method, outdated samples can be prevented from affecting the calculation of policy gradients, thereby improving the training effect of the cluster.

[0070] In step S150, the neural network parameters of the agents in the adversarial cluster may be updated based on the policy gradient to train the adversarial cluster.

[0071] like Figure 2 As shown, the neural network parameters of the agents in the adversarial cluster can be updated based on the policy gradient, and the updated parameters can be input into the agent to update the neural network of the agent to achieve training of the agent.

[0072] The above describes the process of updating the neural network parameters of the agent to achieve training by introducing a policy gradient enhancement mechanism. According to an embodiment of the present disclosure, a hyperparameter adaptive feedback mechanism can also be introduced to update the learning rate and / or training rounds during the training process, thereby achieving better training results.

[0073] Specifically, for the learning rate of cluster training, such as Figure 3 As shown, the training method may also include:

[0074] In step S310, statistics may be collected on the data in the online strategy sample storage to obtain adversarial statistical data of the adversarial cluster.

[0075] Specifically, the adversarial statistical data may be data obtained by statistically analyzing the data in the online strategy sample storage from any dimension, and may include, for example, an average winning rate or the state of the agents in the adversarial cluster. As an example, the state of the agents in the adversarial cluster may include the number of agents currently remaining (or surviving) in the adversarial cluster, or the total life value of the agents in the adversarial cluster (e.g., the total energy of the unmanned device), etc.

[0076] For example, online policy sample storage The data can include multiple rounds of confrontation data, and the average winning rate can be determined based on the results of each round. For another example, in unmanned equipment systems such as drones, unmanned vehicles, and unmanned ships, the confrontation statistics may include the number of currently available equipment, or the total amount of fuel of currently available equipment, etc.

[0077] In step S320, an updated learning rate may be determined based on the adversarial statistical data, a preset maximum learning rate, and a minimum learning rate.

[0078] Here, the maximum learning rate and the minimum learning rate can be set according to actual needs. The updated learning rate can be less than or equal to the maximum learning rate and greater than or equal to the minimum learning rate.

[0079] As an example, the online policy sample storage The average winning rate in the above formula (1) is used to calculate the hyperparameter strategy learning rate. :

[0080] (1)

[0081] in, and Respectively represent the minimum and maximum values ​​of the gradient descent coefficient (i.e. the above learning rate), Indicates the online strategy sample storage The average winning rate in As an example, you can set and , when starting training, the baseline learning rate can be set to .

[0082] In this way, the learning rate of the agent can be updated while considering the average winning rate in the recent adversarial data, so as to further optimize the training of the cluster and increase the training speed.

[0083] For cluster training, the learning rate is Figure 4 As shown, the training method may also include:

[0084] In step S410 , adversarial statistics of the adversarial cluster may be determined based on data in the online strategy sample storage.

[0085] As described above, the adversarial statistical data may be data obtained by statistically analyzing the data in the online strategy sample storage from any dimension, and may include, for example, an average winning rate or the state of an agent in an adversarial cluster. As an example, the state of an agent in an adversarial cluster may include the number of agents currently remaining (or surviving) in the adversarial cluster, or the total life value of agents in the adversarial cluster (e.g., the total energy of an unmanned device), etc.

[0086] For example, online policy sample storage The data can include multiple rounds of confrontation data, and the average winning rate can be determined based on the results of each round. .

[0087] In step S420, an updated number of training rounds may be determined based on the adversarial statistical data, a preset maximum number of training rounds, and a preset minimum number of training rounds.

[0088] Here, the maximum training round and the minimum training round can be set according to actual needs. The updated training round is less than or equal to the maximum training round, and greater than or equal to the minimum training round.

[0089] As an example, the online policy sample storage The average winning rate in the model training rounds is calculated by the following formula (2): :

[0090] (2)

[0091] in, , Respectively represent the minimum and maximum values ​​of the model training rounds, Indicates the online strategy sample storage The average winning rate in As an example, you can set and , when starting training, the benchmark training rounds can be set to Although the average winning rate is used as an example in the above formula (1) and the formula (2) here to illustrate the process of updating the learning rate and the training round, it is not limited to this, and other adversarial statistical data can also be used for update calculation, for example, It can also represent the ratio of the number of agents currently remaining in the adversarial cluster to the number of agents at the beginning of the adversarial cluster.

[0092] When the number of training rounds is determined, it can be determined whether the current number of training rounds exceeds the training round In response to exceeding the training round, the current agent neural network parameters can be output In response to not exceeding the training round, the process may return to executing steps S310 to S320 to recalculate the updated learning rate.

[0093] Through the above method, the training rounds can be dynamically updated during the training process, rather than pre-setting a fixed number of training rounds before training. In this way, adaptive adjustment of the training rounds can be achieved, so that the number of training rounds can be reasonably set while ensuring the training effect, thereby optimizing the training time and training efficiency.

[0094] When an updated learning rate and an updated number of training rounds are obtained, in the above step S140, the neural network parameters of the agents in the adversarial cluster can be updated in the following manner: based on the policy gradient and the updated learning rate, the updated neural network parameters are determined; in response to the current number of training times exceeding the updated number of training rounds, the current neural network parameters of the agents in the adversarial cluster are updated using the updated neural network parameters.

[0095] Specifically, based on the learning rate calculated by equation (2), the neural network parameters of the model can be updated by equation (3):

[0096] (3)

[0097] in, , They represent the agent in , The neural network parameter values ​​at time , Indicates the sample memory from the extension Obtained policy gradient.

[0098] In an embodiment of the present disclosure, the updated agent can continue to perform actions in the confrontation environment. As described above, the actions of the agent can include 4 acceleration actions, 2 rotation shooting directions, and a static action. According to the actions of the agent, the agent can be driven to execute distributedly to continue to generate confrontation data and store it in the online strategy sample storage.

[0099] The above training method according to the exemplary embodiment of the present disclosure can be repeatedly executed in the training of the adversarial cluster until the preset training stop condition is met. Specifically, after the agent performs an action, it can be determined whether the preset training stop condition is currently met. If the training stop condition is met, the neural network parameters of the current agent can be output to complete the training of the adversarial cluster. Otherwise, it can return to step S110 to continue training. The training stop condition may include, for example: the winning rate of the adversarial cluster reaches a preset winning rate value; or, the current number of training steps reaches a preset maximum number of training steps; or, the training reaches a preset maximum number of rounds.

[0100] The above describes an embodiment of the training method of the adversarial cluster according to an embodiment of the present disclosure. As mentioned above, the method can be based on Figure 2 This is achieved by training the adversarial cluster shown.

[0101] like Figure 2 As shown, the training architecture of adversarial clustering can include an online policy sample memory , Offline strategy sample storage , sample filter, extended sample storage , hyperparameter optimization module, and strategy optimizer, etc.

[0102] Online strategy sample storage It can be used to store sample data generated during the agent training process. The maximum amount of data stored is .

[0103] Offline policy sample storage It can be used to continuously receive samples discarded from the online strategy sample memory. Specifically, every time a training cycle is completed, all samples from the online strategy sample memory The samples will be transferred to (and clear the online strategy sample storage ). In order to prevent outdated samples from affecting the policy gradient, if the offline policy sample storage If the capacity is exceeded, the oldest samples will be deleted. Offline strategy sample storage The capacity can be expressed as , Is to adjust the offline strategy sample storage The size hyperparameter.

[0104] Sample filters can be used to filter offline policy sample storage For example, the adversarial data of cluster agents can be ranked according to the round life cycle of the agent, and the proportion of samples passing the sample filter is .

[0105] Extended sample storage Can be used to store samples from online strategy storage All samples and samples from offline strategy storage Filter samples.

[0106] The hyperparameter optimization module can be used to optimize feedback hyperparameters, such as calculating updated learning rates, updated training rounds, etc.

[0107] The policy optimizer can use proximal policy optimization methods to update the policy gradient of the computational agent.

[0108] Figure 5 An example of a confrontation scenario using the training method for adversarial clustering according to an exemplary embodiment of the present disclosure is shown. In this scenario, the red cluster agent may be an agent driven by the training method for adversarial clustering according to an exemplary embodiment of the present disclosure. As an example, these agents may all be local observations and cannot obtain information about all agents in the scenario. The blue cluster agent may, for example, be a cluster confrontation method using self-learning. Here, a benchmark method 1 and a benchmark method 2 for training the red cluster agent may also be set, and the training results of the two methods may be compared with the training results of the training method proposed in the exemplary embodiment of the present disclosure. As an example, the benchmark method 1 may be a multi-agent deep reinforcement learning method based on a concentration network (ConcNet MARL, Concentration networkbased multi-agent reinforcement learning), and the benchmark method 2 may be a method without adding a "policy gradient enhancement mechanism", that is, the policy gradient may be directly calculated based on the data in the online policy sample memory, and the other parts of the benchmark method 2 may be the same as the training method of the embodiment of the present disclosure.

[0109] Figure 6 The simulation results of the above comparative experiment are shown. Figure 6 As shown, compared with the red team cluster agent driven by the network parameters obtained by benchmark methods 1 and 2, the red team cluster agent driven by the network parameters obtained by the training method of an embodiment of the present invention can quickly complete the confrontation with the blue team cluster agent. When the training number exceeds 100,000 rounds, the red team obtains a winning rate of nearly 80%, and has a further upward trend, which further proves the effectiveness of the present invention.

[0110] In the training method of the exemplary embodiment of the present disclosure, the adversarial data of the agent in the adversarial cluster can be obtained, including the observed state, the winning rate, and the reward and punishment value, etc.; based on the policy gradient enhancement mechanism, the adversarial data of the agent is stored; based on the hyperparameter adaptive feedback mechanism, the hyperparameters of the agent neural network are calculated; based on the agent neural network, the action of the agent is calculated; according to the action of the agent, the distributed execution of the agent is driven. In this process, the training stop condition can be judged, and if the condition is met, the agent neural network parameters can be output, otherwise, the step of obtaining adversarial data can be returned.

[0111] According to the training method of the exemplary embodiment of the present disclosure, the problems of weak dynamic confrontation ability, solidified confrontation strategy and poor adaptability in the related technology can be solved, especially the lack of continuous evolution training ability for opponents with self-learning ability, the lack of continuous evolution training ability for intelligent agents, and the inability to effectively realize the continuous evolution of cluster collaborative confrontation ability. The present invention enhances the cluster's confrontation ability against opponents with self-learning ability and improves the continuous learning evolution of the cluster by designing strategy gradient enhancement and hyperparameter adaptive feedback mechanism for cluster confrontation training, which can effectively realize cluster confrontation against self-learning opponents.

[0112] In a second aspect of the exemplary embodiments of the present disclosure, a method for confronting a cluster is provided. The method can be applied to a scenario where a user interacts with software, for example, the software can be loaded on a user terminal, the user can input a cluster confrontation instruction on the user terminal, and the user terminal can implement cluster confrontation by executing the confrontation method according to the exemplary embodiments of the present disclosure.

[0113] like Figure 7 As shown, the confrontation method includes: step S710, obtaining a confrontation cluster, wherein the confrontation cluster is applied to unmanned equipment, the confrontation cluster includes multiple intelligent agents, each intelligent agent corresponds to an unmanned equipment, the intelligent agent can make confrontation decisions in the confrontation process with the opponent, and the confrontation cluster is trained according to the confrontation cluster training method described in the embodiment of the present disclosure; step S720, using the confrontation cluster to confront the target opponent.

[0114] The specific steps and details of the adversarial method according to the exemplary embodiment of the present disclosure can be implemented and understood similarly to the training method of the adversarial cluster described in the first aspect above, and accordingly, the adversarial method of the adversarial cluster has the same or similar beneficial effects as the training method of the adversarial cluster described above, which will not be repeated here.

[0115] In a third aspect of the exemplary embodiments of the present disclosure, a training system for a confrontation cluster is provided. The confrontation cluster is applied to unmanned equipment. The confrontation cluster includes multiple intelligent agents, each of which corresponds to an unmanned equipment. The intelligent agent can make confrontation decisions in the process of confrontation with the opponent, such as Figure 8 As shown, the training system includes a storage unit 810, a data transfer unit 820, a filtering unit 830, a policy gradient determination unit 840 and a training unit 850.

[0116] The storage unit 810 is configured to store the confrontation data of the confrontation cluster in real time into the online strategy sample storage during the confrontation between the confrontation cluster and the opponent, wherein the confrontation data includes the observed state, winning rate and reward and punishment value of each intelligent agent in the confrontation cluster.

[0117] The data transfer unit 820 is configured to transfer the data in the online policy sample storage to the offline policy sample storage whenever the amount of data in the online policy sample storage exceeds a preset first threshold.

[0118] The filtering unit 830 is configured to filter out multiple data with optimal confrontation indicators of the agent from the offline strategy sample storage for the confrontation indicators preset for the agent, and obtain filtered data.

[0119] The policy gradient determination unit 840 is configured to determine the policy gradient of the training adversarial cluster based on the current data in the online policy sample memory and the filtered data.

[0120] The training unit 850 is configured to update the neural network parameters of the agents in the adversarial cluster based on the policy gradient to train the adversarial cluster.

[0121] As an example, the confrontation indicator is the round life cycle of the agent, which refers to the number of rounds that the agent survives during the confrontation process, wherein the filtered data is obtained in the following manner: according to a predetermined filtering ratio, the data with the longest round life cycle of the agent is filtered out from the offline strategy sample memory to obtain the filtered data.

[0122] As an example, the training method also includes: during determining the policy gradient, in response to the amount of data in the offline policy sample memory exceeding a preset second threshold, deleting the earliest preset amount of data stored in the offline policy sample memory, wherein the second threshold is greater than the first threshold.

[0123] As an example, the training method also includes: performing statistics on the data in the online strategy sample memory to obtain adversarial statistics of the adversarial cluster; determining an updated learning rate based on the adversarial statistics, a preset maximum learning rate and a minimum learning rate, wherein the updated learning rate is less than or equal to the maximum learning rate and greater than or equal to the minimum learning rate.

[0124] As an example, the training method also includes: performing statistics on the data in the online strategy sample storage to obtain adversarial statistical data of the adversarial cluster; determining updated training rounds based on the adversarial statistical data, a preset maximum training round and a minimum training round, wherein the updated training round is less than or equal to the maximum training round and greater than or equal to the minimum training round.

[0125] As an example, the neural network parameters of the agents in the adversarial cluster are updated in the following manner: based on the policy gradient and the updated learning rate, the updated neural network parameters are determined; in response to the current number of training rounds exceeding the updated number of training rounds, the current neural network parameters of the agents in the adversarial cluster are updated using the updated neural network parameters.

[0126] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method. Each unit in the adversarial cluster training system can perform the corresponding steps in the method according to the adversarial cluster training method in the method embodiment of the first aspect above, which will not be elaborated here.

[0127] In a fourth aspect of the exemplary embodiments of the present disclosure, a confrontation system for a confrontation cluster is provided, such as Fig. 9 As shown, the confrontation system for confronting a cluster includes a cluster acquisition unit 910 and a confrontation unit 920 .

[0128] The cluster acquisition unit 910 is configured to acquire an adversarial cluster, wherein the adversarial cluster is applied to an unmanned device, the adversarial cluster includes multiple agents, each agent corresponds to an unmanned device, the agent can make adversarial decisions in the process of confronting the opponent, and the adversarial cluster is trained according to the adversarial cluster training method described in the embodiment of the present disclosure. The confrontation unit 920 is configured to use the adversarial cluster to confront the target opponent.

[0129] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0130] Fig.10 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Fig.10As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the training method of the adversarial cluster or the adversarial method of the adversarial cluster as described in the above exemplary embodiments.

[0131] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0132] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0133] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0134] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0135] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0136] In an exemplary embodiment, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of adversarial clustering or the adversarial method of adversarial clustering as described in the above exemplary embodiments. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: read-only memory (ROM), random access memory (RAM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0137] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes computer executable instructions, and when the computer executable instructions are executed by at least one processor, the training method of the adversarial cluster or the adversarial method of the adversarial cluster according to the exemplary embodiment of the present disclosure is implemented.

[0138] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0139] In addition, it should be noted that although several examples of each step are described above with reference to specific drawings, it should be understood that the embodiments of the present disclosure are not limited to the combinations given in the examples, and the steps appearing in different drawings may be combined, and the order of execution of the steps may be changed, which is not exhaustive here.

[0140] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for adversarial clustering, characterized in that: The confrontation cluster is applied to unmanned equipment, the confrontation cluster includes multiple agents, each agent corresponds to an unmanned equipment, and the agent can make confrontation decisions in the process of confrontation with the opponent, wherein the training method includes: During the confrontation between the confrontation cluster and the opponent, the confrontation data of the confrontation cluster is stored in the online strategy sample storage in real time, wherein the confrontation data includes the observation state, winning rate and reward and punishment value of each agent in the confrontation cluster; Whenever the amount of data in the online strategy sample storage exceeds a preset first threshold, transferring the data in the online strategy sample storage to the offline strategy sample storage; According to the confrontation index preset for the intelligent agent, a plurality of data with the best confrontation index of the intelligent agent are selected from the offline strategy sample storage to obtain filtered data; Determining a policy gradient for training the adversarial cluster based on current data in the online policy sample memory and the filtered data; Based on the policy gradient, the neural network parameters of the agents in the adversarial cluster are updated to train the adversarial cluster.

2. The training method according to claim 1, characterized in that: The confrontation indicator is the round life cycle of the agent, which refers to the number of rounds that the agent survives during the confrontation process, wherein the filtered data is obtained by: According to a predetermined screening ratio, the data with the longest round life cycle of the agent is screened out from the offline strategy sample storage to obtain the filtered data.

3. The training method according to claim 1, characterized in that: The training method further comprises: During the determination of the policy gradient, in response to the amount of data in the offline policy sample memory exceeding a preset second threshold, a preset amount of data stored earliest in the offline policy sample memory is deleted, wherein the second threshold is greater than the first threshold.

4. The training method according to claim 1, characterized in that: The training method further comprises: Performing statistics on the data in the online strategy sample storage to obtain confrontation statistics of the confrontation cluster; Based on the adversarial statistical data, a preset maximum learning rate and a preset minimum learning rate, an updated learning rate is determined, The updated learning rate is less than or equal to the maximum learning rate and greater than or equal to the minimum learning rate.

5. The training method according to claim 4, characterized in that: The training method further comprises: Performing statistics on the data in the online strategy sample storage to obtain confrontation statistics of the confrontation cluster; Based on the confrontation statistics, a preset maximum training round and a preset minimum training round, an updated training round is determined, The updated training round is less than or equal to the maximum training round, and greater than or equal to the minimum training round.

6. The training method according to claim 5, characterized in that: The neural network parameters of the agents in the adversarial cluster are updated in the following way: Determining updated neural network parameters based on the policy gradient and the updated learning rate; In response to the current number of training rounds exceeding the updated number of training rounds, the updated neural network parameters are used to update the current neural network parameters of the agents in the adversarial cluster.

7. A method for confronting a cluster, characterized in that: The countermeasure method includes: Acquire a confrontation cluster, wherein the confrontation cluster is applied to unmanned equipment, the confrontation cluster includes multiple agents, each agent corresponds to an unmanned equipment, the agent can make confrontation decisions in the confrontation process with the opponent, and the confrontation cluster is trained according to the confrontation cluster training method according to any one of claims 1 to 6; The adversarial cluster is used to fight against a target opponent.

8. A training system for adversarial clustering, characterized in that: The confrontation cluster is applied to unmanned equipment, the confrontation cluster includes multiple agents, each agent corresponds to an unmanned equipment, and the agent can make confrontation decisions in the process of confrontation with the opponent, wherein the training system includes: A storage unit is configured to store the confrontation data of the confrontation cluster in real time into an online strategy sample storage during the confrontation between the confrontation cluster and the opponent, wherein the confrontation data includes the observed state, winning rate and reward and punishment value of each agent in the confrontation cluster; a data transfer unit, configured to transfer the data in the online policy sample storage to the offline policy sample storage whenever the amount of data in the online policy sample storage exceeds a preset first threshold; A filtering unit is configured to filter out a plurality of data with the best confrontation index of the intelligent agent from the offline strategy sample storage according to the confrontation index preset for the intelligent agent, and obtain filtered data; a policy gradient determination unit, configured to determine a policy gradient for training the adversarial cluster based on current data in the online policy sample memory and the filtered data; A training unit is configured to update the neural network parameters of the agents in the adversarial cluster based on the policy gradient to train the adversarial cluster.

9. A confrontation system for a confrontation cluster, characterized in that: The countermeasure system comprises: A cluster acquisition unit, configured to acquire an adversarial cluster, wherein the adversarial cluster is applied to an unmanned device, the adversarial cluster includes a plurality of intelligent agents, each of which corresponds to an unmanned device, the intelligent agent can make adversarial decisions in a confrontation process with an opponent, and the adversarial cluster is trained according to the adversarial cluster training method according to any one of claims 1 to 6; The confrontation unit is configured to use the confrontation cluster to confront the target opponent.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, Wherein, when the processor executable instructions are executed by the processor, the processor is prompted to execute the training method of the adversarial cluster according to any one of claims 1 to 6 or the adversarial method of the adversarial cluster according to claim 7.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the training method for an adversarial cluster according to any one of claims 1 to 6 or the adversarial method for an adversarial cluster according to claim 7.

12. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the training method for an adversarial cluster according to any one of claims 1 to 6 or the adversarial method for an adversarial cluster according to claim 7 is implemented.

Images