A strategy generation method for unmanned boat swarm confrontation

Abstract

The present invention discloses a strategy generation method for unmanned boat swarm confrontation, comprising the following steps: establishing a scenario-oriented Markov decision process; each unmanned boat i uses an incremental approach to learn the knowledge of its cooperating unmanned boats; combining the rewards after two learning optimizations to revise the strategy of each unmanned boat; using a parameterized neural network to represent the strategy of the unmanned boat, and then using the Actor-Critic learning method to maximize the Q function to learn the optimal strategy; outputting the optimal strategy, and each unmanned boat takes action according to the optimal strategy. This application is oriented towards the constantly changing and evolving situation of unmanned scenes, meeting the needs of multi-unmanned boat joint strategy; the unmanned boats learn the knowledge of other unmanned boats through two-stage incremental learning, and revise the original strategy directly provided by the collaborative decision-making, which is closer to the actual scene situation.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control technology, and in particular relates to a strategy generation method for unmanned boat swarm confrontation. Background Art

[0002] Swarms of intelligent agents (drones or unmanned boats, etc.) have weak transferability across adversarial scenarios. Reinforcement learning is characterized by high sample complexity, and unmanned boats often require extensive training and exploration to excel at a particular task. However, this is limited to a single scenario. However, real-world scenarios are subject to a degree of suddenness and uncertainty. For example, the opponent may suddenly adjust their behavior, or information about the confrontation area may be unknown. Retraining based on the new environmental conditions often misses important opportunities. This is especially true when our swarm is smaller than the enemy's, placing greater demands on the unmanned boats' ability to cope with unexpected situations and unknown environments. Some unpredictable situations are difficult to account for during initial training. Leveraging strategies trained in existing environments to effectively mobilize the unmanned boat swarm to complete tasks, respond to different scenarios and unexpected events, and adaptively adjust their strategies to meet real-world scenarios presents a major challenge.

[0003] Unmanned boat swarms face the challenge of rapidly changing environments. The impact of different environments on individual unmanned boats within the system is difficult to predict. This not only affects the decision-making of the unmanned boats themselves, but can also cause instability in information exchange between them, further impacting their decision-making. Furthermore, the unmanned boat swarm itself faces the risk of structural change. Depending on the scenario or mission requirements, new unmanned boats may join the swarm at any time, or unmanned boats may exit for some reason (such as sinking). Changes in a single unmanned boat can affect the strategy of the entire swarm. Reinforcement transfer learning algorithms have low accuracy and computational efficiency. When the surrounding environment changes rapidly, how to quickly learn knowledge from collaborative unmanned boats becomes critical. Summary of the Invention

[0004] In response to the need for highly flexible and adaptable joint strategies for multiple unmanned boats caused by the ever-changing and evolving situation in unmanned scenarios, this application addresses the current problems of poor real-time performance and complex optimization processes in deep reinforcement transfer learning. It proposes incremental learning, which learns knowledge from collaborative unmanned boats in two stages, improves the strategy of unmanned boats, and solves the problem of unmanned boat clusters quickly making response strategies based on the situation in the scenario. In response to the need for rapid collaboration between new and old unmanned boats caused by the addition and deletion mechanism of unmanned boats in confrontation, and the problem that the unmanned boats that join later have little understanding of the environment and the poor effect of the actions they choose to execute, this application solves the problem of helping new unmanned boats collaborate with old unmanned boats more efficiently and effectively.

[0005] To achieve the above objectives, the present application discloses a strategy generation method for unmanned boat swarm confrontation, comprising the following steps:

[0006] Collect known unmanned boat swarm confrontation scenarios and corresponding strategies as training sets, and establish a scenario-oriented Markov decision process (MDP);

[0007] Each UAV i uses an incremental approach to learn about its cooperating UAVs, i.e., it initially selects some cooperating UAVs as the first set, learns the value and status of the UAVs in the first set, maximizes the reward of each UAV i, then selects more cooperating UAVs as the second set, learns the value and status of the UAVs in the second set, and maximizes the reward of each UAV i a second time.

[0008] Combining the rewards from the two learning optimizations, the strategy of each unmanned boat is revised;

[0009] Use a parameterized neural network to represent the strategy of the unmanned boat, and then use the Actor-Critic learning method to maximize the Q function to learn the optimal strategy;

[0010] Output the optimal strategy, and each unmanned boat takes action according to the optimal strategy.

[0011] Preferably, in the first set, for unmanned boat i, the probability of selecting unmanned boat j is calculated as follows:

[0012]

[0013] Among them, r i and r j are the rewards of UAV i and UAV j respectively, μ is the mean of the rewards of all UAVs, and σ is the variance of the rewards of all UAVs.

[0014] Preferably, the second set includes the first set. For unmanned boats in the second set that are different from the first set, if the reward value obtained by maximizing the unmanned boats in the first set is greater than a first threshold, then from the unmanned boats outside the first set, an unmanned boat whose value similarity with the unmanned boats in the first set is greater than a second threshold is selected and added to the second set; otherwise, an unmanned boat is randomly selected and added to the second set.

[0015] Preferably, the similarity is calculated using a cosine similarity formula.

[0016] Preferably, the reward function for the collaboration between unmanned boat i and unmanned boat j is:

[0017] R i =λ1r co +λ2r en

[0018] Among them, r co Reward for value improvement, r enis the reward for situation improvement, λ1 and λ2 are adjustment factors;

[0019] The value improvement reward is calculated as follows:

[0020]

[0021] is the state action value of unmanned boat i when unmanned boat j cooperates with unmanned boat i, t is the current moment, t-1 is the previous moment, is the state action value of unmanned boat i when unmanned boat j did not cooperate with unmanned boat i at the previous moment;

[0022] The situation improvement reward is a reward given for changes in the number of cooperating unmanned boats. The situation improvement reward is calculated as follows:

[0023]

[0024] For the knowledge of the situation before the change, is the knowledge in the changed situation, θ is the knowledge parameter, is the state-value pair of all unmanned boats in the situation;

[0025] Preferably, the total reward value of a single unmanned boat i is:

[0026]

[0027] L is the total number of unmanned boats cooperating with unmanned boat i, r l is the reward value of the l-th unmanned boat, α l is the weight of the reward value of the lth unmanned boat.

[0028] Preferably, (S, A, p, R) is the state space, action space, choice probability, and reward of the MDP tuple;

[0029] The Q function of the Markov decision process is shown below:

[0030]

[0031] in, is the Q function value, is the action at time t+1, is the value at time t+1, is the total reward value of the unmanned boat i at time t, γ is the discount factor, π (i) For strategy.

[0032] The beneficial effects of this application are as follows:

[0033] This application is aimed at the ever-changing and evolving situation of unmanned scenarios, and can meet the needs of highly flexible and adaptable multi-unmanned boat joint strategies.

[0034] The unmanned boat swarm utilizes other collaborative unmanned boats through incremental learning to assist in decision-making and correct the original actions directly provided by the collaborative decision-making, while meeting the requirements of efficiency and performance, so as to be closer to the actual scenario situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 An overall block diagram of this application. DETAILED DESCRIPTION

[0036] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0037] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0038] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0039] The technical solutions provided in the embodiments of this application involve technologies such as machine learning and natural language processing of artificial intelligence, and are specifically introduced and explained through the following embodiments.

[0040] Before introducing the embodiments of the present application, some terms involved in the present application are explained.

[0041] 1. Value Function: In reinforcement learning, the value function is a core concept that measures the long-term reward an agent can expect to receive from a given state or state-action pair. Value functions are crucial for agent decision-making and policy learning because they reflect the potential impact of different states or actions on future rewards. There are two main types of value functions: 1. State-value function: V(s) represents the average future reward an agent can expect to receive when following a policy in state (s). State-value functions can be expressed and iteratively updated using the Bellman equation. 2. Action-value function, (Q(s,a)): (Q(s,a)) represents the average future reward an agent can expect to receive when following a policy by taking action (a) in state (s). Action-value functions can also be expressed and updated using the Bellman equation. Methods for calculating and updating value functions include Monte Carlo methods, temporal difference learning, Q-Learning, and SARSA. Monte Carlo methods estimate the value function by sampling the entire sequence without a model, approximating the true value function through empirical averaging. Temporal Difference Learning (TD Learning) updates the value estimate of the current state by adding the immediate reward of the current state to the predicted value of the next state, without waiting for the complete sequence. Q-Learning uses the maximum action value to update the Q function, while SARSA uses the actual action value of the next state to update.

[0042] 2. Actor-Critic: Actor-Critic methods are an important class of algorithms in reinforcement learning (RL). They combine the advantages of policy gradient methods and temporal difference learning to more efficiently learn optimal policies. In the actor-critic architecture, the actor and critic play different roles: the actor is responsible for selecting an action based on the current state, with the goal of learning a policy—that is, what action to take in a given state to maximize the long-term reward. The actor is typically a policy function that depends on a set of parameters that are optimized via gradient ascent to maximize the expected cumulative reward. The critic is responsible for evaluating the quality of the actor's actions, that is, estimating the value of a state or state-action pair. The critic uses temporal difference learning or other value function approximation methods (such as Q-Learning or SARSA) to estimate the performance of the policy. The feedback provided by the critic (i.e., the value function estimate) is used to guide the actor's policy updates. The interaction between the actor and the critic can be summarized as follows: 1. The actor selects an action a based on the current state s. 2. The environment receives the action a and returns a new state s' and an immediate reward r. 3. The critic evaluates the quality of the action a based on the new state s' and the immediate reward r, typically by calculating the TD error (i.e., the difference between the actual reward and the expected reward). 4. The actor uses the feedback (TD error) provided by the critic to adjust its policy parameters, hoping to choose a better action in similar situations in the future.

[0043] To address the need for cross-domain strategy formulation for multiple unmanned boats caused by complex and ever-changing scenarios, it is necessary to learn a highly adaptable group collaboration strategy to provide macro-level guidance for autonomous decision-making within the unmanned swarm. On the other hand, by learning and improving strategies, each unmanned boat can correct its specific actions during the micro-level autonomous decision-making process to better coordinate with the group. This application addresses the formulation of cross-domain strategies from the macro to the micro level, with the joint cooperation of these two levels.

[0044] refer to Figure 1 The strategy generation method for unmanned boat swarm confrontation disclosed in this application includes the following steps:

[0045] Collect known unmanned boat swarm confrontation scenarios and corresponding strategies as training sets, and establish a scenario-oriented Markov decision process (MDP);

[0046] Each unmanned boat i adopts a two-stage incremental approach to learn the knowledge of its cooperating unmanned boats, namely: initially select some cooperating unmanned boats as the first set, learn the value and status of the unmanned boats in the first set, maximize the reward of each unmanned boat i, then select more cooperating unmanned boats as the second set, learn the value and status of the unmanned boats in the second set, and maximize the reward of each unmanned boat i a second time; compared with the method of learning the knowledge of all unmanned boats (full learning), the above two-stage incremental learning method can avoid the huge computational complexity and low efficiency of the full learning method, thereby improving computational efficiency.

[0047] Combining the rewards from the two learning optimizations, the strategy of each unmanned boat is revised;

[0048] Use a parameterized neural network to represent the strategy of the unmanned boat, and then use the Actor-Critic learning method to maximize the Q function to learn the optimal strategy;

[0049] Output the optimal strategy, and each unmanned boat takes action according to the optimal strategy.

[0050] In some embodiments, in the first set, for unmanned boat i, the probability of selecting unmanned boat j is calculated as follows:

[0051]

[0052] Among them, r i and r j are the rewards of UAV i and UAV j respectively, μ is the mean of the rewards of all UAVs, and σ is the variance of the rewards of all UAVs.

[0053] In some embodiments, the second set includes the first set. For unmanned boats in the second set that are different from the first set, if the reward value obtained by maximizing the unmanned boats in the first set is greater than a first threshold, then from the unmanned boats outside the first set, an unmanned boat whose value similarity with the unmanned boats in the first set is greater than a second threshold is selected and added to the second set; otherwise, an unmanned boat is randomly selected and added to the second set.

[0054] In some embodiments, the similarity is calculated using a cosine similarity formula.

[0055]

[0056] A and B are value vectors, θ is the angle between A and B, and ||A|| and |B|| represent the module lengths of A and B vectors.

[0057] In some embodiments, the reward function for the collaboration between UAV i and UAV j is:

[0058] R i=λ1r co +λ2r en

[0059] Among them, r co Reward for value improvement, r en is the reward for situation improvement, λ1 and λ2 are adjustment factors;

[0060] The value improvement reward is calculated as follows:

[0061]

[0062] is the state action value of unmanned boat i when unmanned boat j cooperates with unmanned boat i, t is the current moment, t-1 is the previous moment, It is the state action value of unmanned boat i when unmanned boat j did not cooperate with unmanned boat i at the previous moment, and Softmax is a normalized exponential function.

[0063] The situation improvement reward is a reward given for changes in the number of cooperating unmanned boats. The situation improvement reward is calculated as follows:

[0064]

[0065] For the knowledge of the situation before the change, is the knowledge in the changed situation, θ is the knowledge parameter, are the state-value pairs of all unmanned boats in the situation.

[0066] In some embodiments, the total reward value of a single unmanned boat i is:

[0067]

[0068] L is the total number of unmanned boats cooperating with unmanned boat i, r l is the reward value of the l-th unmanned boat, α l is the weight of the reward value of the lth unmanned boat.

[0069] In some embodiments, (S, A, p, R) is the state space, action space, choice probability, and reward of the MDP tuple;

[0070] The Q function of the Markov decision process is shown below:

[0071]

[0072] in, is the Q function value, is the action at time t+1, is the value at time t+1, is the total reward value of the unmanned boat i at time t, γ is the discount factor, π (i) For strategy.

[0073] The beneficial effects of this application are as follows:

[0074] The two-stage incremental learning method of this application, in which each unmanned boat learns knowledge from the collaborative unmanned boat, faces the ever-changing evolution of the unmanned scene situation, and can meet the needs of highly flexible and highly adaptable multi-unmanned boat joint strategies.

[0075] The unmanned boat swarm uses the knowledge learned by other collaborative unmanned boats to assist in decision-making, and corrects the strategies directly provided by the collaborative decision-making, which is closer to the actual scenario situation.

[0076] As used herein, the word "preferred" is intended to serve as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word "preferred" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the naturally inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing examples.

[0077] Moreover, although the present disclosure has been shown and described with respect to one or implementation, those skilled in the art will think of equivalent variations and modifications based on reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if structurally different from the disclosed structure that performs the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that can be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0078] The functional units in the embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or multiple or more units may be integrated into a single module. The aforementioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc. The aforementioned devices or systems may execute the storage method in the corresponding method embodiment.

[0079] In summary, the above embodiment is one implementation method of the present invention, but the implementation method of the present invention is not limited to the described embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A strategy generation method for unmanned boat swarm confrontation, characterized by: The following steps are involved: Collect known unmanned boat swarm confrontation scenarios and corresponding strategies as training sets, and establish a scenario-oriented Markov decision process (MDP); Each unmanned boat i uses an incremental approach to learn the knowledge of its cooperating unmanned boats, namely: initially select some cooperating unmanned boats as the first set, learn the value and status of the unmanned boats in the first set, maximize the reward of each unmanned boat i, then select more cooperating unmanned boats as the second set, learn the value and status of the unmanned boats in the second set, and maximize the reward of each unmanned boat i twice; the second set contains the first set, and for unmanned boats in the second set that are different from the first set, if the reward value obtained by maximizing the unmanned boats in the first set is greater than a first threshold, then from the unmanned boats outside the first set, the unmanned boats whose value similarity with the unmanned boats in the first set is greater than a second threshold are selected and added to the second set; otherwise, an unmanned boat is randomly selected and added to the second set; Combining the rewards from the two learning optimizations, the strategy of each unmanned boat is revised; Use parameterized neural networks to represent the strategy of the unmanned boat, and then use the Actor-Critic learning method to maximize function, thereby learning the optimal strategy; Output the optimal strategy, and each unmanned boat takes action according to the optimal strategy.

2. The strategy generation method for unmanned boat swarm confrontation according to claim 1 is characterized in that: In the first set, for unmanned boat i, the probability of selecting unmanned boat j is calculated as follows: ； in, are the rewards for unmanned boat i and unmanned boat j respectively, is the mean reward of all unmanned boats, is the variance of the rewards of all drones.

3. The strategy generation method for unmanned boat swarm confrontation according to claim 2 is characterized in that: The similarity is calculated using the cosine similarity formula.

4. The strategy generation method for unmanned boat swarm confrontation according to claim 3 is characterized in that: The reward function for the collaboration between unmanned boat i and unmanned boat j is: ； in, Rewards for value improvements, Rewards for improving the situation, and is the regulating factor; The value improvement reward is calculated as follows: ； is the state action value of unmanned boat i when unmanned boat j cooperates with unmanned boat i, t is the current moment, t-1 is the previous moment, is the state action value of unmanned boat i when unmanned boat j did not cooperate with unmanned boat i at the previous moment; The situation improvement reward is a reward given for changes in the number of cooperating unmanned boats. The situation improvement reward is calculated as follows: ； For the knowledge of the situation before the change, For knowledge in a changed situation, is the knowledge parameter, are the state-value pairs of all unmanned boats in the situation.

5. The strategy generation method for unmanned boat swarm confrontation according to claim 4 is characterized in that: The total reward value of a single unmanned boat i is: ； L is the total number of unmanned boats cooperating with unmanned boat i, is the reward value of the lth unmanned boat, is the weight of the reward value of the lth unmanned boat.

6. The strategy generation method for unmanned boat swarm confrontation according to claim 5 is characterized in that: is the state space, action space, choice probability, and reward of the MDP tuple; The Q function of the Markov decision process is shown below: ； in, is the Q function value, is the action at time t+1, is the value at time t+1, is the total reward value of unmanned boat i at time t, is the discount factor, For strategy.

Images