Heterogeneous patrol flight cluster autonomous collaborative planning method based on memory reasoning

Abstract

The invention belongs to the technical field of reinforcement learning. The invention provides a heterogeneous patrol flight cluster autonomous collaborative planning method based on memory reasoning. The method comprises the steps that a simulation environment is trained for agents of a heterogeneous patrol flight cluster, and a Markov decision process is constructed for each agent; constructing an intelligent agent dynamic model; initializing a simulation environment; data are acquired through environment interaction sampling, the double-memory module is updated, data of the long-term memory module are processed through the memory reasoning module, an excitation coefficient is generated in combination with a plot excitation mechanism, the excitation coefficient and data in the short-term memory module jointly calculate a target Q value and a current Q value, and a loss function is weighted to update an action value network. Calculating loss based on the strategy network and the entropy network, updating parameters of the strategy network and the entropy network, and periodically updating the memory reasoning module; and if the ending condition is met, stopping training, and outputting the trained multi-agent reinforcement learning model.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of reinforcement learning technology, and in particular to a method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning. Background Art

[0002] With the increasing complexity of modern battlefield environments, patrol aircraft (ROVs) have gained widespread adoption as low-cost, highly adaptable new weapons. In practical applications, autonomous multi-agent collaborative planning often requires the construction of heterogeneous patrol aircraft swarms to jointly complete missions. However, formation collaborative training presents numerous challenges. Due to the large number of RAVs and their complex formations, the mission environment presents multiple challenges, such as high dimensionality, non-stationarity, and dynamic uncertainty. This complicates the training of multi-ROV collaboration. Currently, multi-agent reinforcement learning technology has been introduced into patrol aircraft swarm collaborative training. RAV swarm reinforcement learning is typically conducted in complex and uncertain environments.

[0003] There are many common technical solutions to the multi-agent collaborative planning problem. Existing technologies include the policy gradient method, which uses state information as input, generates an action probability distribution through a policy network, and uses rewards from environmental feedback to perform gradient optimization on network parameters. For example, the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm. The paper "Multi-UAV Collaborative Task Decision-Making Based on MADDPG" published by Li Bo et al. has improved the learning speed of multi-agents to a certain extent. However, the MADDPG algorithm's use of deterministic policy gradients during training limits its exploration capabilities, and it has high requirements for communication synchronization, resulting in poor training stability in heterogeneous scenarios.

[0004] The particle swarm optimization (PSO) algorithm, a swarm intelligence algorithm, is an optimization algorithm that simulates the behavior of flocks of birds or fish. It iteratively searches for the global optimal solution in the solution space by synergizing the historical best experiences of individuals with the group. Chen Huayi's paper, "Research on Collaborative Route Planning of Multiple Unmanned Aerial Vehicles Based on Intelligent Optimization Algorithms," has achieved some success in this area. However, it has inherent limitations when applied to complex problems. It is highly dependent on the distribution of initial particles and can easily fall into local optimality due to information convergence, thus losing its global search capability.

[0005] Ant Colony Optimization (ACO) is also a type of swarm intelligence algorithm. It simulates the foraging behavior of ant colonies, features parallel search and global optimization capabilities, and uses a positive feedback mechanism of pheromones to find the optimal path. However, in autonomous collaborative planning among multi-agents, it suffers from slow convergence, high computational complexity, and insufficient adaptability to dynamic environments.

[0006] The A* algorithm, a heuristic search algorithm proposed by Peter Hart et al., is a shortest path algorithm that uses a heuristic function to improve path search efficiency. Zhang Zhe et al. published a paper titled "Multi-UAV Collaborative Tactical Planning Based on an Improved A* Algorithm," demonstrating the proposed improved A* algorithm's effectiveness to a certain extent. However, in complex scenarios, it still suffers from inefficiency and computational complexity.

[0007] While these existing technologies have made some progress, they generally suffer from low sample efficiency, high environmental interaction costs, and a tendency to fall into local optima. In particular, they lack effective storage and utilization of historical experience. Training heterogeneous patrol swarms is difficult, requiring a large number of interaction samples to train neural network models, resulting in slow algorithm convergence. Furthermore, traditional algorithms suffer from poor training stability, resulting in performance fluctuations and even non-convergence.

[0008] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0009] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the Invention

[0010] The purpose of the embodiments of the present disclosure is to provide a method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning, thereby overcoming one or more problems caused by the limitations and defects of related technologies, at least to a certain extent.

[0011] According to an embodiment of the present disclosure, a method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning is provided, the method comprising:

[0012] Step S1: for the agent training simulation environment in the heterogeneous patrol swarm game scenario, a Markov decision process is constructed for each agent; wherein the Markov decision process consists of a five-tuple<S,A,P,R,γ> Description, S is the state set, A is the action set, P is the state transition probability, R is the reward function, and γ is the discount factor;

[0013] Step S2, perform two-dimensional dynamic modeling on the intelligent agent, define the dynamic equations of position vector, velocity vector, tangential acceleration and normal acceleration, and set the motion range, velocity range, acceleration range and obstacle constraints;

[0014] Step S3, initializing the agent training simulation environment, including initializing the short-term memory module, long-term memory module, memory reasoning module and network parameters; wherein the network parameters include parameters of the policy network, action value network and entropy network, and the action value network is set to a double Q network;

[0015] Step S4, performing heterogeneous patrol swarm training based on memory and reasoning, includes:

[0016] Step S4-1, in each round, select an action based on the current state and add noise, and update the short-term memory module after interacting with the environment;

[0017] Step S4-2: The environmental interaction data is passed through the encoder in the memory reasoning module to generate an embedding vector, which is normalized and retrieved from the long-term memory module. The neighboring memories are matched by cosine distance, and the long-term memory module is updated according to the cumulative reward threshold.

[0018] Step S4-3, the action value network calculates the target Q value and the current Q value based on the data in the short-term memory module, the data in the long-term memory module, and the plot incentive, and updates the action value network using the weighted loss function;

[0019] Step S4-4, calculating the loss based on the policy network and the entropy network, and updating the policy network and entropy adjustment parameters;

[0020] Step S4-5, periodically training the memory reasoning module to optimize the prediction reward and state reconstruction error of the encoder-decoder;

[0021] Step S5: When the training reaches the preset number of rounds, the multi-agent model is saved.

[0022] Furthermore, dynamic modeling specifically includes:

[0023] The velocity magnitude is adjusted by tangential acceleration, the velocity direction is adjusted by normal acceleration, and the rate of change of heading angle is calculated in combination with the rotation vector;

[0024] Constrain the agent's motion area, speed range, acceleration range, and obstacle avoidance conditions.

[0025] Furthermore, initializing the agent training simulation environment specifically includes:

[0026] Establish a short-term memory module to store the quadruple data of recent action states and update it according to the first-in-first-out principle;

[0027] Build a memory reasoning module, including an encoder and a dual-track decoder. The encoder maps the global state to a low-dimensional embedding vector, and the decoder predicts the highest reward and reconstructs the state.

[0028] Build a long-term memory module to store six-tuple data of high-value experiences, including embedding vector, normalized embedding vector, highest reward, action, reconstructed state and desirability label;

[0029] Initialize the parameters of the policy network, action-value network, and entropy network.

[0030] Furthermore, the encoder is a three-layer fully connected network that uses a self-attention mechanism to capture global state features; among them, the decoder uses a dual-track structure to predict the highest reward and reconstruct the state respectively.

[0031] Furthermore, when updating the long-term memory module, if the cumulative reward of the current trajectory exceeds the preset threshold, it is marked as a desirable state, and the historical embedding vector is matched through the nearest neighbor algorithm to update the access count and desirable count; otherwise, the current data is stored in the long-term memory module.

[0032] Furthermore, during the calculation of the target Q value, the episodic incentive is obtained by multiplying the difference between the highest return of the embedded vector in the long-term memory module and the target optimal Q value by the ratio of the access count to the available count, and is weighted into the target Q value.

[0033] Furthermore, the loss function of the action value network is the weighted sum of short-term memory loss and long-term memory loss, and the sum of the weight coefficients is 1.

[0034] Furthermore, the policy network is updated by minimizing the difference between the policy entropy and the minimum Q value; wherein the policy entropy is dynamically adjusted by the entropy adjustment coefficient.

[0035] Furthermore, the loss function of the memory reasoning module includes the predicted reward error of the predicted reward and the reconstructed state error, and is balanced by the weight of the reconstruction loss.

[0036] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0037] In the disclosed embodiments, the aforementioned memory-based reasoning method for autonomous collaborative planning of heterogeneous patrol swarms utilizes a dual memory module consisting of a short-term memory module and a long-term memory module, incorporating a memory-based reasoning module and an episodic incentive mechanism. The short-term memory module stores action states generated during recent exploration; the long-term memory module continuously stores high-value empirical strategies accumulated during the training phase for subsequent utilization. The memory-based reasoning module, combined with an attention mechanism, constructs a trainable encoder-decoder structure that converts these empirical strategies into embedding vectors and outputs the highest predicted reward for the state, achieving efficient storage of historical empirical strategies. The episodic incentive mechanism generates an incentive coefficient based on the desirability statistics in the long-term memory module combined with the highest predicted reward, guiding the patrol swarm to explore high-reward areas. Furthermore, this method utilizes the dual memory module to obtain high-quality action decisions. Combined with the encoder-decoder, this method enables rapid matching and utilization of historical empirical strategies, guiding the agent to execute action sequences with higher rewards, avoiding repeated exploration of the environment, and improving the utilization efficiency of historical empirical strategies. This significantly reduces the number of training rounds and enhances learning efficiency during network training. In addition, a plot incentive mechanism is provided to achieve additional rewards for high-quality actions, guiding the patrol swarm to prioritize exploring high-return areas, achieving a balance between exploration and utilization, and resolving the shortcomings of the intelligent agent falling into local optimality, being unable to stably complete the target task, or causing performance fluctuations during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0039] Figure 1 A diagram showing the steps of a method for autonomous collaborative planning of a heterogeneous patrol swarm based on memory reasoning in an exemplary embodiment of the present disclosure is shown;

[0040] Figure 2 A diagram showing a structural relationship of network modules in an exemplary embodiment of the present disclosure is shown;

[0041] Figure 3 A schematic diagram of an urban game simulation environment for interactive training of heterogeneous patrol swarms constructed in an exemplary embodiment of the present disclosure is shown;

[0042] Figure 4 A task diagram in an exemplary embodiment of the present disclosure is shown;

[0043] Figure 5A graph showing how the reward value changes with the number of rounds during the city game task training process using the method and the MADDPG algorithm in an exemplary embodiment of the present disclosure;

[0044] Figure 6 A graph showing how the winning rate of the method and the MADDPG algorithm in an exemplary embodiment of the present disclosure changes with the number of steps during the city game task training process is shown. DETAILED DESCRIPTION

[0045] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0046] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0047] This example implementation provides a method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning. Figure 1 As shown in , the autonomous collaborative planning method of heterogeneous patrol swarm based on memory reasoning may include:

[0048] Step S1: for the agent training simulation environment in the heterogeneous patrol swarm game scenario, a Markov decision process is constructed for each agent; wherein the Markov decision process consists of a five-tuple<S,A,P,R,γ> Description, S is the state set, A is the action set, P is the state transition probability, R is the reward function, and γ is the discount factor;

[0049] Step S2, perform two-dimensional dynamic modeling on the intelligent agent, define the dynamic equations of position vector, velocity vector, tangential acceleration and normal acceleration, and set the motion range, velocity range, acceleration range and obstacle constraints;

[0050] Step S3, initializing the agent training simulation environment, including initializing the short-term memory module, long-term memory module, memory reasoning module and network parameters; wherein the network parameters include parameters of the policy network, action value network and entropy network, and the action value network is set to a double Q network;

[0051] Step S4, performing heterogeneous patrol swarm training based on memory and reasoning, includes:

[0052] Step S4-1, in each round, select an action based on the current state and add noise, and update the short-term memory module after interacting with the environment;

[0053] Step S4-2: The environmental interaction data is passed through the encoder in the memory reasoning module to generate an embedding vector, which is normalized and retrieved from the long-term memory module. The neighboring memories are matched by cosine distance, and the long-term memory module is updated according to the cumulative reward threshold.

[0054] Step S4-3, the action value network calculates the target Q value and the current Q value based on the data in the short-term memory module, the data in the long-term memory module, and the plot incentive, and updates the action value network using the weighted loss function;

[0055] Step S4-4, calculating the loss based on the policy network and the entropy network, and updating the policy network and entropy adjustment parameters;

[0056] Step S4-5, periodically training the memory reasoning module to optimize the prediction reward and state reconstruction error of the encoder-decoder;

[0057] Step S5: When the training reaches the preset number of rounds, the multi-agent model is saved.

[0058] The aforementioned memory-based reasoning method for autonomous collaborative planning of heterogeneous patrol swarms utilizes a dual memory module consisting of a short-term memory module and a long-term memory module, integrated with a memory-based reasoning module and an episodic incentive mechanism. The short-term memory module stores action states generated during recent exploration, while the long-term memory module continuously stores high-value empirical strategies accumulated during training for subsequent utilization. The memory-based reasoning module, combined with an attention mechanism, constructs a trainable encoder-decoder structure that converts these empirical strategies into embedding vectors and outputs the highest predicted reward for each state, effectively storing these strategies. The episodic incentive mechanism generates incentive coefficients based on the desirability statistics in the long-term memory module combined with the highest predicted reward, guiding the patrol swarm to explore high-reward areas. Furthermore, this method utilizes the dual memory module to obtain high-quality action decisions. Combined with the encoder-decoder, this method enables rapid matching and utilization of historical strategies, guiding the agent to select action sequences with higher rewards, avoiding repeated exploration of the environment, and improving the utilization of historical strategies. This significantly reduces the number of training rounds and enhances learning efficiency during network training. In addition, a plot incentive mechanism is provided to achieve additional rewards for high-quality actions, guiding the patrol swarm to prioritize exploring high-return areas, achieving a balance between exploration and utilization, and resolving the shortcomings of the intelligent agent falling into local optimality, being unable to stably complete the target task, or causing performance fluctuations during training.

[0059] Below, we will refer to Figures 1 to 6Each step of the above-mentioned memory-reasoning-based heterogeneous patrol swarm autonomous collaborative planning method in this example implementation is described in more detail.

[0060] In step S1, the patrol swarm needs to penetrate the denied area and avoid obstacles such as air defense radars on the one hand, and complete continuous task planning and collaborative decision-making issues such as tracking and encircling the game targets on the other hand.

[0061] The heterogeneous patrol swarm in this application refers to a multi-agent formation containing two different types of patrol aircraft: gaming patrol aircraft and reconnaissance patrol aircraft. Reconnaissance patrol aircraft are primarily responsible for detecting obstacles such as urban air defense radars and gaming targets. They also communicate with gaming patrol aircraft, providing key information for the formation to avoid obstacles and locate mobile targets. Gaming patrol aircraft are responsible for receiving battlefield information transmitted by reconnaissance patrol aircraft and engaging in gaming with located mobile targets.

[0062] For the agent training simulation environment in the heterogeneous patrol swarm game scenario, a Markov decision process is constructed for each agent; the Markov decision process consists of a five-tuple<S,A,P,R,γ> Description, S is the state set, A is the action set, P is the state transition probability, R is the reward function, and γ is the discount factor.

[0063] In step S2, two-dimensional dynamic modeling is performed on the intelligent agent, the dynamic equations of position vector, velocity vector, tangential acceleration and normal acceleration are defined, and the motion range, velocity range, acceleration range and obstacle constraints are set.

[0064] Specifically, the two-dimensional dynamics modeling of the intelligent agent is as follows:

[0065]

[0066] The constraints are:

[0067]

[0068] Among them, i is the index number of the agent. For the i-th agent, S i (t) represents the position vector of the i-th agent at time t, x i (t), y i (t) represents the horizontal position coordinate and vertical position coordinate of the i-th agent at time t; v i (t) represents the velocity vector of the i-th agent at time t, including the velocity magnitude and direction; a i,|| (t) represents the tangential acceleration of the i-th agent at time t, which is used to change the magnitude of the velocity, a i,⊥(t) represents the normal acceleration of the i-th agent at time t, which is used to change the direction of the velocity, R 90 v i (t) is the velocity vector rotated 90 degrees to represent the direction of normal acceleration; θ i (t) represents the heading angle of the i-th agent at time t, which is used to describe the movement direction of the agent; represents the angular velocity of the agent, that is, the rate of change of the heading angle over time. In addition, ‖·‖ represents the vector modulus.

[0069] The symbolic meaning of the constraints: [x i,min ,x i,max ]×[y i,min ,y i,max ] represents the range of motion of the i-th agent, restricting its flight within this area; [v i,min ,v i,max ] represents the speed range of the i-th agent, limiting its minimum and maximum flight speed; p0, a i,max ] represents the acceleration range of the i-th agent, a i,max is the maximum acceleration; the obstacle range can be set to a circular area, where (x o ,y o ) is the obstacle center, R o is the radius.

[0070] In step S3, the agent training simulation environment is initialized, including initializing the short-term memory module, the long-term memory module, the memory reasoning module and the network parameters; wherein the network parameters include the parameters of the policy network, the action value network and the entropy network, and the action value network is set to a double Q network.

[0071] Specifically, initialize the short-term memory module:

[0072] Establish a short-term experience replay buffer D S , mainly stores recent action status. D S Update memory according to the first-in-first-out principle and set the maximum capacity D max , and in 4-tuples<s,s′,a,r> Storage. Where: s represents the current state, s′ represents the next state, a represents the joint action, and r represents the reward value.

[0073] Initialize the memory reasoning module:

[0074] The memory reasoning module is combined with the attention mechanism to construct a trainable encoder-decoder.

[0075] encoder It is defined as a 3-layer fully connected network that can share the dual-track structure of the underlying decoder and can capture the global state s through self-attention tThe key features of . Input global state s t and time step t, outputting a low-dimensional embedding vector The expression is as follows:

[0076]

[0077] Decoder f Ψ : The dual-track structure includes a reward prediction track and a reconstructed state track. The decoder receives the embedding vector x t and time step t, the highest predicted return for the output state and rebuild state

[0078] According to the formula:

[0079]

[0080] Compute the highest predicted reward and reconstruct the state.

[0081] Initialize the long-term memory module:

[0082] Building long-term memory module D L , mainly stores high-value experience strategies. Using six-tuple<x,y,H,a,s,ξ> Storage, where x is the embedding vector generated by the memory reasoning module, y is the normalized embedding vector, H is the highest reward corresponding to the embedding vector, a is the current action, s represents the reconstructed state, and ξ is the desirable label.

[0083] Initialize network parameters:

[0084] Initialize the policy network (Actor network), action value network (Q network) parameters, and entropy network parameters of each agent. The action value network is set to a double Q network.

[0085] The policy network outputs action distribution, the action value network is used to output Q value, and the entropy network outputs entropy adjustment parameters.

[0086] In step S4, Figure 2 The following is a diagram showing the relationship between the network modules. Heterogeneous patrol swarm training based on memory and reasoning.

[0087] Specifically include:

[0088] Step S4-1: In each round:

[0089] Step S4-1-1: Initialize the environment and obtain the current state s.

[0090] Step S4-1-2: In each step t of the round:

[0091] First, for each agent i, according to the current state s i Select action a i :

[0092] According to the formula:

[0093] a i =Actor(s i )+Noise

[0094] Input parameter s to the policy network i Generate action distribution, before t n The first step is to add noise to the action to enhance the agent's exploration.

[0095] The action is further clipped to be limited to the range of [-1, 1]. i =[a1, a2, ..., a N ] to interact with the environment.

[0096] After all agents complete their interactions, the environment returns the next state s′ and the reward value r. The generated experience samples are grouped into 4-tuples.<s,s′,a,r> Stored in the form of short-term experience playback buffer D S , and follow the first-in-first-out mechanism to maintain the buffer capacity.

[0097] Step 4-S2: Update long-term memory module:

[0098] First, the interactive environment<s,a,r> Composing an episodic sequence track, using the encoder in the memory reasoning module Generate embedding vector x t .

[0099] According to the formula:

[0100]

[0101] Calculate the normalized embedding vector y t , where μ x and σ x D L Embed vector x in t The statistical mean and variance of .

[0102] According to the formula:

[0103] R t =r t +γR t-1

[0104] Calculate the cumulative return R of the current round t , where R t-1is the cumulative return calculated in the previous time step, γ is the discount factor, r t is the reward value of the current time step.

[0105] If R t ≥R thr (R thr is the preset target cumulative reward threshold), then mark the current state as a desirable trajectory, i.e., ξ t =1.

[0106] Secondly, the nearest neighbor algorithm is used to retrieve the historical normalized embedding vector y′ stored in the long-term memory module t .

[0107] According to the cosine distance formula:

[0108]

[0109] Calculate y t and y′ t The distance d cos , used to judge the state similarity. Among them, y t Normalized embedding vector generated for the current state, y′ t Normalize the embedding vectors for all history stored in the long-term memory module.

[0110] Furthermore, the parameter K is set to screen out the K neighboring memories with the smallest cosine distance, and then the one with the smallest cosine distance is selected as the nearest neighbor memory.

[0111] Then update and store the long-term memory module:

[0112] ① If the distance d is less than the set threshold δ, the embedding vector x in the nearest neighbor memory t ′’s access count N call (x t ') According to the formula N call (x t ′)=N call (x t ′)+ε is updated, where ε can be preset and is used to quantify the update amplitude of the access count.

[0113] ② If the distance d is less than the set threshold δ, and the current trajectory state is desirable ξ t =1, for x t The possible count N of ′ ξ (x t ') According to the formula N ξ (x t ′)=N ξ (x t′)+ε is updated, where ε can be preset and is used to quantify the update amplitude of the desirable count.

[0114] ③ If the distance d is less than the set threshold δ, the current trajectory is in a desirable state, and x t The historical trajectory corresponding to ′ is not desirable t =0, use x t →x t ′,y t →y t ′,R t →H t ,a t →At t ,S t →s t 'Update long-term memory module D L x in t ′,y t ′,H t ,a′ t ,s t ′, and ξ t Desirability is marked as 1.

[0115] ④ If the distance d is greater than or equal to the set threshold δ, then <x t ,y t ,H t ,a t ,s t ,ξ>stored in the long-term memory module.

[0116] Step S4-3: Calculate the target Q value and current Q value according to the action-value network.

[0117] Step S4-3-1: Calculate the target Q value:

[0118] First, when the short-term experience replay buffer D S The amount of data in exceeds the set capacity D max When D S Randomly extract a batch of data M and separate the state M s , action M a 、Reward M r and the next state M′ s .

[0119] For each agent i:

[0120] Input the next state M' to the policy network s , output the mean μ and standard deviation σ of the action probability distribution, according to the formula:

[0121] a′=tanh(μ+σz)

[0122] Get the next action a′ and limit it to (-1,1). Where tanh is the hyperbolic tangent function, μ+σz is the sampling value, is the standard normal noise.

[0123] Further according to the formula:

[0124]

[0125] Calculates the logarithmic probability of a normal distribution, where ∈ is pre-set to a very small constant.

[0126] Further according to the formula:

[0127]

[0128] Calculate the Q value, and set the action strategy network to a double Q network in steps 3-4. π is the action value function of the Q network in state π, r(s t ,a t ) represents the reward value of state s and action a at time t, γ is the discount factor, and the output is two Q values, which can be seen in step S4-3-2.

[0129] Further according to the formula:

[0130]

[0131] Calculate plot incentive r p , through the encoder The next state s′ t Convert to embedding vector x′ t , then in D L Search x′ t , and obtain the highest reward H′ corresponding to the embedding vector t , desirability statistics N ξ (s′ t ), N call (s′ t ) and the target network in s′ t The Q value corresponding to the optimal action under

[0132] According to the formula:

[0133] Q tar =r(s,a)+γ(min(Q1(s′ t ,a′),Q2(s′ t ,a′))+r p )

[0134] Calculate the target Q value, which includes r pAs a plot incentive. In the formula, r is the reward M in the short-term experience replay buffer r , here only the current reward is taken. s′ t is the next state M in the short-term experience replay buffer s ′, a′ is the next action.

[0135] Step S4-3-2: Calculate the current Q value:

[0136] According to the formula:

[0137] Q cur1 =Q π (s t ,a l )

[0138] Q cur2 =Q π (s t ,a t )

[0139] Calculate the current Q value. Among them, s t Indicates the current state, corresponding to D S State M in s . Through the encoder The state s t Convert to embedding vector x t , then in D L Search x′ t , and obtain the action a corresponding to the six-tuple where the embedding vector is located l . And a t Indicates the current action, corresponding to D S Action M a .

[0140] Generate two different current Q values, Q cur1 represents the Q value calculated by the short-term memory module associated action value network, Q cur2 Represents the Q value calculated by the action-value network corresponding to the better action in the long-term memory module.

[0141] Step S4-4: Update the action-value network:

[0142] According to the formula:

[0143]

[0144] Using mean square error loss, calculate the weighted sum of the action value network loss values. Where a is the weight of the short-term memory association loss L1, b is the weight of the long-term memory association loss L2, minimize L by gradient descent, and soft-update the Q network weight parameters

[0145] Step S4-5: Update the policy network:

[0146] At time t, according to the formula:

[0147]

[0148] Calculate the loss value of the policy network. S Extract s t ,a t , s t represents the state at time t, a t Represents the action. α is the entropy adjustment coefficient, using the gradient descent method, according to L actor Update policy network parameters.

[0149] Step S4-6: Entropy network update:

[0150] At time t, according to the formula:

[0151]

[0152] Calculate the loss value of the entropy network. α is the entropy adjustment coefficient, S target It is the preset minimum limit of policy entropy. The entropy network is adjusted according to this value, and the weight parameters of the entropy network are updated using the gradient descent method.

[0153] Step S4-7: Periodically update the memory reasoning module:

[0154] Every t steps, in the long-term memory module D L Extract a batch of data x of size B at intervals t ,s t To train the encoder-decoder. The loss function is:

[0155]

[0156] It consists of two parts: prediction return error and reconstruction state error. represents the highest predicted reward output by the decoder’s reward prediction track, represents the reconstruction state of the decoder output track, λ rcon is the weight of the reconstruction loss.

[0157] Update network parameters using gradient descent and f Ψ , recalculate D according to the updated encoder-decoder L The embedding vector x in t .

[0158] In step S5, when the number of training rounds is reached, training is stopped and the final trained multi-agent model is saved. Performance indicators such as the reward value that changes with the number of rounds during training are output.

[0159] In a specific embodiment, a reinforcement learning algorithm process based on memory reasoning for heterogeneous patrol swarms in an urban game scenario is provided. This is applicable to situations where heterogeneous patrol swarms perform aerial obstacle avoidance and position game targets in complex urban game environments. The overall process of heterogeneous patrol swarm urban game includes:

[0160] 1. Build a heterogeneous patrol swarm training simulation environment for training tasks:

[0161] like Figure 3 As shown, this embodiment uses the cross-platform Python module Pygame to build a simulation environment, uses Python to write a training algorithm, realizes the communication between the algorithm and the simulation environment, and sends the city game information (the position, speed, azimuth information, air defense radar information, etc. of the patrol aircraft) to the patrol aircraft cluster training algorithm.

[0162] like Figure 4 As shown, the training task of this embodiment is as follows: To adapt to the complex and tactically flexible nature of urban gaming, a heterogeneous patrol aircraft swarm forms a reconnaissance patrol aircraft-gaming patrol aircraft formation in a two-dimensional airspace environment, starting from a random initial position. While avoiding the detection range of air defense radars in the environment, the reconnaissance patrol aircraft locates and tracks gaming targets, transmitting the detected target information to the gaming patrol aircraft, which then surrounds the target location, thereby training the heterogeneous patrol aircraft swarm's urban gaming capabilities.

[0163] The two-dimensional kinematic model of the heterogeneous patrol swarm in this embodiment is as follows:

[0164] There are currently 6 patrol aircraft in total, 4 gaming patrol aircraft of the Feihong-97 model, and 2 reconnaissance patrol aircraft of the Wing Loong-2 model. The continuous-time dynamic equation of the patrol aircraft is:

[0165]

[0166] Constraints

[0167] Among them, i is the index number of the patrol aircraft. For the i-th patrol aircraft, S i (t) represents the position vector of the i-th patrol vehicle at time t, x i (t) represents the lateral position coordinate of the i-th patrol vehicle at time t, y i (t) represents the longitudinal position coordinate of the i-th patrol vehicle at time t; v i(t) represents the velocity vector of the i-th patrol vehicle at time t, including the velocity magnitude and direction; a i,|| (t) represents the tangential acceleration of the i-th rover at time t, which is used to change the magnitude of the velocity; a i,⊥ (t) represents the normal acceleration of the i-th rover at time t, which is used to change the direction of the velocity; R 90 v i (t) is the velocity vector rotated 90 degrees to represent the direction of normal acceleration;

[0168] θ i (t) represents the heading angle of the i-th rover at time t, which is used to describe the movement direction of the rover; represents the angular velocity of the rover, that is, the rate of change of the heading angle over time. In addition, ‖·‖ represents the vector modulus.

[0169] The symbolic meaning of the constraints: [x i,min ,x i,max ]×[y i,min ,y i,max ] represents the range of motion of the i-th patrol vehicle, limiting its flight within this area; [v i,min ,v i,max ] represents the speed range of the i-th patrol vehicle, limiting its minimum and maximum flight speeds; [0,a i,max ] represents the acceleration range of the i-th rover, a i,max The maximum acceleration is set to 6m / s 2 In this embodiment, the air defense radars are centered at (120, 200) and (370, 450), R o =40m is the circular area with detection radius.

[0170] In addition, in this embodiment, the game target is set to move at a constant speed of 6 m / s, and its capture radius is set to 50 m.

[0171] In this embodiment, the simulation environment is defined as a 1000m × 1600m two-dimensional plane. The map parameters are shown in Table 1. The animation refresh rate of the environment is 60 FPS. The initial positions of each patrol vehicle are shown in Table 2. The pseudo code of the control rules of the game-type patrol vehicle is shown in Table 3.

[0172] Table 1 Map parameters of the embodiment

[0173]

[0174] Table 2 Initial position parameters of the patrol vehicle in the embodiment

[0175]

[0176] Table 3 Pseudo code of control rules of game-type patrol aircraft in the embodiment

[0177]

[0178]

[0179] 2. Specify the patroler's state space, action space, and reward function:

[0180] In this embodiment, the patroller state space is set to an 8-dimensional tensor, and the corresponding meanings of each dimension are shown in Table 4; the action space is set to a 3-dimensional tensor, and the corresponding meanings of each dimension are shown in Table 5; the pseudo code of the patroller reward function is shown in Table 6.

[0181] Table 4. State space of patrol aircraft in the embodiment

[0182]

[0183] Table 5: Action space of patrol vehicle in the embodiment

[0184] Dimension meaning 0 Control the tangential acceleration of the rover 1 Control the normal acceleration of the rover 2 Control the angular velocity of the patrol vehicle

[0185] Table 6 Pseudo code of patrol reward function of embodiment

[0186]

[0187]

[0188] 3. Start the simulation environment and initialize the virtual environment of the patrol cluster city game:

[0189] Run this Python script to start the training environment. Over multiple training rounds and rounds, the patrol aircraft interacts in the simulation environment, collecting experience and strategies and storing them in the dual memory module. When the dual memory module has sufficient data, learning is performed to update the neural network parameters. Finally, the training results are saved and plotted.

[0190] 4. Training heterogeneous patrol flight formations:

[0191] Set the number of rounds to 400 and the number of steps per round to 1000.

[0192] Initialize the short-term memory module:

[0193] Establish a short-term experience replay buffer D S , mainly stores recent action status. D S Update memory according to the first-in-first-out principle and set the maximum capacity D max is 20000, in 4-tuple<s,s′,a,r> Storage, where: s represents the current state, s′ represents the next state, a represents the joint action, and r represents the reward value.

[0194] Initialize the memory reasoning module:

[0195] The memory reasoning module is combined with the attention mechanism to construct a trainable encoder-decoder.

[0196] encoder It is defined as a 3-layer fully connected network that can share the dual-track structure of the underlying decoder and can capture the global state s through self-attention t The key features of . Input global state s t and time step t, outputting a low-dimensional embedding vector The expression is as follows:

[0197]

[0198] Decoder f Ψ : The dual-track structure includes a reward prediction track and a reconstructed state track. The decoder receives the embedding vector x t and time step t, the highest predicted return for the output state and rebuild state

[0199] According to the formula:

[0200]

[0201] Compute the highest predicted reward and reconstruct the state.

[0202] Initialize the long-term memory module:

[0203] Building long-term memory module D L , mainly stores high-value experience strategies. Using six-tuple<x,y,H,a,s,ξ> Storage, where x is the embedding vector generated by the memory reasoning module, y is the normalized embedding vector, H is the highest reward corresponding to the embedding vector, a is the current action, s represents the reconstructed state, and ξ is the desirable label.

[0204] Initialize network parameters:

[0205] Initialize the policy network (Actor network), action value network (Q network) parameters, and entropy network parameters of each agent. The action value network is set to a double Q network.

[0206] The policy network outputs action distribution, the action value network is used to output Q value, and the entropy network outputs entropy adjustment parameters.

[0207] 5. Heterogeneous Agent Training Based on Memory and Reasoning:

[0208] In each round:

[0209] The environment is initialized and the current state s is obtained.

[0210] At each step t of the episode:

[0211] First, for each agent i, according to the current state s i Select action a i :

[0212] According to the formula:

[0213] a i =Actor(s i )+Noise

[0214] Use the policy network input parameter s i Generate action distribution, before t n The step is to add OU noise to the action to enhance the agent's exploration. In this embodiment, n Set to 50.

[0215] The action is further clipped to be limited to the range of [-1, 1]. i =[a1, a2, ..., a N ] to interact with the environment.

[0216] After all agents complete their interactions, the environment returns the next state s′ and the reward value r. The generated experience samples are grouped into 4-tuples.<s,s′,a,r> Stored in D S and follow the first-in-first-out mechanism to maintain the buffer capacity.

[0217] Update long-term memory module:

[0218] First, the interactive environment<s,a,r> Composing an episodic sequence track, using the encoder in the memory reasoning module Generate embedding vector x t .

[0219] According to the formula:

[0220]

[0221] Calculate the normalized embedding vector y t , where μ x and σ x D L Embed vector x in t The statistical mean and variance of .

[0222] According to the formula:

[0223] R t =r t +γRt-1

[0224] Calculate the cumulative return R of the current round t , where R t-1 is the cumulative return calculated in the previous time step, γ is the discount factor, set to 0.1, r t is the reward value of the current time step.

[0225] If R t ≥R thr (R thr is the preset target cumulative return threshold), in this embodiment, R is set thr If is 2, the current state is marked as a desirable trajectory, i.e. t =1.

[0226] The nearest neighbor algorithm is used to retrieve the historical normalized embedding vector y′ stored in the long-term memory module t .

[0227] According to the cosine distance formula:

[0228]

[0229] Calculate y t and y′ t The distance d cos , used to judge the state similarity. Among them, y t Normalized embedding vector generated for the current state, y′ t Normalize the embedding vectors for all history stored in the long-term memory module.

[0230] Furthermore, the parameter K is set to 3, and the three neighboring memories with the smallest cosine distance are screened out, and then the one with the smallest cosine distance is selected as the nearest neighbor memory.

[0231] Then update and store the long-term memory module:

[0232] ① If the distance d is less than the set threshold δ (δ is set to 1.3×10 -3 ), the embedding vector x in the nearest neighbor memory t ′’s access count N call (x t ') According to the formula N call (x t ′)=N call (x t ′)+ε is updated, where ε is set to 0.5 to quantify the update amplitude of the desirable count.

[0233] ② If the distance d is less than the set threshold δ, and the current trajectory state is desirable ξ t =1, for xt The possible count N of ′ ξ (x t ') According to the formula N ξ (x t ′)=N ξ (x t ′)+ε is updated, where ε is set to 0.5 to quantify the update amplitude of the desirable count.

[0234] ③ If the distance d is less than the set threshold δ, the current trajectory is in a desirable state, and x t The historical trajectory corresponding to ′ is not desirable t =0, use x t →x t ′,y t →y t ′,R t →H t ,a t →a′ t ,s t →s t 'Update long-term memory module D L x in t ′,y t ′,H t ,a′ t ,S t ′, and ξ t Desirability is marked as 1.

[0235] ④ If the distance d is greater than or equal to the set threshold δ, then <x t ,y t ,H t ,a t ,s t ,ξ>stored in the long-term memory module.

[0236] Calculate the target Q value and current Q value according to the action-value network.

[0237] Calculate the target Q value:

[0238] First, when the short-term experience replay buffer D S When the amount of data in exceeds the set capacity of 20,000, S Randomly extract a batch of data M and separate the state M s , action M a 、Reward M r and the next state M′ s .

[0239] For each agent i:

[0240] Input the next state M' to the policy network s, output the mean μ and standard deviation σ of the action probability distribution, according to the formula:

[0241] a′=tanh(μ+σZ)

[0242] Get the next action a′ and limit it to (-1,1). Where tanh is the hyperbolic tangent function, μ+σz is the sampling value, is the standard normal noise.

[0243] Further according to the formula:

[0244]

[0245] Calculate the logarithmic probability of the normal distribution. Where ∈ is set to 10 -6 .

[0246] Further according to the formula:

[0247]

[0248] Calculate the target Q value, and S204-4 set the action strategy network to a double Q network. π is the action value function of the Q network in state π, r(s t ,a t ) represents the reward value of state s and action a at time t, γ is the discount factor, set to 0.9, and the output is two Q values.

[0249] Further according to the formula:

[0250]

[0251] Calculate plot incentive r p , through the encoder The next state s′ t Convert to embedding vector x′ t , then in D L Search x′ t , and obtain the highest reward H′ corresponding to the embedding vector t , desirability statistics N ξ (s′ t ), N call (s′ t ) and the target network in s′ t The Q value corresponding to the optimal action under

[0252] According to the formula:

[0253] Q tar =r(s,a)+γ(min(Q1(s′ t ,a′),Q2(s′t ,a′))+r p )

[0254] Calculate the target Q value, which includes r p As a plot incentive. In the formula, r is the reward M in the short-term experience replay buffer r , here only the current reward is taken. γ is the discount factor, set to 0.9, s′ t is the next state M in the short-term experience replay buffer s ′, a′ is the next action.

[0255] Calculate the current Q value:

[0256] According to the formula:

[0257] Q cur1 =Q π (s t ,a l )

[0258] Q cur2 =Q π (s t ,a t )

[0259] Calculate the current Q value. Among them, s t Indicates the current state, corresponding to D S State M in s . Through the encoder The state s t Convert to embedding vector x t , then in D L Search x′ t , and obtain the action a corresponding to the six-tuple where the embedding vector is located l . And a t Indicates the current action, corresponding to D S Action M a .

[0260] Generate two different current Q values, Q cur1 represents the Q value calculated by the short-term memory module associated action value network, Q cur2 Represents the Q value calculated by the action-value network corresponding to the better action in the long-term memory module.

[0261] Update the action-value network:

[0262] According to the formula:

[0263]

[0264] The weighted sum of the action-value network loss values ​​is calculated using mean squared error loss. Here, a is the weight of the short-term memory association loss L1, and b is the weight of the long-term memory association loss L2. In this example, a = 0.7 and b = 0.3 are set for the first 100 rounds, and a = 0.3 and b = 0.7 for the next 300 rounds. Gradient descent is used to minimize L, and the Q network weight parameters are soft-updated.

[0265] Update the policy network:

[0266] At time t, according to the formula:

[0267]

[0268] Calculate the loss value of the policy network. Among them, s t Indicates the current state, a t Indicates D S The action extracted from , α is the entropy adjustment coefficient. Using the gradient descent method, according to L actor Update network parameters.

[0269] Entropy adjustment coefficient network update:

[0270] At time t, according to the formula:

[0271]

[0272] Calculate the loss value of the entropy network. S Extract s t and a t , s t Indicates the current state, a t represents the action, α is the entropy adjustment coefficient, S target It is the preset lowest limit value of the policy entropy, which is set to -0.1 in this embodiment. The entropy network is adjusted according to this value, and the weight parameters of the entropy network are updated using the gradient descent method.

[0273] Periodically update the memory reasoning module:

[0274] Every t steps, in the long-term memory module D L Extract a batch of data x of size B at interval t t ,s t To train the encoder-decoder, in this embodiment, t is set to 10000 and B is set to 1024.

[0275] The loss function is:

[0276]

[0277] It consists of two parts: prediction return error and reconstruction state error. represents the highest predicted reward output by the decoder’s reward prediction track, represents the reconstruction state of the decoder output track, λ rcon Set to 0.1.

[0278] 6. When the number of training rounds reaches the maximum number of rounds 400, the training ends.

[0279] The Python program will close the simulation environment, terminate the training, and save the training models of the two different specifications of patrol aircraft as L1.pt and F1.pt files respectively. Figure 5 A curve chart showing how the reward value changes with the number of rounds during a training session of this embodiment is given. Figure 6 A curve diagram showing the change in the winning rate versus the number of steps during a training process of this embodiment is provided. Both trends reflect the effectiveness of the algorithm of the present invention.

[0280] The aforementioned memory-based reasoning method for autonomous collaborative planning of heterogeneous patrol swarms utilizes a dual memory module consisting of a short-term memory module and a long-term memory module, integrated with a memory-based reasoning module and an episodic incentive mechanism. The short-term memory module stores action states generated during recent exploration, while the long-term memory module continuously stores high-value empirical strategies accumulated during training for subsequent utilization. The memory-based reasoning module, combined with an attention mechanism, constructs a trainable encoder-decoder structure that converts these empirical strategies into embedding vectors and outputs the highest predicted reward for each state, effectively storing these strategies. The episodic incentive mechanism generates incentive coefficients based on the desirability statistics in the long-term memory module combined with the highest predicted reward, guiding the patrol swarm to explore high-reward areas. Furthermore, this method utilizes the dual memory module to obtain high-quality action decisions. Combined with the encoder-decoder, this method enables rapid matching and utilization of historical strategies, guiding the agent to select action sequences with higher rewards, avoiding repeated exploration of the environment, and improving the utilization of historical strategies. This significantly reduces the number of training rounds and enhances learning efficiency during network training. In addition, a plot incentive mechanism is provided to achieve additional rewards for high-quality actions, guiding the patrol swarm to prioritize exploring high-return areas, achieving a balance between exploration and utilization, and resolving the shortcomings of the intelligent agent falling into local optimality, being unable to stably complete the target task, or causing performance fluctuations during training.

[0281] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0282] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning, characterized by: The method includes: Step S1: for the agent training simulation environment in the heterogeneous patrol swarm game scenario, a Markov decision process is constructed for each agent; wherein the Markov decision process consists of a five-tuple<S,A,P,R,γ> Description, S is the state set, A is the action set, P is the state transition probability, R is the reward function, and γ is the discount factor; Step S2, perform two-dimensional dynamic modeling on the intelligent agent, define the dynamic equations of position vector, velocity vector, tangential acceleration and normal acceleration, and set the motion range, velocity range, acceleration range and obstacle constraints; Step S3, initializing the agent training simulation environment, including initializing the short-term memory module, long-term memory module, memory reasoning module and network parameters; wherein the network parameters include parameters of the policy network, action value network and entropy network, and the action value network is set to a double Q network; Step S4, performing heterogeneous patrol swarm training based on memory and reasoning, includes: Step S4-1, in each round, select an action based on the current state and add noise, and update the short-term memory module after interacting with the environment; Step S4-2: The environmental interaction data is passed through the encoder in the memory reasoning module to generate an embedding vector, which is normalized and retrieved from the long-term memory module. The neighboring memories are matched by cosine distance, and the long-term memory module is updated according to the cumulative reward threshold. Step S4-3, the action value network calculates the target Q value and the current Q value based on the data in the short-term memory module, the data in the long-term memory module, and the plot incentive, and updates the action value network using the weighted loss function; Step S4-4, calculating the loss based on the policy network and the entropy network, and updating the policy network and entropy network parameters; Step S4-5, periodically training the memory reasoning module to optimize the prediction reward and state reconstruction error of the encoder-decoder; Step S5: When the training reaches the preset number of rounds, the multi-agent model is saved.

2. The autonomous collaborative planning method for heterogeneous patrol swarms based on memory reasoning according to claim 1 is characterized in that: Dynamic modeling specifically includes: The velocity magnitude is adjusted by tangential acceleration, the velocity direction is adjusted by normal acceleration, and the rate of change of heading angle is calculated in combination with the rotation vector; Constrain the agent's motion area, speed range, acceleration range, and obstacle avoidance conditions.

3. The autonomous collaborative planning method for heterogeneous patrol swarms based on memory reasoning according to claim 1 is characterized in that: Initializing the agent training simulation environment specifically includes: Establish a short-term memory module to store the quadruple data of recent action states and update it according to the first-in-first-out principle; Build a memory reasoning module, including an encoder and a dual-track decoder. The encoder maps the global state to a low-dimensional embedding vector, and the decoder predicts the highest reward and reconstructs the state. Build a long-term memory module to store six-tuple data of high-value experiences, including embedding vector, normalized embedding vector, highest reward, action, reconstructed state and desirability label; Initialize the parameters of the policy network, action-value network, and entropy network.

4. The method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning according to claim 3 is characterized in that: The encoder is a three-layer fully connected network that uses a self-attention mechanism to capture global state features; the decoder uses a dual-track structure to predict the highest reward and reconstruct the state respectively.

5. The method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning according to claim 1 is characterized in that: When updating the long-term memory module, if the cumulative reward of the current trajectory exceeds the preset threshold, it is marked as a desirable state, and the historical embedding vector is matched through the nearest neighbor algorithm to update the access count and desirable count; otherwise, the current data is stored in the long-term memory module.

6. The method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning according to claim 1 is characterized in that: During the calculation of the target Q value, the episodic incentive is obtained by multiplying the difference between the highest return of the embedding vector in the long-term memory module and the target optimal Q value by the ratio of the visit count to the available count, and then weighted to the target Q value.

7. The method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning according to claim 1 is characterized in that: The loss function of the action value network is the weighted sum of short-term memory loss and long-term memory loss, and the sum of the weight coefficients is 1.

8. The method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning according to claim 1 is characterized in that: The policy network is updated by minimizing the difference between the policy entropy and the minimum Q value; the policy entropy is dynamically adjusted by the entropy adjustment coefficient.

9. The method for autonomous collaborative planning of heterogeneous patrol swarms based on memory reasoning according to claim 1 is characterized in that: The loss function of the memory reasoning module includes the predicted reward error of the predicted reward and the reconstructed state error, and is balanced by the weight of the reconstruction loss.

Images