Multi-machine cooperative detection route planning method based on reinforcement learning
Through the multi-agent method based on reinforcement learning, the VDN algorithm is used to optimize the multi-aircraft route detection planning, which solves the problem of multi-aircraft coordination, achieves efficient route planning and robustness, and improves detection efficiency and coverage.
Patent Information
- Application Number
- CN202411392556.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2024-12-31
AI Technical Summary
The existing multi-machine path optimization method has major problems in the coordination between multiple reconnaissance aircraft, which is easily disturbed by environmental factors, and lacks processing capabilities, is difficult to converge or be unstable when processing large-scale state space data, and is easily trapped in the local area. Optimal solution.
Using a multi-agent method based on reinforcement learning, the overall airspace is modeled as a reinforcement learning environment, and a multi-agent reinforcement learning algorithm with VDN value decomposition is used to treat the action of reinforcement learning through the flight and mode selection of the reconnaissance aircraft, and the reward function is designed. To optimize multi-aircraft route detection planning.
It realizes strongly robust route planning, can reach 95.31% search range coverage, avoids local optimal problems of traditional algorithms, and has real-time state tracking and distributed decision-making capabilities, avoiding the complete centralized method. The 'lazy' agent problem.
Smart Images
Figure CN119225374A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of antenna and microwave technology, and specifically relates to a multi-machine collaborative detection route planning method based on reinforcement learning. Background Art
[0002] Multi-aircraft collaborative detection route planning is one of the key technologies in the field of detection. An effective navigation path helps to improve the joint detection efficiency of multiple reconnaissance aircraft, achieve ideal airspace coverage, and ensure the successful execution of the mission.
[0003] First, most current research focuses on the optimization of a single aircraft path, but lacks the application of actual optimization strategies for multiple aircraft. Second, the existing multi-aircraft path optimization methods have major problems in the coordination between multiple reconnaissance aircraft. The algorithms currently used mainly rely on heuristic search algorithms or traditional A* algorithms for optimization solutions. Most of these algorithms can only solve the path planning problem of a single aircraft and are easily affected by environmental factors. In a multi-aircraft environment, when processing large-scale state space data, they often face the defects of insufficient processing power, difficulty in convergence or instability, and easy to fall into local optimal solutions. Summary of the invention
[0004] In order to overcome the deficiencies in the prior art, the present invention proposes a multi-aircraft collaborative detection route planning method based on reinforcement learning, focusing on airspace security tasks in order to complete the detection task of the designated airspace. This method adopts the method of multi-agent reinforcement learning, models the airspace as a whole as a reinforcement learning environment, and regards the flight and mode selection of the reconnaissance aircraft as reinforcement learning actions. At the same time, the detection range of the reconnaissance aircraft in the mission airspace is modeled as a reinforcement learning reward function. The multi-agent reinforcement learning algorithm of VDN value decomposition is adopted to successfully solve the problem of coordination difficulty in the multi-aircraft route detection planning problem.
[0005] The multi-machine cooperative detection route planning algorithm based on reinforcement learning of the present invention specifically includes the following process:
[0006] (10) Strengthening the learning environment construction
[0007] (11) Airspace discretization: Discretize the airspace into grid points. Different labels are given to the discretized airspace according to the mission area, no-fly zone, etc., so as to judge the area type in the reinforcement learning process. Figure 1 shown.
[0008] (12) Definition of basic environment for reinforcement learning:
[0009] In this method, the airspace is modeled as a two-dimensional coordinate array, and a third dimension is introduced as a method to distinguish between the task area and the danger area, namely:
[0010]
[0011] Among them, m and n are the corresponding two-dimensional coordinates respectively.
[0012] (13) Observation status of reconnaissance aircraft
[0013] In this method, the current position coordinates of the reconnaissance aircraft are used as its observation state, that is:
[0014] The observation state of the i-th reconnaissance aircraft is:
[0015] obs i =(x i , yi)
[0016] where x i ,y i are the horizontal and vertical coordinates of the i-th reconnaissance aircraft, which specifically represent the corresponding two-dimensional array coordinates in the implementation process.
[0017] (14) Movement of reconnaissance aircraft
[0018] The action of the reconnaissance aircraft includes the direction selected by the reconnaissance aircraft in each reinforcement learning process. The direction settings include five options: stay in place, up, down, left, and right. The corresponding numbers are 0, 1, 2, 3, and 4 respectively.
[0019]
[0020] (15) Rewards for reinforcement learning
[0021] In this task, the task goal is the detection efficiency of the task area, so the reward is designed as the detection range of the task area. In order to distinguish between the detected areas and the undetected areas, the detected task areas are marked.
[0022] The marking method is as follows:
[0023]
[0024] Among them, m and n are the two-dimensional coordinates corresponding to the center point of the airspace.
[0025] Use the map environment designed in (12) to mark whether it is searched.
[0026] (20) Model solution
[0027] The multi-agent reinforcement learning solution method based on the VDN algorithm is adopted, and its algorithm process design is as follows Figure 4As shown, it specifically includes the following sub-steps:
[0028] (21) Obtain specific parameters for task information
[0029] Obtain basic mission information, including the scope of the mission area in the airspace, the scope of the no-fly zone, the location of the boundary points, the working mode of the airborne radar and other specific information.
[0030] (22) Spatial discretization processing modeling
[0031] In order to analyze the mission airspace, this method connects the boundary points set by the user to form a polygonal area. Using the airspace discretization method, in this method, the airspace is divided into small grids with longitude and latitude of 0.5°, and the center point of the grid is abstracted to replace the grid and become the mark point to determine whether the grid can be detected by the radar.
[0032] (23) Calculation of visible points in the airspace
[0033] In order to simplify the environment, the reconnaissance range of the reconnaissance aircraft is changed to a circle around the airspace grid where the reconnaissance aircraft is currently located as the detection range of the reconnaissance aircraft. The specific situation is as follows Figure 2 As shown, the yellow circle is the search range of the reconnaissance aircraft.
[0034] (24) Reward function design
[0035] ① In order to make the reconnaissance aircraft find the mission area as quickly as possible, a time penalty is given to the agent, and the penalty becomes higher with each additional step.
[0036] punish_t=-2*Tm
[0037] Wherein, Tm is the time that the current round has been executed.
[0038] ② In order to avoid collisions between multiple reconnaissance aircraft, a higher penalty is given to multiple reconnaissance aircraft when they are at the same location at the same time.
[0039]
[0040] ③ When the reconnaissance aircraft is in a dangerous area or border, a higher penalty is given
[0041]
[0042] ④ In order to allow the reconnaissance aircraft to know the location of the mission area, a path reward function is set. The center point of the mission area airspace is used as the coordinate to calculate the distance between the current reconnaissance aircraft and the center of the mission area. The path reward is designed as
[0043]
[0044]
[0045] Among them, (x t ,y t ) is the coordinate of the center point of the task area, (x i ,y i ) is the coordinate of the i-th reconnaissance aircraft.
[0046] ⑤ When detecting the task area, different rewards are given according to whether the task area has been explored. When the agent detects an unexplored task area, a higher reward is given, and when it explores a task area that has been explored, a lower reward is given.
[0047]
[0048] The final total reward is
[0049] reward=punish_t+punish_u+punish_r+distance_reward+scan_reward.
[0050] (25) Reconnaissance aircraft action selection
[0051] The action of the reconnaissance aircraft is selected according to the greedy strategy. The specific formula is:
[0052]
[0053] Here, rand(1) is a randomly generated random number between 0 and 1, and epsilon is a preset limit value.
[0054] In this method, it is set to a value that decays gradually over time, that is:
[0055]
[0056] Among them, Ts is the number of iterations of the current algorithm.
[0057] (26) Multi-agent reinforcement learning VDN algorithm
[0058] The specific process of iterative optimization of multi-agent reinforcement learning VDN algorithm is as follows: Figure 6 shown.
[0059] ① Environment modeling: determine the state space and action space of the problem (the specific method is shown in (10) reinforcement learning environment construction), as well as the reward mechanism of the environment (the specific reward scheme is shown in (24) reward function design).
[0060] ② Create an agent neural network: Create a recurrent neural network (RNN) for the agent, which accepts the environment state and action as input and outputs the corresponding Q value estimate. The parameters of each agent neural network are shared.
[0061] ③ Create a combined network: Multi-agent reinforcement learning VDN (Value-Decomposition Networks) is a method that uses neural networks to decompose value functions and is used to deal with collaborative decision-making problems in multi-agent systems. In the VDN method, all agents share the same reward value (or team reward). Because all agents share a reward value, only the overall Q value can be obtained for the update of the Q value, so for the Q value of each agent, the overall Q value needs to be decomposed. The decomposition assumption is to decompose the overall Q value into the sum of the Q values of each sub-agent.
[0062] Create a VDN network that can integrate multiple agent neural networks, combine the Q values output by each agent, and get the overall Q value, such as Figure 5 As shown, the individual neural networks are combined.
[0063] The role of the combined neural network is to combine the individual Q values output by each agent to obtain the total Q value.
[0064]
[0065] Among them, d is the total number of agents, s i is the observed state of the ith agent, a i The action chosen for the ith agent.
[0066] In this way, each agent has its own Q value and can make decisions based on its own local value function.
[0067] The update method of this combined network is similar to the DQN method, and the loss function is:
[0068]
[0069] Q target = reward + γ * maxQ total (s t+1 , a),
[0070] Among them, Q target is the target Q value, which is often referred to as a "label" in neural networks. total is the output of Net2(VDN), maxQ(s t+1 , a) is the maximum Q value of the next state, reward is the reward at the current moment, and γ is the weight for balancing the current reward and the reward at the next moment in reinforcement learning.
[0071] ④ Create the corresponding target calculation neural network (Q network) and action selection neural network (action network) (both the agent neural network and the combination network need to be created), where the Q network is used to calculate the target Q value, the action network is used to select actions, and the Q network is updated by periodically copying the action network.
[0072] ⑤ Create experience tuples: Create experience tuples (state, action, reward, next_state) that store the interaction between the agent and the environment, and randomly sample from them for training the Q network and action network.
[0073] ⑥ Observation period: Each agent performs actions according to its own agent neural network using the reconnaissance aircraft action selection function (25), and stores the reconnaissance aircraft observation status, actions, rewards, etc. in the experience tuple. During this period, the neural network is not updated and only the experience is recorded.
[0074] ⑦ Exploration period: Execute actions according to the reconnaissance aircraft action selection function (25). During this period, after two complete cycles, the action selection neural network action network is updated by randomly extracting tuples from the experience tuples, training them using the method in ③, and updating them using the error gradient descent method. After ten cycles, the Q network is updated by directly copying the neural network weights and biases of the action network.
[0075] The beneficial effects of the present invention are
[0076] (1) Robustness: The traditional A* algorithm and heuristic search algorithm need to re-plan the route when the starting point of the reconnaissance aircraft is modified. However, after the model is trained, the reinforcement learning method still has a certain planning ability even if the starting point is modified. At the same time, it avoids the local optimal situation of the traditional algorithm and has strong robustness in handling problems.
[0077] (2) In terms of route planning: This scheme can achieve a search range of up to 95.31% for the designated airspace. At the same time, since reinforcement learning is a sequential decision-making scheme, it can track the status of the reconnaissance aircraft in real time compared to the heuristic search method.
[0078] (3) In terms of execution: Compared with the fully centralized reinforcement learning method, the VDN algorithm has the advantages of centralized training and distributed execution. Each agent has its own decision network and does not have the problem of "lazy" agents as in the fully centralized method (the lazy agent problem refers to the situation that in a decision, two of the three agents score points and one does not score points. In this way, the total reward is also positive, but there are agents that do not participate in the scoring, which is illogical). BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 This is a schematic diagram of the basic airspace scenario.
[0080] Figure 2 This is a schematic diagram of the reconnaissance aircraft detection scene.
[0081] Figure 3 Schematic diagram of multi-aircraft route optimization scenario based on reinforcement learning.
[0082] Figure 4 This is the main flow chart of the multi-aircraft collaborative detection route planning algorithm based on reinforcement learning.
[0083] Figure 5 This is the neural network structure diagram of the VDN algorithm.
[0084] Figure 6 It is the iterative optimization process of the VDN algorithm.
[0085] Figure 7 The verification results of the two-dimensional path map optimization deployment algorithm for multiple reconnaissance aircraft in a simulation scenario. DETAILED DESCRIPTION
[0086] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0087] 1) Scene information
[0088] Design a typical scenario to verify the method proposed in this invention. The simulation scenario is as follows: Figure 3 The scene details are as follows:
[0089] (1) Mission airspace
[0090] Please refer to the table below for the mission airspace boundary points. The altitude layer range of 20,000 meters must be guaranteed and the target RCS is 1.
[0091] Table 1 Airspace range of simulation scenarios
[0092]
[0093] Table 2 Simulation scenario mission airspace range
[0094]
[0095] Table 3 Range of no-fly zones in simulation scenarios
[0096]
[0097] (2) Algorithm parameter setting
[0098] The number of reconnaissance aircraft num_agents = 3, the maximum number of time steps max_timestep = 50, the observation period T_obs = 100, the training period T_train = 5000, the neural network learning rate learning_rate = 0.1, the reinforcement learning parameter gamma = 0.99, the RNN hidden layer network size hidden_size = 64, the action dimension action_size = 5, the agent observation dimension state_size = 2, the map size map = 20*20, the neural network training frequency T_gap = 4*max_timestep, the target network update frequency T_renew = 5*T_gap, the experience tuple size buffer_size = 200, the batch size batch_size = 64, the starting points of the three reconnaissance aircraft are {[1,1], [1,7], [7,1]}, and the initial value of the action selection limit epsilon = 1.
[0099] 2) Verification process
[0100] Step 1: Model construction
[0101] Taking the parameters of the airspace as the environment for reinforcement learning, the current coordinates of the reconnaissance aircraft as the observation state of the reconnaissance aircraft, and the airspace exploration range as the reward function for reinforcement learning, a multi-aircraft route planning model based on reinforcement learning is established. The specific design of the model refers to Section 3 "Specific technical solutions for achieving the purpose of the invention" and will not be repeated here.
[0102] Step 2: Model solution
[0103] According to the solution steps proposed in this patent, the VDN algorithm is used to complete the iterative optimization. The specific algorithm process is as follows: Figure 6 As shown, the pseudo code of the algorithm is shown in the following table:
[0104] Table 4 Pseudo code of iterative optimization algorithm based on VDN algorithm
[0105]
[0106]
[0107] 3) Verify the results
[0108] The iteration process and results are as follows Figure 7 As shown in the table below, the optimal solution is obtained. From the chart analysis, it can be seen that in the optimization results, each UAV has selected a reasonable flight path, which can effectively cover the mission airspace, and the detection range of the designated airspace has reached 91%, while avoiding the danger zone, and the solution is effective.
[0109] The specific planned paths are shown in the following table:
[0110]
[0111]
[0112] The corresponding longitude and latitude coordinates are:
[0113]
[0114]
[0115] The present invention is not limited to the above specific implementations, and the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made to the above implementations based on the technical essence of the present invention shall be included in the protection scope of the present invention.
Claims
1. A multi-machine collaborative detection route planning method based on reinforcement learning, characterized in that: The specific steps include: (10) Strengthening the learning environment construction (11) Discretize the airspace and label the discretized airspace for use in determining the region type during reinforcement learning; (12) The basic environment setting of reinforcement learning is to determine the airspace environment for the execution of the task, which is defined as a two-dimensional coordinate array. The third dimension is introduced as a method to mark the task area and the danger zone, as follows: (13) Observation status of reconnaissance aircraft The current position of the reconnaissance aircraft is taken as its observation state, that is: obs i (x) i ,and i ); where x i ,y i is the horizontal and vertical coordinates of the i-th reconnaissance aircraft, which specifically represents the corresponding two-dimensional array coordinates in the implementation process; (14) Reconnaissance aircraft motion design The reconnaissance aircraft's action design includes five options: stay still, up, down, left, and right; (15) Environmental Reward Design The environmental rewards should be related to the detection range of the reconnaissance aircraft. In order to allow the reconnaissance aircraft to explore more unknown areas, the explored areas are marked as explored. The specific method is as follows: Among them, m and n are the two-dimensional coordinates corresponding to the center point of the airspace; (20) Reinforcement learning model solution (21) Obtaining specific parameters of the mission information, and obtaining the coordinates of the boundary points of the airspace, the boundary points of the mission area, the boundary points of the danger zone, etc. according to the input information; (22) Airspace discretization processing modeling: the boundary points designed by the user are used as the scope of the entire airspace. The airspace discretization method is applied to divide the airspace into multiple grid points with fixed longitude and latitude. The center of the grid point is abstracted as the coordinates of the path point of whether the airspace can be observed and the route planning; (23) Calculation of visible points in airspace. To simplify the environment, the reconnaissance range of the reconnaissance aircraft is changed to a circle around the airspace grid where the reconnaissance aircraft is currently located as the airspace detectable range of the reconnaissance aircraft. (24) The specific design of the reward function is as follows: reward=punish_t+punish_u+punish_r+distance_reward+scan_reward The variables are: punish_t=-2*Tm Tm is the time the current round has been executed; (x t ,y t ) is the coordinate of the center point of the task area, (x i ,y i ) is the coordinate of the i-th reconnaissance aircraft; (25) Action selection function design: Among them, rand(10 is a randomly generated random number between 0 and 1, and Ts is the number of iterations of the current algorithm; (26) VDN algorithm iteration: ①Environment modeling: determine the state space and action space of the problem, as well as the reward mechanism of the environment; ② Create an agent neural network: Create a recurrent neural network of the agent, which accepts the environment state and action as input and outputs the corresponding Q value estimate. Each agent neural network parameter is shared; ③Create a combined network: Create a VDN network that can integrate multiple agent neural networks, combine the Q values output by each agent, get the overall Q value, and combine the various neural networks; The role of the combined neural network is to combine the individual Q values output by each agent to obtain the total Q value; In this way, each agent has its own Q value and can make decisions based on its own local value function; The update method of this combined network is similar to the DQN method, and the loss function is: Q target =reward+γ*maxQ total (s t+1 ,a); Among them, Q target is the target Q value, which is often referred to as a "label" in neural networks. total is the output of Net2 network, maxQ(s t+1 , a) is the maximum Q value of the next state, reward is the reward at the current moment, and γ is the weight for balancing the current reward and the reward at the next moment in reinforcement learning; ④ Create the corresponding target calculation neural network and action selection neural network, where the Q network is used to calculate the target Q value, the action network is used to select the action, and the Q network is updated by regularly copying the action network; ⑤ Create experience tuples: By storing the experience tuples (state, action, reward, next_state) of the agent’s interaction with the environment, and randomly sampling from them for training the Q network and action network; ⑥Observation period: Each agent performs actions according to the action selection function. During this period, the neural network is not updated and only the experience is recorded; ⑦ Exploration period: Execute actions according to the agent neural network. During this period, after two complete cycles, the action selection neural network action network is updated by randomly extracting tuples from the experience tuples for training and updating them using the error gradient descent method. After ten cycles, the Q network is updated by directly copying the neural network weights and biases of the action network.