An Adaptive Intelligent Path Planning Method for Complex Adversarial Environments
By employing an unmanned swarm collaboration method and dynamic difference calculation, and utilizing a deep deterministic policy gradient algorithm for local path planning, the problems of low accuracy and low intelligence in local path planning in complex adversarial environments are solved, achieving higher adaptability and planning accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA ACAD OF LAUNCH VEHICLE TECH
- Filing Date
- 2023-11-07
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, local path planning suffers from low accuracy, low intelligence, and weak adaptability in complex adversarial environments. It is difficult to effectively adapt between simulation and real-world domains, and decision-making errors are particularly prevalent in unmanned swarms.
An unmanned swarm collaborative approach is adopted, which improves the adaptability and planning accuracy of the simulation domain and the real domain by using global path planning, simulation domain reinforcement learning training and local path replanning, and deep deterministic policy gradient algorithm and dynamic difference calculation.
It significantly improves the accuracy and intelligence level of local path planning, enhances the adaptability of unmanned swarms in complex environments, and solves the problems of low accuracy and weak adaptability in existing technologies.
Smart Images
Figure CN117519284B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent collaboration, and in particular to an adaptive intelligent path planning method for complex adversarial environments. Background Technology
[0002] Research on path planning problems can be divided into two parts: global path planning research and local path planning research. In global path planning research, the solution of optimization problems is relatively mature. However, in local path planning research, the use of reinforcement learning methods to replace optimization problems has become a trend. The research on local path planning using reinforcement learning methods mainly involves two parts: training models in the simulation domain and deploying algorithms in the real domain. For these two parts of research, there are problems such as low model domain adaptability, low accuracy, and low intelligence, making it difficult to use them for practical deployment and operation in the real domain.
[0003] Current research focuses on improving the domain adaptation capabilities of reinforcement learning. Therefore, it is necessary to consider the dynamic differences between the simulation domain and the real domain. At the same time, in local path planning, individual aircraft are prone to decision-making errors, resulting in low accuracy and low intelligence in local path planning. Summary of the Invention
[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose an adaptive intelligent path planning method for complex adversarial environments. By fully considering the dynamic differences between the simulation domain and the real domain, it ensures that each aircraft can perform local path replanning based on the dynamic differences. At the same time, it utilizes the collaborative capabilities of the unmanned swarm to perform collaborative correction of local path planning. This solves the problems of low accuracy, low intelligence, and weak adaptability of local path planning in the architecture of unmanned swarms that is based on "global path planning and centered on local path planning".
[0005] The technical solution of this invention is:
[0006] An adaptive intelligent path planning method for complex adversarial environments includes:
[0007] Perform global path planning for aircraft in an unmanned swarm to find the optimal path;
[0008] A simulation scenario containing obstacles is constructed in the simulation domain and reinforcement learning training is performed. The aircraft in the unmanned swarm make dynamic decisions based on their respective states locally to complete local path planning.
[0009] Based on dynamic environmental variables, the dynamic difference between the simulation domain and the real domain is calculated, and local path replanning is performed for each aircraft based on the optimal path.
[0010] Based on the dynamic differences among the aircraft in the unmanned swarm, the decision weight value of each aircraft is calculated. The mean value of the action state parameters of the local path planning of each aircraft in the unmanned swarm is then calculated based on the decision weight value. The mean value of the action state parameters is then used to perform collaborative correction of the local path planning of each aircraft in the unmanned swarm.
[0011] The A* algorithm is used to perform global path planning for each of the n unmanned aircraft in the swarm, and the optimal path is initially found.
[0012] Reinforcement learning training is performed using a deep deterministic policy gradient algorithm. In this unmanned swarm, aircraft dynamically make decisions based on their individual states to complete local path planning, including:
[0013] The Actor network selects actions based on the laser_state of the current local position of aircraft i; the Critic network evaluates the state and calculates the reward value of the next state laser_state; the reward value r of the laser_state is... i The calculation formula is:
[0014]
[0015] Where v i Let ω be the linear velocity value in the decision-making action state parameters of aircraft i in the unmanned swarm. i Let θ be the angular velocity value in the decision-making action state parameters of aircraft i in the unmanned swarm. i Let r be the pitch angle value in the decision-making action state parameters of aircraft i in the unmanned swarm. i ε is the maximum reward value in the decision-making action state parameters of aircraft i in the unmanned swarm, and ε is the minimum threshold of radar detection data.
[0016] The parameters of the Critic network are updated using the reward value of state 'laser_state' and the policy gradient method, while the parameters of the Actor network are also updated.
[0017] The updated Actor network outputs the action state a. i =[v i ,ω i ,θ i ,r i Then, execute the next action to perform local path planning.
[0018] The process involves calculating the dynamic difference between the simulation domain and the real domain based on dynamic environmental variables, and performing local path replanning for each aircraft based on the optimal path, including:
[0019] Calculate the dynamic differences between the three dynamic environmental variables in the simulation domain and the real domain;
[0020] Based on the dynamic differences of three dynamic environmental variables in the simulation and real domains, the dynamic difference value DIFF between the simulation and real domains for aircraft i in the unmanned swarm is calculated. i ;
[0021] Based on the dynamic difference value DIFF between the simulation domain and the real domain of aircraft i in the unmanned swarm i For aircraft i, a local path replanning is performed based on the optimal path.
[0022] The calculation of the dynamic differences between the three dynamic environmental variables in the simulation domain and the real domain is as follows:
[0023]
[0024]
[0025]
[0026] Where diff n,i Let n be the dynamic difference value of the radar detection characteristics of aircraft i in the real domain and the simulation domain. i Let n′ be the radar detection characteristics of aircraft i in the real domain. i For the radar detection characteristics of aircraft i in the simulation domain, diff s,i Let s be the dynamic difference in sensor sensitivity between the real and simulated domains for aircraft i. i Let s′ be the sensor sensitivity of aircraft i in the real domain. i For the sensor sensitivity of aircraft i in the simulation domain, diff b,i Let b be the dynamic difference value of the position calibration deviation vector of aircraft i in the real domain and the simulation domain. i To calibrate the position deviation vector of aircraft i in the real domain, b′ i The position calibration deviation vector for aircraft i in the simulation domain.
[0027] The dynamic difference value DIFF between the simulation domain and the real domain for aircraft i in the unmanned swarm is calculated based on the dynamic difference values of three dynamic environmental variables in the simulation domain and the real domain. i Specifically, it is expressed as:
[0028] DIFF i =1-exp(-(diff) n,i +diff s,i +diff b,i )).
[0029] The dynamic difference value DIFF between the simulation domain and the real domain of aircraft i in the unmanned swarm is used. i The aircraft i performs local path replanning to obtain a new motion state a′. i =[v′ i ,ω′ i ,θ′ i ,r′ i '], where v′ i =(1-DIFF) i )×v i , ω′ i =(1-DIFF) i )×ω i , θ′ i =(1-DIFF) i )×θ i ,r′ i =(1-DIFF) i )×r i .
[0030] Based on the dynamic difference value DIFF of each aircraft in the unmanned swarm i The decision weight value W for each aircraft was calculated. i :
[0031]
[0032] Calculate the mean motion state parameters of the local path planning for n unmanned aircraft in an unmanned swarm based on the decision weight values:
[0033]
[0034] in The mean linear velocity is the state parameter of decision-making actions for all aircraft in the unmanned swarm. The angular velocity is the average value among the decision-making action state parameters of all aircraft in the unmanned swarm. The pitch angle is the average value among the state parameters of decision-making actions of all aircraft in the unmanned swarm.
[0035] The mean value of this action state parameter is used to perform collaborative correction of the local path planning of each aircraft in the unmanned swarm, including:
[0036]
[0037] Where ε v ε ω and ε θ These are the set linear velocity, angular velocity, and pitch angle thresholds, respectively. i "ω represents the linear velocity of the aircraft after coordinated correction." i "θ represents the angular velocity of the aircraft i after collaborative correction."i "This is the pitch angle of the aircraft after collaborative correction."
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] This invention takes into account the dynamic differences between the simulation domain and the real domain under complex adversarial environments, significantly improving the domain adaptability and the accuracy of intelligent path planning. It solves the problems of low accuracy, low intelligence, and weak adaptability of local path planning in the existing architecture of unmanned swarms that is based on "global path planning and centered on local path planning". Attached Figure Description
[0040] Figure 1 A flowchart of an adaptive intelligent path planning method for complex adversarial environments;
[0041] Figure 2 Flowchart of a local path replanning method for a single aircraft;
[0042] Figure 3 A flowchart of collaborative correction of local path planning in unmanned clusters. Detailed Implementation
[0043] The present invention addresses the intelligent path planning method for aircraft. The current challenge is the dynamic difference between the simulation domain and the real domain, which leads to problems such as low accuracy and weak adaptability of the real domain model deployment.
[0044] like Figure 1 As shown, this invention proposes an adaptive intelligent path planning method for complex adversarial environments, comprising:
[0045] The first step involves each of the n unmanned aircraft in the swarm using the A* algorithm for global path planning, selecting the optimal node from the global pool for expansion, and initially finding the best path.
[0046] The second step involves constructing a scenario in the simulation domain and training a reinforcement learning model using the Deep Deterministic Policy Gradient (DDPG) algorithm. In this unmanned swarm, the aircraft make dynamic decisions based on their respective states locally to complete local path planning.
[0047] The third step is to consider the dynamic differences between the simulation domain and the real domain, calculate the dynamic difference values between the simulation domain and the real domain based on the dynamic environmental variables, and then perform local path replanning for a single aircraft.
[0048] The fourth step involves considering the probability of errors in the local path replanning decision of a single aircraft. Based on the dynamic differences between the aircraft in the unmanned swarm and the maximum reward value of the decision action, the decision weight value of each aircraft is calculated, and the local path planning of each aircraft in the unmanned swarm is collaboratively corrected.
[0049] The following is a detailed introduction:
[0050] In the first step, the process of global path planning for each of the n unmanned aircraft in the swarm using the A* algorithm is as follows:
[0051] Step 1: When the aircraft uses the A* algorithm for global path planning, it first clarifies the starting node and the target node, initializes the cost and heuristic estimate of the starting node, and sets the starting node as the current node;
[0052] Step 2: Iteratively search from the priority queue to select the node with the lowest cost as the current node. Check if the current node is the target node. If it is, the search ends and the best path has been found. If the current node is not the target node, expand the current node and generate its neighboring nodes, i.e., successor nodes. For each successor node, calculate its path cost and heuristic estimate and add it to the priority queue.
[0053] In the second step, the Deep Deterministic Policy Gradient (DDPG) algorithm is used, and the specific decision-making process includes:
[0054] Step 1: The Actor network selects the action output based on the current local position of aircraft i, and the Critic network evaluates the state and calculates the reward value for the next possible local state.
[0055] Step 2: Use the policy gradient method to update the parameters of the Critic network to maximize the expected cumulative reward value, and at the same time update the parameters of the Actor network;
[0056] Step 3: Perform local path planning by maximizing the reward value to execute the next action, resulting in the action state a. i =[v i ,ω i ,θ i ,r i ];
[0057] Where v i Let ω be the linear velocity value in the decision-making action state parameters of aircraft i in the unmanned swarm. i Let θ be the angular velocity value in the decision-making action state parameters of aircraft i in the unmanned swarm. i Let r be the pitch angle value in the decision-making action state parameters of aircraft i in the unmanned swarm. i In an unmanned swarm, the decision-making action state parameters of aircraft i are used to maximize the reward value of the action.
[0058] In the third step described above, such as Figure 2As shown, considering the dynamic differences between the simulation domain and the real domain, based on dynamic environmental variables such as radar detection characteristics, sensor sensitivity, and position calibration deviation vector, the dynamic differences between the simulation domain and the real domain for three dynamic environmental variables are calculated, specifically expressed as follows:
[0059]
[0060]
[0061]
[0062] Where diff n,i Let n be the dynamic difference value of the radar detection characteristics of aircraft i in the real domain and the simulation domain. i Let n′ be the radar detection characteristics of aircraft i in the real domain. i For the radar detection characteristics of aircraft i in the simulation domain, diff s,i Let s be the dynamic difference in sensor sensitivity between the real and simulated domains for aircraft i. i Let s′ be the sensor sensitivity of aircraft i in the real domain. i For the sensor sensitivity of aircraft i in the simulation domain, diff b,i Let b be the dynamic difference value of the position calibration deviation vector of aircraft i in the real domain and the simulation domain. i To calibrate the position deviation vector of aircraft i in the real domain, b′ i The position calibration deviation vector of aircraft i in the simulation domain is given, and the length of the position calibration deviation vector in the aircraft is 3.
[0063] Then, based on the dynamic differences of the three dynamic environmental variables in the simulation domain and the real domain, the dynamic difference value DIFF between the simulation domain and the real domain for aircraft i in the unmanned swarm is calculated. i Specifically, it is expressed as:
[0064] DIFF i =1-exp(-(diff) n,i +diff s,i +diff b,i ))
[0065] Then, based on the dynamic difference value DIFF between the simulation domain and the real domain of aircraft i in the unmanned swarm, i The aircraft i performs local path replanning to obtain a new motion state a′. i =[v′ i ,ω′ i ,θ′ i ,r′ i The specific representation method is as follows:
[0066] v′i =(1-DIFF) i )×v i ,
[0067] ω′ i =(1-DIFF) i )×ω i ,
[0068] θ′ i =(1-DIFF) i )×θ i ,
[0069] r i '=(1-DIFF i )×r i ,
[0070] In the fourth step described above, such as Figure 3 As shown, the dynamic difference value DIFF of each aircraft is... i Maximizing the reward value r of the action state i Calculate the decision weight value W for each aircraft. i Specifically, it is expressed as follows:
[0071]
[0072] The greater the maximum reward value of the action made by aircraft i in the decision network, and the smaller the dynamic difference value of aircraft i, the greater its decision weight value.
[0073] Then, based on the decision weight values, the mean values of the action state parameters for the local path planning of the n unmanned aircraft in the swarm are calculated, as shown below:
[0074]
[0075] in The mean linear velocity is the state parameter of decision-making actions for all aircraft in the unmanned swarm. The angular velocity is the average value among the decision-making action state parameters of all aircraft in the unmanned swarm. The pitch angle is the average value among the state parameters of decision-making actions of all aircraft in the unmanned swarm.
[0076] like Figure 3 As shown, the decision-making action state parameters of each aircraft in the unmanned swarm are then collaboratively corrected based on the average value of the decision-making action state parameters, as specifically shown below:
[0077]
[0078] Where ε v ε ω and ε θThese are the set linear velocity, angular velocity, and pitch angle thresholds, respectively. i "ω represents the linear velocity of the aircraft after coordinated correction." i "θ represents the angular velocity of the aircraft i after collaborative correction." i "The pitch angle of aircraft i after collaborative correction is determined by comparing the difference between the decision action state parameters and the mean value to decide whether to perform collaborative correction of the decision action state parameters."
[0079] The contents not described in detail in this specification are common knowledge to those skilled in the art.
Claims
1. An adaptive intelligent path planning method for complex adversarial environments, characterized in that... include: Perform global path planning for aircraft in an unmanned swarm to find the optimal path; A simulation scenario containing obstacles is constructed in the simulation domain and reinforcement learning training is performed. The aircraft in the unmanned swarm make dynamic decisions based on their respective states locally to complete local path planning. Based on dynamic environmental variables, the dynamic difference between the simulation domain and the real domain is calculated, and local path replanning is performed for each aircraft based on the optimal path. Based on the dynamic differences of each aircraft in the unmanned swarm, the decision weight value of each aircraft is calculated. Based on the decision weight value, the mean value of the action state parameters of the local path planning of each aircraft in the unmanned swarm is calculated. The mean value of the action state parameters is used to perform collaborative correction of the local path planning of each aircraft in the unmanned swarm. The process involves calculating the dynamic difference between the simulation domain and the real domain based on dynamic environmental variables, and performing local path replanning for each aircraft based on the optimal path, including: Calculate the dynamic differences between the three dynamic environmental variables in the simulation domain and the real domain; Based on the dynamic differences of three dynamic environmental variables in the simulation domain and the real domain, calculate the dynamic differences between the simulation domain and the real domain for aircraft i in the unmanned swarm. ; Based on the dynamic difference between the simulation domain and the real domain of aircraft i in the unmanned swarm For aircraft i, a local path replanning is performed based on the optimal path. The calculation of the dynamic differences between the three dynamic environmental variables in the simulation domain and the real domain is as follows: in Let be the dynamic difference value of the radar detection characteristics of aircraft i in the real domain and the simulation domain. For the radar detection characteristics of aircraft i in the real domain, For the radar detection characteristics of aircraft i in the simulation domain, Let be the dynamic difference in sensor sensitivity between the real and simulated domains for aircraft i. The sensor sensitivity of aircraft i in the real domain. Let i be the sensor sensitivity of aircraft i in the simulation domain. Let be the dynamic difference value of the position calibration deviation vector of aircraft i in the real domain and the simulation domain. The position deviation vector of aircraft i in the real domain is calibrated. The position calibration deviation vector for aircraft i in the simulation domain is calculated. The dynamic difference between the simulation domain and the real domain of aircraft i in the unmanned swarm is calculated based on the dynamic difference values of three dynamic environmental variables in the simulation domain and the real domain. Specifically, it is expressed as: 。 2. The adaptive intelligent path planning method for complex adversarial environments according to claim 1, characterized in that: For each of the n unmanned aircraft in the swarm, use A The algorithm performs global path planning and initially finds the optimal path.
3. The adaptive intelligent path planning method for complex adversarial environments according to claim 2, characterized in that: Reinforcement learning training is performed using a deep deterministic policy gradient algorithm. In this unmanned swarm, aircraft dynamically make decisions based on their individual states to complete local path planning, including: The Actor network considers the state of aircraft i at its current local position. Output action selection; the Critic network performs state evaluation and calculates the next state. Reward value; status Reward value The calculation formula is: in Let the linear velocity value be the state parameter of the decision-making action of aircraft i in the unmanned swarm. Let be the angular velocity value in the decision-making action state parameters of aircraft i in the unmanned swarm. Let the pitch angle value be one of the decision-making action state parameters for aircraft i in the unmanned swarm. Let i be the aircraft in the unmanned swarm whose decision-making action state parameters maximize the reward value. This is the minimum threshold for radar detection data; Adopting status The reward value and policy gradient method are used to update the parameters of the Critic network, and at the same time, the parameters of the Actor network are also updated. The updated Actor network outputs the action state. Then, execute the next action to perform local path planning.
4. The adaptive intelligent path planning method for complex adversarial environments according to claim 3, characterized in that: The dynamic difference between the simulation domain and the real domain of aircraft i in the unmanned swarm is used as a basis. The aircraft i performs local path replanning to obtain a new motion state. ,in , , , .
5. The adaptive intelligent path planning method for complex adversarial environments according to claim 4, characterized in that: Based on the dynamic differences among the various aircraft in the unmanned swarm The decision weight values of each aircraft were calculated. : 。 6. The adaptive intelligent path planning method for complex adversarial environments according to claim 5, characterized in that: Calculate the mean motion state parameters of the local path planning for the n unmanned aircraft swarm based on the decision weight values: ， ， in The mean linear velocity is the state parameter of decision-making actions for all aircraft in the unmanned swarm. The angular velocity is the average value among the decision-making action state parameters of all aircraft in the unmanned swarm. The pitch angle is the average value among the state parameters of decision-making actions of all aircraft in the unmanned swarm.
7. The adaptive intelligent path planning method for complex adversarial environments according to claim 6, characterized in that: The mean value of the action state parameters is used to perform collaborative correction of the local path planning of each aircraft in the unmanned swarm, including: ， ， in , and These are the set threshold values for linear velocity, angular velocity, and pitch angle, respectively. For the coordinated correction of the aircraft's linear velocity, For the coordinated correction of the aircraft's i-angular velocity, The pitch angle of the aircraft after collaborative correction.