Path planning method based on double Q-Learning

Abstract

The invention discloses a path planning method based on double Q-Learning, and the method comprises the following steps: constructing a probability graph of an unknown moving target to the ground: carrying out the probability estimation of the escaping direction of the unknown moving target through introducing ground meshing, a passability region and a random point mode, and estimating the probability graph of the moving target as an observation space of reinforcement learning; and reconnaissance is carried out based on reinforcement learning: by introducing space gridding, motion models of the unmanned aerial vehicle, the target to be identified and the threat target are established, and a path planning method based on double-Q-Learning coverage search is constructed, so that reconnaissance of the unknown moving target to the ground is realized. According to the method, a consensus voting mechanism is fused, and distributed path planning is realized. Robot cooperation and form verification are combined, a logic integral formula provides quantitative constraints, and especially in multi-unmanned aerial vehicle cooperative reconnaissance, the decision stability in an uncertain environment is improved.

Description

Technical Field

[0001] This invention belongs to the field of adversarial simulation technology, and particularly relates to a path planning method based on dual Q-Learning. Background Technology

[0002] In scenarios where multiple UAVs collaborate to complete reconnaissance missions, the inherent uncertainties of the reconnaissance environment lead to a lack of decision-making stability. To accomplish reconnaissance missions, it is necessary to leverage existing simulation technologies and training resources to construct networked, systematic, and intelligent simulation training equipment and systems. This involves conducting system-on-system confrontation simulation training, establishing motion models for UAVs, targets to be identified, and threat targets, developing path planning methods for multiple UAVs, and ultimately completing reconnaissance missions against unknown moving targets on the ground.

[0003] In the prior art, Chinese patent application CN120010549A proposes a collaborative reconnaissance and electronic countermeasures control method based on multiple UAVs. This method refines electronic countermeasures tasks into reconnaissance and jamming sub-tasks according to task type and adjusts priorities in real time based on the environment. The reinforcement learning decision module uses DQN and PPO deep reinforcement learning algorithms to optimize reconnaissance and jamming strategies, ensuring optimal task execution. Through multi-agent reinforcement learning and bee swarm algorithms, it coordinates UAV task allocation and flight paths to resolve task conflict issues. This method implements different reconnaissance and jamming strategies according to task type, improving task execution efficiency. However, in real-world reconnaissance environments, UAVs, targets to be identified, and threat targets are all in dynamic motion, requiring the establishment of models for UAVs, targets to be identified, and threat targets, as well as path planning for multiple UAVs. Summary of the Invention

[0004] This invention discloses a path planning method based on dual Q-Learning, combining robot collaboration with formal verification. Logical integral formulas provide quantitative constraints, particularly improving decision-making stability in uncertain environments during multi-UAV collaborative reconnaissance. By introducing spatial gridding, motion models of UAVs, the target to be identified, and the threat target are established, and a path planning method based on dual Q-Learning coverage search is constructed to achieve reconnaissance of unknown moving targets on the ground.

[0005] Specifically, to achieve the above objectives, the path planning method based on dual Q-Learning disclosed in this application includes the following steps: Constructing a probability map of unknown moving targets on the ground: By introducing ground gridding, traversable areas, and random points, the probability of the escape direction of unknown moving targets is estimated. The probability map of the moving targets is used as the observation space for reinforcement learning. Reinforcement learning-based target reconnaissance: By introducing spatial gridding, motion models of UAVs, targets to be identified, and threat targets are established, and a path planning method based on dual Q-Learning coverage search is constructed to achieve reconnaissance of unknown moving targets on the ground.

[0006] Furthermore, a probability map of the target's location is constructed, where a darker color indicates that the target is more likely to be located at that position. The specific steps are as follows: (1) Step 1: Initialize the coordinates of the starting point S0 of the target to be measured as follows: The area where the target may move is divided into hexagonal grids, with the radius of the hexagonal grid being the width of the target to be identified. (2) Step 2: Mark the areas where movement is possible as traversable areas and impassable areas; (3) Step 3: Randomly mark n points in the passable area, and denote these n points as D(d1,...,d2). i ,…,d n (as the endpoint) (4) Step 4: Using a parallel approach, obtain the path from the starting point S0 to any endpoint D(d1,…,d) according to the path planning A* algorithm. i ,…,d n If a path is given, then the set of points traversed by each path is denoted as SD. i (sd1,…,sd i ,…,sd m ), where m is the total number of points visited in the path; (5) Step 5: Statistically analyze these sets. The darker the color, the more likely the target to be identified is in that location. Use 0 to 255 to describe it. "0" means impassable and "255" means the target is most likely to be in this area. Use matrix A to represent the target location probability map. Use B to represent the corresponding point coordinates.

[0007] Furthermore, a probability map of the target location to be identified is constructed, and the specific steps are as follows: Step 1: Initialize the starting coordinates of the target to be measured , The x and y coordinates of the starting point are used to divide the area where the target may move into a hexagonal grid, with the grid radius equal to the target width. Meanwhile, using knowledge graph methods, regional nodes are labeled as traversable and intraversable, and graph edges represent traversal costs. Step 2: Model the target motion as a nonlinear diffusion process, starting from the starting point. Diffusion begins, probability density over time Evolution; Diffusion damping coefficient of impassable regions Passable area ; Step 3: In the passable area, randomly sample using the knowledge graph. Potential endpoints For each destination, the set of basic paths is computed using the parallel A* algorithm. ; Step 4: Calculate the probability graph and probability matrix. Each grid point The value is calculated quantitatively using the following formula: The above formula realizes probabilistic evolution prediction from static to dynamic, where, It is a location In time The probability density; It is time; It is an integral variable; It is the update interval. Denotes the divergence operator, It is attention weight; It is a position-dependent diffusion coefficient; It is an attention mechanism that calculates the attention given to the current point by the set of paths; The calculation is as follows: in yes The query vector, It is the path point of and the A key vector, It is a dimension; For traversable areas, the location-dependent diffusion coefficient The calculation is as follows: It's about cost, obtained from knowledge graphs. It is a cost decay parameter; Step 5: Normalized to 0~255, where 0 represents impassable and 255 represents the highest probability of being passable. Corresponding coordinate matrix Each element in the graph; the probability graph is dynamically updated every time step. Re-diffusion, incorporating real-time observation data for correction. .

[0008] Furthermore, the motion models of the drone, the target to be identified, and the threat target are established. The specific steps are as follows: (1) Divide the area where the UAV flies into a hexagonal grid. The radius of the hexagonal grid is the radius of the area where the UAV flies. The minimum distance traveled in a given time; (2) Simplify the drone into a point mass, its The coordinates of the time are The velocity angle is The speed of the drone is , angle is There are six directions, corresponding to 0°, 60°, 120°, 180°, 240°, and 360° respectively. This represents the radius of the drone's ground search. The distance between the drone and the target point to be identified. For a weighted average, Where A is the probability map of the target to be tested, and B is the corresponding coordinate value in the probability map. These are weighting coefficients; (3) The distance between the drone and the target point of the threat; (4) The continuous trajectory of the UAV can be modeled as a series of continuous discrete points. Let the position of the UAV at the next moment be... The velocity angle is The time interval of the movement is Then the drone is The state at that moment is: in, , The angle of the drone changes, assuming the threat target is stationary.

[0009] Furthermore, for the problem of reconnaissance of unknown moving targets on Earth based on reinforcement learning, a joint space is established, including the agent's observation space, action space, and reward function; 1) Observation space: The observation space of the agent is represented by a tuple ( , , This indicates that the observed values ​​are normalized within a certain range; 2) Action Space: The action space of the agent is represented by a tuple. express; 3) Reward Function: The base reward is the negative of the distance to the nearest target point; entering the threat zone results in an additional deduction. The reward function is as follows: T0 is a preset threshold; 4) Reinforcement learning: A path planning method based on double Q-Learning coverage search is adopted, and the task is terminated based on whether the search is completed.

[0010] Furthermore, the steps of the dual-Q-Learning path planning method are as follows: The observation space consists of multi-agent observation tuples. in It is a sequential logic state, indicating that "all high-probability points will eventually be covered". This is the current location; It is a distance term. It is the distance between the object's current position and the agent. It is the distance between the object's current position and the target. It represents the distance between the object's current position and the obstacle; the observations are normalized to the range [-1, 1] and incorporated into the probability graph. As a global feature; The actions of drones form a continuous spatial tuple , It's the drone's movement, speed. , It is the maximum speed; angle , It is a consensus signal, that is, a collaborative instruction between multiple agents, ranging from 0 to 1, indicating the degree of uniformity of actions among multiple agents; The reward function is as follows: in, It is the minimum distance, representing the distance to the nearest target point; It is a threat to the area of ​​punishment. It is a sequential logic function. Indicates coverage, Indicating conflict, This represents logical AND. Indicates logical NOT; This indicates a "final" operator; It is a path trajectory; It is logical weight; It refers to the task duration; It is an indicator function, which is 1 when the area is within the threat zone; Reinforcement learning algorithm: Each agent maintains two Q-tables During updates, actions are used alternately and selected through consensus voting: in , It is the number of intelligent agents. It is the vote of the m-th agent, obtained based on the local Q value; It is the learning rate; It is a discount factor. This is the drone's next move. This is the state of the drone at the next moment; the mission ends when the coverage of all high-probability points is > 95% or the timing logic is satisfied. Attached Figure Description

[0011] Figure 1 is a probability map of the location of the target to be identified; Figure 2 shows the motion model of the drone, the target to be identified, and the threat target; Figure 3 shows the path planning based on dual Q-Learning coverage search. Detailed Implementation

[0012] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structure, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0013] like Figure 1 The image shown is a probability map of the location of the target to be identified. The darker the color, the more likely the target is to be located at that location. The specific steps are as follows: (1) Step 1: Initialize the coordinates of the starting point S0 of the target to be measured as follows: The area where the target may move is divided into hexagonal grids, with the radius of the hexagonal grid being the width of the target to be identified. (2) Step 2: Mark the possible movement areas as passable areas and impassable areas. Passable areas refer to roads, bridges, lawns, etc., while impassable areas refer to buildings, rivers, forests, etc. (3) Step 3: Randomly mark n points in the passable area, and denote these n points as D(d1,...,d2). i ,…,d n (as the endpoint) (4) Step 4: Using a parallel approach, obtain the path from the starting point S0 to any endpoint D(d1,…,d) according to the path planning A* algorithm. i ,…,d n If a path is given, then the set of points traversed by each path is denoted as SD. i (sd1,…,sd i ,…,sd m ); (5) Step 5: Statistically analyze these sets. The darker the color, the more likely the target to be identified is in that location. Use 0 to 255 to describe it. "0" means impassable and "255" means the target is very likely to be in this area. Use matrix A to represent the target location probability map and use B to represent the corresponding coordinates.

[0014] like Figure 1 The numbers (1), (2), (3), (4), and (5) shown in the probability map of the location of the target to be identified correspond to steps 1 to 5, respectively.

[0015] like Figure 2 The diagram shows the motion model of the drone, the target to be identified, and the threat target. The specific steps are as follows: (1) Divide the area where the UAV flies into a hexagonal grid. The radius of the hexagonal grid is the radius of the area where the UAV flies. The minimum distance traveled in a given time; (2) Simplify the drone into a point mass and consider it as an intelligent agent. The coordinates of the time are The velocity angle is The speed of the drone is The angle is There are six directions, corresponding to 0°, 60°, 120°, 180°, 240°, and 360° respectively. This represents the radius of the drone's ground search. The distance between the drone and the target point to be identified. For a weighted average, Where A is the probability map of the target object, and B is the corresponding coordinate value in the probability map. These are the weighting coefficients.

[0016] (3) This refers to the distance between the drone and the target location.

[0017] (4) The continuous trajectory of the UAV can be modeled as a series of continuous discrete points. Let the position of the UAV at the next moment be... The velocity angle is The time interval of the movement is Then the drone is The state at that moment is: in: , The angle of the drone changes, assuming the threat target is stationary.

[0018] like Figure 3The diagram illustrates path planning based on dual Q-Learning coverage search. In reinforcement learning problems, Markov decision models are the most widely used environment models. For reinforcement learning-based ground target reconnaissance of unknown moving targets, a joint space needs to be established, including the agent's observation space, action space, and reward function.

[0019] 1) Observation Space: Considering the scenario where the agent can only observe local environmental information, the agent's observation space stores local environmental observation information. The agent receives an observation as input, including its current position, distance to other agents, distance to the target, and distance to obstacles. Therefore, the agent's observation space is represented by a tuple (...). , , This indicates that the observations are continuous, therefore they are normalized to a certain range.

[0020] 2) Action Space: The actions of a drone agent are determined by its velocity and velocity angle. The agent can choose a velocity between 0 and its maximum velocity, and an angle. Therefore, the agent's action space is represented by a tuple. express.

[0021] 3) Reward Function: Rewards play a crucial role in guiding agents to learn effective navigation strategies. The reward is calculated based on the agent's position and its distances from the target, obstacles, and other agents. The base reward is a negative number of the distance to the nearest target point; entering a threat zone results in an additional deduction. The reward function is as follows: 4) Reinforcement learning: This invention adopts a path planning method based on dual Q-Learning coverage search, and determines whether to end the task based on whether the search is completed.

[0022] In one embodiment, a diffusion model and the target location of a random walk are used to construct a dynamic probability graph. The specific steps are as follows: Step 1: Initialize the starting coordinates of the target to be measured The area where the target may move is divided into hexagonal grids with a grid radius equal to the target width. Meanwhile, using knowledge graph methods, regional nodes are labeled as traversable and intraversable, with graph edges representing travel costs. Traversable nodes include roads, bridges, etc., while intraversable nodes include buildings, rivers, etc.

[0023] Step 2: Drawing inspiration from diffusion models in computer vision, simulate the random walk process of the target. Model the target motion as a nonlinear diffusion process, starting from the initial point... Diffusion begins, probability density over time Evolution; Setting diffusion damping coefficient for impassable areas Passable area .

[0024] Step 3: In the passable area, randomly sample using the knowledge graph. Potential endpoints For each destination, the set of basic paths is computed using the parallel A* algorithm. However, by introducing diffusion weighting, path probabilities are not just statistical frequencies, but dynamic densities calculated through diffusion equations.

[0025] Step 4: Introduce the "diffusion-attention fusion equation" to calculate the probability map and probability matrix. Each grid point The value is calculated quantitatively using the following formula: The above equation integrates physical diffusion with visual attention, achieving probabilistic evolution prediction from static to dynamic. In the above equation, It is a location In time The probability density satisfies the initial conditions. ; It is time; It is an integral variable; It is the update interval. Represents the divergence operator, simulating physical diffusion. It is the attention weight, and its value ranges from 0 to 1; It is a position-dependent diffusion coefficient; It is an attention mechanism that calculates the attention given to the current point by the set of paths.

[0026] The calculation is as follows: in yes The query vector, It is a path point key vector. It is a dimension.

[0027] For traversable areas, the location-dependent diffusion coefficient The calculation is as follows: It's about cost, obtained from knowledge graphs. This is the cost attenuation parameter, with a default value of 0.5.

[0028] Step 5: Normalized to 0~255, where 0 represents impassable and 255 represents the highest probability of being passable, corresponding to the coordinate matrix. The probability graph is updated dynamically, every time step. Re-diffusion, incorporating real-time observation data such as drone imagery, to correct... .

[0029] This embodiment integrates nonlinear diffusion equations and attention mechanisms to achieve dynamic temporal evolution of the probability map; it simulates the physical process of target "diffusion escape". In complex land environments, the probability map transforms from static to predicting future positions, improving reconnaissance accuracy.

[0030] In one embodiment, the steps of the dual-Q-Learning path planning method are as follows: The observation space consists of multi-agent observation tuples. ,in It is a sequential logic state, indicating that "all high-probability points will eventually be covered". This is the current location; It is a distance term. It is the distance between the object's current position and the agent. It is the distance between the object's current position and the target. This is the distance between the object's current position and the obstacle. The observations are normalized to the range [-1, 1] and incorporated into the probability graph. As a global feature.

[0031] Action Space: The continuous spatial tuples that compose the actions of a drone. , It's the drone's movement, speed. , It is the maximum speed; angle , It is a consensus signal, that is, a collaborative instruction between multiple agents, ranging from 0 to 1, indicating the degree of uniformity of actions among multiple agents.

[0032] The reward function is as follows: in, It is the minimum distance, representing the distance to the nearest target point; It is a threat to the area of ​​punishment. It is a sequential logic function. Indicates coverage, Indicating conflict, This represents logical AND. Indicates logical NOT; This indicates a "final" operator; It is a path trajectory; It is the logical weight, ranging from 0.1 to 1; It refers to the task duration; It is an indicator function, which is 1 when the area is within the threat zone.

[0033] Each agent maintains two Q-tables. During updates, actions are used alternately and selected through consensus voting: in , It is the number of intelligent agents. It is the vote of the m-th agent, obtained based on the local Q value; This is the learning rate, with a default value of 0.01. This is the discount factor. The task ends when the coverage of all high-probability points is > 95% or the timing logic is satisfied.

[0034] This step transforms abstract logical formulas into integral rewards and integrates a consensus voting mechanism to achieve distributed path planning. By combining robot collaboration with formal verification, the logical integral formula provides quantitative constraints, which improves decision-making stability in uncertain environments, especially in multi-UAV collaborative reconnaissance.

[0035] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A path planning method based on double Q-Learning, characterized in that, Includes the following steps: Constructing a probability map of unknown moving targets on the ground: By introducing ground gridding, traversable areas, and random points, the probability of the escape direction of unknown moving targets is estimated. The probability map of the moving targets is used as the observation space for reinforcement learning. Reinforcement learning-based target reconnaissance: By introducing spatial gridding, motion models of UAVs, targets to be identified, and threat targets are established, and a path planning method based on dual Q-Learning coverage search is constructed to achieve reconnaissance of unknown moving targets on the ground.

2. The dual Q-Learning based path planning method of claim 1, wherein, Construct a probability map of the target's location, where darker colors indicate a higher likelihood of the target being located at that position. The specific steps are as follows: (1) Step 1: initialize the starting point S0 coordinate of the target to be tested as The region where the target to be tested may move is hexagonally meshed, and the radius of the hexagonal mesh is the width of the target to be recognized. (2) Step 2: Mark the areas where movement is possible as traversable areas and impassable areas; (3) Step 3: Randomly mark n points in the passability region, and the n points are denoted as D(d1,..., dn) as the end points. i ,…,d n ) as the end points. (4) Step 4: Using a parallel approach, obtain the path from the starting point S0 to any endpoint D(d1,…,d) according to the path planning A* algorithm. i ,…,d n If a path is given, then the set of points traversed by each path is denoted as SD. i (sd1,…,sd i ,…,sd m ), where m is the total number of points visited in the path; (5) Step 5: Statistically analyze these sets. The darker the color, the more likely the target to be identified is in that location. Use 0 to 255 to describe it. "0" means impassable and "255" means the target is most likely to be in this area. Use matrix A to represent the target location probability map and use B to represent the corresponding point coordinates.

3. The dual Q-Learning based path planning method of claim 1, wherein, The specific steps for constructing the probability map of the target location to be identified are as follows: Step 1: initialize the starting point coordinates of the target to be tested , is the horizontal and vertical coordinates of the starting point, and the region where the target to be tested may move is hexagonally meshed, with a mesh radius of the target width ; at the same time, by means of a knowledge graph method, the region nodes are labeled as passability and impassability, and the graph edges represent the pass cost; Step 2: Model the target motion as a nonlinear diffusion process from the starting point The probability density evolves with time The diffusion damping coefficient of the infeasible region The feasible region ; Step 3: Randomly sample in the navigable region, incorporating the knowledge graph Potential end points ; Use parallel A* algorithm to compute the base path set for each end point ; Step 4: Compute probability map, probability matrix the value of each grid point is quantitatively computed by the following equation: The above formula realizes the evolution prediction of probability from static to dynamic, wherein, is the position at time ; the probability density is ; time is ; the integral variable is ; the update interval is ; the divergence operator is represented by ; the attention weight is ; the position-dependent diffusion coefficient is ; the attention mechanism is the attention degree of the path set to the current point; The calculation is as follows: wherein is a query vector, is the and the key vector of the path point, is the dimension; For the passable region, the position-dependent diffusion coefficient is calculated as follows: is a cost, obtained from a knowledge graph, is a cost decay parameter; Step 5: Normalized to 0~255, where 0 represents impassable and 255 represents the highest probability of being passable. Corresponding coordinate matrix Each element in the graph; the probability graph is dynamically updated every time step. Re-diffusion, incorporating real-time observation data for correction. .

4. The path planning method based on dual Q-Learning according to claim 2, characterized in that, The specific steps for establishing motion models of the drone, the target to be identified, and the threat target are as follows: (1) Divide the area where the UAV flies into a hexagonal grid. The radius of the hexagonal grid is the radius of the area where the UAV flies. The minimum distance traveled in a given time; (2) Simplify the drone into a point mass, its The coordinates of the time are The velocity angle is The speed of the drone is , angle is There are six directions, corresponding to 0°, 60°, 120°, 180°, 240°, and 360° respectively. This represents the radius of the drone's ground search. The distance between the drone and the target point to be identified. For a weighted average, Where A is the probability map of the target to be tested, and B is the corresponding coordinate value in the probability map. These are weighting coefficients; (3) The distance between the drone and the target point of the threat; (4) The continuous trajectory of the UAV can be modeled as a series of continuous discrete points. Let the position of the UAV at the next moment be... The velocity angle is The time interval of the movement is Then the drone is The state at that moment is: in, , The angle of the drone changes, assuming the threat target is stationary.

5. The path planning method based on dual Q-Learning according to claim 3, characterized in that, For the problem of reconnaissance of unknown moving targets on Earth based on reinforcement learning, a joint space is established, including the agent's observation space, action space, and reward function; 1) Observation space: The observation space of the agent is represented by a tuple ( , , This indicates that the observations are normalized; 2) Action Space: The action space of the agent is represented by a tuple. express; 3) Reward Function: The base reward is the negative of the distance to the nearest target point; entering the threat zone results in an additional deduction. The reward function is as follows: T0 is a preset threshold; 4) Reinforcement learning: A path planning method based on double Q-Learning coverage search is adopted, and the task is terminated based on whether the search is completed.

6. The path planning method based on dual Q-Learning according to claim 1, characterized in that, The steps of the dual-Q-Learning path planning method are as follows: The observation space consists of multi-agent observation tuples. in It is a sequential logic state, indicating "eventually covering all high-probability points". This is the current location; It is a distance term. It is the distance between the object's current position and the agent. It is the distance between the object's current position and the target. It represents the distance between the object's current position and the obstacle; the observations are normalized to the range [-1, 1] and incorporated into the probability graph. As a global feature; The actions of drones form a continuous spatial tuple , It's the drone's movement, speed. , It is the maximum speed; angle , It is a consensus signal, that is, a collaborative instruction between multiple agents, ranging from 0 to 1, indicating the degree of uniformity of actions among multiple agents; The reward function is as follows: in, It is the minimum distance, representing the distance to the nearest target point; It is a threat to the area of ​​punishment. It is a sequential logic function. Indicates coverage, Indicating conflict, This represents logical AND. Indicates logical NOT; This indicates a "final" operator; It is a path trajectory; It is logical weight; It refers to the task duration; It is an indicator function, which is 1 when the area is within the threat zone; Reinforcement learning algorithm: Each agent maintains two Q-tables During updates, actions are used alternately and selected through consensus voting: in , It is the number of intelligent agents. It is the vote of the m-th agent, obtained based on the local Q value; It is the learning rate; It is a discount factor. This is the drone's next move. This is the state of the drone at the next moment; the mission ends when the coverage of all high-probability points is > 95% or the timing logic is satisfied.

Images