Ground unit cooperative path planning method based on adaptive Q learning

Abstract

The application discloses a ground unit cooperative path planning method based on adaptive Q learning and belongs to the field of maneuverability.The application changes the state space division mode from the traditional grid method to the mode of determining the space state according to the relative distance between the initial node and the target node and the fixed state interval d, designs a reward and punishment function according to the degree of the ground unit approaching the target, thereby relieving the risk that the Q table containing environment information cannot be applied to a new environment due to the change of the target position, and the efficiency and safety of the ground unit cooperative path planning can be improved.The application changes the action space division mode from the traditional fixed action to the mode of determining the action space according to the connection relationship of the road network nodes, reduces the useless action space, improves the applicability of the action space to the road network environment, can relieve the "dimension disaster" problem in the large-scale ground unit path planning problem, and improves the cooperative path planning efficiency under the complex road network environment.

Description

TECHNICAL FIELD

[0001] The application relates to a ground unit cooperative path planning method based on adaptive Q learning, and belongs to the field of maneuverability. BACKGROUND

[0002] The ground units represented by armored vehicles, ground command vehicles and missile launch vehicles are widely used in key fields such as intelligence reconnaissance, communication command and combat confrontation, and the survival ability of the ground units has a very important influence on the combat effectiveness of the whole combat system. The survival ability of the ground unit refers to the ability of the ground unit to continuously and effectively perform the intended task under the premise that the tactical and technical performance is not significantly lost in the confrontation process. The ground unit improves its survival ability by avoiding threats and finally safely reaches the target area. Therefore, improving the timeliness of the cooperative path planning of multiple ground units in a complex road network environment can effectively improve the survival ability of the multiple ground units. The ground unit path planning algorithm can be mainly divided into graph search algorithms, artificial potential field methods and intelligent optimization algorithms. The graph search algorithms include Dijkstra algorithm, A* algorithm, artificial potential field method, rapid search random tree (RRT) algorithm, genetic algorithm (GA), particle swarm optimization algorithm (PSO) and the like. The graph search algorithms all need to discretize the map, so a large number of nodes need to be traversed and calculated in a complex environment. The artificial potential field method guides the ground unit to approach the target point by constructing a virtual force field, has the advantages of simple algorithm and good real-time performance, but is easy to fall into a local optimal solution. The swarm intelligence optimization algorithm often has a large amount of calculation when solving the path planning problem, and it is difficult to quickly process high-dimensional path problems. With the increasing complexity of the battlefield environment, the future ground unit will mainly engage in large-scale combat, and the above-mentioned classical path planning algorithms are difficult to quickly solve the cooperative path planning problem of multiple ground units under road network constraints. In recent years, the reinforcement learning method has been gradually applied to solve the path planning problem. The Q learning algorithm as a typical model-independent reinforcement learning algorithm is widely used in path planning. The Q learning algorithm trains through the interaction between the agent and the environment, makes decisions based on the Q table completed by training, and is similar to the idea of quickly looking up the table, so the path planning using the Q learning has high planning efficiency. In view of the problems such as the need for multiple ground units to adapt to the fast-changing environment in the cooperative path planning in a complex road network environment, the classical Q learning algorithm causes the Q table trained to be only applicable to the scene with fixed target state and threat state, and the adaptability to the complex fast-changing road network environment is poor. In view of the above problems, an adaptive Q learning cooperative path planning method for enhancing the survival ability of multiple ground units is proposed. The relative distance between the beginning and the end of the planning is taken as the state input, a space state division mechanism of the Q learning is designed, so that the Q table is applicable to any target position. A road network environment model is established based on graph theory, and the action space is determined according to the node relationship of the road network. A reward and punishment function is designed based on the threat information and the target distance, and an adaptive Q table containing task information and environment information is customized.

[0003] (1) Reinforcement learning driven multi-ground unit path planning

[0004] The essence of the ground unit path planning problem is to find one or more collision-free paths with small path cost and less time consumption under the condition that the ground unit meets the road network constraint and obstacle constraint. In a complex road network environment, the classic path planning algorithm faces the problem of long search time consumption. The Q-learning algorithm is a kind of efficient learning algorithm, which can be used to solve the problem of long time consumption of path planning in a complex road network constraint environment. The ground unit explores the state space by taking actions in the task environment, and makes judgments through environmental rewards.

[0005] Based on the road network model, the reinforcement learning elements of the path planning problem are designed. The ground unit path planning problem meets the Markov Decision Process (MDP), so the Q-learning algorithm can be used to solve it. It is assumed that the current state of the ground unit is s i , the next state is s i+1 , and s i+1 is only related to s i , and has nothing to do with the previous state. In the path planning process, the MDP is composed of a five-tuple {S, A, P, R, γ}. The meaning of each component is as follows: S={s1, s2, s3,..., s n} represents the set of states that the ground unit is in, A={a1, a2, a3,..., a n} represents the action performed by the ground unit in a certain state, P={s t , a t , s t+1} represents the transition probability of the ground unit from the current state s t to the next state s t by performing action a t+1 . R={r1, r2, r3,..., r n} represents the reward value obtained by the ground unit in the current state by performing a certain action, γ∈(0,1), γ represents the discount factor, and the larger the γ, the more the Q-learning tends to take the action that maximizes the reward of the current state. SUMMARY

[0006] In order to alleviate the disadvantages of the Q table under the classic Q learning algorithm that is no longer applicable when the target position and the threat area change, the ground unit cooperative path planning under the road network constraint faces problems of large amount of calculation and increasing planning time, and the technical problem to be solved by the ground unit cooperative path planning method based on adaptive Q learning disclosed by the application is that in a complex road network environment, by changing the division mode of the state space and the action space, the Q table containing the road network environment state information has good adaptability to the fast-changing environment, the 'dimension disaster' problem faced in the large-scale ground unit path planning problem can be alleviated, and the efficiency of path planning is improved. The application is suitable for the path planning problem under the road network constraint that the target changes and the threat area updates.

[0007] The object of the application is achieved by the following technical solutions.

[0008] The ground unit cooperative path planning method based on adaptive Q learning disclosed by the application changes the division mode of the state space from the traditional grid method to the relative distance between the beginning and the end of planning as the state input, and changes the division mode of the action space from the traditional fixed action to determining the action space according to the connection relationship of the road network nodes. The Q table containing the road network environment state information is used to quickly plan a collision-free and low-cost path for each starting point of the ground unit, so that the efficiency of the ground unit cooperative path planning under the road network constraint is improved.

[0009] The ground unit cooperative path planning method based on adaptive Q learning disclosed by the application comprises the following steps:

[0010] Step A: respectively establishing a road network model and a cooperative path planning model, giving initial information to the road network model and the cooperative path planning model, representing environment information through the road network model, and determining task information through the path planning model.

[0011] Step A-1: establishing a road network model, adopting nodes and paths to mathematically logically describe the road network, and a basic digital road network model diagram can be expressed as:

[0012]

[0013] In the formula, G is a road network model, N is a set of path nodes, n i represents the i th node, represents the position of the arbitrary node, R is a set of paths, r l represents the l th path, and f is a weight value of two road nodes or an arbitrary path.

[0014] Step A-2: establishing a cooperative path planning model, the cooperative path planning task is to plan a collision-free path after the starting position, the target position and the threat information are given. The ground unit cooperative path planning model is established as follows:

[0015]

[0016] wherein: Φ is the total task of the ground unit path planning; k i represents the i-th starting point, d m represents the m-th target point. represents the position of the starting point, represents the position of the target point; T is a set of threat areas, N t is the number of threat areas, r t , x t , y t represents the radius and position of the threat area.

[0017] Step B: taking the target position as the center, the fixed state interval d is determined by using the distance of adjacent nodes, and the number of space states M is defined by combining the relative distance of the initial node and the target node and the fixed state interval d. The range of the space state is a concentric circle with the target position as the center and the fixed state interval d as the radius. By defining the state space, the planning space is discretized, and the applicability of the Q table to the dynamic target environment is improved.

[0018] The fixed state space interval d is determined, taking the target position as the center, and the number of space states M is defined by combining the relative distance of the initial node and the target node and the fixed state interval d. r m represents the range radius of the m-th space state with the target point as the center, r m which is calculated by formula (3):

[0019] r m = m * d (m = 1, 2,..., M) (3)

[0020] By changing the state space division method from the traditional grid method to the relative distance between the beginning and end of the planning as the state input, the problem that the Q table is no longer applicable when the target changes under the traditional Q learning algorithm can be solved, and the applicability of the Q table to the dynamic target environment is improved.

[0021] Step C: determine the connection relationship of the path nodes according to the road network model established in step A, determine the action space according to the connection relationship of the road network nodes, change the division method of the action space from the traditional fixed action to determine the action space according to the connection relationship of the road network nodes, reduce the useless action space, improve the applicability of the action space to the road network environment, and can alleviate the "dimension disaster" problem faced in large-scale ground unit path planning, and improve the path planning efficiency in complex road network environment.

[0022] Determine the adaptive action space set of each node. Determine the action set according to the connection relationship of the constructed road network model. Determine the current node i, j represents the node having a connection relationship with the current node i, Define A for the action of the current node i to the next node j i for the action set of the current node i, N represents the node set of the current road network. The current node i can be extended to the union of the directions of the next node, which satisfies the following relationship:

[0023]

[0024] By changing the division method of the action space from the traditional fixed action to the action space determined according to the connection relationship of the road network node, the disadvantages of the fixed action space being difficult to adapt to the road network environment can be solved, and the adaptability of the action space to the road network environment can be improved.

[0025] Step D: Design a reward and punishment function according to the degree of the ground unit approaching the target. When the node is in the safe area, the reward value obtained by the ground unit gradually increases as the current node gradually approaches the target node, guiding the ground unit to expand towards the target node. When the node is in the threat area, the ground unit is punished, guiding the ground unit to avoid obstacles.

[0026] Design a reward and punishment function to guide the ground unit to approach the target and avoid obstacles. The definition of the reward and punishment function is shown in formula (5):

[0027]

[0028] In the formula, R(n i ,n i+1 ) represents the reward and punishment value of the current node i moving to the next node i+1, k1 and k2 are both positive numbers to be designed, dis(n i ,n g ) represents the distance value from the current node i to the target node g; d is the state interval; Ω safe represents the safe area; and Ω threat represents the threat area.

[0029] When the node is in the safe area, the reward value gradually increases as the current node gradually approaches the target node, guiding the ground unit to expand towards the target node. When the node is in the threat area, the ground unit is punished, guiding the ground unit to avoid obstacles.

[0030] Step E: Generate a Q table containing environment and task information by combining task information, state space, action space and reward and punishment function. The row of the Q table represents the state space, the column represents the action that can be selected by the current node, and the value of each cell in the Q table represents the reward and punishment value obtained by the ground unit in the current state by taking a certain action.

[0031] Step E is implemented as follows:

[0032] Step E-1: initialize the Q table to all zeros in combination with the task information.

[0033] Step E-2: set a suitable learning rate a and a discount factor g, and determine the target node to start updating the Q table, and the updating formula of the Q table is shown in equation (6):

[0034] Q(s,a)←Q(s,a)+a[R+gmaxQ(s',a')-Q(s,a)] (6)

[0035] In the equation, s represents the state of the current node, a represents the action selected by the current node, Q(s,a) represents the reward value after the current node selects the action a; s' represents the state of the next node, a' represents the action selected by the next node, and Q(s',a') represents the reward value after the next node selects the action.

[0036] Step E-3: the ground unit selects an action in the action space and executes it, and judges whether the next node transferred to is located in a threat area, if yes, the action is discarded; compare the reward values of different actions executable by the current node, and take the action with the larger reward value to reach the next node;

[0037] Step E-4: judge whether the current node of the ground unit reaches the target node, if yes, end the updating of the Q table; if not, continue step E-3.

[0038] Step F: based on the Q table updated in step E, quickly plan a collision-free path for each starting node of the ground unit, and improve the efficiency of the cooperative path planning of the ground unit under road network constraints.

[0039] Step F is implemented as follows:

[0040] Step F-1: set the starting node and the target node of the ground unit;

[0041] Step F-2: select the action a that maximizes the reward value of the current node according to the updated Q table in step E, store the node where the ground unit is located and update the current node;

[0042] Step F-3: judge whether the current node reaches the target node, if yes, complete the path planning; if not, repeat step F-2.

[0043] Beneficial effects:

[0044] 1. The ground unit cooperative path planning method based on adaptive Q learning discloses a kind of based on adaptive Q learning ground unit cooperative path planning method, the division mode of state space is changed into with the relative distance of initial node and target node and fixed state interval d determine space state from traditional grid method, to relieve the risk that Q table containing environmental information cannot be applicable to new environment due to target position change, can improve ground unit cooperative path planning efficiency.

[0045] 2, The ground unit cooperative path planning method based on adaptive Q learning discloses a kind of based on adaptive Q learning ground unit cooperative path planning method, the division mode of action space is changed into according to the action space of determining connection relationship of road network node from traditional fixed action, reduce useless action space, improve the applicability of action space to road network environment, and can relieve the "dimension disaster" problem faced in large-scale ground unit path planning problem, improve path planning efficiency under complex road network environment.

[0046] 3, The ground unit cooperative path planning method based on adaptive Q learning discloses a kind of based on adaptive Q learning ground unit cooperative path planning method, according to the degree of ground unit approaching target design reward and punishment function, when node is in safe area, with current node gradually approaching target node, the reward value obtained by ground unit gradually increases, guide ground unit to expand towards target node. When node is in threat area, ground unit is punished, guide ground unit to avoid obstacle, can improve safety on the basis of improving path planning efficiency.

[0047] 4, The ground unit cooperative path planning method based on adaptive Q learning discloses a kind of based on adaptive Q learning ground unit cooperative path planning method, since adaptive Q learning mechanism has universality and universality, can be applied to the following engineering fields: path planning problem solution containing large-scale ground unit, ground unit fast path planning problem containing dynamic target, ground unit path planning problem solution containing dynamic threat area, can relieve the "dimension disaster" problem faced in large-scale ground unit path planning problem. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 It is the flow chart of the ground unit cooperative path planning method based on adaptive Q learning of the application;

[0049] Figure 2 It is road network environment model;

[0050] Figure 3 It is path planning result-2 car;

[0051] Figure 4 It is path planning result-4 car;

[0052] Figure 5 It is path planning result-6 car;

[0053] Figure 6A path planning total cost comparison chart;

[0054] Figure 7 A path planning time consumption comparison chart. DETAILED DESCRIPTION

[0055] In order to better illustrate the purposes and advantages of the present application, the present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0056] Embodiments

[0057] As Figure 1 shown, the embodiment discloses a ground unit cooperative path planning method based on adaptive Q learning, which has high calculation efficiency for multi-ground unit cooperative path planning problem, and can improve the efficiency of multi-ground unit path planning under road network constraints.

[0058] Step A: respectively establishing a road network model and a cooperative path planning model, giving the road network model and the cooperative path planning model initial information, representing environmental information through the road network model, and determining task information through the path planning model.

[0059] Step A-1: establishing a road network model, using nodes and paths to mathematically logically describe the road network, and a basic digital road network model diagram can be represented as:

[0060]

[0061] In the formula: G is the road network model; N is the set of path nodes, n i represents the i-th node, represents the position of the arbitrary node; R is the set of paths, r l represents the l-th path; f is the weight value of two road nodes or any path. The road network model composed of 81 path nodes N and 170 paths l is selected, the road network area is located between north latitude 32° to 38° and east longitude 113° to 119°. The test road network environment model is as shown in Figure 2 .

[0062] Step A-2: establishing a cooperative path planning model, the cooperative path planning task is to plan a collision-free path after the start position, target position, and threat information are given. The ground unit cooperative path planning model is established as follows:

[0063]

[0064] In the formula: Φ is the total task of ground unit path planning; k i represents the i-th starting point, d m represents the m-th target point. represents the position of the starting point, represents the position of the target point; T is a set of threat areas, N t is the number of threat areas, r t , x t , y t represents the radius and position of the threat area.

[0065] It is assumed that the number of ground units is consistent with the number of target points and they are located at the nodes of the road network. Three sets of test ground unit starting nodes and target nodes are set as shown in Table 1:

[0066] Table 1 Ground unit task setting

[0067]

[0068] Step B: Taking the target position as the center, the fixed state interval d is determined by using the distance of adjacent nodes. The number of space states M is defined in combination with the relative distance between the initial node and the target node and the fixed state interval d. The range of the space state is a concentric circle with the target position as the center and the fixed state interval d as the radius. By defining the state space, the planning space is discretized, and the applicability of the Q table to the dynamic target environment is improved.

[0069] The fixed state space interval d is determined, and the number of space states M is defined in combination with the relative distance between the initial node and the target node and the fixed state interval d, taking the target position as the center. r m represents the range radius of the mth space state with the target point as the center, r m is calculated by formula (9). In this case, M is taken as 10-20, and d = 1 / 2.

[0070] r m = m * d (m = 1, 2,..., M) (9)

[0071] By changing the state space partition method from the traditional grid method to the relative distance between the beginning and end of the planning as the state input, the problem that the Q table is no longer applicable when the target changes under the traditional Q learning algorithm can be solved, and the applicability of the Q table to the dynamic target environment is improved.

[0072] Step C: The connection relationship of the path nodes is determined according to the road network model established in step A, and the action space is determined according to the connection relationship of the road network nodes. The action space partition method is changed from the traditional fixed action to the action space determined according to the connection relationship of the road network nodes, which reduces the useless action space, improves the applicability of the action space to the road network environment, and can alleviate the "dimension disaster" problem faced in large-scale ground unit path planning, and improve the path planning efficiency in complex road network environment.

[0073] Determine the adaptive action space set of each node. Determine the action set according to the connection relationship of the constructed road network model. Determine the current node i, j represents the node having a connection relationship with the current node i, Define A for the action of the current node i to the next node j i N represents the action set of the current node i, and N represents the node set of the current road network. The current node i can be extended to the union of the directions of the next node, which satisfies the following relationship:

[0074]

[0075] By changing the division method of the action space from the traditional fixed action to the action space determined according to the connection relationship of the road network node, the disadvantages of the fixed action space being difficult to adapt to the road network environment can be solved, and the adaptability of the action space to the road network environment can be improved.

[0076] Step D: Design a reward and punishment function to guide the ground unit to the target and avoid obstacles. The definition of the reward and punishment function is shown in formula (11):

[0077]

[0078] In the formula, R(n i ,n i+1 ) represents the reward and punishment value of the current node i to the next node i+1, k1 and k2 are both positive numbers that need to be designed, dis(n i ,n g ) represents the distance value of the current node i to the target node g; d is the state interval; Ω safe represents the safe area; and Ω threat represents the threat area. The parameter setting is shown in formula (12):

[0079]

[0080] When the node is in the safe area, the reward value gradually increases as the current node gradually approaches the target node, guiding the ground unit to expand towards the target node. When the node is in the threat area, the ground unit is punished, guiding the ground unit to avoid obstacles.

[0081] Step E: Generate a Q table containing environment and task information by combining task information, state space, action space and reward and punishment function. The row of the Q table represents the state space, the column represents the action that can be selected by the current node, and the value of each cell in the Q table represents the reward and punishment value obtained by the ground unit in the current state by taking a certain action.

[0082] The implementation method of step E is as follows:

[0083] Step E-1: Initialize the Q table to all zeros by combining the task information.

[0084] Step E-2: Set learning rate α and discount factor γ, and determine the target node to start updating the Q table. The updating formula of the Q table is shown in equation (13):

[0085] Q(s,a)←Q(s,a)+α[R+γmaxQ(s',a')-Q(s,a)] (13)

[0086] In the formula, s represents the state of the current node, a represents the action selected by the current node, Q(s,a) represents the reward value after the current node selects action a; s' represents the state of the next node, a' represents the action selected by the next node, and Q(s',a') represents the reward value after the next node selects the action. The parameter setting is shown in equation (14):

[0087]

[0088] Step E-3: The ground unit selects an action in the action space and executes it. It is judged whether the next node transferred to is located in the threat area. If yes, the action is discarded. The reward values of different actions executable by the current node are compared, and the action with the larger reward value is taken to reach the next node.

[0089] Step E-4: It is judged whether the current node of the ground unit reaches the target node. If yes, the Q table updating is ended. If not, step E-3 is continued.

[0090] Step F: Based on the Q table updated in step E, the collision-free path of each starting node of the ground unit is quickly planned, and the efficiency of the cooperative path planning of the ground unit under the road network constraint is improved.

[0091] The implementation method of step F is as follows:

[0092] Step F-1: Set the starting node and the target node of the ground unit.

[0093] Step F-2: According to the Q table updated in step E, select the action a that makes the reward value of the current node maximum, store the node where the ground unit is located, and update the current node.

[0094] Step F-3: It is judged whether the current node reaches the target node. If yes, the path planning is completed. If not, step F-2 is repeated.

[0095] The path planning result is as follows: Figures 3-5As shown, in the process of multi-ground unit cooperative path planning, by comparing the planning time and path cost of the adaptive Q learning algorithm and the classic sparse A* algorithm in the path planning task, the results show that in the cooperative path planning process of 2, 4, 6 ground units, the average calculation time of the algorithm is reduced by 55.92%, 76.48% and 83.25% respectively than the sparse A* algorithm. The total path cost is increased by 10.42%, 10.13% and 11.86% respectively than the sparse A* algorithm. The above results further illustrate that the method has better planning efficiency.

[0096] In order to verify the applicability and feasibility of the application, 50 Monte Carlo simulation experiments are carried out for the task setting of the above case, and compared with the classic sparse A*(spare A*). The starting position and target position of the ground unit in each experiment are not changed as shown in Table 1, and the statistical results are shown in Figure 6 and Figure 7 .

[0097] According to the above results, the optimality of the adaptive Q learning algorithm is slightly lower than that of the sparse A* algorithm, since the sparse A* algorithm is a global search, while the adaptive Q learning algorithm takes the action that maximizes the current node reward at each node, which is a local search algorithm. However, with the increase of the number of ground units, the planning time of the sparse A* algorithm increases linearly, and the planning time of the adaptive Q learning algorithm remains basically unchanged, about 0.04s. By comparison, the algorithm has a significant effect on improving the efficiency of path planning, and has good application prospect for large-scale ground unit cooperative path planning in complex road network environment.

[0098] The above specific description further illustrates the purpose, technical scheme and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application, which is used to explain the application and does not limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A ground unit cooperative path planning method based on adaptive Q-learning, characterized in that: Comprising the following steps, Step A: respectively establishing a road network model and a cooperative path planning model, giving the road network model and the cooperative path planning model initial information, representing environmental information through the road network model, and determining task information through the path planning model; Step B: taking the target position as the center, determining the fixed state interval d by using the distance of adjacent nodes, defining the number of space states M in combination with the relative distance between the initial node and the target node and the fixed state interval d; the range of the space state is a concentric circle with the target position as the center and the fixed state interval d as the radius; by defining the state space, the planning space is discretized, and the applicability of the Q table to the dynamic target environment is improved; Step C: determining the connection relationship of the path nodes according to the road network model established in step A, and determining the action space according to the connection relationship of the road network nodes, changing the division method of the action space from the traditional fixed action to the action space determined according to the connection relationship of the road network nodes, reducing the useless action space, improving the applicability of the action space to the road network environment, and alleviating the "dimension disaster" problem in large-scale ground unit path planning, and improving the path planning efficiency in complex road network environment; Step D: designing a reward and punishment function according to the degree of the ground unit approaching the target, when the node is in the safe area, the reward value obtained by the ground unit gradually increases as the current node gradually approaches the target node, guiding the ground unit to expand towards the target node; when the node is in the threat area, the ground unit is punished, guiding the ground unit to avoid obstacles; Step E: generating a Q table containing environmental and task information in combination with task information, state space, action space and reward and punishment function; the row of the Q table represents the state space, the column represents the action that can be selected by the current node, and the value of each cell in the Q table represents the reward and punishment value obtained by the ground unit in the current state by taking a certain action; Step F: based on the Q table updated in step E, quickly planning a collision-free path for each starting point of the ground unit, and improving the cooperative path planning efficiency of the ground unit under road network constraints.

2. The ground unit cooperative path planning method based on adaptive Q-learning according to claim 1, characterized in that: The implementation method of step A is, Step A-1: establishing a road network model, which can be mathematically described by nodes and paths, and the basic digital road network model can be represented as: where: G is a road network model; N is a set of path nodes, n i ∈N represents the ith node, represents the position of the arbitrary node; R is a set of paths, r l ∈R represents the ith path; and f is a weight value of two road nodes or an arbitrary path. Step A-2: establishing a cooperative path planning model, which plans a collision-free path after the starting position, target position and threat information are given; the ground unit cooperative path planning model is established as follows: Where: Φ is the total task of ground unit path planning; k i ∈K represents the ith starting point, d m ∈D represents the mth target point; represents the position of the starting point, represents the position of the target point; T is a set of threat areas, N t is the number of threat areas, r t , x t , y t represents the radius and position of the threat area.

3. The ground unit cooperative path planning method based on adaptive Q-learning according to claim 2, characterized in that: The implementation method of step B is, A fixed state space interval d is determined, and a number of space states M is defined in combination with the relative distance between the initial node and the target node and the fixed state interval d with the target position as the center of a circle. m r represents a range radius of the mth space state with the target point as the center of a circle. m The formula (3) is calculated. r m = m * d (m = 1, 2,..., M) (3) By changing the division method of the state space from the traditional grid method to the relative distance between the planning start and end as the state input, the disadvantage that the Q table is no longer applicable when the target changes under the traditional Q learning algorithm can be solved, and the applicability of the Q table to the dynamic target environment is improved.

4. The ground unit cooperative path planning method based on adaptive Q-learning according to claim 3, characterized in that: The implementation method of step C is, determining the adaptive action space set of each node; determining the action set according to the connection relationship of the constructed road network model; determining the current node i, j represents the node having a connection relationship with the current node i, for the action of the current node i to the next node j, A is defined i for the action set of the current node i, N represents the node set of the current road network; the direction set of the current node i which can be extended to the next node satisfies the following relationship: By changing the division method of the action space from the traditional fixed action to the action space determined according to the connection relationship of the road network nodes, the disadvantage that the fixed action space is difficult to adapt to the road network environment can be solved, and the applicability of the action space to the road network environment is improved.

5. The ground unit cooperative path planning method based on adaptive Q-learning according to claim 4, characterized in that: The implementation method of step D is, The reward-punishment function is designed to guide the ground unit to approach the target and avoid the obstacles. The reward-punishment function is defined as shown in equation (5): In the formula, R(n i ,n i+1 ) represents the reward or punishment value of the current node i to the next node i+1, k1 and k2 are both positive numbers to be designed, dis(n i ,n g ) represents the distance value of the current node i to the target node g; d is the state interval; Ω safe represents a safety area; Ω threat represents a threat area; When the node is in the safe area, the reward value gradually increases as the current node gradually approaches the target node, guiding the ground unit to expand towards the target node; when the node is in the threat area, the ground unit is punished, guiding the ground unit to avoid obstacles.

6. The ground unit cooperative path planning method based on adaptive Q-learning according to claim 5, characterized in that: Step E is implemented by, Step E-1: initializing the Q table to all zeros in combination with the task information; Step E-2: setting a suitable learning rate α and a discount factor γ, and determining the start of the update of the Q table by the target node, and the update formula of the Q table is shown in equation (6): Q(s,a)←Q(s,a)+α[R+γmaxQ(s',a')-Q(s,a)] (6) In the equation, s represents the state of the current node, a represents the action selected by the current node, Q(s,a) represents the reward value after the current node selects the action a; s' represents the state of the next node, a' represents the action selected by the next node, and Q(s',a') represents the reward value after the next node selects the action; Step E-3: the ground unit selects an action in the action space and executes it, and judges whether the next node transferred to is located in the threat area, if yes, the action is discarded; the reward values of different actions executable by the current node are compared, and the action with the larger reward value is taken to reach the next node; Step E-4: judging whether the current node of the ground unit reaches the target node, if yes, the update of the Q table is ended; if not, step E-3 is continued.

7. The ground unit cooperative path planning method based on adaptive Q-learning according to claim 6, wherein: Step F is implemented by, Step F-1: setting the start node and the target node of the ground unit; Step F-2: selecting the action a with the maximum reward value of the current node according to the updated Q table of step E, storing the node where the ground unit is located and updating the current node; Step F-3: judging whether the current node reaches the target node, if yes, the path planning is completed; if not, step F-2 is repeated.

Images