Multi-unmanned aerial vehicle SAR path planning method based on reinforcement learning

Abstract

The invention provides a multi-unmanned aerial vehicle SAR path planning method based on reinforcement learning, and relates to the technical field of unmanned aerial vehicle path planning, and the method comprises the steps: simulating a mountain environment through a digital elevation model generated by a function, and representing the positions of unmanned aerial vehicles through discrete coordinates; establishing a multi-unmanned aerial vehicle SAR resource allocation system; constructing a task allocation objective function under the constraint condition; adopting a differential evolution algorithm to convert a task allocation objective function into a multi-objective optimization problem for solving; constructing a target function of path planning, and ensuring that the unmanned aerial vehicle completes an imaging task while avoiding collision; based on a task allocation result, performing path planning on the multiple unmanned aerial vehicles by adopting a Q table method; and based on the obtained unmanned aerial vehicle path, point target imaging simulation is carried out by adopting a BP algorithm. According to the method, multi-unmanned aerial vehicle SAR cooperative path planning can be efficiently completed in a complex environment, and in post-disaster emergency rescue, the unmanned aerial vehicles can effectively cooperate with a rescue team to execute early reconnaissance and information support tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle path planning, and particularly relates to a multi-unmanned aerial vehicle SAR path planning method based on reinforcement learning. BACKGROUND

[0002] In the prior art, the path planning of a multi-unmanned aerial vehicle synthetic aperture radar (SAR) system usually adopts a traditional algorithm (such as Dijkstra, A*, etc.) or an intelligent optimization algorithm (such as a genetic algorithm, a particle swarm algorithm, etc.). Although these methods are effective in specific scenarios, they generally have the problem of low computational efficiency, especially in complex unknown mountainous environments, it is difficult to simultaneously consider the needs of multi-objective optimization (such as flight distance, time length and imaging performance) and real-time obstacle avoidance. In addition, the traditional method has weak coupling between task allocation and path planning under the constraint of multi-machine cooperation, resulting in low overall task execution efficiency.

[0003] Specifically, the existing related technology is difficult to achieve efficient cooperative path planning in complex environments, and the problems mainly lie in: First, in the task allocation stage, there is a lack of a multi-objective optimization method that can flexibly adjust the weight to adapt to different scenarios; Second, in the path planning stage, obstacles, imaging benefits and cooperative constraints between unmanned aerial vehicles are not effectively integrated into a real-time decision-making framework, resulting in a possible non-optimal planning path and a large amount of computation, which cannot meet the high real-time requirements in post-disaster rescue scenarios. SUMMARY

[0004] In order to solve the technical problems in the related art, the present application provides a multi-unmanned aerial vehicle SAR path planning method based on reinforcement learning.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is: A multi-unmanned aerial vehicle SAR path planning method based on reinforcement learning, comprising the following steps: Step S1: Simulate a mountainous environment through a function-generated digital elevation model, and use discrete coordinates to represent the position of the unmanned aerial vehicle; Step S2: Establish a multi-unmanned aerial vehicle SAR resource allocation system, in which N unmanned aerial vehicles serve as receiving platforms, a high-orbit satellite serves as a transmitting platform, and all unmanned aerial vehicles fly at the same speed in three-dimensional space to form N groups of bistatic SAR pairs to image a target area; Step S3: Based on the established multi-unmanned aerial vehicle SAR resource allocation system, a task allocation objective function under the constraint condition is constructed; Step S4: The objective function of task allocation is transformed into a multi-objective optimization problem by using the differential evolution algorithm. The multi-objectives include flight distance, flight duration and imaging performance, and different weights are assigned to each objective. Step S5: Construct the objective function for path planning to ensure that the UAV completes the imaging task while avoiding collisions; Step S6: Based on the task allocation results of step S4, the Q-table method is used to plan the paths for multiple UAVs; Step S7: Based on the UAV path obtained in step S6, point target imaging simulation is performed using the BP algorithm.

[0006] Optionally, in step S1, the function simulating the mountainous environment in the digital elevation model is: in, This is the original digital terrain model function, where x and y are the planar projected coordinates. In the formula, a is the phase offset coefficient along the y-axis; b is the amplitude along the x-axis; c, These represent the amplitude of the radially symmetrical cosine wave and the frequency of the radial ripple, respectively. The height of the parallel ridge; f and g are the amplitude of the radially symmetrical sine wave and the frequency of the radial ripple, respectively, used to modify the terrain; For mountain topography functions, ; In the formula, Indicates the number of hills. Indicates the first The peak height of the hill, Indicates the first The center of the hill coordinate, Indicates the first The center of the hill coordinate.

[0007] Optionally, in step S2, the multi-UAV SAR resource allocation system satisfies the following constraints: All drones fly in a three-dimensional space [X,Y,Z]. The coordinates of the launch platform drone are: The coordinates of the i-th drone are And satisfy: The distance between any two drones satisfies: In the formula, For safe distance.

[0008] Optionally, in the step S3, the task allocation objective function comprises: wherein, is the total flight distance of the i-th UAV after allocating k tasks, denotes the distance between the i-th UAV from the j-th task to the k-th task, denotes the number of tasks allocated to the i-th UAV, is the total flight range of the system, is the resolution cell area of the k-th imaging task of the i-th UAV, denotes the average resolution cell area of the total imaging tasks of the i-th UAV, is the total running time of the system, is the imaging performance benefit, denotes the average resolution cell area of the N groups of dual-base SAR. Optionally, in the step S4, the expression of the multi-objective optimization problem is:

[0009] The constraint condition is: wherein, denotes the optimization function, denotes the total number of tasks in the flight range, denotes the maximum flight distance of the UAV, is the decision variable, when the i-th UAV executes the k-th task =1, otherwise =0, , , and are the weights of the three task allocation objective functions, respectively. Optionally, in the step S4, the implementation process of the differential evolution algorithm specifically comprises: Step S41: initializing the population size Pop, the scaling factor F and the crossover probability Cr, and randomly generating an initial population;

[0010] ​​​​​​​​Step S42: evaluating the initial population, performing mutation operation on each individual to generate a mutated individual, and then performing crossover operation to obtain a trial individual, and storing the trial individual into an intermediate population before the number of iterations reaches the population size; Step S43: comparing the individuals in the original population and the intermediate population, retaining the individuals with better fitness to the next generation population, updating the population and iterating, and finally outputting the optimal solution.

[0011] Optionally, in the step S5, the objective function of the path planning includes: wherein, represents the collision threat of the unmanned aerial vehicle located at the point in the scene, represents the threat degree of the unmanned aerial vehicle to other unmanned aerial vehicles, represents the probability that the obstacle can be detected, represents the distance between the current unmanned aerial vehicle and the kth unmanned aerial vehicle, represents the coordinates of the kth unmanned aerial vehicle, represents the global threat matrix of the unmanned aerial vehicle at the point ; represents the imaging benefit of the unmanned aerial vehicle to the target point, respectively represent the resolution unit area of the unmanned aerial vehicle to each target.

[0012] Optionally, in the step S6, when the Q-table method is used for path planning, the calculation formula of the reward matrix is: wherein, represents the reward of the unmanned aerial vehicle from the point pi to pr, is the geometric distance between two points, and K and M are respectively the tolerance degree to the obstacle and the emphasis coefficient on the imaging effect.

[0013] Optionally, in the step S6, the specific process of the path planning includes: determining the initial flight path according to the task allocation; in the running process, each unmanned aerial vehicle continuously judges the current position, and if the end point is reached, the task is terminated, otherwise the obstacles in the detection range are scanned.​​​​​ replanning the path immediately when an obstacle is found; obtaining the optimal path to the destination by iteratively updating the cost matrix.

[0014] Optionally, in the step S7, the BP algorithm is used to calculate the optimal imaging position for each imaging point to meet the imaging requirements, and the point target imaging simulation is completed.

[0015] Beneficial effects: 1、Through the above technical solution, first, the method of the application can effectively improve the environmental adaptability and planning reliability. Specifically, the method of the application converts the complex and continuous real mountain environment into a discrete digital model that can be processed by computer algorithms through "generating a digital elevation model by a function" and "using discrete coordinates to represent the position of the unmanned aerial vehicle". In this way, a stable, unified and quantifiable calculation basis can be provided for all subsequent optimization algorithms (differential evolution, reinforcement learning), significantly improving the feasibility and reliability of the path planning algorithm in complex three-dimensional terrain and avoiding planning failure caused by inaccurate environment model.

[0016] Second, the method of the application can realize system-level coordination and multi-objective optimization. Specifically, "establishing a multi-unmanned aerial vehicle SAR resource allocation system" and "constructing a task allocation objective function under constraints" establish the framework for system coordination. The effect is to improve the actions of multiple unmanned aerial vehicles from "each for itself" to "systematic planning" and clearly define the optimization target. Then, by "adopting a differential evolution algorithm to convert the task allocation objective function into a multi-objective optimization problem, including flight distance, flight time and imaging performance, and giving different weights", the optimization effect is that instead of finding a local optimal solution of a single indicator (such as only considering the shortest path), it can systematically balance multiple conflicting performance indicators (range, time, imaging quality). By assigning different weights, the method of the application obtains flexibility in dealing with different application scenarios, for example, in emergency rescue, the flight time can be given a higher weight to pursue speed, and in detailed survey, the imaging performance can be given a higher weight to pursue quality. In this way, the problem of low efficiency in complex multi-objective trade-off of traditional methods can be fundamentally solved.

[0017] Thirdly, the method of the present application can ensure the intelligence and task completion degree of online execution. Specifically, after obtaining the optimized task allocation scheme, a decision mechanism of reinforcement learning of "adopting Q table method to plan path for multiple unmanned aerial vehicles" is introduced. In this way, the unmanned aerial vehicle can have the ability of online decision and real-time obstacle avoidance in unknown or dynamic environment. It is different from static and one-time path planning, but the unmanned aerial vehicle can dynamically adjust the path according to the real-time perceived environmental information (such as sudden obstacles), thereby greatly improving the success rate and robustness of task execution. Finally, through "adopting BP algorithm to simulate point target imaging", a technical closed loop from path planning to final task verification is formed. In this way, the method of the present application not only considers the imaging requirement in the planning stage (through the objective function), but also verifies the effectiveness of the planned path in the final output stage, ensuring that the whole method serves the final purpose of the core task of SAR imaging, that is, the generated path is indeed "imaging".

[0018] Overall, the technical effect of the present application lies in constructing a complete technical chain from environment modeling to system cooperation and multi-target offline optimization, to online intelligent planning and task verification. Through the organic combination of each step, the scheme cooperates and finally realizes the overall technical effect of efficiently, flexibly and reliably completing the multi-unmanned aerial vehicle SAR cooperative path planning in a complex environment, which is the core advantage of the scheme over the traditional method.

[0019] 2, Other beneficial effects or advantages of the present application will be described in detail in the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Among them: Figure 1 is a step flowchart of a multi-unmanned aerial vehicle SAR path planning method based on reinforcement learning provided by an exemplary embodiment of the present application; Figure 2 is a flowchart of a multi-unmanned aerial vehicle SAR path planning method based on reinforcement learning provided by an exemplary embodiment of the present application; Figure 3 is a schematic diagram of a target scene arrangement adopted by an exemplary embodiment of the present application; Figure 4 is a task allocation schematic diagram adopted by an exemplary embodiment of the present application; Figure 5 is a schematic diagram of the imaging result of target-2, target-3 and target-8 in the target scene shown in FIG. 8; Figures 6 to 8 is Figure 3 is a schematic diagram of the imaging result of target-2, target-3 and target-8 in the target scene shown in FIG. 8; Figure 6 is a schematic diagram of the imaging result of target-2, Figure 7 is a schematic diagram of the imaging result of target-3, Figure 8 is a schematic diagram of the imaging result of target-8. DETAILED DESCRIPTION

[0022] In order for the relevant technical personnel to have a clearer and more accurate understanding of the technical solutions of the present application, the prior related technologies and the technical problems existing therein will be described in more detail as follows.

[0023] Synthetic Aperture Radar (SAR) is a radar that uses synthetic aperture to achieve high-resolution imaging. Due to its all-weather, all-day, and high-resolution characteristics, synthetic aperture radar is widely used in military, civilian, and other fields. Synthetic aperture radar can be used on various platforms, and traditional synthetic aperture radar is mostly used on large platforms such as airplanes and satellites. Airborne synthetic aperture radar systems are usually used for short-term tasks, such as mapping disaster areas or measuring large construction sites. Compared with the above-mentioned synthetic aperture radar systems, synthetic aperture radar systems based on unmanned aerial vehicles have the advantages of low cost, flexible deployment, and easy operation.

[0024] In recent decades, SAR imaging algorithms have been continuously developed, mainly divided into time-domain and frequency-domain methods. Frequency-domain imaging algorithms process echo data in the frequency domain, improving computational efficiency. Typical frequency-domain algorithms include: Rang Doppler (RD) algorithm, Chirpscaling (CS) and Omega-K algorithm. However, each algorithm has its own scope of application, and introduces approximations in the calculation process, reducing imaging accuracy.

[0025] Compared with traditional single-base / dual-base synthetic aperture radar, multi-unmanned aerial vehicle SAR system based on multi-unmanned aerial vehicle platform has many advantages. First, multi-unmanned aerial vehicle SAR system can obtain target scattering information from multiple angles, enabling the system to suppress noise and obtain better resolution, thereby having better image interpretation, terrain classification and target recognition capabilities. Second, multi-unmanned aerial vehicle SAR system has high flexibility and autonomy, can be configured according to specific task requirements and application scenarios, has strong resistance to unexpected accidents, and is suitable for complex military and civilian tasks. Multi-unmanned aerial vehicle SAR system also has the advantages of low cost, small size, light weight and fast movement, which makes it easy to build and quickly deploy. Multi-unmanned aerial vehicle SAR system can also perform imaging tasks on multiple targets in complex scenes, making the task execution more efficient.

[0026] Traditional path planning algorithms include Dijkstra, A*, D* and so on, which need to rasterize the space, and the obtained path is not suitable for UAV flight and needs to be smoothed. Intelligent algorithms such as genetic algorithm, evolutionary algorithm and ant colony algorithm are also often used to solve path planning problems. For example, the document Path Planning Algorithm Using the Particle Swarm Optimization and the Improved Dijkstra Algorithm combines the particle swarm optimization algorithm with the Djikstra algorithm, and the obtained path is shorter than the path obtained by the Djikstra algorithm. For example, the document Genetic Algorithm Based Approach for Autonomous Mobile Robot Path Planning improves the crossover operator and fitness function in the genetic algorithm, and the convergence of the algorithm is improved. For the path planning problem of GEO-UAV bistatic SAR, Z Sun et al. of University of Electronic Science and Technology of China in the document Path Planning for GEO-UAV Bistatic SAR Using Constrained Adaptive Multiobjective Differential Evolution model the path planning problem as a constrained multi-objective optimization problem, and propose a path planning algorithm based on a constrained adaptive multi-objective differential evolution algorithm to solve the problem.

[0027] Although these methods are effective in specific scenarios, there is a common problem of low computational efficiency, especially in complex unknown mountainous environments, it is difficult to simultaneously consider multi-objective optimization (such as flight distance, time and imaging performance) and real-time obstacle avoidance requirements.

[0028] Therefore, how to more efficiently realize UAV path planning in complex and unknown environments has become a problem to be solved.

[0029] The technical concept of the present application is that: first, the multi-UAV cooperative imaging problem is modeled as a multi-objective optimization problem by using a differential evolution algorithm for offline task allocation, considering flight range, task duration and imaging performance to obtain a globally optimal initial task scheme; then, on this basis, reinforcement learning (Q table method) is used for online path planning, a reward function is constructed by fusing collision threat, obstacle distance and imaging benefit, so that each UAV can dynamically adjust the path according to the real-time environmental information during flight, thereby realizing obstacle avoidance and efficiently completing the imaging task, and finally the imaging effect is verified by BP algorithm.

[0030] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0031] As shown in Figure 1 and Figure 2 The present application provides a multi-UAV SAR path planning method based on reinforcement learning, comprising the following steps: Step S1: Simulate the mountainous environment by a function-generated digital elevation model, and use discrete coordinates to represent the position of the UAV; Step S2: Establish a multi-UAV SAR resource allocation system, in which N UAVs serve as receiving platforms and high-orbit satellites serve as transmitting platforms, all UAVs fly at the same speed in three-dimensional space to form N groups of bistatic SAR to image the target area; Step S3: Based on the established multi-UAV SAR resource allocation system, construct a task allocation objective function under constraints; Step S4: Use the differential evolution algorithm to convert the task allocation objective function into a multi-objective optimization problem for solving, the multi-objective including flight distance, flight time and imaging performance, and different weights are assigned to each objective; Step S5: Construct a path planning objective function to ensure that the UAV completes the imaging task while avoiding collision; Step S6: Based on the task allocation result of step S4, use the Q-table method to plan the path of the multi-UAV; Step S7: Based on the UAV path obtained in step S6, use the BP algorithm to perform point target imaging simulation.

[0032] Through the above technical solutions, first, the method of the present application can effectively improve the environmental adaptability and planning reliability. Specifically, the method of the present application converts the complex and continuous real mountainous environment into a discrete digital model that can be processed by computer algorithms by "simulating the mountainous environment by a function-generated digital elevation model" and "using discrete coordinates to represent the position of the UAV". In this way, a stable, unified and quantifiable calculation basis can be provided for all subsequent optimization algorithms (differential evolution, reinforcement learning), significantly improving the feasibility and reliability of the path planning algorithm in complex three-dimensional terrain, and avoiding planning failure caused by inaccurate environment model.

[0033] Secondly, the method of the present application can realize system-level coordination and multi-objective optimization. Specifically, the "establishing a multi-UAV SAR resource allocation system" and "constructing a task allocation objective function under constraints" establish the framework for system coordination. The effect is to upgrade the actions of multiple UAVs from "each for itself" to "system planning", and to clearly define the optimization target. Then, by "adopting a differential evolution algorithm to convert the task allocation objective function into a multi-objective optimization problem, including flight distance, flight time and imaging performance, and giving different weights", the optimization effect is achieved: instead of finding a local optimal solution of a single indicator (such as only considering the shortest path), it can systematically balance multiple conflicting performance indicators (range, time, imaging quality). By assigning different weights, the method of the present application obtains flexibility in dealing with different application scenarios, for example, in emergency rescue, flight time can be given a higher weight to pursue speed, and in detailed survey, imaging performance can be given a higher weight to pursue quality. In this way, the problem of low efficiency in complex multi-objective trade-off in traditional methods can be fundamentally solved.

[0034] Thirdly, the method of the present application can ensure the intelligence and task completion degree of online execution. Specifically, after obtaining the optimized task allocation scheme, the "adopting Q table method for path planning of multiple UAVs" reinforcement learning decision mechanism is introduced. In this way, the UAV can have online decision-making and real-time obstacle avoidance capability in unknown or dynamic environment. It is different from static and one-time path planning, but allows the UAV to dynamically adjust the path according to the real-time perceived environmental information (such as sudden obstacles), thereby greatly improving the success rate and robustness of task execution. Finally, by "adopting BP algorithm for point target imaging simulation", a technical closed loop from path planning to final task verification is formed. In this way, the method of the present application not only considers the imaging requirements in the planning stage (through the objective function), but also verifies the effectiveness of the planned path in the final output stage, ensuring that the entire method serves the ultimate purpose of SAR imaging, i.e. the generated path is indeed "imaging".

[0035] Overall, the technical effect of the present application lies in constructing a complete technical chain from environment modeling to system coordination and multi-objective offline optimization, and then to online intelligent planning and task verification. Through the organic combination of each step, the scheme realizes the overall technical effect of efficiently, flexibly and reliably completing multi-UAV SAR cooperative path planning in complex environments, which is the core advantage of the present application over traditional methods.

[0036] It is understood that the method of this invention specifically employs reinforcement learning algorithms for task allocation and path planning in multi-UAV SAR. First, a dual-multi-base SAR system is established using N UAVs and a high-orbit satellite. Then, a differential evolution algorithm is used to transform the objective function into a multi-objective optimization problem, which is then solved. The resulting path is used as the initial path for the UAVs. Based on this, a reward matrix for path planning is constructed. Using the Q-table method, during operation, each UAV continuously determines its current position: if it has reached the destination, the task is terminated; otherwise, it scans for obstacles within its detection range, and if an obstacle is detected, the path is immediately replanned. Compared with existing path planning algorithms, this significantly reduces the computational load and achieves multi-UAV collaboration, showing broad application prospects. The method of this invention can be used in disaster relief, autonomous landing, and autonomous navigation.

[0037] The method of the present invention will be further described below with reference to an exemplary implementation method (it should be noted that the present invention is mainly verified by simulation experiments, and all steps and conclusions have been verified to be correct on Matlab2018).

[0038] Step 1: The simulation parameters of the multi-UAV SAR system in this method are shown in Table 1 below. A coordinate system is established with the digital elevation model as the center.

[0039] Table 1 Simulation Parameter Table The mountainous environment in this method is simulated using a generated digital elevation model, with the UAV's position represented by discrete coordinates. The original digital terrain model function is: Where x and y are the projected coordinates of the plane. This is the height at the corresponding position. In the formula, a is the phase offset coefficient along the y-axis; b is the amplitude along the x-axis; c, These represent the amplitude of the radially symmetrical cosine wave and the frequency of the radial ripple, respectively. The height of the parallel ridge; f and g are the amplitude of the radially symmetrical sine wave and the frequency of the radial ripple, respectively, used to change the terrain.

[0040] The mountain topographic function is: In the formula, Indicates the number of hills. Indicates the first The peak height of the hill, Indicates the first The center of the hill coordinate, Indicates the first The center of the hill coordinate.

[0041] Therefore, the simulated mountain environment function is: Step 2: Establish a multi-UAV SAR resource allocation system. In this system, N UAVs act as receiving platforms to receive signals, and a high-orbit satellite acts as a transmission platform. All UAVs fly in a three-dimensional space [X,Y,Z]. The coordinates of the UAVs on the transmission platform are... The coordinates of the i-th drone are To ensure that all drones fly within this three-dimensional space, the following spatial constraints must be satisfied: For the i-th drone, it is related to any other drone The distance between the two drones is: ; The distance between any two drones must not be less than the safe distance. Otherwise, collisions between drones will occur, therefore drones must meet cooperative constraints: .

[0042] Step 3: Based on the multi-UAV system established in Step 2, construct the objective function for task allocation under constraints: in, The total flight distance of the i-th UAV after task assignment is given. Indicates the first The drone from the first -1 task to the The distance between tasks Indicates the first The number of tasks assigned to each drone. It is the total flight range of the system. It is the first The resolution cell area of ​​the k-th imaging task of the UAV. Indicates the first The average resolution cell area of ​​the entire imaging mission of the UAV. This is the total system uptime. It is an improvement in imaging performance. This represents the average resolution cell area of ​​N groups of bistatic SAR.

[0043] Step four: According to step three, the multi-objective function is converted into a multi-objective optimization problem, which is expressed as: wherein, represents the optimization function, represents the total number of tasks within the flight range, represents the maximum flight distance of the UAV, is the decision variable, when the kth target is executed by the i th UAV, = 1, otherwise = 0.

[0044] The key parameters of the differential evolution algorithm are initialized, wherein the population size , the scale factor F = 0.7, the crossover probability , and the initial population is randomly generated on this basis. The algorithm enters the iterative optimization loop: before reaching the preset number of iterations, the algorithm will continuously perform mutation operations on each individual in the population to generate mutant individuals, then generate trial individuals through crossover operations, and all trial individuals are collected to form an intermediate population. Then, a strict selection operation is performed, and the individuals in the intermediate population are compared with the corresponding individuals in the original population one by one. If the fitness of the trial individual is better, it is retained in the new generation population, otherwise the original individual is still retained, thereby completing the update of the population and entering the next iteration, and this process is repeated until the termination condition is met, and finally the global optimal solution is output.

[0045] The optimization model solved by the algorithm aims to balance multiple objectives through a weighted sum, wherein the weights can be adjusted according to different application scenarios, and the model also includes a key constraint to ensure that each imaging target is only responsible for one UAV, thereby effectively avoiding repeated allocation of tasks and waste of resources.

[0046] Step five: The objective function expression of path planning is: wherein, represents the mountain collision threat of the UAV located at point in the scene, represents the threat level of the UAV received from other N-1 UAVs; represents the probability that the obstacle can be detected, and the obstacle threat level changes with the interval distance between UAVs, ​represents the distance between the current UAV and the kth UAV, , , represents the coordinate of the kth UAV, represents the global threat matrix that the UAV is subjected to; represents the imaging yield of the UAV to the target point, , , respectively represent the resolution unit area of the UAV to each target.

[0047] To measure the reward matrix under the factors of collision risk, geometric distance and imaging demand, the reward and punishment of the UAV to any other point in the scene are evaluated. The three-dimensional mountain map is represented by a 10km x 10km x 10km grid. The UAV moves from any point pi in the map to another point pr , and the reward can be represented as: wherein, represents the reward of the UAV from point pi to pr, is the geometric distance between the two points. If exceeds the set step length, then: .

[0048] K and M reflect the tolerance to obstacles and the emphasis on imaging effect. The higher the value of K, the farther the UAV will be from the obstacles, and the path length will increase. The higher the value of M, the closer the UAV will be to the ideal imaging position point, and the path length will increase. When applied to actual scenarios, K and M can be adjusted according to actual requirements. For each point in the map, the reward matrix is composed of all associated with .

[0049] To adapt to actual situations, a safety distance , i.e. an obstacle observation radius, is introduced. A circular area with a radius of centered on the UAV position point is the observation area of the UAV. Only when an obstacle enters the observation area of the UAV, the UAV will detect the obstacle and obtain the risk information. Therefore, only the observed obstacles are counted as risks when calculating the reward matrix.

[0050] ​Step six: according to the task allocation initial flight path. In the running process, each UAV will continue to determine the current position: if the end point is reached, the task is terminated; otherwise, the obstacles in the detection range are scanned, and if the obstacles are found, the path is immediately re-planned. A cost matrix is designed to obtain the best path to the end point, and the cost matrix is obtained by iteration of the reward matrix. After iteration, the cost matrix will converge and remain stable. In a 10km*10km*10km three-dimensional map, the update mechanism of the cost matrix is as follows: First, initialize a cost matrix G, where the corresponding position value of the UAV flight end point is set to 0, and the rest of the points are set to . Then enter the iterative update stage: randomly select a point in the map, and compare the current cost value of each point in the map with the correction value provided by the reference reward matrix, so as to update the value of the point in the matrix G. Repeat this update process until the preset maximum iteration number is reached, or the cost matrix G does not change in the last ten iterations, at which time the iteration is terminated.

[0051] After generating the final cost matrix G, select the position point with the minimum cost value from it, add it to the path node list Route, and reset the value of the point in the matrix G to . Repeat the above process until the path extends to the flight end point or the path node list length exceeds the preset upper limit. The final generated path node list Route is the path sequence followed by the UAV, and each UAV has its own path node list.

[0052] Step seven: according to the path generated in step six, use the BP algorithm to perform point target imaging, and the imaging results are as shown in Figures 5 to 8 , Figures 6 to 8 are the imaging results of target-2, target-8 and target-3 points in Figure 4 , respectively. From the imaging results, it can be seen that the target outline is clear, the two-dimensional resolution is orthogonal, the dynamic range is reasonable, the 3dB main lobe width is relatively narrow, and the peak side lobe ratio (PSLR) and integral side lobe ratio (ISLR) are at excellent level, which reflects the excellent resolution and signal focusing ability of the imaging system. The point target imaging effect of the path planning algorithm for target points is good.

[0053] As can be seen from the specific embodiments of the present application, the present application can realize multi-UAV SAR path planning. By using the differential evolution algorithm for task allocation and the reinforcement learning for path planning, the computational amount is greatly reduced, and multi-UAV cooperation is realized. The method of the present application can be used in disaster relief, autonomous landing, autonomous navigation and other fields.

[0054] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed by the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for multi-UAV SAR path planning based on reinforcement learning, characterized in that, The method comprises the following steps: Step S1: Simulate a mountainous environment by a function generated digital elevation model, and use discrete coordinates to represent the position of the unmanned aerial vehicle; Step S2: Establish a multi-unmanned aerial vehicle SAR resource allocation system, wherein N unmanned aerial vehicles serve as receiving platforms, a high-orbit satellite serves as a transmitting platform, and all the unmanned aerial vehicles fly at the same speed in a three-dimensional space to form N groups of bistatic SAR to image a target region; Step S3: Based on the established multi-unmanned aerial vehicle SAR resource allocation system, construct a task allocation objective function under constraints; Step S4: Convert the task allocation objective function into a multi-objective optimization problem for solving by using a differential evolution algorithm, wherein the multi-objective includes flight distance, flight time and imaging performance, and different weights are allocated to each objective; Step S5: Construct a path planning objective function to ensure that the unmanned aerial vehicle completes the imaging task while avoiding collision; Step S6: Based on the task allocation result of step S4, use a Q table method to plan paths for the multi-unmanned aerial vehicles; Step S7: Based on the paths of the unmanned aerial vehicles obtained in step S6, use a BP algorithm to perform point target imaging simulation.

2. The method of claim 1, wherein, In step S1, the function for simulating a mountainous environment in the digital elevation model is: wherein is the original digital terrain model function, x and y are the plane projection coordinates, wherein a is a phase shift coefficient in the y-axis direction; b is an amplitude in the x-axis direction; c, are the amplitude of the radially symmetric cosine wave and the frequency of the radial undulation, respectively; is the height of the parallel ridges; f, g are the amplitude of the radially symmetric sine wave and the frequency of the radial undulation, respectively, for changing the terrain; is a mountainous terrain function, wherein represents the number of hills, represents the peak height of the hills, represents the coordinates of the center of the hills, represents the coordinates of the center of the hills.

3. The method of claim 1, wherein, In step S2, the multi-unmanned aerial vehicle SAR resource allocation system satisfies the following constraints: All UAVs fly in three-dimensional space [X, Y, Z], the launch platform UAV coordinates are , the i-th UAV coordinates are , and satisfy: The distance between any two unmanned aerial vehicles satisfies: In the formula, is the safety distance.

4. The method of claim 1, wherein, In step S3, the task allocation objective function includes: In the formula, is the total flight distance of the i-th UAV after it is assigned k tasks, represents the i-th UAV from the j-th task to the k-th task, represents the i-th UAV assigned task number, is the total flight range of the system, is the resolution unit area of the k-th imaging task of the i-th UAV, represents the average resolution unit area of the total imaging task of the i-th UAV, is the total running time of the system, is the imaging performance benefit, represents the average resolution unit area of the N groups of dual-base SAR.​​​ 5. The method of claim 4, wherein, In step S4, the expression of the multi-objective optimization problem is: The constraints are: ; In the formula, Describes the optimization function. Indicates the total number of missions within the flight range. Let $\mathbf{a}$ represent the maximum flight distance of the drone, and $\mathbf{a}$ be the decision variable. When the drone executes the k-th target =1, otherwise =0, , and These are the weights of the objective function for assigning tasks to the three tasks.

6. The method of claim 5, wherein, In step S4, the implementation process of the differential evolution algorithm specifically includes: Step S41: initialize population size Pop, scale factor F and crossover probability , randomly generate an initial population; Step S42: Evaluate the initial population, and before the number of iterations reaches the population size, perform a mutation operation on each individual to generate a mutated individual, and then obtain a trial individual through a crossover operation and store it in an intermediate population; Step S43: Compare the individuals in the original population and the intermediate population, retain the individuals with better fitness to the next generation population, update the population and iterate, and finally output the optimal solution.

7. The method of claim 5, wherein, In step S5, the path planning objective function includes: wherein, represents the collision threat of the mountain to the UAV at point represents the threat degree of the UAV to other UAVs, represents the probability that the obstacle can be detected, represents the distance between the current UAV and the kth UAV, , , ) represents the coordinates of the kth UAV, represents the global threat matrix at which the UAV is subjected to at point represents the imaging revenue of the UAV to the target point, , , respectively represent the resolution unit area of the UAV to each target.​​​ 8. The method of claim 7, wherein, In step S6, when the Q table method is used for path planning, the calculation formula of the reward matrix is: wherein, represents the reward of the UAV from point pi to pr, is the geometric distance between two points, and K and M are the tolerance to obstacles and the emphasis on imaging effect coefficients, respectively.

9. The method of claim 8, wherein, In step S6, the specific process of path planning includes: Determine the initial flight path according to the task allocation; During the running process, each unmanned aerial vehicle continuously judges the current position, and if the end point is reached, the task is terminated, otherwise the obstacles in the detection range are scanned; Immediately re-plan the path when an obstacle is found; Update the cost matrix through iteration to obtain the optimal path to the end point.

10. The method of claim 1, wherein, In step S7, the BP algorithm is used to calculate the best imaging position for each imaging point to meet the imaging requirements, and complete the point target imaging simulation.

Images