Reinforced learning unmanned aerial vehicle route planning method and system based on safe flight corridor

Abstract

The invention discloses a reinforcement learning unmanned aerial vehicle route planning method and system based on a safe flight corridor, and relates to the technical field of route planning. A safe flight corridor is generated through a grid map of a target scene and starting point information; calculating a plurality of initial B spline control points through observation information of the unmanned aerial vehicle; iteratively generating a new B-spline control point based on the plurality of initial B-spline control points, the key features extracted based on the safe flight corridor and the time variable through a reinforcement learning algorithm; and generating a B spline path of the unmanned aerial vehicle according to all B spline control points. According to the invention, a feasible flight area in a complex environment is abstracted as a safe flight corridor. Environment characteristics are hidden in the safe flight corridor, and common application in different complex environments can be achieved through training in one environment. A B-spline control point is iteratively generated by using a reinforcement learning algorithm, and environment change can be iteratively adapted in real time, so that the generated B-spline path better meets the actual flight demand of the unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of route planning, in particular to a reinforcement learning unmanned aerial vehicle route planning method and system based on a safe flight corridor. BACKGROUND

[0002] Unmanned aerial vehicles are increasingly widely used, from logistics distribution, environmental monitoring to disaster relief, and unmanned aerial vehicles are required to safely and efficiently complete flight tasks, and path planning is a key technology to ensure that unmanned aerial vehicles achieve the above goals. The end-to-end unmanned aerial vehicle route planning method based on neural networks can fully utilize the end-to-end characteristics of neural networks, improve the real-time performance of route planning and reduce the power consumption, so it has become a hot research direction in the field of unmanned aerial vehicles. Reinforcement learning method has long-term and short-term balance ability, and can consider both near obstacles and distant obstacles, so it is the most suitable method for neural network route planning.

[0003] At present, there are many methods of applying reinforcement learning technology to realize unmanned aerial vehicle route planning, but these methods have certain limitations. Reinforcement learning method needs to collect experience first, and train neural network based on these experience, so as to realize effective route planning. However, if the environment samples collected by reinforcement learning during training are greatly different from the environment during application, the reinforcement learning method cannot make effective response. In other words, the samples collected in the training stage cannot cover all types of environment, so it is impossible to realize the generalization application in different environments. On the other hand, most unmanned aerial vehicle reinforcement learning route planning methods depend on the dynamics parameters of unmanned aerial vehicles, and it is difficult for the same method to be used on two unmanned aerial vehicles with different sizes and structures.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a reinforcement learning unmanned aerial vehicle route planning method and system based on a safe flight corridor, aiming at solving the problem that the existing unmanned aerial vehicle route planning method is difficult to realize the generalization application in different environments.

[0006] The technical solution adopted by the present application to solve the problem is as follows:

[0007] In a first aspect, the present application provides a reinforcement learning unmanned aerial vehicle route planning method based on a safe flight corridor, which comprises:

[0008] Obtaining a grid map and a starting point information of a target scene, and generating a safe flight corridor according to the grid map and the starting point information; the grid map is used to reflect the obstacle distribution; the safe flight corridor is used to reflect the safe flight area;

[0009] a plurality of initial B-spline control points are calculated based on observation information of the UAV; the observation information comprises position, velocity and acceleration of the UAV; the B-spline control points are key parameters for determining the shape of the B-spline curve;

[0010] a plurality of new B-spline control points are iteratively generated based on the initial B-spline control points, key features extracted from the safe flight corridor and a time variable through a reinforcement learning algorithm;

[0011] a B-spline path of the UAV is generated according to all the B-spline control points.

[0012] In a second aspect, the embodiments of the present application further provide a reinforcement learning UAV path planning system based on a safe flight corridor, which comprises:

[0013] a map analysis module, configured to acquire a grid map and start point information of a target scene, and generate a safe flight corridor based on the grid map and the start point information; the grid map is used to reflect the distribution of obstacles; the safe flight corridor is used to reflect a safe flight area;

[0014] an initial calculation module, configured to calculate a plurality of initial B-spline control points based on observation information of the UAV; the observation information comprises position, velocity and acceleration of the UAV; the B-spline control points are key parameters for determining the shape of the B-spline curve;

[0015] a reinforcement learning module, configured to iteratively generate a plurality of new B-spline control points based on the initial B-spline control points, key features extracted from the safe flight corridor and a time variable through a reinforcement learning algorithm;

[0016] a path output module, configured to generate a B-spline path of the UAV according to all the B-spline control points.

[0017] In a third aspect, the embodiments of the present application further provide a terminal, which comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the reinforcement learning UAV path planning method based on a safe flight corridor as described above; the processors are configured to execute the programs.

[0018] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a plurality of instructions; the instructions are adapted to be loaded and executed by a processor to implement the steps of the reinforcement learning UAV path planning method based on a safe flight corridor as described above.

[0019] The beneficial effects of the present application: the embodiment of the present application generates a safe flight corridor through the grid map and the starting point information of the target scene; calculates the initial B-spline control points through the observation information of the unmanned aerial vehicle; generates new B-spline control points through the reinforcement learning algorithm, based on the initial B-spline control points, the key features extracted based on the safe flight corridor and the time variable; and generates the B-spline path of the unmanned aerial vehicle according to all the B-spline control points. The present application abstracts the feasible flight area in a complex environment into a continuous safe flight corridor. Since the safe flight corridor hides the environmental features, the experience samples in different environments are basically similar, and training in one environment can realize the universal application in different complex environments. The reinforcement learning algorithm is used to iteratively generate B-spline control points, which can adapt to environmental changes in real time, improve the adaptability to complex environments, and make the generated B-spline path more in line with the actual flight requirements of the unmanned aerial vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a flowchart of the reinforcement learning unmanned aerial vehicle path planning method based on a safe flight corridor provided by the embodiment of the present application.

[0022] Figure 2 is a whole schematic diagram of the reinforcement learning unmanned aerial vehicle path planning system based on a safe flight corridor provided by the embodiment of the present application.

[0023] Figure 3 is a schematic diagram of a safe flight corridor provided by the embodiment of the present application.

[0024] Figure 4 is a schematic diagram of a simulation training environment of an unmanned aerial vehicle provided by the embodiment of the present application.

[0025] Figure 5 is an outdoor path planning effect diagram provided by the embodiment of the present application.

[0026] Figure 6 is an indoor path planning effect diagram provided by the embodiment of the present application.

[0027] Figure 7 is a basic module schematic diagram of the reinforcement learning unmanned aerial vehicle path planning system based on a safe flight corridor provided by the embodiment of the present application.

[0028] Figure 8is a principle block diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] The application discloses a reinforcement learning unmanned aerial vehicle route planning method and system based on a safe flight corridor, and the purpose, technical scheme and effects of the application are more clear and definite. The application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0030] Those skilled in the art can understand that the singular forms "a," "an," and "the" used herein include plural forms unless specifically stated otherwise. It should be further understood that the use of the term "includes" in the specification of the application means that the stated features, integers, steps, operations, elements, and / or components are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0031] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0032] In view of the above defects of the prior art, the present application provides a reinforcement learning UAV route planning method based on a safe flight corridor, which acquires a grid map of a target scene and starting point information, generates a safe flight corridor according to the grid map and the starting point information; the grid map is used to reflect the obstacle distribution situation; the safe flight corridor is used to reflect the safe flight region; a plurality of initial B-spline control points are calculated from observation information of a UAV; the observation information includes the position, speed and acceleration of the UAV; the B-spline control points are key parameters for determining the shape of a B-spline curve; a new B-spline control point is iteratively generated based on the initial B-spline control points, key features extracted from the safe flight corridor and a time variable by a reinforcement learning algorithm; and a B-spline path of the UAV is generated according to all the B-spline control points. First, the present application generates a safe flight corridor from a grid map, and abstracts the feasible flight region in a complex environment into a continuous safe flight corridor. Since the safe flight corridor hides the environmental features, the experience samples under different environments are basically similar, and training in one environment can achieve universal application in different complex environments. Second, the present application takes the initial control points, key features of the safe flight corridor and the time variable as inputs, iteratively generates B-spline control points by using the self-optimizing characteristics of reinforcement learning, and quickly plans a smooth B-spline UAV route. The method not only integrates the constraints of prior environmental information, but also adapts to environmental changes through real-time iteration, improves the adaptability to complex environments, and makes the generated path more in line with the actual flight requirements of the UAV.

[0033] As shown in Figure 1 the method specifically comprises the following steps:

[0034] Step S100, a grid map of a target scene and starting point information are acquired, and a safe flight corridor is generated according to the grid map and the starting point information; the grid map is used to reflect the obstacle distribution situation; the safe flight corridor is used to reflect the safe flight region.

[0035] Specifically, the route planning method of the present embodiment is divided into two parts: navigation and path planning. First, for the navigation part, this part needs to rely on a grid map, which includes the obstacle information around the UAV and can reflect the obstacle distribution situation. Second, a safe flight corridor is generated based on the grid map and starting point information, which can reflect the safe flight region around the UAV.

[0036] For example, based on the data collected by sensors such as a depth camera or a laser radar carried by the UAV, a grid map is generated by combining a simultaneous localization and mapping algorithm. The grid map is used as input data for UAV navigation, and the start point and the end point of the UAV navigation are known. In actual application scenarios, a three-dimensional map can be used to construct a grid map: the three-dimensional grid map uses a certain resolution (generally 0.1 meters to 0.5 meters) to represent the three-dimensional space in pixel blocks, and if there is an obstacle in a pixel block, the pixel block is marked as occupied. For example, the three-dimensional space is divided into small blocks similar to pixel blocks, and if a small block contains an obstacle, the value of the small block is set to 1 (i.e., marked as occupied), otherwise, the value is 0.

[0037] In an implementation manner, the safe flight corridor is generated according to the grid map and the start point information, including:

[0038] The obstacle avoidance reference trajectory is generated based on the grid map and the start point information by using a preset heuristic search algorithm.

[0039] The safe flight corridor is generated according to the grid map and the obstacle avoidance reference trajectory. The safe flight corridor is used to reflect the safe flight area on both sides of the obstacle avoidance reference trajectory.

[0040] Specifically, first, the start point and the end point are marked in the grid map. Then, a preset heuristic search algorithm (for example, an A* heuristic search algorithm or a local three-dimensional A* search algorithm) is used to find a feasible path from the start point to the end point, that is, to obtain the obstacle avoidance reference trajectory. The feasible path can be a polyline composed of a series of points on the grid map and line segments between the points. Finally, the safe flight corridor is defined based on the obstacle avoidance reference trajectory, which is used to indicate the safe flight area on both sides of the obstacle avoidance reference trajectory.

[0041] For example, in the navigation stage, first, the start point (generally the current position of the UAV) and the end point of the UAV route planning are located on the grid map, and a straight line from the start point to the end point is used as an initial reference trajectory. The initial reference trajectory can have multiple intersecting straight lines with obstacles (i.e., the parts of the initial reference trajectory that collide with the obstacles). For each intersecting straight line, the start point and the end point of the intersecting straight line are located, a preset heuristic search algorithm is used to find an obstacle avoidance trajectory from the local start point to the end point, and the obstacle avoidance trajectory is used to replace the intersecting straight line on the global initial reference trajectory. Until all the intersecting straight lines are replaced by the obstacle avoidance trajectories, a global obstacle avoidance reference trajectory is obtained, and a safe flight corridor is generated around the obstacle avoidance reference trajectory.

[0042] In an implementation manner, the safe flight corridor is generated according to the grid map and the obstacle avoidance reference trajectory, including:

[0043] Dividing the obstacle avoidance reference trajectory into a plurality of line segments; the length of each line segment is determined based on the resolution of the grid map;

[0044] For each side of each line segment, determine a safety distance based on the obstacle closest to that side, and construct a sub-safe flight corridor on that side based on the safety distance;

[0045] A safe flight corridor is generated based on all the sub-safe flight corridors.

[0046] Specifically, the global obstacle avoidance reference trajectory is first divided into a series of equal-length line segments, with the length of each line segment equal to the resolution of the grid map. The obstacle avoidance reference trajectory can be represented as a broken line. For each line segment in the broken line, the safety area around the line segment is calculated based on the distance to the closest obstacles on both sides of the line segment, thus obtaining the sub-safe flight corridor of the line segment. The sub-safe flight corridors of each line segment are combined to obtain the complete safe flight corridor. The safe flight corridor can be simplified by a series of feature data and used as input data for the subsequent reinforcement learning neural network.

[0047] For example, the global obstacle avoidance reference trajectory can be composed of k trajectory points and k-1 line segments, expressed as u = <u0→u1→...→u k >. Among them, u represents the set of trajectory points, u0 represents the 0th trajectory point (i.e., the starting point); u1 represents the first trajectory point, u k Indicates the last track point (i.e., the end point). Each line segment is represented by L i = i-1 →u i >, L i In the calculation of the safe flight corridor, only the horizontal information of the obstacle avoidance reference trajectory can be considered. For example, the i-th trajectory point u i =[u i ,u i,y ] T ,i∈[0,k], represents the i-th trajectory point u i Contains only the x-axis position u i,x and the y-axis position u i,y , T represents matrix transpose, u i Set to a column vector to facilitate matrix multiplication operations.

[0048] Based on the obstacle avoidance reference trajectory represented by trajectory points and line segments, a sub-safe flight corridor is generated around each line segment based on the distance of the obstacle closest to the obstacle avoidance reference trajectory. All sub-safe flight corridors are merged to obtain the global safe flight corridor. i ​First, the line segment is located in the grid map, and then the nearest obstacle is searched on the left and right side of the line segment respectively. The distance from the obstacle to the line segment is taken as the safety distance on the left side of the line segment or the safety distance on the right side Each line segment peripheral sub-safety flight corridor is divided into a left safety flight corridor and a right safety flight corridor.

[0049] The left safety flight corridor is defined as follows:

[0050]

[0051] where p represents the projection of the unmanned aerial vehicle position on the horizontal plane; dist(p, L i ) represents the Euclidean distance between p and the A* reference trajectory segment L i on the horizontal plane; n i represents the normal vector perpendicular to L i , n i = [u i,y-1 -u i,y u i,x-1 -u i,x ] T / ||L i ||2, and ||L i ||2 represents the length of L i . The mathematical meaning of the above formula is that all regions located on the left side of L i and having a distance not more than from L i , represents the left side safety distance of the line segment.

[0052] Similarly, the right side safety flight corridor and the L i peripheral sub-safety flight corridor SFC i composed of the left and right sides can be defined as follows:

[0053]

[0054] The global safety flight corridor SFC is composed of the sub-safety flight corridors SFC i around each line segment L i , i∈[0, k]:

[0055]

[0056] where represents the union set from SFC i to SFC k .

[0057] AsFigure 3 As shown, there is a sub-safe flight corridor. The lines therein correspond to the obstacle avoidance reference trajectory, the dark gray is the obstacle, and the light gray is the obstacle inflation layer, i.e., the obstacle is enlarged to a certain size to further avoid collision.

[0058] In step S200, initial B-spline control points are calculated based on observation information of the UAV; the observation information includes position, velocity and acceleration of the UAV; and the B-spline control points are key parameters for determining the shape of the B-spline curve.

[0059] Specifically, according to the position, velocity and acceleration of the UAV at the beginning of the route planning, a set of B-spline control points (for example, 3 B-spline control points) can be reversely mapped and taken as the initial state of the UAV route planning. The B-spline control points refer to the control points of the B-spline curve, and a smooth and continuous B-spline curve can be generated through the B-spline control points. Since the B-spline curve has good smoothness, its application in the route planning task of the UAV can make the generated path more in line with the actual flight requirements of the UAV.

[0060] In step S300, new B-spline control points are iteratively generated based on the initial B-spline control points, key features extracted from the safe flight corridor and a time variable through a reinforcement learning algorithm.

[0061] In step S400, a B-spline path of the UAV is generated according to all the B-spline control points.

[0062] Specifically, the neural network is used in the embodiment to generate B-spline control points in an end-to-end manner. Then, the B-spline control points are combined to obtain a complete B-spline path of the UAV that is fused with the safe flight corridor. The planned route of the UAV in the embodiment is represented by a three-dimensional 3-order B-spline. The three-dimensional 3-order B-spline is composed of a series of B-spline control points, and each control point includes three-dimensional position information of x, y and z axes. The 3 B-spline control points can be uniquely mapped to a set of UAV dynamics states including position, velocity and acceleration, and the 4 B-spline control points can be mapped to a small section of the UAV curve route. The 3 B-spline control points are combined with the key features of the safe flight corridor, and a time variable representing the time from the start of the route planning to the current time is added to form the state space of the reinforcement learning, i.e., all variables that need to be input to the neural network during the route planning. Through the reinforcement learning algorithm, new B-spline control points are constantly output based on the safe flight corridor information and the state information of the UAV itself, so as to obtain the B-spline path.

[0063] In an implementation, the reinforcement learning algorithm comprises a policy network; through the reinforcement learning algorithm, based on the initial several B-spline control points, the key features extracted based on the safe flight corridor, and the time variable, new B-spline control points are iteratively generated, comprising:

[0064] State information is generated according to the initial several B-spline control points, the key features, and the time variable;

[0065] The state information is input into the policy network to obtain action information, and new B-spline control points are generated according to the action information;

[0066] It is judged whether the path planning termination condition is reached at present, if not, the latest several B-spline control points are selected, and the state information is updated;

[0067] The step of inputting the state information into the policy network to obtain action information, and generating new B-spline control points according to the action information is continuously executed until the path planning termination condition is reached.

[0068] Specifically, the reinforcement learning neural network responsible for outputting B-spline control points is called a policy network. The policy network receives state information including several initial B-spline control points (for example, 3 B-spline control points, and the B-spline control points have a time sequence from front to back), safe flight corridor states, and time variables, and generates a subsequent B-spline control point. At the beginning of path planning, this embodiment will maintain a B-spline control point storage, which has 3 B-spline control points that are inversely mapped from the position, speed, and acceleration of the unmanned aerial vehicle. After the policy network generates a new B-spline control point each time, the B-spline control point is added to the front end of the storage, and the 3 B-spline control points at the front end of the storage are taken as new state information and continuously input into the policy network to generate a new B-spline control point and add it to the front end of the storage. This cycle continues, and after several steps (for example, 10 steps), a smooth unmanned aerial vehicle B-spline path lasting for several seconds can be generated, which is taken as the output result of the unmanned aerial vehicle route planning, and the route planning of the unmanned aerial vehicle is completed.

[0069] Further, when reinforcement learning is used for unmanned aerial vehicle path planning, key features of the global safe flight corridor need to be taken as input quantities of the policy network. The key features of the global safe flight corridor are composed of the track points of the obstacle avoidance reference track and the distances of the nearest obstacles on both sides of each line segment in the obstacle avoidance reference track, and the safe flight corridor is defined by these key features. When there are k+1 points (and k line segments between the points) in the obstacle avoidance reference track, the corresponding safe flight corridor is determined by 3(k+1)+2k=5k+3 features. Since the length of the obstacle avoidance reference track is not determined and the number of input features of the policy network is determined, a segment of the global safe flight corridor can be taken as the input data of the policy network. For example, the length of the segment can be set to 14 line segments corresponding to 15 obstacle avoidance reference track points, and the total number of features corresponding to the segment is 73. The starting point u of the segment can be set to the point closest to the current position of the unmanned aerial vehicle among all the obstacle avoidance reference track points. i i∈[0,k] can be set to the point closest to the current position of the unmanned aerial vehicle among all the obstacle avoidance reference track points.

[0070] In an implementation manner, to reduce the number of features of the safe flight corridor, the safe flight corridor defined above does not contain z-axis information. In this embodiment, the height of the safe flight corridor on the z-axis is a determined value h, including a space with a height of h / 2 above the obstacle avoidance reference track and a space with a height of h / 2 below the obstacle avoidance reference track. Since this value is a determined constant that can be learned by the policy network, it can not be taken as a feature input to the policy network.

[0071] For example, the working principle of the policy network in this embodiment is as follows:

[0072] The policy network takes state information defined in the state space as input, including safe flight corridor features, unmanned aerial vehicle dynamics features, and time; and outputs action information defined in the action space.

[0073] First, the construction process of the state space includes:

[0074] When a reinforcement learning method is used to plan a B-spline track of an unmanned aerial vehicle, the unmanned aerial vehicle track planning problem needs to be converted into a reinforcement learning problem. The essence of the reinforcement learning problem is a Markov decision process, which is defined by a four-tuple:

[0075] {S,A,P,r};

[0076] Wherein, S is a state space, A is an action space, p(s t+1 |s t ,a t ) is a state transition mapping, s t represents state information at the t-th step, a t represents action information at the t-th step. r t =r(s t+1) is the reward function. The following sections will construct each element of the quadruple in turn.

[0077] State space {S: s t ∈ R 83} represents the dimension of the policy network input, which includes the 73-dimensional safety flight corridor features defined above, 9-dimensional UAV dynamics features, and a one-dimensional time quantity:

[0078] s t : = {SFC, p t-3 , p t-2 , p t-1 , τ} ;

[0079] where p t-1 represents the B-spline control point at the t-1 step; τ represents time. The 9-dimensional UAV dynamics features are represented by 3 three-dimensional three-order B-spline control points.

[0080] The definition of the three-order B-spline is as follows:

[0081]

[0082] where,

[0083] τ t = tΔτ;

[0084] where t is the step number of reinforcement learning, which is also the serial number of the B-spline control point p t . In reinforcement learning, a B-spline control point is generated at each step, so the above two variables can be represented by the same symbol; τ represents time; τ t is the time corresponding to the tth B-spline control point; p(τ) is the position of the B-spline curve at time τ; B t,k is the k-order B-spline basis function with serial number t; Δτ is the time interval between two adjacent B-spline control points. According to the definition of the three-order B-spline, at time τ t , the UAV B-spline trajectory is only affected by three B-spline control points. Therefore, the task of reinforcement learning at any step is set as follows: at time tΔτ, with the safety flight corridor information and the three three-dimensional B-spline control points p t-3 , p t-2 , p t-1 at times (t-1)Δτ, (t-2)Δτ, and (t-3)Δτ as information input, the B-spline control point at time tΔτ is output. In the next step, reinforcement learning will add the B-spline control point at time tΔτ to the input state according to the first-in-first-out principle, and then output the B-spline control point at time (t+1)Δτ. This cycle is repeated, and after several steps, a complete B-spline trajectory will be generated.

[0085] Further, at the beginning of the planning (i.e. the step number t = 0 of reinforcement learning), three known B-spline control points can be derived from the UAV position, velocity, acceleration observed by the UAV sensors, and the calculation formula is as follows:

[0086]

[0087] a0= a(0);

[0088] Wherein, v t ,a t ,j t are the velocity control point, acceleration control point and jerk control point respectively, which can constitute the velocity p(t), acceleration v(t) and jerk a(t) of the UAV B-spline trajectory, wherein t represents the t step. According to the above formula, the first three control points p -2 , p -1 and p0of the UAV B-spline path can be calculated according to p(0), v(0) and a(0).

[0089] The construction process of the action space includes:

[0090] The action space A can be defined as a three-dimensional vector a t = [a x,t a y,t a z,t ] T ∈ [-1, 1] 3 , which is not a position control point, wherein a x,t a y,t a z,t represent the acceleration components in the x-axis, y-axis and z-axis directions at time t respectively; T represents the matrix transpose operation. The acceleration control point needs to be transformed into a position control point through a series of transformations, and the transformation formula is as follows:

[0091]

[0092] a z,t = a max a z ;

[0093] Wherein, a x,t , a y,t , a z,t are three variables representing the acceleration in three axes, and t represents the t step; a maxrepresents the maximum acceleration that needs to be set (which can be set according to the actual environment needs); ε is a small number close to 0, which is set to prevent zero division problems in the case of acceleration being 0. The above steps first convert the action output of the policy network into an acceleration control point a t = [a x,t a y,t a z,t ]. The above formula makes the maximum value of a t equal in different acceleration directions. a t After the following formula, it can be gradually converted into a speed control point v t = [v x,t ,v y,t ,v z,t ] and a position control point p t , t represents the t-th step:

[0094] v t = v t-1 +a t Δτ;

[0095]

[0096] p t = p t-1 +v t Δτ;

[0097] wherein v x,t ,v y,t ,v z,t three variables represent the speed on the three axes respectively; Δτ represents the time interval of two adjacent B-spline control points; v max represents the maximum speed. Since the state information s t+1 input by the policy network can be completely determined by p t and known p t-1 , p t-2 , the above action transformation also realizes the definition of the state transition mapping P(s t+1 |s t ,a t ) of reinforcement learning by outputting p t .

[0098] In an implementation mode, the reinforcement learning algorithm further comprises an evaluation network; and the training method of the policy network comprises:

[0099] obtaining training state information of the unmanned aerial vehicle by sampling a simulation training environment;

[0100] inputting the training state information into the policy network to obtain training action information;

[0101] input the training state information and the training action information into the evaluation network to obtain a state action value;

[0102] The policy network is updated in parameters through the state action value and a gradient descent method to train the policy network.

[0103] Specifically, the policy network needs to be trained before being applied to output an effective obstacle avoidance route. The embodiment designs a dedicated simulation training system for the policy network, such as a simulation training environment running on an ubuntu+ROS system. Figure 4 The simulation training environment is for a UAV, wherein the dark block-shaped objects are obstacles, and different lines represent different obstacle avoidance trajectories generated. The policy network of reinforcement learning is trained in the simulation training environment to obtain a reinforcement learning B-spline path planning strategy that can be directly applied to a real UAV. Further, in the simulation training environment, the embodiment adopts a continuous reinforcement learning algorithm combining a policy network and an evaluation network, such as a soft policy-evaluation algorithm or a double-delay deep deterministic policy gradient algorithm. The algorithm has two neural networks, one being a policy network and the other being an evaluation network. The evaluation network adopts a time difference method to fit a state action value (i.e., a Q function value), and updates the policy network through a gradient descent method (or a policy gradient method) according to the state action value as an evaluation standard. After the training in the simulation training environment, the UAV can directly apply the trained policy network to the route planning of the real UAV without further training in the real environment, and achieve the obstacle avoidance effect in a dense environment.

[0104] For example, the embodiment needs to apply three neural networks: a policy network π(α t |s t ;θ π ), an evaluation network Q(s t ,α t ;θ Q ), and a target evaluation network Q(s t ,α t ;θ Q ′), wherein θ π represents all neural network parameters in the policy network; θ Q represents all neural network parameters in the evaluation network; and θ Q ′ represents all neural network parameters in the target evaluation network. The reinforcement learning path planning problem can be trained by applying different reinforcement learning algorithms, including a deep deterministic policy gradient (DDPG), a double-delay deep deterministic policy gradient (TD3), and a soft policy-evaluation (SAC). Different algorithms have different neural network structures.

[0105] Taking DDPG as an example, the policy network takes the state information defined by the state space as input, including the safe flight corridor characteristics, the unmanned aerial vehicle dynamics characteristics and the time; and outputs the action information defined by the action space. The policy network has 83 inputs and 3 outputs, and the network structure is a multi-layer fully connected neural network, which has two hidden layers, each of which has 256 neurons. The evaluation network takes the state information and the action information as input, and outputs the state-action value corresponding to the state information and the action information, i.e. Q(s t ,α t ;θ Q ). The evaluation network has 83+3-dimensional input and 1-dimensional output, and is also a multi-layer fully connected neural network with two 256-unit hidden layers. The target evaluation network is a copy of the evaluation network and has the same structure. The state-action value output by the evaluation network is combined with the gradient descent method to update the policy network, so as to train the policy network.

[0106] In an implementation manner, the evaluation network and the policy network adopt an alternating training manner, and a training method of the evaluation network comprises the following steps.

[0107] inputting the training state information into the policy network without exploration noise to obtain an obstacle avoidance trajectory without exploration noise;

[0108] inputting the training state information into the policy network with exploration noise to obtain an obstacle avoidance trajectory with exploration noise;

[0109] sending the obstacle avoidance trajectory without exploration noise to an actuator of a simulation system, and generating experience samples in an experience pool according to the obstacle avoidance trajectory with exploration noise and a reward function;

[0110] updating parameters of the evaluation network through the experience pool and a one-step time difference method, so as to train the evaluation network.

[0111] Generally, the policy network and the evaluation network are in an alternating training mode, the evaluation network is updated first, and then the policy network is updated. This embodiment can collect experience samples from a simulation training environment, store the experience samples in an experience pool, and replay the experience samples in the experience pool to realize offline reinforcement learning training of the evaluation network, and then update the policy network through the gradient descent method.

[0112] Specifically, in the simulation training phase, this embodiment designs an exploration strategy to improve the effectiveness of experience samples. During the reinforcement learning training process, setting a larger exploration rate can improve training efficiency, but it will also increase the probability of the drone colliding with obstacles in the simulation. The exploration method of this embodiment is: when planning the route, first plan a non-explored route, and make the drone execute this route in the simulation training environment; then, use the same set of state information and use a policy network with exploration noise to generate multiple drone routes. These routes will not be executed in the simulation training environment, but will be added to the experience pool as experience samples (or called training experience). This exploration method can increase the probability of the drone flying into dense environmental obstacles in the simulation training environment, thereby training a more effective policy network for executing drone route planning.

[0113] Furthermore, the policy network with exploration noise is an exploration strategy that decouples exploration from execution, so that the exploration process does not affect the execution process of the trajectory in the simulation training environment, thereby improving the efficiency of the reinforcement learning training process and the stability of the policy network obtained through training. The exploration strategy that decouples exploration from execution means that during the training phase, a certain amount of noise needs to be superimposed on the policy output of the policy network, so that the policy can explore more possible optimal actions through trial and error. Taking DDPG as an example, its policy π is fixed, that is, α t =π(α t |s t θ π ); During the exploration process of reinforcement learning training, a Gaussian white noise is added to π:

[0114] α t =π(α t |s t θ π )+π norm ,π norm ~N(0,σ);

[0115] Among them, σ represents the standard deviation of noise, which is generally set to 0.1; π norm represents the added Gaussian white noise; N represents the normal distribution.

[0116] This noise can enable reinforcement learning to explore more potential optimal actions, but it can also lead to instability during training. For example, in a simulation, π normTaking a larger value may affect the planned B-spline path of the strategy, resulting in collision between the UAV and the obstacle, and the simulation terminates. In the simulation training environment designed in this embodiment, the instability brought by such exploration makes it difficult for the UAV to reach the right side of the simulation training environment, thereby failing to explore the obstacle avoidance strategy in a dense environment. Therefore, when planning the B-spline path of the UAV each time, a set of states is first sampled from the environment, and a strategy network without exploration noise is used to plan an obstacle avoidance trajectory, i.e., to obtain an obstacle avoidance trajectory without exploration noise, and the trajectory is output to the actuator of the simulation system, so that the UAV flies along the trajectory. Subsequently, using the same state information, a strategy network with exploration noise is used to plan multiple obstacle avoidance trajectories, i.e., to obtain multiple obstacle avoidance trajectories with exploration noise. These obstacle avoidance trajectories with exploration noise are not output to the actuator of the simulation system, but are stored in an experience pool. All experience samples in the experience pool are stored in the form of single-step state transition, i.e., each experience sample includes [s t ,a t ,r(t),s t+1 ,D(t)], wherein s t represents the state information of the t-th step, a t represents the action information of the t-th step, r(t) represents the reward value calculated by the reward function of the t-th step, and D(t) represents the termination value calculated by the termination function of the t-th step. Using the experience samples in the experience pool, the evaluation network is updated in a single-step time difference manner. Taking the DDPG algorithm as an example, the evaluation network is updated by minimizing the following loss function Ω Q (t) :

[0117] Ω Q (t) = [Q(s t , a t ; θ Q ) - r(t) - γQ(s t+1 , π(a t+1 | s t+1 ; θ π ) ; θ Q '] 2 ;

[0118] wherein γ is the discount rate of the reward function.

[0119] After updating the evaluation network using the above formula, the strategy network of the reinforcement learning path planning can be updated in a gradient descent manner, wherein β is the learning rate:

[0120]

[0121] wherein β represents the learning rate.

[0122] After updating the parameters of the evaluation network and the strategy network, the parameters of the target evaluation network are updated:

[0123] θ Q ′←0.99θ Q ′+0.01θ Q ;

[0124] The above are all steps of upgrading the neural network in the reinforcement learning training process. While continuously collecting new experience samples, the above steps are cycled, and the reinforcement learning training of the unmanned aerial vehicle B-spline planning strategy can be completed. After the training is completed, the strategy network can be directly deployed on the real unmanned aerial vehicle to perform unmanned aerial vehicle reinforcement learning B-spline path planning.

[0125] In an implementation manner, a method for constructing a reward function of an unmanned aerial vehicle reinforcement learning path planning problem comprises:

[0126] The reward function is constructed through a flight speed reward item, a collision penalty item, and a planning success reward item.

[0127] Specifically, the reward function mainly has two functions. The core function is to score the action in the training process of the strategy network, and to indicate the good and bad of the action, thereby effectively guiding the path planning strategy. The secondary function is that when the reinforcement learning route planner is deployed on the real machine, in a few cases, the path planned by the strategy network may be unreasonable. At this time, the reward function can score the output path. If the path is not ideal, the output result is discarded, and the path is re-planned.

[0128] The design goal of the reward function is to make the path planned by the strategy network meet three standards: not leaving the safe flight corridor range (obstacle avoidance), the jerk of the trajectory cannot be too high (dynamic constraint), and the flight speed is as fast as possible. Considering the above factors, the reward function r(t) can be expressed as:

[0129] r(t)=k v r v (t)+k p r p (t)+k s r s (t);

[0130] Among them, the three sub-items of the reward function are: the flight speed reward r v (t), the corresponding gain coefficient is k v ; the collision penalty item r p (t), the corresponding gain coefficient is k p ; the planning success reward r s (t), and the corresponding gain coefficient is k s .

[0131] In the calculation of the reward function, firstly, the closest obstacle avoidance reference trajectory segment to the current position of the UAV is found, and the position of this segment in the obstacle avoidance reference trajectory is an important reference for calculating the forward distance of the UAV. For the i-th segment of the obstacle avoidance reference trajectory at the t-th step of reinforcement learning (corresponding to the physical time τ t = tΔτ), the distance between it and the UAV is calculated as The distance can be calculated using the following formula:

[0132]

[0133] Wherein, each physical quantity has been defined in the foregoing part. The minimum The segment L i in the obstacle avoidance reference trajectory corresponding to it is the closest segment to the UAV. The size of the flight speed reward r v (t) depends on L i The position of L i in the global obstacle avoidance reference trajectory is that the UAV is closest to L i-3 at time t, and closest to L v at time t-1. The value of r v (t) is 3.

[0134] r v (t) can be expressed by the formula:

[0135]

[0136] Wherein, B represents the B-spline, and the position of the UAV at time t is approximated by the position of the B-spline path of the UAV at time t; argmin represents the serial number corresponding to the minimum value.

[0137] Further, the present embodiment does not separately set a penalty term for the UAV dynamics constraint, but combines it with r v (t). The present embodiment mainly constrains the jerk, and sets the maximum value of the jerk as j max When the value of the jerk j(τ t ) is greater than j max / 2, the reward value obtained will be discounted on r v (t):

[0138]

[0139] The collision penalty term r p (t) mainly depends on whether the UAV collides with the obstacle. After the path planning strategy network generates p t at the t-th step, the position of the UAV at τ t-1 to τ t can be calculated. From the time period τt-1 to τ t Uniformly sample 10 points in [0, τ], and calculate whether the UAV is in the safe flight corridor at these 10 time points. If it is not, then r p (t) is negative:

[0140]

[0141] wherein, represents the k-th sub-reward at the k-th time point, represents the product of the ten sub-rewards.

[0142] Further, the success reward item gives a positive reward only when the B-spline path of the UAV reaches the end point of the obstacle avoidance reference trajectory, i.e., the last line segment of the obstacle avoidance reference trajectory is closest to the current position of the UAV:

[0143]

[0144] wherein n is the total number of line segments of the obstacle avoidance reference trajectory. Thus, the construction of the reward function is completed.

[0145] Further, while constructing the reward function, the embodiment also constructs a termination function D(t), which is affected by three factors. When the jerk exceeds the upper limit, or the UAV exceeds the safe flight corridor, or the UAV reaches the end point, the current UAV path planning process is terminated:

[0146]

[0147] The reward function r(t) is only used in the training process of reinforcement learning, while D(t) is applied to both the training process and the deployment on the real UAV. When D(t) = 1, the current path planning process is terminated, which can avoid planning a failed path. Thus, the Markov decision process corresponding to the UAV reinforcement learning path planning problem is constructed.

[0148] In an implementation manner, the simulation training environment includes a plurality of simulation obstacle walls; one side of the simulation training environment is taken as the take-off side of the UAV, and the other side is taken as the target side of the UAV, and the spacing of the obstacle walls in the simulation training environment decreases in turn from the take-off side to the target side.

[0149] Specifically, the simulation training environment and the generated experience sample can effectively guide the optimization of the path planning strategy and accelerate the training process. The simulation training environment is composed of multiple rows of simulation obstacle walls, with the left side as the take-off side and the right side as the target side. For example, the UAV starts from the left side of the simulation training environment and aims to fly to the right side of the simulation training environment. The distance between the obstacle walls on the left side is large, and the distance between the obstacle walls decreases from left to right. At the beginning of the training, the simulation training environment is relatively easy for path planning, which can help the reinforcement learning algorithm quickly train an initial effective strategy. After obtaining the initial effective strategy, the path planning based on the strategy network can effectively support the simulation UAV to fly to the right side with high obstacle density, thereby training the obstacle avoidance strategy in the dense environment with high difficulty.

[0150] For example, the simulation training environment is a 70m*70m three-dimensional scene, and the initial position of the UAV is on the left side, and the target is to fly to the right side of the simulation training environment. A series of obstacle walls are arranged from left to right in the simulation training environment, and the width of the obstacle walls is 1.5m to 2.5m, the thickness is 0.5m to 1m, and the distance between the walls decreases from left to right. During the simulation process, the UAV flies along the B-spline path planned by the reinforcement learning according to the ideal state. At the beginning of the training, the UAV does not form an effective obstacle avoidance strategy, and the relatively wide spacing on the left side makes it easier for the UAV to explore an effective obstacle avoidance strategy, which can effectively accelerate the training of the initial strategy. After a period of training, the UAV obtains an initial obstacle avoidance strategy, which can support it to fly to the right side of the simulation training environment. The smaller spacing on the right side can enable the UAV to effectively learn the obstacle avoidance strategy in the dense obstacle environment.

[0151] In an implementation manner, the strategy network after training can be directly transferred and deployed to a real UAV. The hardware of the deployed UAV needs to meet the following conditions:

[0152] 1. The UAV needs to have a rotor structure, which can be a quadcopter, a hexacopter, an octocopter, etc., and has the ability to hover in the air;

[0153] 2. The UAV needs to be equipped with an on-board computing module and be able to communicate with the flight controller of the UAV. The present embodiment method has a small amount of demand for computing power, and the UAV should at least be equipped with a micro single-board computer carrying a quad-core Arm Cortex-A72 processor;

[0154] 3. The UAV needs to be equipped with environmental sensors to realize the detection and mapping of obstacles around itself, and the sensors that can be selected include depth cameras and laser radars, etc.;

[0155] 4. The battery and power kit of the UAV can enable the UAV to reach the maximum speed v max , acceleration amax jerk max The above three values can be arbitrarily adjusted before the reinforcement learning training begins, but cannot be adjusted after the training ends. If changes are needed, the path planning strategy needs to be retrained.

[0156] The software carried by the single-board computer on the deployed drone needs to meet the following conditions:

[0157] 1. The single-board computer needs to carry an Ubuntu + ROS development environment, as well as a simultaneous localization and mapping algorithm (SLAM algorithm), which can accurately locate the current position of the drone and establish a grid map based on surrounding obstacle information.

[0158] 2. The single-board computer also needs to carry a path tracking algorithm module, which can receive the B-spline flight path output by the strategy network and make the drone fly according to the path through a control algorithm.

[0159] After meeting the above conditions, the method of the embodiment can be deployed on a physical drone and realize path planning for the drone.

[0160] In actual use, a computer is needed to remotely control the drone, and the two are connected through a wireless network. The ROS software system relied on by the method of the embodiment can display the path planned by the system and the obstacles around the drone in real time, and support selecting a target point on the map.

[0161] In actual use, the drone is first connected to the computer, and then takes off from the ground. After taking off, the drone will map the surrounding obstacles and display the map. On the map, the drone will plan a B-spline path from the current position to the target point.

[0162] For example, in actual use, the key steps to generate a complete B-spline path are as follows:

[0163] Step 1. Based on the grid map, a local three-dimensional A* search algorithm is used to generate an obstacle avoidance reference trajectory;

[0164] Step 2. Based on the grid map and the obstacle avoidance reference trajectory, a safe flight corridor is generated;

[0165] Step 3. Based on the drone position, speed, and acceleration provided by the sensor, three initial B-spline control points are calculated, which are combined with the safe flight corridor and the time variable to form the state information input to the strategy network, and a storage sequence including the three initial B-spline control points is constructed;

[0166] Step 4. The state information is input to the strategy network, and the strategy network outputs action information; the action information is converted into a B-spline position control point, which is added to the storage sequence;

[0167] Step 5. Call the termination function D(t) to verify whether the B-spline path planning process is terminated, if not, update the state space of reinforcement learning, and return to step 4 to continue generating B-spline control points;

[0168] Step 6. After the planning process is terminated, all B-spline control points in the storage sequence are connected to form a complete B-spline path and output.

[0169] After reaching the target point, stop the B-spline path planning, and then select a new target point on the map, and the UAV will continue to plan a B-spline path to the new target point. Figure 5 and Figure 6 The effect of the method of the embodiment on path planning in an actual physical environment is shown, Figure 5 which shows the effect of outdoor path planning, Figure 6 which shows the effect of indoor path planning. The method of the embodiment can be flexibly applied in various environments, and supports a quadcopter UAV flying at a speed of 12 m / s with obstacle avoidance, and is compatible with different UAV structures, including quadcopter, hexacopter, octocopter, and fixed-wing UAVs.

[0170] Based on the above embodiment, the application further provides a reinforcement learning UAV route planning system based on a safe flight corridor, as shown in Figure 7 The system comprises:

[0171] a map analysis module 01 for obtaining a grid map of a target scene and starting point information, and generating a safe flight corridor according to the grid map and the starting point information; the grid map is used to reflect the distribution of obstacles; and the safe flight corridor is used to reflect the safe flight area;

[0172] an initial solution module 02 for solving an initial number of B-spline control points through observation information of a UAV; the observation information includes the position, speed and acceleration of the UAV; and the B-spline control point is a key parameter for determining the shape of the B-spline curve;

[0173] a reinforcement learning module 03 for iteratively generating new B-spline control points based on the initial number of B-spline control points, key features extracted based on the safe flight corridor, and a time variable through a reinforcement learning algorithm;

[0174] a path output module 04 for generating a B-spline path of the UAV according to all the B-spline control points.

[0175] Further, in an actual application scenario, a complete system overall diagram as shown in Figure 2 may be used.

[0176] Based on the above embodiments, the application further provides a terminal, a principle block diagram of which can be shown in Figure 8 The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the safe flight corridor-based reinforcement learning UAV route planning method. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0177] Those skilled in the art can understand that Figure 8 The principle block diagram shown in the above embodiments is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0178] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, including instructions for performing the safe flight corridor-based reinforcement learning UAV route planning method.

[0179] Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above embodiments.

[0180] In summary, the application discloses a safe flight corridor-based reinforcement learning UAV route planning method and system, and relates to the technical field of route planning. First, the application generates a safe flight corridor through a grid map, and abstracts the feasible flight area in a complex environment into a continuous safe flight corridor. Since the safe flight corridor hides the environmental features, the experience samples in different environments are basically similar, and training in one environment can achieve universal application in different complex environments. Second, the application takes an initial control point, key features of the safe flight corridor, and a time variable as inputs, iteratively generates B-spline control points by using the self-optimization characteristics of reinforcement learning, and quickly plans a smooth B-spline UAV route. The method not only integrates the constraints of prior environmental information, but also adapts to environmental changes through real-time iteration, improves the adaptability to complex environments, and makes the generated path more consistent with the actual flight requirements of the UAV.

[0181] It is to be understood that the application is not limited to the examples described above, which can be modified or adapted in several ways by those skilled in the art without departing from the scope of the present application, as defined by the appended claims.

Claims

1. A reinforcement learning UAV route planning method based on safe flight corridor, characterized by: The method comprises: Obtaining a grid map and starting point information of a target scene, and generating a safe flight corridor based on the grid map and the starting point information; the grid map is used to reflect the distribution of obstacles; and the safe flight corridor is used to reflect the safe flight area; Calculating initial B-spline control points using observation information from the drone; the observation information includes the drone's position, velocity, and acceleration; the B-spline control points are key parameters for determining the shape of the B-spline curve; Iteratively generating new B-spline control points based on the initial B-spline control points, the key features extracted from the safe flight corridor, and a time variable through a reinforcement learning algorithm; Generate the B-spline path of the drone based on all the B-spline control points.

2. The reinforcement learning UAV route planning method based on safe flight corridor according to claim 1 is characterized in that: Generating a safe flight corridor according to the grid map and the starting point information includes: Generate an obstacle avoidance reference trajectory based on the grid map and the starting point information through a preset heuristic search algorithm; A safe flight corridor is generated according to the grid map and the obstacle avoidance reference trajectory; the safe flight corridor is used to reflect the safe flight areas on both sides of the obstacle avoidance reference trajectory.

3. The reinforcement learning UAV route planning method based on safe flight corridor according to claim 2 is characterized in that: Generating a safe flight corridor according to the grid map and the obstacle avoidance reference trajectory, including: Dividing the obstacle avoidance reference trajectory into a plurality of line segments; the length of each line segment is determined based on the resolution of the grid map; For each side of each line segment, determine a safety distance based on the obstacle closest to that side, and construct a sub-safe flight corridor on that side based on the safety distance; A safe flight corridor is generated based on all the sub-safe flight corridors.

4. The reinforcement learning UAV route planning method based on safe flight corridor according to claim 1 is characterized in that: The reinforcement learning algorithm includes a policy network; through the reinforcement learning algorithm, new B-spline control points are iteratively generated based on the initial number of B-spline control points, the key features extracted from the safe flight corridor, and the time variable, including: generating state information according to the initial plurality of B-spline control points, the key features, and the time variable; Inputting the state information into the strategy network to obtain action information, and generating new B-spline control points according to the action information; Determine whether the path planning termination condition is currently met. If not, select the latest several B-spline control points and update the state information; Continue to execute the steps of inputting the state information into the strategy network to obtain action information, and generating new B-spline control points according to the action information until the path planning termination condition is reached.

5. The reinforcement learning UAV route planning method based on safe flight corridor according to claim 4 is characterized in that: The reinforcement learning algorithm also includes an evaluation network; the training method of the policy network includes: The training status information of the UAV is obtained by sampling the simulation training environment; Inputting the training state information into the strategy network to obtain training action information; Inputting the training state information and the training action information into the evaluation network to obtain a state behavior value; The policy network is parameter updated by using the state behavior value and gradient descent method to train the policy network.

6. The reinforcement learning UAV route planning method based on safe flight corridor according to claim 5 is characterized in that: The evaluation network and the policy network are trained in an alternating manner. The training method of the evaluation network includes: Inputting the training state information into the policy network without exploration noise to obtain an obstacle avoidance trajectory without exploration noise; Inputting the training state information into the strategy network with exploration noise to obtain an obstacle avoidance trajectory with exploration noise; Sending the non-exploratory noise obstacle avoidance trajectory to an actuator of a simulation system, and generating experience samples in an experience pool according to the obstacle avoidance trajectory with exploration noise and a reward function; The evaluation network is trained by updating parameters of the evaluation network through the experience pool and the single-step temporal difference method.

7. The reinforcement learning UAV route planning method based on safe flight corridor according to claim 5 is characterized in that: The simulation training environment includes several rows of simulated obstacle walls; one side of the simulation training environment is used as the take-off side of the drone, and the other side is used as the target side of the drone, and the spacing of the obstacle walls in the simulation training environment decreases from the take-off side to the target side.

8. A reinforcement learning UAV route planning system based on safe flight corridors, characterized by: The system comprises: A map analysis module is configured to obtain a grid map and starting point information of a target scene, and generate a safe flight corridor based on the grid map and the starting point information; the grid map is configured to reflect the distribution of obstacles; and the safe flight corridor is configured to reflect a safe flight area; An initial solution module is used to solve several initial B-spline control points using observation information from the drone; the observation information includes: the position, velocity and acceleration of the drone; the B-spline control points are key parameters for determining the shape of the B-spline curve; A reinforcement learning module, configured to iteratively generate new B-spline control points based on the initial B-spline control points, the key features extracted from the safe flight corridor, and a time variable through a reinforcement learning algorithm; The path output module is used to generate a B-spline path of the UAV based on all the B-spline control points.

9. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program includes instructions for executing the reinforcement learning drone path planning method based on a safe flight corridor as described in any one of claims 1 to 7; and the processor is used to execute the program.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the reinforcement learning drone path planning method based on a safe flight corridor as described in any one of claims 1-7.

Images