End-to-end UAV autonomous navigation method based on differential theory

Abstract

The present invention discloses an end-to-end UAV autonomous navigation method based on differential theory, including: obtaining several depth maps and starting point information and end point information of trajectories, using a trajectory generation network based on motion primitives to generate several groups of flight corridors, constructing a constrained spatiotemporal optimization problem based on the lowest energy trajectory class, solving the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors; calculating the trajectory loss of each optimal trajectory, and using the optimal trajectory with the lowest trajectory loss as the trajectory required to be executed by the UAV, thereby realizing autonomous navigation of the UAV. This method makes the spatiotemporal optimization problem differentiable through differential theory, so that a neural network can be used to learn corridors in an end-to-end manner, extracting a safe guidance area from a depth image and reconstructing it into a geometric space constraint for trajectory optimization, and then combining the kinematic constraints specified by the user to efficiently and robustly converge (about 1 millisecond) to a high-quality trajectory.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle autonomous navigation, and in particular relates to an end-to-end unmanned aerial vehicle autonomous navigation method based on differential theory. Background Art

[0002] Drones have been widely used in various fields such as aerial photography, exploration, and search and rescue due to their compact and simple hardware structure combined with flexible and high maneuverability. As a key component to achieve these tasks, efficient and robust autonomous navigation modules have received widespread attention from both industry and academia. Traditionally, navigation modules use sensors such as depth cameras to perceive the environment, explicitly build occupancy maps, and calculate environmental representations that are conducive to motion planning, such as Euclidean signed distance fields. Subsequently, search-based or optimization-based motion planning algorithms are used on these maps to calculate the optimal trajectory considering the starting and final states and obstacle avoidance. Although this is intuitive from an engineering perspective, this modular decomposition framework inevitably introduces additional physical delays. In addition, this modular approach often leads to a lack of cohesion between sub-modules and requires engineers to perform a lot of manual parameter tuning. Recently, learning-based navigation has attracted widespread attention due to its effective integration of perception and planning modules. This end-to-end approach directly outputs trajectories from raw sensor data, bypassing explicit mapping. However, this approach strongly relies on the capabilities of neural networks, resulting in a risky black-box system that poses an obstacle to debugging and reduces the scalability and interpretability of the system. In addition, considering the limitations of the physical platform or specific mission scenarios, various custom constraints need to be imposed on the trajectory, such as the kinematic constraints of the robot. This poses a significant challenge to the network. Researchers often design more complex strategies to solve these constraints of the network, but these strategies may sometimes affect the optimality or fail to fully ensure the satisfaction of the constraints, thus affecting the completeness of the entire system.

[0003] Existing navigation methods are divided into classical methods and learning-based methods. For classical methods, optimization-based methods use gradient information to efficiently converge to feasible trajectories in continuous space, balancing quality and time. These methods usually require explicit environment modeling and manual safety constraint extraction through depth information, such as using ESDF to build safety constraints. However, building ESDF incurs additional computational costs and involves a trade-off between efficiency and accuracy. Although some planners avoid the construction of ESDF by iteratively generating safe guidance paths for obstacle avoidance gradients, they lack convergence guarantees and may fall into unsafe local minima in complex environments. Compared with classical methods, learning-based methods, as a promising method in local planning, eliminate the need for explicit mapping and reduce latency. Some methods use deep convolutional neural networks to learn flight trajectories from depth images, using human pilot trajectories as supervision. However, this method requires high-quality and large-scale datasets. Recently, some methods combine networks with numerical optimization. Some scholars use LSTM to learn the time distribution of piecewise polynomial trajectories to achieve optimal solutions under ideal conditions. However, this approach requires an offline convex decomposition of a known global map and safe areas, which makes it impractical for end-to-end local vision-based planners. Others have used neural networks to learn points for closed cubic spline interpolation directly from depth maps to approximate the original trajectory planning problem, but this lightweight approximation does not strictly guarantee compliance with dynamic constraints, which may lead to trajectories that are difficult to execute during high-speed flight and ultimately cause collisions. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention combines the advantages of neural networks and numerical optimization to disclose an end-to-end UAV autonomous navigation method based on differential theory. This method uses the network to extract depth information and model the mixed distribution of the optimal trajectory, extracts the safe space constraints required for optimization, and finally uses numerical optimization to plan the optimal space-time trajectory. This fundamentally ensures the feasibility of optimality and dynamic constraints while eliminating the delay caused by the perception and mapping module.

[0005] According to a first aspect of an embodiment of the present application, there is provided an end-to-end UAV autonomous navigation method based on differential theory, comprising:

[0006] Obtaining several depth maps and the starting point and end point information of the trajectory, using a trajectory generation network based on motion primitives to generate several groups of flight corridors, constructing a constrained spatiotemporal optimization problem based on the lowest energy trajectory class, and solving the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors;

[0007] The trajectory loss of each optimal trajectory is calculated, and the optimal trajectory with the lowest trajectory loss is used as the trajectory that the UAV needs to execute, thereby realizing autonomous navigation of the UAV.

[0008] Furthermore, before the depth map and the starting point information and the end point information of the trajectory are input into the trajectory generation network based on the motion primitive, the depth map and the starting point information and the end point information of the trajectory are preprocessed. The preprocessing operation includes cropping and depth normalization of all depth maps, and converting the starting point information and the end point information of the trajectory to the body coordinate system of the drone.

[0009] Furthermore, in the trajectory generation network based on motion primitives:

[0010] Use the image encoder to encode the depth map to obtain obstacle features;

[0011] The state encoder is used to encode the starting point information, end point information and past state estimation information of each local replanning to obtain the state characteristics of the UAV.

[0012] Based on the pre-built motion primitive library and the motion primitive features obtained by splicing the obstacle features and the drone state features, the probability distribution of the motion primitive features to the motion primitive library is obtained by using the primitive probability output layer;

[0013] Based on the probability distribution, a number of motion primitives with the highest probability are selected using a primitive selection layer;

[0014] The corridor refinement layer is used to adjust the position of each point on each selected motion primitive, the adjusted position is set as the center of the sphere, and a corresponding safety radius is assigned to each sphere, so as to obtain the flight corridor corresponding to each selected motion primitive;

[0015] The trajectory optimization layer is used to construct a constrained spatiotemporal optimization problem, and the spatiotemporal optimization problem is solved to obtain the optimal trajectory corresponding to each group of flight corridors.

[0016] Furthermore, the spatiotemporal optimization problem is:

[0017]

[0018] in, is the objective function of the spatiotemporal optimization problem, is the activation function, and is the weight corresponding to the penalty term, The physical meaning of is that the trajectory point violates the spatial corridor constraint. is the temporal regularization parameter, For the trajectory No. Derivatives, Define constraints for user-defined task scenarios, and are the time and number of segments of the trajectory, is the number of spheres in a set of flight corridors.

[0019] Furthermore, the motion primitive-based trajectory generation network is an offline trained network. During the training process, the parameter update gradient is calculated by the following formula:

[0020]

[0021] in is the evaluation function of the trajectory Relative to network parameters The gradient of It's a flight corridor Relative to network parameters The gradient of is the optimal trajectory calculated by differential theory Relative to the gradient of the flight corridor, is the gradient of the evaluation function of the trajectory relative to the optimal trajectory.

[0022] Furthermore, the trajectory loss of the optimal trajectory is calculated based on the trajectory quality and the probability weighting of the corresponding motion primitive.

[0023] According to a second aspect of an embodiment of the present application, there is provided an end-to-end UAV autonomous navigation device based on differential theory, comprising:

[0024] A trajectory generation module is used to obtain several depth maps and the starting point information and the end point information of the trajectory, generate several groups of flight corridors using a trajectory generation network based on motion primitives, construct a constrained spatiotemporal optimization problem based on the lowest energy trajectory class, and solve the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors;

[0025] The trajectory selection module is used to calculate the trajectory loss of each optimal trajectory, and use the optimal trajectory with the lowest trajectory loss as the trajectory that the UAV needs to execute, thereby realizing autonomous navigation of the UAV.

[0026] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0027] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0028] one or more processors;

[0029] A memory for storing one or more programs;

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0031] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0032] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0033] The present application can use a neural network to directly output the optimal spatiotemporal trajectory from a depth map in an end-to-end manner without the need to build a map; the present application ensures the network's exploration of multi-environment topologies by modeling the optimal topological mixed distribution, thereby increasing the robustness and optimality of the system; the present application makes trajectory optimization differentiable so that the trajectory gradient can be directly returned to the corridor, thereby ensuring the coherent training of the network; the present application can be applicable to completely unknown environments and does not rely on the robot's global positioning system.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] Figure 1 The present invention is a flow chart of an end-to-end UAV autonomous navigation method based on differential theory according to an exemplary embodiment.

[0037] Figure 2 is a schematic diagram of a trajectory generation network based on motion primitives according to an exemplary embodiment.

[0038] Figure 3 Schematic diagram of the trajectory and flight corridor of a navigating UAV in a simulation environment.

[0039] Figure 4 The present invention is a block diagram of an end-to-end UAV autonomous navigation device based on differential theory according to an exemplary embodiment.

[0040] Figure 5 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0041] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0042] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] The present invention proposes an end-to-end UAV autonomous navigation method based on differential theory, such as Figure 1 As shown, the method may include the following steps:

[0045] Step S1: obtaining several depth maps and the starting point information and the end point information of the trajectory, using the trajectory generation network based on motion primitives to generate several groups of flight corridors, constructing a constrained spatiotemporal optimization problem based on the lowest energy trajectory class, and solving the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors;

[0046] Step S2: Calculate the trajectory loss of each optimal trajectory, and use the optimal trajectory with the lowest trajectory loss as the trajectory that the UAV needs to execute, so as to achieve autonomous navigation of the UAV.

[0047] The goal of visual navigation is to stay in a safe area Find a dynamically feasible trajectory within the range that satisfies the initial and final state constraints. It will be represented by a class of minimum energy trajectories, which is a special piecewise polynomial consisting of the time of each trajectory , and a series of joint points Parameterization, where is the number of segments of the trajectory. With this compact representation, the trajectories of our method inherently satisfy the initial and terminal state conditions and ensure high-order continuity at landmarks between adjacent polynomials. These properties provide a solid foundation for smooth and coherent motion. Subsequently, our method uses flight corridors To represent the free space, it is modeled as a safety constraint to restrict the shape of the trajectory space. The trajectory optimization that minimizes the control energy and includes first-order temporal regularization is expressed as follows:

[0048]

[0049] The constraints are as follows:

[0050]

[0051]

[0052]

[0053]

[0054] It is the energy term that measures the loss of the quadcopter motor, and is expressed here as the integral sum of jerk. is the temporal regularization parameter, Indicates control input Relative to the trajectory The derivative order of . It often refers to various user-defined constraints tailored for specific mission scenarios, such as speed, acceleration, thrust, torque, etc. =3 is the control order, which means that the jerk of the trajectory is used as a measure of smoothness. The trajectory of Derivatives, Refers to a safe area in space. In this application, Represents the flight corridor, modeled as is the center and the radius is sphere. It is a spatial constraint that constrains the trajectory within the flight corridor. In traditional algorithms, the flight corridor is usually predetermined and does not participate in the optimization process. These algorithms usually involve using a front-end path planning algorithm to find a path on a grid map, and then deriving the corridor along the path through computational geometry methods. However, this modular layer introduces delays and reduces the optimality of the solution. In order to ensure efficiency, the cost function of the front-end path planning algorithm is usually different from the evaluation index of the final trajectory, resulting in the generated corridor may not be conducive to subsequent trajectory optimization. In addition, this explicit decoupling strategy eliminates the possibility of dynamically adjusting the corridor according to the quality of the optimized trajectory. This method uses a neural network to learn the corridor in an end-to-end manner. By combining joint training with trajectory optimization, this method directly propagates the gradient related to the trajectory quality back to the corridor network to ensure adaptive dynamic adjustment.

[0055] Specifically, this method decouples the original problem expressed by formula (1) and solves the two-level optimization problem:

[0056]

[0057]

[0058]

[0059] As can be seen from the formula, this two-layer optimization is divided into inner loop optimization and outer loop optimization. Intuitively speaking, the inner loop optimization is responsible for trajectory optimization under given flight corridor conditions, which will be achieved through real-time online numerical optimization. The outer loop is responsible for optimizing a high-quality flight corridor. Specifically, the outer loop will be managed by a neural network, which continuously learns on offline data sets and optimizes its internal network parameters to meet this requirement.

[0060] Here, are the parameters of the neural network, Represents the target under given spatial constraints Intuitively, the model-free neural network undertakes the external optimization to extract corridors from the depth information, while the model-based trajectory planning performs the internal optimization to determine the best trajectory in space and time. After obtaining the trajectory, the corresponding trajectory metrics are back-propagated through the differentiable trajectory optimization layer to update the parameters of the network. This collaborative process of network update and trajectory optimization constitutes a nested and coupled two-layer optimization framework that enables the network to directly output the safe region that is most conducive to trajectory generation.

[0061] In the specific implementation of step S1, several depth maps and starting point information and end point information of the trajectory are obtained, and several groups of flight corridors are generated using a trajectory generation network based on motion primitives. A constrained spatiotemporal optimization problem is constructed based on the lowest energy trajectory class, and the spatiotemporal optimization problem is solved to obtain the optimal trajectory corresponding to each group of flight corridors;

[0062] In this application, each trajectory will be discretized into Constraint points are used to process the original time-continuous constraints, so the network needs to assign a safety sphere to each constraint point, that is, a total of To achieve the above goals, this application designs a trajectory generation network based on motion primitives. , effectively estimates the mixed distribution of the optimal trajectory in the workspace and further refines the corridor. This step can include the following sub-steps:

[0063] S11: obtaining several depth maps and the starting point information and the end point information of the trajectory and performing preprocessing, wherein the preprocessing operation includes cropping and depth normalization of all depth maps, and converting the starting point information and the end point information of the trajectory to the current local coordinate system of the robot;

[0064] Specifically, in order to reduce the risk of overfitting due to the absolute coordinates in the world coordinate system, the starting and ending information of the trajectory planning problem (including position, attitude and speed) are converted to the body coordinate system of the drone. In addition, in order to ensure more stable training and enhance the efficiency of gradient descent, this method preprocesses the depth value of the image into a range of 0 to 1 by cropping and normalizing it, and the maximum depth value is set to 10 meters.

[0065] S12: Based on the preprocessed depth map and the starting point information and the end point information of the trajectory, a trajectory generation network based on motion primitives is used to generate several groups of flight corridors;

[0066] like Figure 2 As shown, the trajectory generation network based on motion primitives First, a graph encoder (residual convolution) is used to encode the multi-frame depth map continuously taken by the depth camera to obtain obstacle features. At the same time, a state encoder (multi-layer perceptron, MLP) is used to encode the starting point and end point information of each local replanning. It is also responsible for encoding the state estimation information (position and posture) of the past multiple frames (10 frames are selected in this work), including the position and posture of the robot, to obtain the state features of the drone. The encoded information and obstacle features are spliced ​​into motion primitive features , based on a pre-built library of offline motion primitives representing spatial topology ,feature is fed into a primitive probability output layer (a multilayer perceptron) to output a probability distribution over the library of primitives . In order to be consistent with the subsequent corridor parameters, each motion primitive All by The primitive selection layer is based on the probability distribution Pick a few probabilities The maximum motion primitive. To improve accuracy, the motion primitives selected Further and potential characteristics coupled and fed into the final corridor refinement layer, which is also composed of multi-layer perceptrons. This layer performs a Every point on Apply precise position adjustments , and regard the adjusted result as the corresponding sphere center . In addition, the corridor refinement layer is responsible for assigning a corresponding safety radius to each sphere. This is essentially an approximation of the spatial mixture distribution of the optimal trajectory, thereby matching the inherent multimodal nature of the local planning problem. In addition, this unique network structure has the potential to give the planner the ability to explore multiple topological spaces. After this, the differentiable trajectory optimization layer optimizes each flight corridor to obtain the corresponding optimal trajectory, which will be described in detail in step S13.

[0067] Step S13: For each group of flight corridors, the trajectory optimization layer constructs a constrained spatiotemporal optimization problem based on the lowest energy trajectory class, and solves the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors;

[0068] Before discussing gradient backpropagation, we first reconstruct the internal optimization in equation (8) and introduce its solution method (forward process). Here, this method uses the time discrete penalty term to relax the original time continuous constraint, that is, equation (2), and then the inner loop optimization problem of the above two-level optimization problem will be reconstructed as an unconstrained nonlinear optimization problem:

[0069]

[0070] here, is the objective function of the reconstructed unconstrained nonlinear optimization, It is a built-in activation function in Pytorch. When the input is less than 0, the output is 0. When the input is greater than 0, the output is equal to the input. and is the weight corresponding to the penalty term. The physical meaning is that the trajectory point violates the flight corridor constraint, denoted by This reformulated problem can be solved robustly using common gradient-based numerical solvers such as L-BFGS. Here, we define is the optimal solution to this optimization problem, is the evaluation loss applied on the trajectory.

[0071] It should be noted that the trajectory generation network based on motion primitives is a neural network that has been trained offline. During the training process, the parameter update gradient can be calculated as follows:

[0072]

[0073] here is the evaluation function of the trajectory Relative to network parameters The gradient of is the flight corridor relative to the network parameter The gradient of is the gradient of the optimal trajectory relative to the flight corridor, is the gradient of the evaluation function of the trajectory relative to the optimal trajectory.

[0074] Among them, all gradient calculations follow the denominator layout. Generally speaking, can be calculated analytically, and It can be calculated by automatic differentiation of the network. Therefore, this method mainly discusses the gradient calculation of the optimal solution relative to the constraints . Since gradient-based numerical solvers are used, an intuitive way to estimate parameter gradients, called unfolded computation, involves maintaining the entire computational graph throughout the iterations. However, this approach has significant challenges in terms of memory usage and efficiency, especially when dealing with complex problem formulations. In addition, it may also encounter problems related to gradient divergence or vanishing. In this work, since the optimal solution to the problem has been obtained, , this method uses the implicit function differentiation theorem to analytically derive the gradient without explicitly unfolding the entire iterative process. According to the first-order optimality conditions of nonlinear programming, the optimal solution of the optimization problem should satisfy the following equation:

[0075]

[0076] Here, as an example, It represents the high-order gradient of the constraint point relative to the gradient of the trajectory ( is the order), is the gradient of the constraint at the optimal trajectory point relative to the optimal trajectory point. Then, this method uses the total differential algorithm for this first-order condition:

[0077]

[0078] Each term can be derived analytically. Then, through matrix operations, the equation can be equivalently converted into the following matrix compact form:

[0079]

[0080] in, is the center point of the flight corridor, is the spherical radius of the flight corridor, is the optimality condition (i.e., the objective function Derivative with respect to the trajectory) Gradient with respect to the center point of the sphere, is the gradient of the optimality condition with respect to the sphere radius, is the parameter representing the flight corridor, so the optimal trajectory Gradient relative to flight corridor It can be deduced as follows:

[0081]

[0082] in is the gradient of the optimality condition with respect to the trajectory. Substituting the result of this formula into equation (9), this method can analytically obtain the Jacobian matrix required in equation (9), thereby deriving the gradient of the loss function with respect to the parameters:

[0083]

[0084] In the specific implementation of step S2, the trajectory loss of each optimal trajectory is calculated, and the optimal trajectory with the lowest trajectory loss is used as the trajectory to be executed by the drone, thereby realizing autonomous navigation of the drone;

[0085] Specifically, for the multiple trajectories obtained in step S1, the method generates the probability of each motion primitive predicted by the trajectory generation network based on motion primitives. and the corresponding trajectory quality , the loss of this trajectory is comprehensively weighted .here is the corresponding probability weight, and the final loss will be selected The lowest trajectory is taken as the trajectory the drone will actually execute.

[0086] This method deploys the algorithm in a random forest simulation environment for simulation testing. The results show that this method can effectively extract safe areas and use them as a basis for rapid obstacle avoidance and navigation. Figure 3 shown.

[0087] In summary, in order to solve the error accumulation caused by the module decomposition of the traditional method and the black box problem of the learning-based method, this method combines the traditional trajectory optimization with the neural network to create an end-to-end UAV autonomous navigation method, which directly generates the spatiotemporal optimal trajectory that meets the kinematic requirements from the depth image without explicit mapping. Compared with the traditional learning-based motion planning algorithm, this method embeds the trajectory optimization into the neural network by using implicit differentiation, thus realizing coupled training. This method reduces the burden of the network while ensuring optimality, enhances the interpretability and scalability of the system, and fundamentally guarantees the feasibility of kinematic constraints. Specifically, the neural network in this application does not simply generate a series of trajectory points, but extracts the safe guidance area from the depth image and reconstructs it into the geometric space constraints of the trajectory optimization, and then combines the user-specified kinematic constraints to converge efficiently and robustly (about 1 millisecond) to a high-quality trajectory. Turning the numerical optimization into a differentiable form allows it to be modeled as a layer in the neural network, so that the gradient of the trajectory evaluation loss can be directly back-propagated, thereby encouraging the network to focus on the area that produces the optimal trajectory. In order to fully explore the environment and meet the multimodal characteristics of local motion planning, the neural network in this application outputs a mixed distribution covering an offline regularized lightweight motion primitive library. According to probability, specific motion primitives are selected and allocated a safe feasible space, which is then input into the optimization module to generate smooth and flexible motion.

[0088] Corresponding to the aforementioned embodiment of the end-to-end UAV autonomous navigation method based on differential theory, the present application also provides an embodiment of an end-to-end UAV autonomous navigation device based on differential theory.

[0089] Figure 4 is a block diagram of an end-to-end UAV autonomous navigation device based on differential theory according to an exemplary embodiment. Figure 4 , the device may include:

[0090] The trajectory generation module 21 is used to obtain a plurality of depth maps and the starting point information and the end point information of the trajectory, generate a plurality of groups of flight corridors by using the trajectory generation network based on the motion primitive, construct a constrained spatiotemporal optimization problem based on the minimum energy trajectory class, and solve the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors;

[0091] The trajectory selection module 22 is used to calculate the trajectory loss of each optimal trajectory, and use the optimal trajectory with the lowest trajectory loss as the trajectory that the drone needs to execute, thereby realizing autonomous navigation of the drone.

[0092] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0093] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0094] Accordingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the end-to-end UAV autonomous navigation method based on differential theory as described above.

[0095] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned end-to-end UAV autonomous navigation method based on differential theory. Figure 5 As shown in FIG. 1 , a hardware structure diagram of an end-to-end UAV autonomous navigation device based on differential theory provided by an embodiment of the present invention is provided in any device with data processing capability, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0096] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the end-to-end drone autonomous navigation method based on differential theory as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0097] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. An end-to-end UAV autonomous navigation method based on differential theory, characterized in that: include: Obtaining several depth maps and the starting point and end point information of the trajectory, using a trajectory generation network based on motion primitives to generate several groups of flight corridors, constructing a constrained spatiotemporal optimization problem based on the lowest energy trajectory class, and solving the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors; Calculate the trajectory loss of each optimal trajectory, and use the optimal trajectory with the lowest trajectory loss as the trajectory that the UAV needs to execute, so as to achieve autonomous navigation of the UAV; Among them, in the trajectory generation network based on motion primitives: Use the image encoder to encode the depth map to obtain obstacle features; The state encoder is used to encode the starting point information, end point information and past state estimation information of each local replanning to obtain the state characteristics of the UAV. Based on the pre-built motion primitive library and the motion primitive features obtained by splicing the obstacle features and the drone state features, the probability distribution of the motion primitive features to the motion primitive library is obtained by using the primitive probability output layer; Based on the probability distribution, a number of motion primitives with the highest probability are selected using a primitive selection layer; The corridor refinement layer is used to adjust the position of each point on each selected motion primitive, the adjusted position is set as the center of the sphere, and a corresponding safety radius is assigned to each sphere, so as to obtain the flight corridor corresponding to each selected motion primitive; Using the trajectory optimization layer to construct a constrained spatiotemporal optimization problem, the spatiotemporal optimization problem is solved to obtain the optimal trajectory corresponding to each group of flight corridors; The motion primitive-based trajectory generation network is an offline trained network. During the training process, the parameter update gradient is calculated by the following formula: ， in is the evaluation function of the trajectory Relative to network parameters The gradient of It's a flight corridor Relative to network parameters The gradient of It is the optimal trajectory derived by differential theory Relative to the gradient of the flight corridor, is the gradient of the evaluation function of the trajectory relative to the optimal trajectory.

2. The method according to claim 1, characterized in that Before the depth map and the starting point information and the end point information of the trajectory are input into the trajectory generation network based on the motion primitive, the depth map and the starting point information and the end point information of the trajectory are preprocessed. The preprocessing operation includes cropping and depth normalization of all depth maps, and converting the starting point information and the end point information of the trajectory to the body coordinate system of the drone.

3. The method according to claim 1, characterized in that The spatiotemporal optimization problem is: ， in, is the objective function of the spatiotemporal optimization problem, is the activation function, and is the weight corresponding to the penalty term, The physical meaning of is that the trajectory point violates the spatial corridor constraint. is the temporal regularization parameter, For the trajectory No. Derivatives, Define constraints for user-defined task scenarios, and are the time and number of segments of the trajectory, is the number of spheres in a set of flight corridors.

4. The method according to claim 1, characterized in that: The trajectory loss of the optimal trajectory is calculated based on the trajectory quality and the probability weighting of the corresponding motion primitive.

5. An end-to-end UAV autonomous navigation device based on differential theory, characterized in that: include: A trajectory generation module is used to obtain several depth maps and the starting point information and the end point information of the trajectory, generate several groups of flight corridors using a trajectory generation network based on motion primitives, construct a constrained spatiotemporal optimization problem based on the lowest energy trajectory class, and solve the spatiotemporal optimization problem to obtain the optimal trajectory corresponding to each group of flight corridors; A trajectory selection module is used to calculate the trajectory loss of each optimal trajectory, and use the optimal trajectory with the lowest trajectory loss as the trajectory that the UAV needs to execute, thereby realizing autonomous navigation of the UAV; Among them, in the trajectory generation network based on motion primitives: Use the image encoder to encode the depth map to obtain obstacle features; The state encoder is used to encode the starting point information, end point information and past state estimation information of each local replanning to obtain the state characteristics of the UAV. Based on the pre-built motion primitive library and the motion primitive features obtained by splicing the obstacle features and the drone state features, the probability distribution of the motion primitive features to the motion primitive library is obtained by using the primitive probability output layer; Based on the probability distribution, a number of motion primitives with the highest probability are selected using a primitive selection layer; The corridor refinement layer is used to adjust the position of each point on each selected motion primitive, the adjusted position is set as the center of the sphere, and a corresponding safety radius is assigned to each sphere, so as to obtain the flight corridor corresponding to each selected motion primitive; Using the trajectory optimization layer to construct a constrained spatiotemporal optimization problem, the spatiotemporal optimization problem is solved to obtain the optimal trajectory corresponding to each group of flight corridors; The motion primitive-based trajectory generation network is an offline trained network. During the training process, the parameter update gradient is calculated by the following formula: ， in is the evaluation function of the trajectory Relative to network parameters The gradient of It's a flight corridor Relative to network parameters The gradient of It is the optimal trajectory derived by differential theory Relative to the gradient of the flight corridor, is the gradient of the evaluation function of the trajectory relative to the optimal trajectory.

6. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images