End-to-end autonomous flight planning method and system

Abstract

The invention relates to an end-to-end autonomous flight planning method and a system thereof in the field of automatic driving, a sensing unit performs environment sensing and obtains environment information, the information is displayed and implicitly processed, a three-dimensional vectorized map is obtained, instance-level information is included, and the three-dimensional vectorized map is displayed and displayed; the scene vector learning unit carries out information processing through proxy query and self-query, carries out prediction and decision making and delivers a final map vector and a final motion vector to the planning unit, and the final planning normal form training unit carries out end-to-end training through constraint of an instance level and adjusts different proxy weights, so that a final planning normal form is obtained. And the control model adapts to different environments and flexibly adapts to and completes the target. Compared with a Transform architecture based on a self-attention mechanism, the large-kernel convolutional neural network is adopted, the real-time performance of operation perception reasoning is higher, the timeliness and errors of a detection result are remarkably improved, the system runs faster, and the system is more suitable for the environment with the light weight requirement.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous flight planning method and system thereof. Background Art

[0002] With the continuous advancement of science and technology, drones have been widely used in many fields, such as military reconnaissance, aerial photography, agricultural plant protection, logistics and distribution. The autonomy and intelligence level of drones have become key factors in their development. Traditional drone control methods are usually based on pre-programmed flight paths and manual remote control operations. In simple, stable and predictable environments, these methods can meet basic flight mission requirements. However, when faced with complex and changing environments, such as complex terrain, dynamic obstacles and uncertain meteorological conditions, the limitations of traditional methods gradually emerge. Traditional drones mainly rely on a single type of sensor (such as GPS for positioning and simple cameras for visual assistance), and the environmental information they obtain is often incomplete and lacks in-depth analysis. It is difficult to accurately perceive and process tiny obstacles in complex environments, dynamically changing objects, and the impact of meteorological factors on flight, which limits the drone's ability to fly safely and operate effectively in complex environments.

[0003] Decision-making systems based on pre-programmed flight paths and simple rules are difficult to quickly adjust to real-time environmental changes. For example, when encountering sudden obstacles or wind direction changes, the drone cannot re-plan a reasonable flight path in time, which may increase the risk of collision or fail to complete the scheduled mission. Existing drone systems lack an effective mechanism that can adaptively adjust their own behavior when facing different environments and mission requirements. It is usually necessary to manually reset parameters or adjust programs, which is not only time-consuming and labor-intensive, but also difficult to achieve the desired effect through manual intervention in some complex environments and mission scenarios.

[0004] End-to-end technology has made some progress in the field of autonomous driving. Its core lies in unified perception, prediction, decision-making and planning through comprehensive processing of information obtained by multiple sensors, and continuous optimization of system performance through a large amount of training data. It can integrate information from multiple sensors such as IMU (inertial measurement unit), GPS, camera, etc. to achieve a more comprehensive and in-depth perception of the environment; make real-time predictions and decisions through advanced algorithms to improve the flexibility of drones in dealing with complex environmental changes; and with the help of effective training mechanisms, drone systems can adapt to different environments and mission requirements to achieve efficient collaborative work.

[0005] However, the architecture of the existing end-to-end autonomous driving model has significant disadvantages: first, the encoder design of the perception unit relies heavily on the Transformer based on the self-attention mechanism, and its operation requires efficient computing support. However, efficient computing is usually difficult to miniaturize, and efficient computing loads suitable for light and small drones are difficult to achieve; second, excessive reliance on the Transformer based on the self-attention mechanism will lead to insufficient real-time computing perception reasoning, large time lag in detection results, and significant errors in detection positions; third, the Transformer series structure in the existing end-to-end autonomous driving model is bloated in architecture, has low operating efficiency, and processes a large amount of information per unit time when running at high speed, resulting in the planning unit of the Transformer series architecture being unable to run in real time on the embedded recording platform. Summary of the invention

[0006] In view of the defects in the prior art, an object of the present invention is to provide an end-to-end autonomous flight planning method and system thereof.

[0007] According to 1. an end-to-end autonomous flight planning method provided by the present invention, it is characterized in that it comprises the following steps:

[0008] S1, encodes the multi-dimensional information obtained by the drone, generates end-to-end PIP dynamic and static trajectory prediction through the forward feedback network to obtain the trajectory prediction of the surrounding moving objects, and obtains the map vectorization representation through the end-to-end vectorized map converter.

[0009] S2, receives the trajectory prediction of the surrounding moving objects and the explicit and implicit information of the map vectorization, re-encodes it for interactive processing, processes it through the implicit information of the agent query and the map query and the explicit information of the motion vector and the map vector, performs decision reasoning through the trained feedforward decision network and obtains the decision reasoning result;

[0010] S3, based on the received explicit and implicit information, combined with the instructions required by the actual map and the state of the UAV obtained by dynamic prediction, integrates and analyzes it, adopts a fast feedforward network planner with a dynamically restricted altitude range, and uses the encoder output fusion data as input to obtain the UAV's own vector;

[0011] S4, performs paradigm training on the received motion vectors, map vectors and self vectors, optimizes and adjusts various weights after iterative training through instance-level constraints.

[0012] In some implementations, in step S1, multi-dimensional information is encoded and processed by a large-core convolutional neural network to form two-dimensional feature information in a unified format, and a forward feedback network processes the two-dimensional feature information in the same format.

[0013] In some implementations, in step S2, the decision reasoning result includes obstacle information and scene description, wherein the obstacle environment information and the drone's own information are used for trajectory planning, and the scene description information is used for decision assistance and decision interpretation.

[0014] In some implementations, in step S4, the constraints of the paradigm training include collision constraints, boundary crossing constraints, and heading constraints;

[0015] Collision constraint: The position of the drone is divided into three dimensions: width, length, and height. During training, the distance between the drone itself and the map and obstacles is checked to see if they are reasonable. Incorrect training is penalized, and the loss function is used to modify the weights.

[0016] Boundary crossing restriction: The boundary where the drone is traveling is clearly divided to ensure that the drone moves forward within the designated area and never crosses the boundary;

[0017] Heading constraint: Ensure that the direction of the drone is consistent with the indicated self-vector direction, and guide the drone to travel correctly.

[0018] In some implementations, in step S4, paradigm training is end-to-end training based on vectorized planning constraints. The overall loss of end-to-end learning is the weighted sum of vectorized scene learning loss, vectorized planning constraint loss, and imitation learning loss. The total map loss is obtained by calculating the Manhattan distance and focal loss. In the neural network training, labeling and iterative training are continuously performed, and the weights are continuously adjusted under the constraints of paradigm training.

[0019] The present invention also provides an end-to-end autonomous flight planning system, comprising:

[0020] The perception unit is used to encode and process the multi-dimensional information obtained by the drone, generate end-to-end PIP dynamic and static trajectory predictions through the forward feedback network to obtain the trajectory predictions of the surrounding moving objects, and obtain the map vectorization representation through the end-to-end vectorized map converter.

[0021] The scenario-based vector learning unit is used to receive the trajectory prediction of the surrounding moving objects and the explicit and implicit information of the map vectorization, re-encode it for interactive processing, process it through the implicit information of the agent query and the map query and the explicit information of the motion vector and the map vector, perform decision reasoning through the trained feedforward decision network and obtain the decision reasoning result;

[0022] The planning unit is used to integrate and analyze the received implicit information, the instructions required by the actual map, and the state of the UAV obtained by dynamic prediction, and adopt a fast feedforward network planner with a dynamically limited altitude range, and use the encoder output fusion data as input to obtain the UAV's own vector;

[0023] The planning paradigm training unit is used to perform paradigm training on the received motion vectors, map vectors and self-vectors, and adjust and optimize various weight values ​​after iterative training through instance-level constraints.

[0024] In some implementations, in the perception unit, multi-dimensional information is encoded and processed through a large-core convolutional neural network to form two-dimensional feature information in a unified format, and a forward feedback network processes the two-dimensional feature information in the same format.

[0025] In some embodiments, in the scenario-based vector learning unit, the decision reasoning result includes obstacle information and scene description, wherein the obstacle environment information and the drone's own information are transmitted to the planning unit for trajectory planning, and the scene description information is used for decision assistance and decision interpretation.

[0026] In some implementations, in the planning paradigm training unit, the constraints of the paradigm training include collision constraints, boundary crossing constraints, and heading constraints;

[0027] Collision constraint: The position of the drone is divided into three dimensions: width, length, and height. During training, the boundary distance between the drone itself and the map and other moving objects is examined to see whether it is reasonable. Incorrect training is penalized, and the loss function is used to modify the weights.

[0028] Boundary crossing restriction: The boundary where the drone is traveling is clearly divided to ensure that the drone moves forward within the designated area and never crosses the boundary;

[0029] Heading constraint: Ensure that the direction of the drone is consistent with the indicated self-vector direction, and guide the drone to travel correctly.

[0030] In some implementations, in the planning paradigm training unit, paradigm training is end-to-end training based on vectorized planning constraints. The overall loss of end-to-end learning is the weighted sum of vectorized scene learning loss, vectorized planning constraint loss, and imitation learning loss. The total map loss is obtained by calculating the Manhattan distance and focal loss. In the neural network training, labeling and iterative training are continuously performed, and the weights are continuously adjusted under the constraints of paradigm training.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The end-to-end autonomous flight planning method and system of this application adopts a large-core convolutional neural network. Compared with the Transformer architecture based on the self-attention mechanism, its operation perception and reasoning are more real-time, the timeliness and error of the detection results are significantly improved, the system runs faster, and is more suitable for lightweight environments.

[0033] 2. The end-to-end autonomous flight planning method and system of the present application, compared with the prior art where the end-to-end autonomous driving model adopts a Transformer series structure, which is difficult to run in real time on an embedded recording platform due to the large amount of information processed per unit time during high-speed operation, the present application adopts a feedforward network planner in the planning unit, which greatly simplifies the system architecture and greatly improves the system operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0035] Figure 1 This is a schematic diagram of the operation of the end-to-end autonomous flight planning system for this application;

[0036] Figure 2 This is a flow chart of the end-to-end autonomous flight planning system for this application;

[0037] Figure 3 Schematic diagram of neural network training for the end-to-end autonomous flight planning system of this application. DETAILED DESCRIPTION

[0038] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0039] Example 1

[0040] The present invention provides an end-to-end autonomous flight planning method, such as Figure 1-3 As shown, the following steps are included:

[0041] S1, encodes and processes the multi-dimensional information obtained by the drone through sensors such as IMU, GPS and camera through a large-core convolutional neural network, and uniformly encodes various multimodal information. The encoded multimodal information is two-dimensional feature information in a unified format, and the forward feedback network generates end-to-end PIP dynamic and static trajectory predictions to obtain trajectory predictions of surrounding moving objects. Similarly, the vectorized representation of the map is obtained through an end-to-end vectorized map converter. Finally, the display and implicit vectorized representations of both are obtained. In this embodiment, a large-core convolutional neural network is used. Compared with the Transformer architecture based on the self-attention mechanism, its operation perception reasoning is more real-time, the timeliness and error of the detection results are significantly improved, the system runs faster, and it is more suitable for lightweight requirements.

[0042] S2 receives the trajectory prediction of the surrounding moving objects and the displayed and implicit information of the map vectorization, re-encodes it for interactive processing, processes it through the implicit information of the proxy query and map query, and the displayed information of the motion vector and map vector, and performs decision reasoning through the trained feedforward decision network and obtains the decision reasoning result. The decision reasoning result includes obstacle information and scene description, among which the obstacle environment information and the drone's own information are used for trajectory planning, and the scene description information is used for decision assistance and decision interpretation.

[0043] S3, based on the implicit information received, combined with the instructions required by the actual map, and the state of the UAV obtained by dynamic prediction, an integrated analysis is performed, and a fast feedforward network planner with a dynamically restricted altitude range is used. The encoder output fusion data is used as input to obtain the UAV's own vector to guide the UAV's trajectory. The state of the above-mentioned UAV includes state information such as speed, angle, direction, and power. Compared with the problem that the end-to-end autonomous driving model in the prior art adopts a Transformer series structure, which is difficult to run in real time on an embedded recording platform due to the large amount of information processed per unit time during high-speed operation, the feedforward network planner is used in the planning unit in this application, which greatly simplifies the system architecture and greatly improves the system operation efficiency.

[0044] S4, the received motion vector, map vector and self-vector are trained in a paradigm through instance-level constraints. Paradigm training is end-to-end training based on vectorized planning constraints. The overall loss of end-to-end learning is the weighted sum of vectorized scene learning loss, vectorized planning constraint loss and imitation learning loss. The total map loss is obtained by calculating the Manhattan distance and focal loss. In the neural network training, labeling and iterative training are continuously performed, and various weights are continuously adjusted under the constraints of paradigm training. The constraints of paradigm training include collision constraints, boundary crossing constraints and heading constraints;

[0045] Collision constraint: The position of the drone is divided into three dimensions: width, length, and height. During training, the boundary distance between the drone itself and the map and other moving objects is examined to see whether it is reasonable. Incorrect training is penalized, and the loss function is used to modify the weights.

[0046] Boundary crossing constraints: The boundaries where the drone is traveling are clearly demarcated to ensure that the drone moves forward within the designated area and never crosses the boundary.

[0047] Heading constraint: Ensure that the direction of the drone is consistent with the indicated self-vector direction, and guide the drone to travel correctly.

[0048] Through iterative training, the end-to-end model of the drone that is finally trained can better adapt to various environmental changes and actively make corresponding decisions. Path planning improves the computing speed of the drone's autonomous driving process, and increases stability and accuracy, making it easier for the drone to carry out work in various processes, which plays a great role in practical applications.

[0049] Example 2

[0050] This embodiment provides an end-to-end autonomous flight planning system. Figure 1-3 As shown, including:

[0051] The perception unit is used to encode and process the multi-dimensional information obtained by the drone through sensors such as IMU, GPS and camera through a large-core convolutional neural network, and uniformly encode various multimodal information. The encoded multimodal information is two-dimensional feature information in a unified format. The forward feedback network generates end-to-end PIP dynamic and static trajectory predictions to obtain trajectory predictions of surrounding moving objects. Similarly, the vectorized representation of the map is obtained through an end-to-end vectorized map converter. Finally, the display and implicit vectorized representations of both are obtained. In this embodiment, a large-core convolutional neural network is used. Compared with the Transformer architecture based on the self-attention mechanism, its computational perception reasoning is more real-time, the timeliness of the detection results and the error are significantly improved, the system runs faster, and it is more suitable for lightweight requirements.

[0052] Through comprehensive multi-dimensional information processing, 3D-BEV images and environmental conditions (such as wind direction and size, etc.) are obtained.

[0053] The scenario-based vector learning unit is used to receive the trajectory prediction of the surrounding moving objects and the displayed and implicit information of the map vectorization, re-encode it for interactive processing, process it through the implicit information of the proxy query and map query, and the displayed information of the motion vector and map vector, and perform decision reasoning and obtain the decision reasoning results through the trained feedforward decision network. The decision reasoning results include obstacle information and scene description, among which the obstacle environment information and the drone's own information are used for trajectory planning, and the scene description information is used for decision assistance and decision interpretation.

[0054] The planning unit integrates and analyzes the received implicit information, the instructions required by the actual map, and the state of the UAV obtained by dynamic prediction, and adopts a fast feedforward network planner with a dynamically restricted altitude range, and uses the encoder output fusion data as input to obtain the UAV's own vector to guide the UAV's trajectory. The state of the above-mentioned UAV includes state information such as speed, angle, direction, and power. Compared with the problem that the end-to-end autonomous driving model in the prior art adopts a Transformer series structure, which is difficult to run in real time on an embedded recording platform due to the large amount of information processed per unit time during high-speed operation, the feedforward network planner is used in the planning unit in this application, which greatly simplifies the system architecture and greatly improves the system operation efficiency.

[0055] The planning paradigm training unit is used to perform paradigm training on the received motion vectors, map vectors, and self-vectors through instance-level constraints. Paradigm training is end-to-end training based on vectorized planning constraints. The overall loss of end-to-end learning is the weighted sum of vectorized scene learning loss, vectorized planning constraint loss, and imitation learning loss. The total map loss is obtained by calculating the Manhattan distance and focal loss. In the neural network training, labeling and iterative training are continuously performed, and various weights are continuously adjusted under the constraints of paradigm training. The constraints of paradigm training include collision constraints, boundary crossing constraints, and heading constraints;

[0056] Collision constraint: The position of the drone is divided into three dimensions: width, length, and height. During training, the boundary distance between the drone itself and the map and other moving objects is examined to see whether it is reasonable. Incorrect training is penalized, and the loss function is used to modify the weights.

[0057] Boundary crossing constraints: The boundaries where the drone is traveling are clearly demarcated to ensure that the drone moves forward within the designated area and never crosses the boundary.

[0058] Heading constraint: Ensure that the direction of the drone is consistent with the indicated self-vector direction, and guide the drone to travel correctly.

[0059] Through iterative training, the end-to-end model of the drone that is finally trained can better adapt to various environmental changes and actively make corresponding decisions. Path planning improves the computing speed of the drone's autonomous driving process, and increases stability and accuracy, making it easier for the drone to carry out work in various processes, which plays a great role in practical applications.

[0060] In the description of the present application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0061] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0062] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. An end-to-end autonomous flight planning method, characterized in that it comprises the following steps: S1, encodes the multi-dimensional information obtained by the drone, generates end-to-end PIP dynamic and static trajectory prediction through the forward feedback network to obtain the trajectory prediction of the surrounding moving objects, and obtains the map vectorization representation through the end-to-end vectorized map converter. S2, receives the trajectory prediction of the surrounding moving objects and the explicit and implicit information of the map vectorization, re-encodes it for interactive processing, processes it through the implicit information of the agent query and the map query and the explicit information of the motion vector and the map vector, performs decision reasoning through the trained feedforward decision network and obtains the decision reasoning result; S3, based on the received explicit and implicit information, combined with the instructions required by the actual map and the state of the UAV obtained by dynamic prediction, integrates and analyzes it, adopts a fast feedforward network planner with a dynamically restricted altitude range, and uses the encoder output fusion data as input to obtain the UAV's own vector; S4, performs paradigm training on the received motion vectors, map vectors and self vectors, optimizes and adjusts various weights after iterative training through instance-level constraints.

2. The end-to-end autonomous flight planning method according to claim 1, characterized in that: In step S1, the multi-dimensional information is encoded by a large-core convolutional neural network to form two-dimensional feature information in a unified format, and the forward feedback network processes the two-dimensional feature information in the same format.

3. The end-to-end autonomous flight planning method according to claim 1, characterized in that: In step S2, the decision reasoning result includes obstacle information and scene description, wherein the obstacle environment information and the drone's own information are used for trajectory planning, and the scene description information is used for decision assistance and decision interpretation.

4. The end-to-end autonomous flight planning method according to claim 1, characterized in that: In step S4, the constraints of paradigm training include collision constraints, boundary crossing constraints, and heading constraints; Collision constraint: The position of the drone is divided into three dimensions: width, length, and height. During training, the distance between the drone itself and the map and obstacles is checked to see if they are reasonable. Incorrect training is penalized, and the loss function is used to modify the weights. Boundary crossing restriction: The boundary where the drone is traveling is clearly divided to ensure that the drone moves forward within the designated area and never crosses the boundary; Heading constraint: Ensure that the direction of the drone is consistent with the indicated self-vector direction, and guide the drone to travel correctly.

5. The end-to-end autonomous flight planning method according to claim 4, characterized in that: In step S4, paradigm training is end-to-end training based on vectorized planning constraints. The overall loss of end-to-end learning is the weighted sum of vectorized scene learning loss, vectorized planning constraint loss and imitation learning loss. The total map loss is obtained by calculating the Manhattan distance and focal loss. In the neural network training, labeling and iterative training are continuously performed, and the weights are continuously adjusted under the constraints of paradigm training.

6. An end-to-end autonomous flight planning system, characterized in that: include: The perception unit is used to encode and process the multi-dimensional information obtained by the drone, generate end-to-end PIP dynamic and static trajectory predictions through the forward feedback network to obtain the trajectory predictions of the surrounding moving objects, and obtain the map vectorization representation through the end-to-end vectorized map converter. The scenario-based vector learning unit is used to receive the trajectory prediction of the surrounding moving objects and the explicit and implicit information of the map vectorization, re-encode it for interactive processing, process it through the implicit information of the agent query and the map query and the explicit information of the motion vector and the map vector, perform decision reasoning through the trained feedforward decision network and obtain the decision reasoning result; The planning unit is used to integrate and analyze the received implicit information, the instructions required by the actual map, and the state of the UAV obtained by dynamic prediction, and adopt a fast feedforward network planner with a dynamically limited altitude range, and use the encoder output fusion data as input to obtain the UAV's own vector; The planning paradigm training unit is used to perform paradigm training on the received motion vectors, map vectors and self-vectors, and adjust and optimize various weight values ​​after iterative training through instance-level constraints.

7. The end-to-end autonomous flight planning system according to claim 6, characterized in that: In the perception unit, multi-dimensional information is encoded and processed through a large-core convolutional neural network to form two-dimensional feature information in a unified format, and the forward feedback network processes the two-dimensional feature information in the same format.

8. The end-to-end autonomous flight planning system according to claim 6, characterized in that the scenario In the vector learning unit, the decision reasoning result includes obstacle information and scene description. The obstacle environment information and the drone's own information are passed to the planning unit for trajectory planning, and the scene description information is used for decision assistance and decision interpretation.

9. The end-to-end autonomous flight planning system according to claim 6, characterized in that the planning In the paradigm training unit, the constraints of paradigm training include collision constraints, boundary crossing constraints, and heading constraints; Collision constraint: The position of the drone is divided into three dimensions: width, length, and height. During training, the boundary distance between the drone itself and the map and other moving objects is examined to see whether it is reasonable. Incorrect training is penalized, and the loss function is used to modify the weights. Boundary crossing restriction: The boundary where the drone is traveling is clearly divided to ensure that the drone moves forward within the designated area and never crosses the boundary; Heading constraint: Ensure that the direction of the drone is consistent with the indicated self-vector direction, and guide the drone to travel correctly.

10. The end-to-end autonomous flight planning system according to claim 9, characterized in that the planning In the paradigm training unit, paradigm training is end-to-end training based on vectorized planning constraints. The overall loss of end-to-end learning is the weighted sum of vectorized scene learning loss, vectorized planning constraint loss and imitation learning loss. The total map loss is obtained by calculating the Manhattan distance and focal loss. In the neural network training, labeling and iterative training are continuously performed, and the weights are continuously adjusted under the constraints of paradigm training.

Images