An unmanned aerial vehicle motion planning method and system based on abstract module processing
By introducing abstract modules and the SAC algorithm into UAV path planning, the motion planning features of UAVs are extracted, solving the problems of poor convergence and sensor robustness of UAVs in complex environments, and realizing efficient planning for autonomous navigation and obstacle avoidance.
Patent Information
- Application Number
- CN202310283625.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-17
- Publication Date
- 2026-03-03
- Estimated Expiration
- 2043-03-17
AI Technical Summary
Existing UAV path planning algorithms have slow convergence speed in complex environments and cannot respond quickly. Furthermore, the high state dimension of deep reinforcement learning networks leads to poor convergence, large differences between virtual and real-world sensor environments, and insufficient robustness.
Abstract modules are introduced into the deep reinforcement learning model. Features related to UAV motion planning are extracted through YOLOv5s, FCOS3D and VINS-Mono networks. The UAV motion planning model is trained using the SAC algorithm to reduce the state dimension and optimize the reward function to improve convergence and robustness.
In environments without GPS signals and complete map information, the system enables autonomous navigation and obstacle avoidance for UAVs, balancing the conflict between exploration and development experience. This enhances the exploratory nature and robustness of UAVs and prevents premature convergence to local optima.
Smart Images
Figure CN116307323B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) path planning technology, and more specifically to a UAV motion planning method and system based on abstract module processing. Background Technology
[0002] In recent years, urban warfare has become one of the main forms of modern warfare. Using drones as reconnaissance and strike weapons is a common tactic in urban operations. In environments lacking GPS signals and complete map information, autonomous motion planning for drones is particularly crucial. Currently, path planning algorithms can be mainly divided into traditional and intelligent algorithms. These algorithms have strong path search capabilities but lack the ability to respond quickly to complex environments, have slow convergence speeds, and require pre-marked trajectories for path planning, making them unsuitable for complex environments. Deep reinforcement learning methods, on the other hand, have advantages such as not requiring a global map, strong learning capabilities, and low dependence on sensor accuracy, making them highly suitable for solving the autonomous motion planning problem for drones performing high-speed maneuvers in complex and unknown environments.
[0003] Deep reinforcement learning can be divided into model-based reinforcement learning and model-free reinforcement learning based on whether the agent can obtain an environmental model. For urban combat conditions, model-free reinforcement learning algorithms are more widely used in intelligent equipment. However, model-free reinforcement learning algorithms face the problem of how to balance the contradiction between the agent's exploration of the environment and the development of acquired experience. Furthermore, the higher the state dimension of a deep reinforcement learning network, the more difficult it is to design a suitable reward function to make the entire Markov decision process approximate a convex problem that is easy to optimize using gradient descent. The network's convergence performance and robustness will also be significantly reduced.
[0004] Therefore, proposing a UAV motion planning method and system based on abstract module processing to solve the difficulties existing in the prior art is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the present invention provides a UAV motion planning method and system based on abstract module processing. By adding an abstract module at the perception end on the basis of using the SAC algorithm as the framework of the UAV deep reinforcement learning motion planning model, the state dimension of the motion planning model is reduced, thereby solving the problems of poor convergence, differences between virtual and real environment sensors, and sensor robustness caused by the high state dimension of the deep reinforcement learning motion planning model.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A UAV motion planning method based on abstract module processing includes the following steps:
[0008] Step 1: Acquire raw sensor information from the drone;
[0009] Step 2: Use an abstract module to process the raw sensor information and extract features that are strongly correlated with the UAV motion planning as intermediate representation information;
[0010] Step 3: Input the intermediate representation information processed by the abstract module as the state input to the UAV motion planning model based on the algorithm;
[0011] Step 4: Train the UAV motion planning model based on the SAC algorithm to obtain the trained UAV motion planning model based on the SAC algorithm.
[0012] Step 5: Use the trained UAV motion planning model based on the SAC algorithm to obtain the UAV motion planning path.
[0013] Optionally, in the above method, the raw sensor information of the UAV in step 1 includes: depth image and IMU data.
[0014] Optionally, the intermediate representation information in step 2 of the above method includes: two-dimensional features of the target, three-dimensional features of the obstacle, and motion trajectory features of the UAV.
[0015] Optionally, step 2 of the above method includes the following steps:
[0016] Step 2-1: Use the YOLOv5s network model to identify the depth image and extract the two-dimensional feature information of the target;
[0017] Step 2-2: Use the FCOS3D network model to identify the depth image and extract the three-dimensional feature information of the obstacle;
[0018] Steps 2-3: Use the VINS-Mono network model to identify two-dimensional and three-dimensional feature information and IMU data to extract the feature information of the UAV's motion trajectory.
[0019] Optionally, in the above method, the YOLOv5s network model in step 2-1 includes an input terminal, a baseline network, a Neck network, and a Head output terminal.
[0020] Optionally, the FCOS3D model in step 2-2 of the above method consists of three main parts: a backbone for feature extraction, a neck for multi-level branch construction, and a detection head for dense prediction.
[0021] Optionally, in the VINS-Mono initialization stage of the above method, images are acquired from a monocular camera, and adaptive histogram equalization is performed on the acquired images. Harris feature points in the images are extracted based on the FAST algorithm, and the KLT pyramid optical flow algorithm is used to track the feature points.
[0022] Optionally, step 4 of the above method includes the following steps:
[0023] Step 4-1: Define the environment reward function and initialize the parameters of the SAC policy network and value network. The reward function is as follows:
[0024] (1)
[0025] in, For the current environmental reward, r arrive A reward for the drone reaching the target. r collide A reward for drone collisions; The distance between the drone and the target. This represents the current distance between the drone and the target. This is the relative position of the drone and the target; this state is used to define the target orientation of the drone.
[0026] Step 4-2: Generate actions based on the motion planning strategy of the SAC algorithm;
[0027] Step 4-3: Execute the action in 4-2, and transition to the next state according to the environment, storing the state transition into the experience replay pool;
[0028] Step 4-4: Update the network parameters to update both the value network and the policy network; update the value network using the soft Bellman equation, the formula of which is:
[0029] (2)
[0030] Where α is a temperature coefficient used to control the proportion of policy entropy. The value network output at time t. The value network output at time t+1. Let π be the entropy of the action, and π be the policy network. Let be the expected sum of the value network output and the entropy of the action after taking an action in state t+1. Let γ be the reward at time t, and γ be the discount factor for the reward;
[0031] The policy network update employs the method of minimizing the KL divergence between the policy and the Q function, making the policy distribution approximate the Q function distribution. The formula is as follows:
[0032] (3)
[0033] in, For the new policy network, For the old policy network, Let be the value network output distribution at time t. Used to normalize the distribution. For the policy network output distribution, Strategies to minimize KL divergence That is, the optimal strategy chosen for this algorithm. Each action is assigned a specific probability to conform to the Q-value distribution, thereby satisfying the requirements of a stochastic strategy.
[0034] A UAV motion planning system based on abstract module processing, employing the aforementioned UAV motion planning method based on abstract module processing, includes, in sequence, an image processing module, a raw information processing module, a representation information processing module, a training module, and a path planning module; wherein...
[0035] Image processing module: Acquires raw sensor information from the UAV;
[0036] Raw information processing module: The abstract module processes the raw sensor information and extracts features that are strongly correlated with the UAV motion planning as intermediate representation information;
[0037] Representation information processing module: The intermediate representation information processed by the abstract module is used as the state input to the UAV motion planning model based on the algorithm;
[0038] Training module: Trains the UAV motion planning model based on the SAC algorithm to obtain the trained UAV motion planning model based on the SAC algorithm;
[0039] Path planning module: Using the trained UAV motion planning model based on the SAC algorithm, the motion planning path of the UAV is obtained.
[0040] As can be seen from the above technical solution, compared with the prior art, this invention discloses a UAV motion planning method and system based on abstract module processing: under the obstruction conditions of no GPS signal and no complete map information, the deep reinforcement learning SAC algorithm is used as the action selection strategy for the UAV to complete autonomous navigation and obstacle avoidance tasks. This effectively balances the contradiction between the reinforcement learning agent exploring unknown situations and developing existing experience, ensuring that the UAV has strong exploratory ability and strong robustness, and preventing the UAV from converging to a local optimum too early. At the same time, an abstract module is added to the perception end of the motion planning model to process the original sensor information, extracting the two-dimensional features of the target, the three-dimensional features of the obstacle, and the motion trajectory features of the UAV that are strongly related to the UAV motion planning as the state of the deep reinforcement learning motion planning model, effectively reducing the dimensionality of the original sensor information, thereby solving the problems of poor convergence, differences between virtual and real environment sensors, and sensor robustness caused by the high state dimension of the deep reinforcement learning motion planning model. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0042] Figure 1 A flowchart of a UAV motion planning method based on abstract module processing provided by the present invention;
[0043] Figure 2 This invention provides a structural diagram of an unmanned aerial vehicle (UAV) motion planning system based on abstract module processing.
[0044] Figure 3 This is a diagram of a drone training and testing environment built in AirSim, provided as an embodiment of the present invention.
[0045] Figure 4 This is a diagram of the training environment built for Turtlebot3 in the Gazebo simulation environment, as provided in an embodiment of the present invention.
[0046] Figure 5 The experimental trajectory diagram of the algorithm test of combining the UAV abstract module with SAC provided in the embodiment of the present invention. Detailed Implementation
[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] See Figure 1 As shown, this invention discloses a UAV motion planning method based on abstract module processing, comprising the following steps:
[0049] Step 1: Acquire raw sensor information from the drone;
[0050] Step 2: Use an abstract module to process the raw sensor information and extract features that are strongly correlated with the UAV motion planning as intermediate representation information;
[0051] Step 3: Input the intermediate representation information processed by the abstract module as the state input to the UAV motion planning model based on the algorithm;
[0052] Step 4: Train the UAV motion planning model based on the SAC algorithm to obtain the trained UAV motion planning model based on the SAC algorithm.
[0053] Step 5: Use the trained UAV motion planning model based on the SAC algorithm to obtain the UAV motion planning path.
[0054] Specifically, in step S5, the policy network is used to output continuous actions in four dimensions: throttle amount in one dimension and attitude angle in three dimensions, and finally the planned path is obtained.
[0055] Specifically, the proof that using this abstract module can reduce the gap between virtual and real sensing information is as follows:
[0056] Without considering differences in implementing agencies, for a Lipsitz continuous strategy The upper limit of the distance between its virtual and real worlds is as follows:
[0057] (4)
[0058] In the formula, For the agent's strategy in a virtual environment, For the agent's strategy in the real environment, The objective function of the virtual environment, For the objective function in the real-world environment, K represents the update delay of the policy network for the agent in a real-world environment, used to delay updates to the target network. K is the Lipsitz constant. For probability distribution, for Expected value This represents the observation model in a virtual environment. This represents an observation model in the real-world environment. Let be the lower bound distance between the virtual environment observation space and the real environment observation space for the strategy. Equation (4) can be proved from equation (5):
[0059] (5)
[0060] In the formula, This represents the lower bound distance between the action space of the virtual environment and the action space of the real environment for the strategy. Let γ be the lower bound distance between the virtual and real-world observation spaces, and let γ be the reward discount factor. The strategy is based on the raw sensor information data. d p The distance in the action space. d o For the distance in the observation space, K is the Lipsitz constant.
[0061] Next, consider the impact of the abstraction module on the input observations. Let f be the mapping of the observations, which is usually a method designed based on task requirements and is unaffected by interference factors. The impact of the abstraction module on the input observations satisfies the following relationship:
[0062] (6)
[0063] Based on equations (4) and (6), it can be proven that a strategy incorporating abstract modules can be used in both virtual and real environments. ,Right now (M), (L), compared to the strategy of directly using raw sensor information data. That is, the difference between M and L should be smaller.
[0064] Furthermore, the raw sensor information from the UAV in step 1 includes: depth images and IMU data.
[0065] Furthermore, the intermediate representation information in step 2 includes: two-dimensional features of the target, three-dimensional features of the obstacle, and motion trajectory features of the UAV.
[0066] Furthermore, step 2 includes the following steps:
[0067] Step 2-1: Use the YOLOv5s network model to identify the depth image and extract the two-dimensional feature information of the target;
[0068] Step 2-2: Use the FCOS3D network model to identify the depth image and extract the three-dimensional feature information of the obstacle;
[0069] Steps 2-3: Use the VINS-Mono network model to identify two-dimensional and three-dimensional feature information and IMU data to extract the feature information of the UAV's motion trajectory.
[0070] Specifically, each network mainly consists of six types of modules: CBL, Res unint, CSP1X, CSP2X, FOCUS, and SPP. The CBL module is composed of Conv+BN+Leakyrelu activation functions. The Res unint module borrows the residual structure from ResNet to construct deep networks; CBM is a sub-module within the residual module. The CSP1X module consists of the CBL module, the Res unint module, convolutional layers, and a concatenation function. The CSP2X module consists of convolutional layers and X concatenated Res unint modules. The Focus module first concatenates multiple slice results and then feeds them into the CBL module. The SPP module uses 1×1, 5×5, 9×9, and 13×13 max pooling methods for multi-scale feature fusion.
[0071] Specifically, the backbone uses a pre-trained ResNet101 model and deformable convolutions for feature extraction, with the parameters of the first convolutional block fixed. The neck region uses a feature pyramid network for detecting objects at different scales. The design of the shared detection heads follows the conventional design of RetinaNet and FCOS. Each shared head consists of four shared convolutional blocks and smaller detection heads for different targets.
[0072] Specifically, the extracted feature points are stored in a queue, the essential matrix is calculated using the five-point method, and mismatched feature points in the queue are removed using random consistency sampling. IMU pre-integration is performed to transform the world coordinate system to the ontology coordinate system. After initialization, a sliding window-based nonlinear optimization method is used to estimate the ontology's state, i.e., the position, velocity, rotation, acceleration deviation, and angular velocity deviation of the IMU in the keyframes of the image within the sliding window. When the system detects a loop closure, i.e., when a pose constraint relationship is established with a previous frame, the visual measurement residual obtained from the loop closure detection is added to the cost function to relocalize the ontology's state and reduce accumulated errors.
[0073] Furthermore, the YOLOv5s network model in step 2-1 includes an input, a baseline network, a Neck network, and a Head output.
[0074] Furthermore, the FCOS3D model in step 2-2 mainly consists of three parts: a backbone for feature extraction, a neck for multi-level branch construction, and a detection head for dense prediction.
[0075] Furthermore, in the VINS-Mono initialization stage of steps 2-3, images are acquired from the monocular camera, and adaptive histogram equalization is performed on the acquired images. Harris feature points in the images are extracted based on the FAST algorithm, and the KLT pyramid optical flow algorithm is used to track the feature points.
[0076] Furthermore, step 4 includes the following steps:
[0077] Step 4-1: Define the environment reward function and initialize the parameters of the SAC policy network and value network. The reward function is as follows:
[0078] (1)
[0079] in, For the current environmental reward, r arrive A reward for the drone reaching the target. r collide A reward for drone collisions; The distance between the drone and the target. This represents the current distance between the drone and the target. This is the relative position of the drone and the target; this state is used to define the target orientation of the drone.
[0080] Step 4-2: Generate actions based on the motion planning strategy of the SAC algorithm;
[0081] Step 4-3: Execute the action in 4-2, and transition to the next state according to the environment, storing the state transition into the experience replay pool;
[0082] Step 4-4: Update the network parameters to update both the value network and the policy network; update the value network using the soft Bellman equation, the formula of which is:
[0083] (2)
[0084] Where α is a temperature coefficient used to control the proportion of policy entropy. The value network output at time t. The value network output at time t+1. Let π be the entropy of the action, and π be the policy network. Let be the expected sum of the value network output and the entropy of the action after taking an action in state t+1. Let γ be the reward at time t, and γ be the discount factor for the reward;
[0085] The policy network update employs the method of minimizing the KL divergence between the policy and the Q function, making the policy distribution approximate the Q function distribution. The formula is as follows:
[0086] (3)
[0087] in, For the new policy network, For the old policy network, Let be the value network output distribution at time t. Used to normalize the distribution. For the policy network output distribution, Strategies to minimize KL divergence That is, the optimal strategy chosen for this algorithm. Each action is assigned a specific probability to conform to the Q-value distribution, thereby satisfying the requirements of a stochastic strategy.
[0088] and Figure 1 Corresponding to the method described above, the present invention also provides a UAV motion planning system based on abstract module processing, for use in... Figure 1 The specific implementation of the method is shown in the following structural diagram. Figure 2 As shown, it includes an image processing module, a raw information processing module, a representation information processing module, a training module, and a path planning module connected in sequence.
[0089] Image processing module: Acquires raw sensor information from the UAV;
[0090] Raw information processing module: The abstract module processes the raw sensor information and extracts features that are strongly correlated with the UAV motion planning as intermediate representation information;
[0091] Representation information processing module: The intermediate representation information processed by the abstract module is used as the state input to the UAV motion planning model based on the algorithm;
[0092] Training module: Trains the UAV motion planning model based on the SAC algorithm to obtain the trained UAV motion planning model based on the SAC algorithm;
[0093] Path planning module: Using the trained UAV motion planning model based on the SAC algorithm, the motion planning path of the UAV is obtained. Specific implementation examples:
[0095] The simulation environment is: Ubuntu 18.04 LST operating system, Airsim simulation platform.
[0096] This invention considers a three-dimensional map model, with a planar coordinate system. Figure 3 This invention provides a training scenario and testing environment for a UAV motion planning model built in Arisim. Before developing the UAV motion planning algorithm, this invention pre-studied a path planning method for ground robots based on deep reinforcement learning. This research primarily involved training the Turtlebot3 ground robot path planning model using the Gazebo simulator. Figure 4 The images show the renderings of three scenes of varying complexity built for Turtlebot3 in the Gazebo simulation environment. Finally, the trained motion planning model was tested in a virtual test scene built with Airsim; the experimental trajectory is shown below. Figure 5 As shown, black represents obstacles, orange represents timeout positions, blue represents the motion planning model test trajectory, green represents the starting position, yellow represents the destination position, and purple crosses represent collision positions.
[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0098] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for unmanned aerial vehicle (UAV) motion planning based on abstract module processing, the method comprising: Comprising the following steps: Step 1: obtaining original sensing information of the unmanned aerial vehicle; Step 2: processing the original sensing information by using an abstraction module to extract strong correlation features of unmanned aerial vehicle motion planning as intermediate representation information; Step 3: inputting the intermediate representation information processed by the abstraction module into an unmanned aerial vehicle motion planning model based on an algorithm as a state; Step 4: training the unmanned aerial vehicle motion planning model based on the SAC algorithm to obtain a trained unmanned aerial vehicle motion planning model based on the SAC algorithm; Step 5: obtaining an unmanned aerial vehicle motion planning path by using the trained unmanned aerial vehicle motion planning model based on the SAC algorithm; Step 4 comprises the following steps: Step 4-1: defining an environment reward function, initializing SAC policy network and value network parameters, and the reward function is as follows: (1) wherein, is an environmental reward at the current time, r arrive is a reward when the UAV reaches the target, r collide is a reward when the UAV collides; is a distance between the UAV and the target, is a distance between the UAV and the target at the current time, is a relative relationship between the UAV and the target position, and this state is used to define a target direction of the UAV; Step 4-2: generating an action according to the SAC algorithm motion planning strategy; Step 4-3: executing the action of 4-2 and transferring to the next state according to the environment, and storing the state transition into an experience replay pool; Step 4-4: updating the network parameters to update the value network and the policy network; the value network is updated using the soft Bellman equation, and the formula is as follows: (2) wherein a is a temperature coefficient used to control the proportion of strategy entropy, is the value network output at time t, is the value network output at time t+1, is the entropy of action, and π is the strategy network, is the expectation of the sum of the value network output and the entropy of action after taking action in state at time t+1, is the reward at time t, and γ is the discount factor of reward. The policy network is updated by minimizing the KL divergence of the policy and the Q function, so that the distribution of the policy is close to the distribution of the Q function, and the formula is as follows: (3) where, is the new policy network, is the old policy network, is the value network output distribution at time t, is used to normalize the distribution, is the policy network output distribution, is the policy that minimizes the KL divergence is the optimal policy chosen by this algorithm, is to assign each action a certain probability that matches the Q-value distribution, thus satisfying the requirement of a random policy. 2.The unmanned aerial vehicle motion planning method based on abstract module processing of claim 1, wherein, The original sensing information of the unmanned aerial vehicle in step 1 includes depth images and IMU data. 3.The unmanned aerial vehicle motion planning method based on abstract module processing of claim 1, wherein, The intermediate representation information in step 2 includes two-dimensional features of the target, three-dimensional features of the obstacle, and motion trajectory features of the unmanned aerial vehicle. 4.The unmanned aerial vehicle motion planning method based on abstract module processing of claim 1, wherein, Step 2 comprises the following steps: Step 2-1: using a YOLOv5s network model to identify the depth image to extract two-dimensional feature information of the target; Step 2-2: using a FCOS3D network model to identify the depth image to extract three-dimensional feature information of the obstacle; Step 2-3: using a VINS-Mono network model to identify the two-dimensional feature information and the three-dimensional feature information and the IMU data to extract feature information of the motion trajectory of the unmanned aerial vehicle.
5. The unmanned aerial vehicle motion planning method based on abstract module processing according to claim 4, characterized in that, The YOLOv5s network model in step 2-1 includes an input end, a reference network, a Neck network and a Head output end.
6. The unmanned aerial vehicle motion planning method based on abstract module processing according to claim 4, wherein, The FCOS3D model in step 2-2 mainly consists of three parts: a backbone part for feature extraction, a neck part for multi-level branch construction, and a detection head for dense prediction.
7. The unmanned aerial vehicle motion planning method based on abstract module processing according to claim 4, wherein, In the VINS-Mono initialization stage in step 2-3, pictures are obtained from a monocular camera, and the collected pictures are subjected to adaptive histogram equalization processing, Harris feature points in the pictures are extracted based on the FAST algorithm, and the feature points are tracked by using the KLT pyramid optical flow algorithm.
8. A UAV motion planning system based on abstract module processing, characterized in that, The unmanned aerial vehicle motion planning method based on the abstraction module processing according to any one of claims 1-7 comprises an image processing module, an original information processing module, a representation information processing module, a training module and a path planning module connected in sequence; wherein The image processing module obtains original sensing information of the unmanned aerial vehicle; The original information processing module processes the original sensing information by using an abstraction module to extract strong correlation features of unmanned aerial vehicle motion planning as intermediate representation information; The characteristic information processing module inputs the intermediate characteristic information processed by the abstraction module into an unmanned aerial vehicle motion planning model based on an algorithm as a state; The training module trains the unmanned aerial vehicle motion planning model based on the SAC algorithm to obtain a trained unmanned aerial vehicle motion planning model based on the SAC algorithm; The path planning module obtains an unmanned aerial vehicle motion planning path by using the trained unmanned aerial vehicle motion planning model based on the SAC algorithm.