Path planning method and device, electronic equipment and storage medium
By constructing the state vector of the drone and using the strategy and reward network dynamic programming paths, the problem of low path planning efficiency in the environment changes is solved, and the efficiency of drone task processing is improved.
Patent Information
- Application Number
- CN202410992273.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional artificial intelligence algorithms require a lot of time to re-plan drone paths when environmental changes are changed, resulting in low path planning efficiency and unable to meet high-performance obstacle avoidance and navigation needs.
By obtaining environmental image and location information, the drone's state vector is constructed, and the preset policy network and reward network output candidate operations and rewards are used to dynamically plan the flight path and avoid duplicate path planning.
It improves the efficiency of drone path planning, reduces the path planning time caused by environmental changes, and improves task processing efficiency.
Smart Images

Figure CN118938985A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a path planning method, device, electronic device and storage medium. Background Art
[0002] Drones have recently become increasingly popular as sophisticated monitoring instruments for many applications. Advances in communications technology and the increasing miniaturization of onboard sensors, as well as the development of new algorithms and software, have opened the way for the expanded use of drones in many fields, thus opening up new avenues in scientific research, environmental monitoring, and geohazard monitoring, among others.
[0003] Beyond the Visual Line of Sight (BVLOS) has become a focus of the drone industry as it allows drone flights to have a certain degree of autonomy and efficiency while reducing costs and increasing the granularity of surveillance or delivery. Due to these aspects, such systems are gradually gaining their share in the drone market. With the demand for extended levels of autonomy, BVLOS drones meet the need for high-performance obstacle avoidance and navigation algorithms that will allow autonomous drones to operate with minimal or no human intervention.
[0004] Traditional artificial intelligence algorithms are widely used in the literature to find the shortest path in a two-dimensional or three-dimensional environment, avoid obstacles, and successfully navigate drones in a known and stable environment. However, when the environment changes, it takes a lot of time to replan the route, which shows that the current path planning efficiency is low. Summary of the invention
[0005] The embodiments of the present application provide a path planning system, method, electronic device, and storage medium, which can improve the efficiency of path planning, thereby improving the task processing efficiency of the drone.
[0006] In a first aspect, an embodiment of the present application provides a path planning method, including:
[0007] Get the environment image of the preset range and the current location information;
[0008] Constructing a state vector corresponding to the drone according to the environment image, location information, and a preset target endpoint;
[0009] Based on the preset strategy network and the state vector, outputting candidate operations corresponding to the drone;
[0010] The reward corresponding to the candidate operation is estimated according to a preset reward network and the environment image, and a flight path from the current position to the target end point is planned based on the reward.
[0011] Optionally, in some embodiments of the present application, estimating the reward corresponding to the candidate operation according to a preset reward network includes:
[0012] Extracting depth information of the environment image;
[0013] The depth information and the candidate operation are input into a preset reward network to obtain a reward corresponding to the candidate operation.
[0014] Optionally, in some embodiments of the present application, inputting the depth information and the candidate operation into a preset reward network to obtain a reward corresponding to the candidate operation includes:
[0015] The depth information and the candidate operation are input into a preset reward network, and the immediate reward and long-term cumulative reward corresponding to the candidate operation are output.
[0016] Optionally, in some embodiments of the present application, the planning of a flight path from the current position to the target destination based on the reward includes:
[0017] Get preset strategy information;
[0018] Determine the policy weight of each preset policy in the preset policy information;
[0019] Adjusting the reward corresponding to the candidate operation based on the strategy weight;
[0020] A flight path is planned from the current position to the target destination according to the adjusted reward.
[0021] Optionally, in some embodiments of the present application, the planning of a flight path from the current position to the target destination based on the reward includes:
[0022] Based on the reward, determining a target operation among the candidate operations;
[0023] Obtaining an updated position after performing the target operation;
[0024] According to the updated position and the environment image, a flight path from the updated position to the target end point is planned.
[0025] Optionally, in some embodiments of the present application, the outputting of candidate operations corresponding to the drone based on the preset policy network and the state vector includes:
[0026] determining executable operations of the drone;
[0027] Based on the preset strategy network, the state vector and the executable operations, the candidate operations corresponding to the drone are output.
[0028] Optionally, in some embodiments of the present application, the method further includes:
[0029] Acquire an initial path planning model, wherein the initial path planning model includes a basic strategy network and a basic reward network;
[0030] Collect sample environment information and sample location of the sample drone in a preset environment;
[0031] The initial path planning model is trained according to the sample environment information and the sample position to obtain a strategy network and a reward network.
[0032] In a second aspect, an embodiment of the present application provides a path planning device, including:
[0033] An acquisition module is used to acquire an environmental image within a preset range and current location information;
[0034] A construction module, used to construct a state vector corresponding to the drone according to the environment image, location information and a preset target end point;
[0035] An output module, used for outputting candidate operations corresponding to the drone based on a preset strategy network and the state vector;
[0036] An estimation module, used for estimating a reward corresponding to the candidate operation according to a preset reward network and the environment image;
[0037] A planning module is used to plan a flight path from a current position to the target destination based on the reward.
[0038] Correspondingly, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of any of the above methods when executing the program.
[0039] The present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements any of the above methods when the computer program is executed by a processor.
[0040] The present application provides a path planning method, device, electronic device and storage medium. After acquiring an environmental image of a preset range and current location information, a state vector corresponding to a drone is constructed according to the environmental image, location information and a preset target destination. Then, based on a preset strategy network and the state vector, a candidate operation corresponding to the drone is output. Finally, a reward corresponding to the candidate operation is estimated according to a preset reward network and the environmental image, and a flight path from the current position to the target destination is planned based on the reward. In the path planning scheme provided in the present application, a state vector corresponding to a drone can be constructed according to the environmental image, location information and a preset target destination. Then, based on the preset strategy network, reward network and state vector, a flight path from the current position to the target destination is planned. Thus, when the environment changes, a large amount of time spent on re-planning the route can be avoided, thereby improving the efficiency of path planning and thus improving the task processing efficiency of the drone. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0042] Figure 1 It is a flowchart of a path planning method provided in an embodiment of the present application;
[0043] Figure 2 It is a schematic diagram of a scenario of a path planning method provided in an embodiment of the present application;
[0044] Figure 3 is a schematic diagram of a Markov decision process;
[0045] Figure 4 is a schematic diagram of the structure of an A2C network provided in an embodiment of the present application;
[0046] Figure 5 is a schematic diagram of the structure of a path planning device provided in an embodiment of the present application;
[0047] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0048] Figure 7 It is another structural schematic diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of systems and methods consistent with the examples detailed in the attached claims, or with some aspects of the present application.
[0050] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.
[0051] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0052] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0053] Embodiments of the present application provide a path planning method, device, electronic device, and storage medium.
[0054] The path planning device can be deployed in a drone. A drone (UAV), also known as a remote-controlled aircraft or an unmanned aircraft system (UAS), is an aircraft that does not require a pilot to operate in the cabin. The drone can communicate with a server via wired or wireless means. The server can include an independently operated server or a distributed server, or a server cluster consisting of multiple servers. The terminal can include a mobile phone, a tablet computer or a personal computer (PC).
[0055] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.
[0056] The present application provides a path planning method, including: obtaining an environmental image of a preset range and current position information; constructing a state vector corresponding to a drone according to the environmental image, position information and a preset target endpoint; outputting candidate operations corresponding to the drone based on a preset strategy network and the state vector; estimating rewards corresponding to the candidate operations according to a preset reward network and the environmental image, and planning a flight path from the current position to the target endpoint based on the reward.
[0057] See also Figure 1 , Figure 1 : is a flow chart of the path planning method provided by this application, and the path planning method specifically includes the following process:
[0058] 101. Obtain an environment image within a preset range and current location information.
[0059] Environmental images refer to visual data of the surrounding environment captured by the drone through its onboard sensors. Environmental images are used to identify and locate obstacles on the flight path, such as buildings, trees, wires, etc. When a depth camera or lidar is used, the environmental image can provide distance information of obstacles, which is essential for accurate obstacle avoidance. Optionally, in some embodiments of the present application, the captured image needs to be processed to extract useful information, such as the shape, size, and position of the obstacle. Environmental images can be combined with other sensor data (such as infrared, sonar) to obtain a more comprehensive environmental perception. The current position information refers to the positioning information collected by the positioning sensor mounted on the drone, and the current position information can be expressed as a set of coordinates, including longitude, latitude, and altitude.
[0060] For example, the visual sensors (such as cameras, depth cameras, and lidar) carried by the drone can be used to capture images of the surrounding environment; the precise latitude, longitude, and altitude information of the drone (i.e., the current location information) can be obtained through the positioning system.
[0061] 102. Construct the state vector corresponding to the drone based on the environment image, location information and the preset target end point.
[0062] The target endpoint refers to the predetermined location of the drone when the mission is completed. The target endpoint is usually defined in the form of coordinates, which can be geographic coordinates (longitude, latitude, altitude) or a position vector relative to the starting point. Before the mission begins, the planner or automatic mission planning system sets the target endpoint as the destination of the drone flight mission. The state vector is a core concept in reinforcement learning and other machine learning tasks. It contains a set of features or variables of the state of an agent (in this case, a drone) at a specific time step. The state vector provides the agent with the necessary information to evaluate the environment and make decisions. The state vector is a mathematical representation of the state of the agent at a certain moment, usually a multi-dimensional feature vector. The state vector can include the current position, speed, altitude, flight direction, attitude angle (pitch, roll, yaw), remaining battery power, etc. of the drone.
[0063] Construct a state vector based on the environment image, location information, and the preset target endpoint. The state vector usually includes but is not limited to the following elements:
[0064] Current location coordinates (x, y, z)
[0065] The relative position or coordinate difference of the target end point
[0066] Flight velocity vector (vx,vy,vz)
[0067] Attitude angles (pitch, roll, yaw)
[0068] Obstacle location and depth information
[0069] Other factors that may affect flight decisions, such as battery power
[0070] Optionally, in some embodiments of the present application, dimension reduction or feature selection may be performed on the state vector as needed to improve algorithm efficiency.
[0071] During flight, the drone needs to update its state vector in real time to respond to environmental changes and adjust its flight path.
[0072] In some cases, the state vector may contain a large number of features, which may lead to increased computational complexity and reduced algorithm efficiency. Dimensionality reduction is the process of simplifying the model by reducing the number of features in the state vector while retaining important information as much as possible. Feature selection is the process of selecting the most useful feature subset for the UAV mission from the original feature set. This helps to improve the operating efficiency of the algorithm and may improve the generalization ability of the model. Commonly used dimensionality reduction techniques include principal component analysis (PCA), linear discriminant analysis (LDA), t-distributed stochastic neighbor embedding (t-SNE), etc. The drone will continuously collect sensor data during flight, and the state vector needs to be updated in real time to reflect the latest environment and flight status.
[0073] 103. Based on the preset strategy network and state vector, output the candidate operations corresponding to the drone.
[0074] The state vector is input to the policy network (Actor network). The policy network is a deep learning model that is used to decide the best action based on the current state. The policy network receives the state vector as input and processes it through its internal neural network structure. This usually involves multiple hidden layers and non-linear activation functions. The policy network outputs the probability of each possible action (i.e. candidate action). These candidate actions include forward, backward, up, down, turn left, turn right, etc.
[0075] Optionally, in some embodiments of the present application, the step of “outputting candidate operations corresponding to the drone based on a preset policy network and a state vector” may specifically include:
[0076] Determine the actions that can be performed by the drone;
[0077] Based on the preset policy network, state vector, and executable operations, the candidate operations corresponding to the drone are output.
[0078] Determine all the operations that the drone can perform in the current environment and flight conditions, such as forward, backward, ascend, descend, turn left, turn right, etc. Use the drone's sensors to collect environmental information, including obstacle locations, terrain features, etc., which are critical to determining executable operations. Integrate the drone's current flight state and the collected environmental information into a state vector, which is the basis for decision-making. Input the state vector into the preset policy network (Actor network). The policy network is a trained deep learning model used to generate candidate operations. The policy network processes the input state vector through its internal neural network structure to generate a probability distribution corresponding to executable operations. The policy network outputs the probability of each executable operation, which reflects the preference for performing each operation in the current state.
[0079] 104. Estimate rewards corresponding to the candidate operations according to a preset reward network and the environment image, and plan a flight path from the current position to the target end point based on the reward.
[0080] The reward network (Critic network) is a key component in reinforcement learning. Its role is to evaluate the expected value of the actions selected by the policy network (Actor network). The reward network can use time difference learning to update its predictions, that is, to adjust the value estimate by comparing the actual reward obtained with the predicted reward. The reward network uses a discount factor when calculating the expected reward to reduce the importance of future rewards relative to immediate rewards. In this application, a reward is a feedback signal used to guide the behavior of an agent (such as a drone) to progress towards a specific goal or task. The goal of an agent is usually to maximize the cumulative reward it obtains in a series of steps, which is usually achieved by discounting future rewards. Rewards are used to evaluate the effectiveness of the current strategy, and the agent uses rewards to adjust its strategy in the hope of obtaining higher cumulative rewards.
[0081] For example, the reward network receives the state vector and candidate actions as input and estimates the expected reward for each action. If there are multiple reward objectives (such as shortest path, minimum energy consumption, maximum monitoring coverage), the weights and balances of these objectives need to be considered in the reward estimation. The path with the highest expected cumulative reward and the lowest risk is selected as the optimal flight path.
[0082] Optionally, in some embodiments of the present application, the step of “estimating the reward corresponding to the candidate operation according to the preset reward network and the environment image” may specifically include:
[0083] Extracting depth information from the environment image;
[0084] The depth information and candidate operations are input into the preset reward network to obtain the rewards corresponding to the candidate operations.
[0085] For example, depth information is extracted from the captured environment image, including the distance, size, and shape of each object in the scene, and the extracted depth information is integrated into the state vector of the drone. Then, the policy network (Actor network) outputs the corresponding candidate actions based on the updated state vector, and the reward network evaluates the expected immediate reward and long-term cumulative reward for each candidate action. The reward network uses a discount factor to balance the importance of immediate rewards and future rewards. If there are multiple reward targets, the reward network needs to consider these targets comprehensively, such as the shortest path, minimum energy consumption, maximum monitoring coverage, etc. In addition, when estimating rewards, the reward network also needs to consider the risk and safety of the operation to avoid potential collisions and dangerous behaviors.
[0086] Optionally, in some embodiments of the present application, the step of “inputting the depth information and the candidate operation into a preset reward network to obtain a reward corresponding to the candidate operation” may specifically include:
[0087] The depth information and candidate operations are input into the preset reward network, and the immediate reward and long-term cumulative reward corresponding to the candidate operations are output.
[0088] The depth information and candidate operations are provided as input data to the reward network (Critic network).
[0089] The reward network evaluates the immediate reward for each candidate action. It is usually based on the immediate effect of the action on the current state of the drone, such as avoiding an obstacle or moving towards a target. The reward network further predicts the long-term cumulative reward for each candidate action, which involves predicting the effect of a sequence of actions on future states. A discount factor (γ) is used to calculate the long-term cumulative reward, ensuring that more recent rewards have a higher weight than more distant rewards. When estimating the reward, the risk and safety of the action are taken into account to avoid potential negative consequences, such as collisions. For each candidate action, the reward network outputs an immediate reward value and a long-term cumulative reward prediction. The overall value of each candidate action is evaluated by taking into account the immediate reward and the long-term cumulative reward. The candidate action with the highest expected total reward (immediate reward plus long-term cumulative reward) is selected as the action to be executed.
[0090] Optionally, in some embodiments of the present application, the step of “planning a flight path from the current position to the target destination based on the reward” may specifically include:
[0091] Get preset strategy information;
[0092] Determine the policy weight of each preset policy in the preset policy information;
[0093] Adjusting the reward corresponding to the candidate operation based on the strategy weight;
[0094] Plan a flight path from the current position to the target destination based on the adjusted reward.
[0095] The preset strategy information may include the shortest path, minimum energy consumption, maximum monitoring coverage, etc. A strategy weight is assigned to each preset strategy. These weights reflect the importance of each strategy goal in the overall task. The reward of each candidate operation is weighted and adjusted according to the weight of the preset strategy. Each candidate operation sequence is simulated to evaluate the cumulative reward of the entire path from the current state to the goal. The path with the highest expected cumulative reward and the lowest risk after reward adjustment is selected as the optimal flight path.
[0096] Optionally, in some embodiments of the present application, the step of “planning a flight path from the current position to the target destination based on the reward” may specifically include:
[0097] Based on the reward, determine the target operation among the candidate operations;
[0098] Get the updated position after executing the target operation;
[0099] According to the updated position and the environment image, a flight path from the updated position to the target end point is planned.
[0100] For example, specifically, based on the estimated reward, the candidate operation with the highest expected reward can be selected as the target operation. The drone performs the selected target operation, such as moving, turning, raising or lowering, etc. After the operation is performed, the state of the drone changes, and at this time, the updated position information is obtained. Then, the sensor is used to capture the environmental image at the updated position, including obstacles, terrain and other information. According to the new environmental image and the updated position information, the state vector of the drone is updated. Based on the updated state vector, a new sequence of candidate operations is regenerated and evaluated. If there is new environmental information or the policy weight is updated, the estimated reward of the candidate operation may need to be adjusted. Using the updated state information and the adjusted reward, the optimal flight path from the current position to the target end point is planned. Optionally, in some embodiments of the present application, dynamic programming techniques, such as Monte Carlo tree search or time difference learning, can be applied to optimize path selection.
[0101] Monte Carlo Tree Search (MCTS) is a heuristic search algorithm used in certain decision-making processes, especially in game theory, reinforcement learning, and robotics. It evaluates potential actions by running multiple simulations and selects those that appear to be optimal. The key steps and concepts of MCTS: Starting from the root node, according to a specific strategy (usually UCT, Upper Confidence bounds applied to Trees), select the most promising child nodes to expand. If the selected node is not a terminal node, expand this node and add one or more child nodes, which represent possible subsequent states. Starting from the expanded node, Monte Carlo simulations are performed until a terminal state is reached. This process usually does not consider the strategy used in the tree search, but randomly selects actions. The results of the simulation (such as the reward obtained) are propagated back to the nodes in the tree, and the statistics of these nodes, such as the number of visits and cumulative rewards, are updated. The selection, expansion, simulation, and backtracking steps are repeated until a predetermined number of iterations or computing time is reached. After multiple iterations, the child node with the highest weighted average reward is selected as the action. In the selection step, the UCT formula is usually used to balance exploration (selecting nodes with fewer visits) and exploitation (selecting nodes with higher average rewards). MCTS can be easily parallelized because each simulation is independent. MCTS is suitable for problems that can be decomposed into a series of decision steps, such as board games, video games, or robot path planning. MCTS does not rely on a specific model or function, which makes it useful even without an explicit model. In reinforcement learning, MCTS can be used to plan policies, especially when the cost of interacting with the environment is high.
[0102] Temporal Difference Learning (TD learning) is an important algorithm in reinforcement learning. It is used to estimate the value function of a state or state-action pair, that is, the expected return of taking a specific strategy in a given state. TD learning is an algorithm that combines the Monte Carlo method and dynamic programming ideas. It can learn from interactions with the environment without complete model information.
[0103] Optionally, in some embodiments of the present application, the path planning method may further include:
[0104] Obtain an initial path planning model, which includes a basic strategy network and a basic reward network;
[0105] Collect sample environment information and sample location of the sample drone in a preset environment;
[0106] The initial path planning model is trained according to the sample environment information and sample positions to obtain the policy network and reward network.
[0107] Create an initial path planning model, including a basic policy network (Actor network) and a basic reward network (Critic network). Set the neural network structure of the policy network and reward network, including the number of layers, the number of neurons in each layer, and the activation function. Determine the preset environment for training, which may be a simulated environment or a real environment with specific characteristics. Run the initial model in the preset environment and collect sample environment information and sample locations of the drone. Clean and format the collected sample data for training. Use the sample data to train the policy network so that it learns the strategy of selecting the best action in different states. Also use the sample data to train the reward network so that it can accurately predict the expected rewards of different actions. Specify the reward function to ensure that it can reasonably evaluate the actions generated by the policy network. Optimize the parameters of the policy network and reward network through multiple iterations. Apply a temporal difference learning algorithm to update the network weights, adjusting them according to the difference between the actual reward obtained and the predicted reward. Use an exploration strategy (such as ε-greedy) during training to discover better strategies. Regularly evaluate the performance of the model to ensure that the policy network and reward network can work together effectively. Adjust the learning rate according to the progress of training to ensure stable update of the network weights. Use a discount factor to balance immediate rewards and long-term cumulative rewards. Test the model on the validation set to ensure the model's generalization ability.
[0108] To understand the path planning scheme of this application, please refer to Figure 2 This application proposes a reinforcement learning-based autonomous UAV path planning method for geological disaster monitoring, which uses depth information and GPS input to safely navigate the UAV through obstacles. This application proposes a reinforcement learning model that can navigate the UAV from the starting point A to the target point B by determining a non-predefined path P(A,B) and not colliding with any obstacles during flight. In this direction, this application assumes that the final path P is decomposed into a series of sub-steps p i ,…,p N , where p i ,i∈1,…,N, is the obstacle avoidance maneuver, such as Figure 1 As shown. Obstacles on the flight path can be buildings or any other objects that appear in the drone's flight path. To this end, the drone must detect obstacles early, sense their location, position and possible movement, and avoid collisions. This application tries to use the minimum number of sensors to minimize the cost and load of the drone.
[0109] like Figure 3As shown in Figure 1, reinforcement learning algorithms are usually modeled as a Markov decision process, which consists of: i) a set of environment and agent states, ii) a set of actions for the agent, iii) the probability of the agent choosing an action in a given state, and iv) an immediate reward given to the agent after choosing an action to transition from one state to another.
[0110] The A2C reinforcement learning algorithm consists of two neural networks, the Actor and the Critic. The Actor network is responsible for the behavior of the agent (all possible moves) at each given time based on the learning policy. The Critic network is value-based and is responsible for judging how good the Actor's behavior is. During training, the actor model tries to map each possible state to the action with the best possible outcome. In the case of a drone, the state contains information about the agent's surroundings, which is collected by a stereo camera or other sensors. The output of the actor model is a probability distribution corresponding to each action. At the same time, the Critic model tries to map each state to a corresponding value, which represents the quality of the Actor's behavior in the future. Update the two models
[0111] Compared with other reinforcement learning algorithms, the Actor-Critic algorithm updates the weights of the two networks at each step, rather than at the end of a predefined number of episodes, so α t is the action taken at step t, s t is the state of the same step. t ,α t ) is the subtraction between the predicted feature reward (critic’s output) and the actual reward using the function (time difference error), and finally π θ log(α t |s t ) is defined for the action step t. Also very different from other algorithms, A2C uses only a small batch of states at each training step and this batch is replaced with a new batch at each step. The characteristics of this algorithm make it a non-memory intensive algorithm.
[0112] Following the general definition of the A2C reinforcement learning algorithm, it is important to formulate the UAV navigation task in A2C terms and identify the main ingredients of the reinforcement learning solution, namely the observation space, action space, and reward function.
[0113] Observation Space: The observation space corresponds to a formal representation of the world and the drones in it, and captures the various parameters that define the drone state. The drone state is the value of all of these parameters at a specific moment in time. The state must contain as much information as possible about the world of the mission.
[0114] according to Figure 2As shown, you can see that the drone can move on any of the three axes: x, y, and z. In this case, use:
[0115] ① The relative position between the current position K and the target position B, expressed as the distance of each axis of the ternary body
[0116]
[0117] ② The speed of the drone on three axes (V x ,V y ,V y )
[0118] ③ The direction of the drone on the three axes of pitch, yaw and roll (P, Y, R)
[0119] ④Current altitude
[0120] ⑤ Two depth information (D front ,D bottom )
[0121] Altitude is used to give the agent an acceptable flying height, and speed is used to allow the agent to approach obstacles and targets at an appropriate speed to avoid collisions or overshoots. Direction is used to know where the drone is facing at any time. Combined with depth information, it will gradually let the agent understand where obstacles are and how to carefully avoid them.
[0122] Action Space: The action space contains all the available actions for the agent to perform at any time. Given that the drone can move in all three dimensions, this application defines six different actions, two for each axis. More specifically, this application chooses to increase the default values for the drone's speed in the following axes: -Z and +Z for altitude control, -X and +X for moving forward or backward, and finally -Y and +Y for left and right movement.
[0123] Given an observation space and an action space, the goal of the agent is to choose the most appropriate action at each moment by evaluating the drone state. The training process evaluates each action after execution and uses a reward function.
[0124] Reward Function: The reward function is one of the hardest parts of any reinforcement learning implementation, and its purpose is to help the agent learn a policy for the task. In the case of a drone, the reward function forces the drone to minimize the distance between its current position and the goal position without colliding with any objects or the ground. Therefore, the reward for an action that does not result in a collision is a function of the distance to the goal achieved by the action. If a collision occurs, a negative reward is given, which is related to the distance of the drone from the goal when the collision occurred. The maximum positive reward is given when the drone reaches the goal area. Two negative rewards are given when the drone flies for a long time without reaching the goal, and when it exceeds the limits in any direction.
[0125] See also Figure 4 ,The architecture of the A2C network includes two identical networks, the policy network (Actor) and the target network (Critic), which differ only in the last layer output. Specifically, the Actor network includes an input layer, a network layer (Antirectifier layer), and a fully connected layer (Dense) layer. The Actor network is usually used to generate decisions or actions. In the Actor network, the Antirectifier layer is used to adjust the decision of the output layer to make it more robust or avoid extreme behavior. In the Actor network, the Critic network is used to evaluate the actions generated by the Actor network. In the Critic network, the Antirectifier layer is used to adjust the evaluation results to ensure that the evaluation is more accurate or avoid misleading the Actor network. In the Dense layer, each neuron is connected to all neurons in the previous layer. This connection method enables the network to learn complex relationships between input data. The Dense layers are stacked to form a multi-layer network structure. Each layer further processes and abstracts the output of the previous layer, and gradually extracts higher-level features.
[0126] The policy and goal networks share the same inputs, and while the policy has 6 identical outputs (one for each possible movement), the value network has a single output that represents how good the action chosen from the Actor model was.
[0127] In the experiments of the method described in this application, Microsoft's Air-sim simulator was used. Based on the references, the Air-sim simulator is commonly used by many researchers for reinforcement learning tasks similar to this application. In the training of the drone, the "Blocks" Air-sim environment was selected based on its specificity. The world is composed of multiple objects of different dimensions, some objects are similar to buildings, and some objects are smaller in size and shaped like a ball or cylinder. Two areas are also assumed in the simulation, the starting point and the end point, and we randomly choose our starting point and end point to fall on each route. The two areas are placed before and after the "building" to force the drone to pass through it.
[0128] Training parameters: For A2C training, several parameters need to be set, such as rewards and penalties, exploration rate, and learning rate optimization strategy. The reward function uses a penalty of -100 for collision with obstacles, collision with the floor, pause, and overshoot.
[0129] Evaluation Metrics and Baselines: To evaluate our method, we used the percentage of successfully completed or failed test routes (1000 routes in total. For example, crashing on the floor, crashing on an obstacle, long pile-ups on a loop, or exceeding boundary limits), and measured the time required to complete the route, i.e., flight time, and calculated the average time and standard deviation of all successful routes. The baseline for comparison with our method is the A* algorithm.
[0130] As can be seen from the above, the embodiment of the present application provides a path planning method. After obtaining the environmental image of the preset range and the current position information, the state vector corresponding to the drone is constructed according to the environmental image, the position information and the preset target end point. Then, based on the preset strategy network and the state vector, the candidate operation corresponding to the drone is output. Finally, the reward corresponding to the candidate operation is estimated according to the preset reward network and the environmental image, and the flight path from the current position to the target end point is planned based on the reward. In the path planning scheme provided by the present application, the state vector corresponding to the drone can be constructed according to the environmental image, the position information and the preset target end point. Then, based on the preset strategy network, the reward network and the state vector, the flight path from the current position to the target end point is planned. Therefore, when the environment changes, it can avoid spending a lot of time re-planning the route, improve the path planning efficiency, and then improve the task processing efficiency of the drone.
[0131] In order to better implement the path planning method of the embodiment of the present application, the embodiment of the present application also provides a path planning device, wherein the meanings of the terms are the same as those in the above-mentioned path planning system, and the specific implementation details can refer to the description in the system embodiment.
[0132] See also Figure 5 , Figure 5 A schematic diagram of the structure of a path planning device provided in an embodiment of the present application, wherein the path planning device may specifically include an acquisition module 201, a construction module 202, an output module 203, an estimation module 204, and an identification module 205, which may be specifically as follows:
[0133] The acquisition module 201 is used to acquire an environment image within a preset range and current location information.
[0134] The construction module 202 is used to construct a state vector corresponding to the drone according to the environment image, location information and a preset target end point.
[0135] The output module 203 is used to output candidate operations corresponding to the drone based on the preset strategy network and the state vector.
[0136] Optionally, in some embodiments of the present application, the output module 203 may be specifically used for:
[0137] Determine the actions that can be performed by the drone;
[0138] Based on the preset policy network, state vector, and executable operations, the candidate operations corresponding to the drone are output.
[0139] The estimation module 204 is used to estimate the reward corresponding to the candidate operation according to the preset reward network and the environment image.
[0140] Optionally, in some embodiments of the present application, the estimation module 204 may specifically include:
[0141] An extraction unit, used for extracting depth information of an environment image;
[0142] The input unit is used to input the depth information and the candidate operation into a preset reward network to obtain the reward corresponding to the candidate operation.
[0143] Optionally, in some embodiments of the present application, the input unit may be specifically used to: input the depth information and the candidate operation into a preset reward network, and output an immediate reward and a long-term cumulative reward corresponding to the candidate operation.
[0144] The planning module 205 is used to plan a flight path from the current position to the target destination based on the reward.
[0145] Optionally, in some embodiments of the present application, the planning module 205 can be specifically used to: obtain preset strategy information; determine the strategy weight of each preset strategy in the preset strategy information; adjust the reward corresponding to the candidate operation based on the strategy weight; and plan a flight path from the current position to the target destination based on the adjusted reward.
[0146] Optionally, in some embodiments of the present application, the planning module 205 can be specifically used to: determine a target operation among candidate operations based on a reward; obtain an updated position after performing the target operation; and plan a flight path from the updated position to the target destination based on the updated position and the environmental image.
[0147] Optionally, in some embodiments of the present application, the path planning device may further include a training module 206, and the training module 206 may be specifically used for:
[0148] Obtain an initial path planning model, which includes a basic strategy network and a basic reward network;
[0149] Collect sample environment information and sample location of the sample drone in a preset environment;
[0150] The initial path planning model is trained according to the sample environment information and sample positions to obtain the policy network and reward network.
[0151] The embodiment of the present application provides a path planning device. After the acquisition module 201 acquires the environment image of the preset range and the current position information, the construction module 202 constructs the state vector corresponding to the drone according to the environment image, the position information and the preset target end point. Then, the output module 203 outputs the candidate operation corresponding to the drone based on the preset strategy network and the state vector. Finally, the estimation module 204 estimates the reward corresponding to the candidate operation according to the preset reward network and the environment image. The recognition module 205 plans the flight path from the current position to the target end point based on the reward. In the path planning scheme provided in the present application, the state vector corresponding to the drone can be constructed according to the environment image, the position information and the preset target end point. Then, based on the preset strategy network, the reward network and the state vector, the flight path from the current position to the target end point is planned. Therefore, when the environment changes, it can avoid spending a lot of time to re-plan the route, improve the path planning efficiency, and then improve the task processing efficiency of the drone.
[0152] In addition, the present application also provides an electronic device, such as Figure 7 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0153] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0154] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.
[0155] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and path planning methods by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0156] The electronic device also includes a power supply 303 for supplying power to each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 303 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0157] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0158] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby realizing various functions, as follows:
[0159] Acquire an environmental image of a preset range and current position information; construct a state vector corresponding to the drone based on the environmental image, position information and a preset target endpoint; output candidate operations corresponding to the drone based on a preset strategy network and the state vector; estimate rewards corresponding to the candidate operations based on a preset reward network and the environmental image, and plan a flight path from the current position to the target endpoint based on the reward.
[0160] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0161] After acquiring the environmental image of the preset range and the current location information, the embodiment of the present application constructs the state vector corresponding to the drone according to the environmental image, location information and the preset target end point, then outputs the candidate operation corresponding to the drone based on the preset strategy network and state vector, and finally estimates the reward corresponding to the candidate operation based on the preset reward network and environmental image, and plans the flight path from the current position to the target end point based on the reward. In the path planning scheme provided in the present application, the state vector corresponding to the drone can be constructed according to the environmental image, location information and the preset target end point, and then the flight path from the current position to the target end point can be planned based on the preset strategy network, reward network and state vector, thereby, when the environment changes, it can avoid spending a lot of time re-planning the route, improve the path planning efficiency, and then improve the task processing efficiency of the drone.
[0162] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0163] To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any path planning method provided in the embodiment of the present application. For example, the instructions can execute the following steps:
[0164] Acquire an environmental image of a preset range and current position information; construct a state vector corresponding to the drone based on the environmental image, position information and a preset target endpoint; output candidate operations corresponding to the drone based on a preset strategy network and the state vector; estimate rewards corresponding to the candidate operations based on a preset reward network and the environmental image, and plan a flight path from the current position to the target endpoint based on the reward.
[0165] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0166] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0167] Since the instructions stored in the storage medium can execute the steps in any path planning method provided in the embodiments of the present application, the beneficial effects that can be achieved by any path planning method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0168] The above is a detailed introduction to a path planning method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A path planning method, characterized in that: include: Get the environment image of the preset range and the current location information; Constructing a state vector corresponding to the drone according to the environment image, location information, and a preset target endpoint; Based on the preset strategy network and the state vector, outputting candidate operations corresponding to the drone; The reward corresponding to the candidate operation is estimated according to a preset reward network and the environment image, and a flight path from the current position to the target end point is planned based on the reward.
2. The path planning method according to claim 1, characterized in that: The estimating the reward corresponding to the candidate operation according to a preset reward network includes: Extracting depth information of the environment image; The depth information and the candidate operation are input into a preset reward network to obtain a reward corresponding to the candidate operation.
3. The path planning method according to claim 2, characterized in that: The inputting the depth information and the candidate operation into a preset reward network to obtain a reward corresponding to the candidate operation includes: The depth information and the candidate operation are input into a preset reward network, and the immediate reward and long-term cumulative reward corresponding to the candidate operation are output.
4. The path planning method according to claim 1, characterized in that: The flight path from the current position to the target destination is planned based on the reward, including: Get preset strategy information; Determine the policy weight of each preset policy in the preset policy information; Adjusting the reward corresponding to the candidate operation based on the strategy weight; A flight path is planned from the current position to the target destination according to the adjusted reward.
5. The path planning method according to claim 1, characterized in that: The flight path from the current position to the target destination is planned based on the reward, including: Based on the reward, determining a target operation among the candidate operations; Obtaining an updated position after performing the target operation; According to the updated position and the environment image, a flight path from the updated position to the target end point is planned.
6. The path planning method according to claim 1, characterized in that: The outputting of candidate operations corresponding to the drone based on the preset strategy network and the state vector includes: determining executable operations of the drone; Based on the preset strategy network, the state vector and the executable operations, the candidate operations corresponding to the drone are output.
7. The path planning method according to any one of claims 1 to 6, characterized in that: Also includes: Acquire an initial path planning model, wherein the initial path planning model includes a basic strategy network and a basic reward network; Collect sample environment information and sample location of the sample drone in a preset environment; The initial path planning model is trained according to the sample environment information and the sample position to obtain a strategy network and a reward network.
8. A path planning device, characterized in that: include: An acquisition module is used to acquire an environmental image within a preset range and current location information; A construction module, used to construct a state vector corresponding to the drone according to the environment image, location information and a preset target end point; An output module, used for outputting candidate operations corresponding to the drone based on a preset strategy network and the state vector; An estimation module, used for estimating a reward corresponding to the candidate operation according to a preset reward network and the environment image; A planning module is used to plan a flight path from a current position to the target destination based on the reward.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program according to the steps of the path planning method as claimed in any one of claims 1 to 7.
10. A storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the path planning method according to any one of claims 1 to 7.
Citation Information
Cited By
Unmanned aerial vehicle autonomous navigation method, device, equipment and medium
CN121430635A
