Navigation strategy optimization method in intensive obstacle environment of reinforcement learning unmanned aerial vehicle based on state entropy excitation
By introducing state entropy as an intrinsic reward in drone navigation, the navigation strategy is optimized, and the traditional navigation method explores the problems of low efficiency and local optimal solutions in dense obstacle environments, achieving more efficient, safe and adaptable navigation ability.
Patent Information
- Application Number
- CN202411003909.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional drone navigation methods are difficult to adapt to dynamic changes and complex environments in dense obstacle environments, and are inefficient in exploration, easily fall into local optimal solutions, and lack generalization capabilities, so they cannot work effectively in a diversified environment.
Introducing state entropy as an intrinsic reward, enhancing the exploration and navigation capabilities of the drone in dense obstacle environments, optimizing navigation strategies by combining external rewards and intrinsic rewards, avoiding local optimal solutions, and improving exploration efficiency and adaptability.
It significantly improves the navigation efficiency and security of drones in complex environments, enhances the ability to adapt to environmental complexity, realizes more overall navigation optimization and generalization capabilities, and can effectively navigate in different environments.
Smart Images

Figure CN118938988A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of autonomous navigation algorithms for unmanned aerial vehicles, and in particular relates to a method for optimizing navigation strategies for unmanned aerial vehicles in dense obstacle environments using reinforcement learning based on state entropy excitation. Background Art
[0002] With the rapid development of drone technology, its application areas are constantly expanding, from traditional military and aerial photography fields to today's logistics distribution, environmental monitoring and disaster relief scenarios. However, in actual applications, drones often face complex flight environments, especially in dense obstacle areas such as cities or forests. How to navigate safely and efficiently has become a huge challenge.
[0003] Traditional navigation methods usually rely on pre-programmed path planning and simple obstacle avoidance strategies, which cannot adapt to dynamic changes and complex environments. These methods are prone to fall into local optimal solutions when facing dense obstacles, which greatly reduces the efficiency and safety of path planning. In addition, traditional methods lack effective responses to environmental uncertainties and are prone to losing direction in complex environments.
[0004] Reinforcement learning, as a technology that can self-learn and optimize in complex environments, has received extensive attention in the field of robot navigation in recent years. In particular, deep reinforcement learning combines the powerful perception ability of deep learning with the decision-making ability of reinforcement learning, providing a new way to solve navigation problems in complex environments. However, the performance of standard reinforcement learning methods in dense obstacle environments still faces several significant problems. First, the exploration efficiency is low: in such environments, standard reinforcement learning algorithms require a lot of trial and error in the exploration phase to find an effective navigation path, which greatly increases the training time and computational cost. Second, the problem of local optimal solution: standard RL algorithms tend to optimize paths in known reward structures, but in complex environments, they are prone to fall into local optimal solutions and cannot find better navigation strategies. In addition, the generalization ability is insufficient: in a diverse real-world environment, the navigation strategy of drones needs to have strong generalization ability and be able to work effectively under different environmental conditions. When training, standard reinforcement learning algorithms often only perform well in specific training environments, but they are not effective in new environments that have not been seen, and they lack the ability to adapt to different scenarios. Therefore, a new algorithm is needed to achieve the navigation of drones in dense obstacle environments to solve the problems of existing algorithms. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention proposes a method for optimizing the navigation strategy of a UAV in a dense obstacle environment based on reinforcement learning based on state entropy incentive. This method enhances the exploration and navigation capabilities of UAVs in dense obstacle environments by introducing state entropy as an intrinsic reward in the standard reinforcement learning framework. State entropy provides an intrinsic incentive mechanism by measuring the complexity and uncertainty of the state space, so that UAVs give priority to exploring areas with high uncertainty, thereby reducing ineffective exploration behaviors and improving overall exploration efficiency. Combining the advantages of external rewards and intrinsic rewards, the method of the present invention not only enables UAVs to learn the optimal path faster, but also effectively avoids the dilemma of local optimal solutions. Finally, a safe, efficient and adaptable UAV autonomous navigation solution is provided, which significantly enhances the navigation capability and application potential of UAVs in real complex environments.
[0006] To achieve the above technical objectives, the present invention provides the following technical solution: a method for optimizing the navigation strategy of a UAV in a dense obstacle environment based on reinforcement learning based on state entropy excitation, which specifically includes the following steps:
[0007] Step 1: Initialize the environment, including initializing the physical and dynamic parameters of the drone, setting the map information of the dense obstacle environment, and configuring the neural network model for the reinforcement learning algorithm, wherein the neural network model includes a policy network and a value network;
[0008] Step 2: At time step j, obtain the current state s of the drone from the initialization environment j , input it into the policy network to get the action vector a j ′, and add random noise to determine the execution action a j ;
[0009] Step 3: Execute action a j , update the next state s of the drone j+1 , calculate the external reward And determine whether the termination condition of the environment is met, and record the corresponding end mark;
[0010] Step 4: At time step j, the current state s j , perform action a j , External Rewards Next state s j+1 The end mark is stored in the experience replay pool; the end of the round is judged according to the end mark: if the current round ends, return to step 1; if the current round does not end, samples are drawn from the experience replay pool and the state entropy is calculated to obtain the intrinsic reward With external rewards Combined into total reward
[0011] Step 5: Update the parameters of the value network with the total reward obtained in step 4, and optimize the policy network through policy gradient;
[0012] Step 6: Update the parameters of the target policy network and the target value network, and regularly evaluate the performance of the strategy in a dense obstacle environment; adjust the parameters based on the evaluation results and substitute back to step 2, and repeat steps 2-6 for iterative training;
[0013] Step 7: When the predetermined number of training rounds or the total reward converges, the training is terminated to obtain the parameters of the final policy network and value network, that is, the model parameters for the drone to perform the navigation task.
[0014] Furthermore, step 1 includes the following sub-steps:
[0015] Step 11: Set the initial position and attitude of the drone, including the (x, y, z) position coordinates and the direction angles (θ x ,θ y ,θ z ); configure the initial speed and acceleration parameters of the drone (v x ,v y ,v z ) and (a x ,a y ,a z ) ; define the dynamic model of the drone, including parameters such as torque, thrust and air resistance;
[0016] Step 12: Set map information of dense obstacle environment, such as location, size and shape of obstacles;
[0017] Step 13: Configure the neural network model for the reinforcement learning algorithm; define the structure of the policy network, including the number and connection method of neurons in the input layer, hidden layer, and output layer, and generate the actions that the drone should perform in a given state; configure the architecture of the value network to evaluate the value under a specific state and action combination; initialize the weights and biases of the neural network and set the training parameters of the network, including learning rate, batch size, and discount factor, for subsequent training and optimization.
[0018] Furthermore, step 2 includes the following sub-steps:
[0019] Step 21: At time step j, obtain the current state s of the drone from the environment j =[x,y,z,v x ,v y ,v z ,d 1 ,d 2 ,…,d 20 ] T, where [x, y, z] represents the position coordinates of the drone, [v x ,v y ,v z ] represents the speed of the drone along the X-axis, Y-axis and Z-axis, [d 1 ,d 2 ,…,d 20 ] represents the distance between the point detected by the laser radar ray carried by the drone and the drone;
[0020] Step 22: Input the vector into the policy network to obtain the action vector a at the current time step j j ′=[v x ,v y ,v z ] T , v x ,v y ,v z Represents the speed of the drone along the X-axis, Y-axis, and Z-axis respectively;
[0021] Step 23: Increase the exploration capability. In the above step 22, the action vector a j Add random noise on the basis of ′ to determine the final execution action a j .
[0022] Furthermore, step 3 includes the following sub-steps:
[0023] Step 31: Set the selected action a j Input into the UAV's dynamic model to update and calculate the next state s of the UAV j+1 ;
[0024] Step 32: Calculate external reward r ext =r 1 +r 2 +r 3 , r 1 is the distance reward, and the formula is:
[0025]
[0026] where d 0 Indicates the distance between the current drone and the target point;
[0027] r 2 For the arrival reward, when the drone reaches the target point, a larger reward item is given:
[0028]
[0029] r 3 For collision penalty, when the drone collides with an obstacle, a larger penalty item is given:
[0030]
[0031] Step 33: determine whether the termination condition is met and record the corresponding end mark; if the drone reaches the target point or collides, it is determined that the current round is over, otherwise the current round is not over.
[0032] Furthermore, step 4 includes the following sub-steps:
[0033] Step 41: At each time step j, the current state s j , Action performed a j , external rewards from the environment Next state s j+1 and whether the identifier is finished are combined into a tuple, and the tuple is stored in the experience replay pool B;
[0034] Step 42: When the amount of data in the experience replay pool reaches a certain amount or the accumulated number of rounds reaches a preset value, a batch of tuple samples are randomly extracted from the pool; these extracted samples are used for batch update in the training process;
[0035] Step 43: The extracted batch of tuple samples is recorded as For each tuple in the current state s j , calculate it and the set The distance between all state representations s in the calculation; among all the calculated distances, select the one with the current state s j The k nearest states are used as neighbors, and the state s closest to the current state is used. k-nn Calculate distance||s j -s k-nn || 2 , and the intrinsic reward is recorded as The formula is:
[0036]
[0037] Step 44: Calculate intrinsic rewards Then, combine it with external rewards Combine to get the final total reward The formula is:
[0038]
[0039] Among them, λ is a weight parameter;
[0040] Furthermore, the above step 5 includes the following sub-steps:
[0041] Step 51: Use the total reward obtained to minimize the TD error δ j To update the parameters of the value network, the formula is expressed as:
[0042]
[0043] Among them, δ j is the TD error, Q is the value network, θ Q is its parameter, μ is the policy network, θ μ is the parameter of the policy network, γ is the discount factor, Q′ and μ′ correspond to the target value network and the target policy network respectively, θ Q′ and θ μ′ They are target value network parameters and target strategy network parameters respectively;
[0044] Step 52: Use the policy gradient method to update the parameters of the policy network to maximize the long-term cumulative reward. The formula is:
[0045]
[0046] in, Represents relative to the policy network parameter θ μ The objective function J represents the expected value of the long-term cumulative reward. s is the state and a is the action. This policy gradient formula takes into account the total reward. Updated value network.
[0047] Furthermore, the above step 6 includes the following sub-steps:
[0048] Step 61: Regularly update the parameters of the target strategy network and the target value network, using a soft update strategy, and the formula is:
[0049] θ Q′ ←τθ Q +(1-τ)θ Q′ ;
[0050] θ μ′ ←τθ μ +(1-τ)θ μ′ ;
[0051] Among them, θ Q is the parameter of the current value network, θ Q′ is the parameter of the target value network, θ μ is the parameter of the current policy network, θ μ′ is the parameter of the target policy network, τ is the soft update coefficient, which is a small value;
[0052] Step 62: Evaluate the performance of the strategy in a dense obstacle environment, and adjust hyperparameters such as the learning rate, noise parameter, and weight of the intrinsic reward based on the evaluation results.
[0053] Furthermore, the specific process of step 7 is: when the training process reaches a predetermined number of training rounds or the total reward converges stably within several consecutive rounds, the training is terminated; in this process, it is first necessary to pay attention to the changing trend of the total reward. When the change in the total reward is within the set convergence threshold and continues for a predetermined period of time or number of rounds, it is considered that the algorithm has converged, that is, the training is terminated. At this time, the weights of the final policy network and value network are saved as model parameters used by the drone in navigation tasks.
[0054] Compared with the prior art, the present invention has the following advantages:
[0055] This paper proposes a method for optimizing the navigation strategy of a UAV in a dense obstacle environment based on state entropy incentives. The invention introduces state entropy as an intrinsic reward, which enables the UAV to preferentially explore areas with high complexity. This method effectively guides the exploration behavior of the UAV, enables it to find the best path more quickly, and significantly improves the exploration efficiency in a dense obstacle environment. Compared with traditional methods, this algorithm reduces unnecessary exploration time and resource consumption.
[0056] In addition, the present invention enhances the UAV's pursuit of the global optimal solution during the exploration process by using state entropy as an intrinsic reward. Traditional reinforcement learning algorithms are prone to fall into local optimal solutions in specific reward structures, while state entropy incentives prompt UAVs to continue exploring areas of high uncertainty, thereby more effectively avoiding the dilemma of local optimal solutions and achieving a more global optimized navigation strategy. State entropy incentives help UAVs maintain high navigation performance in complex and changing environments by measuring the uncertainty in the state space. This method allows the UAV to flexibly adjust its navigation strategy when facing dynamically changing obstacles and complex terrain, thereby enhancing its adaptability to environmental complexity and allowing the UAV to operate stably in various complex scenarios.
[0057] The method of the present invention not only performs well in a specific training environment, but also has strong generalization capabilities and can maintain effective navigation performance in new environments that have not been seen before. State entropy excitation improves the universality of the learned strategy by prompting the drone to explore different areas and states during the training process, so that it can adapt to different environmental conditions in a variety of practical application scenarios. In summary, these advantages of the method for optimizing the navigation strategy of drones in dense obstacle environments based on reinforcement learning based on state entropy excitation make the present invention have broad application potential and value in the field of autonomous navigation of drones, and provide a safer, more efficient and adaptable solution for drones in complex and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 This is a flow chart of the method for optimizing the navigation strategy of a UAV in a dense obstacle environment based on state entropy excitation by reinforcement learning of the present invention;
[0059] Figure 2 A schematic diagram of a dense obstacle environment of the present invention;
[0060] Figure 3 This is a comparison chart of the training results of the deep deterministic policy gradient (DDPG) algorithm used in the present invention and the training results of the deep deterministic policy gradient (DDPG) algorithm using state entropy (SE) excitation. DETAILED DESCRIPTION
[0061] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.
[0062] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0063] Please refer to Figure 1-3 , shows a specific implementation of this embodiment. This embodiment introduces state entropy as an intrinsic reward, so that the drone can give priority to exploring those areas with high complexity, significantly improving the exploration efficiency of the drone in dense obstacle environments and reducing unnecessary exploration time and resource consumption; at the same time, the state entropy incentive is used to measure the uncertainty in the state space, helping the drone to maintain high navigation performance in complex and changing environments. It allows the drone to flexibly adjust its navigation strategy when facing dynamically changing obstacles and complex terrain, thereby enhancing its ability to adapt to environmental complexity and allowing the drone to pursue the global optimal solution during the exploration process; in addition, the method involved in this implementation also has a strong generalization ability. The use of state entropy incentives improves the universality of the learned strategy by prompting the drone to explore different areas and states during the training process, so that it can also adapt to different environmental conditions in a variety of practical application scenarios.
[0064] Please refer to Figure 1This embodiment proposes a method for optimizing the navigation strategy of a UAV in a dense obstacle environment based on reinforcement learning based on state entropy excitation, which specifically includes the following steps:
[0065] Step 1: Initialize the environment, including initializing the physical and dynamic parameters of the drone, setting the map information of the dense obstacle environment, and configuring the neural network model for the reinforcement learning algorithm, wherein the neural network model includes a policy network and a value network;
[0066] As a preferred implementation of step 1, the following steps are specifically included:
[0067] Step 11: Set the initial position and attitude of the drone, including the (x, y, z) position coordinates and the direction angles (θ x ,θ y ,θ z ); configure the initial speed and acceleration parameters of the drone (v x ,v y ,v z ) and (a x ,a y ,a z ) to reflect the dynamic characteristics of the UAV in space; define the UAV's dynamic model, including parameters such as torque, thrust and air resistance, to accurately simulate the UAV's flight behavior and response;
[0068] Step 12: Set map information of dense obstacle environment, such as the location, size and shape of obstacles. These obstacles are densely distributed on the map, forming a complex navigation environment.
[0069] Step 13. Configure the neural network model for the reinforcement learning algorithm and define the structure of the policy network, including the number and connection method of the neurons in the input layer, hidden layer, and output layer, which are used to generate the actions that the drone should perform in a given state; configure the architecture of the value network to evaluate the value under a specific state and action combination to guide the optimization of the policy network; initialize the weights and biases of the neural network and set the training parameters of the network, including the learning rate, batch size, and discount factor, etc., to facilitate the subsequent training and optimization process.
[0070] Step 2: At time step j, obtain the current state s of the drone from the initialization environment j , input it into the policy network to get the action vector a j ′, and add random noise to determine the execution action a j ;
[0071] As a preferred implementation of step 2, the following steps are specifically included:
[0072] Step 21: At time step j, obtain the current state s of the drone from the environment j =[x,y,z,v x ,v y ,v z ,d 1 ,d 2 ,…,d 20 ] T , where [x, y, z] represents the position coordinates of the drone, [v x ,v y ,v z ] represents the speed of the drone along the X-axis, Y-axis and Z-axis, [d 1 ,d 2 ,…,d 20 ] represents the distance between the point detected by the laser radar ray carried by the drone and the drone;
[0073] Step 22: Input the vector into the policy network to obtain the action vector a at the current time step j j ′=[v x ,v y ,v z ] T , v x ,v y ,v z Represents the speed of the drone along the X-axis, Y-axis, and Z-axis respectively;
[0074] Step 23: In order to increase the exploration capability, the action vector a described in step 22 above is j Add random noise on the basis of ′ to determine the final execution action a j , which can prompt the algorithm to explore more possible actions in different states, thereby improving the effectiveness of the navigation strategy.
[0075] Step 3: Update the next state of the drone according to the execution action, calculate the external reward, determine whether the termination condition of the environment is met, and record the corresponding end mark;
[0076] As a preferred implementation of step 3, the following steps are specifically included:
[0077] Step 31: Action a will be executed j Input into the UAV's dynamic model to update and calculate the next state s of the UAV j+1 This step adjusts the UAV’s dynamic characteristics such as speed, position, and attitude according to the current action, simulating the UAV’s flight behavior in space;
[0078] Step 32: Calculate external reward r ext =r 1 +r 2+r 3 , r 1 For distance rewards:
[0079]
[0080] where d 0 Indicates the distance between the current drone and the target point;
[0081] r 2 For the arrival reward, when the drone reaches the target point, a larger reward item is given:
[0082]
[0083] r 3 For collision penalty, when the drone collides with an obstacle, a larger penalty item is given:
[0084]
[0085] Step 33: Determine whether the termination condition is met, such as whether the drone reaches the target point or collides with an obstacle, and record the corresponding end mark. If the drone reaches the target point or collides, the current round ends.
[0086] Step 4: At time step j, the current vector s j , perform action a j , external rewards ext , next state s j+1 The end flag is stored in the experience replay pool; the end of the round is determined based on the end flag: if the current round ends, return to step 1; if the current round does not end, samples are drawn from the experience replay pool, and the state entropy is calculated to obtain the intrinsic reward, which is combined with the external reward as the total reward;
[0087] As a preferred implementation of step 4, the following steps are specifically included:
[0088] Step 41: At each time step j, the current state s j , Action performed a j , external rewards from the environment Next state s j+1 The identifier of whether the experience is over is combined into a tuple, and the tuple is stored in the experience replay pool B. These data are used for subsequent network training to improve the learning efficiency and stability of the model;
[0089] Step 42: When the amount of data in the experience replay pool reaches a certain amount or the accumulated number of rounds reaches a preset value, a batch of tuple samples are randomly extracted from the pool. These extracted samples are used for batch updates during the training process. The purpose of this step is to break the time correlation of the data so that the model can better generalize and adapt to different environments;
[0090] Step 43: The extracted batch of tuple samples is recorded as For each tuple in the current state s j , calculate it and the set The distance between all states s in . This process measures the similarity between the current state and other states; among all the distances calculated, select the state with the current state s j The k states closest to the current state are used as neighbors. This step mainly selects the k states closest to the current state, reflecting the density and distribution of the current state in the state space; the state s closest to the current state is used. k-nn Calculate distance||s j -s k-nn || 2 , and the intrinsic reward is recorded as:
[0091]
[0092] This intrinsic reward motivates the drone to explore high uncertainty regions in the state space, thus enhancing its exploration capability;
[0093] Step 44: Calculate intrinsic rewards Then, combine it with external rewards Combine to get the final total reward Specifically, the total reward calculation formula is:
[0094]
[0095] Among them, λ is a weight parameter used to balance the influence of external rewards and intrinsic rewards. In this way, the navigation algorithm can comprehensively utilize the external feedback of the environment and the incentive of intrinsic exploration to learn more effective navigation strategies, especially in dense obstacle environments, and achieve global optimization of efficient paths.
[0096] Step 5: Update the parameters of the value network with the total reward obtained in step 4, and optimize the policy network through policy gradient;
[0097] As a preferred implementation of step 5, the following steps are specifically included:
[0098] Step 51: Use the total reward obtained to minimize the TD error δ j To update the parameters of the value network, the formula is expressed as:
[0099]
[0100] Among them, δ j is the TD error, Q is the value network, θ Q is its parameter, μ is the policy network, θ μ is the parameter of the policy network, γ is the discount factor, Q′ and μ′ correspond to the target value network and the target policy network respectively, θ Q′ and θ μ′ They are target value network parameters and target strategy network parameters respectively;
[0101] Step 52: Use the policy gradient method to update the parameters of the policy network to maximize the long-term cumulative reward. The formula is:
[0102]
[0103] in, Represents relative to the policy network parameter θ μ The objective function J represents the expected value of the long-term cumulative reward. s is the state and a is the action. This policy gradient formula takes into account the total reward. Updated value network.
[0104] Step 6: Update the parameters of the target policy network and the target value network, and regularly evaluate the performance of the strategy in a dense obstacle environment; adjust the parameters based on the evaluation results and substitute back to step 2, and repeat steps 2-6 for iterative training;
[0105] As a preferred implementation of step 6, the following specific steps are performed:
[0106] Step 61: Regularly update the parameters of the target policy network and the target value network, usually using a soft update strategy to ensure that the target network can smoothly follow the changes of the main network:
[0107] θ Q′ ←τθ Q +(1-τ)θ Q′ ;
[0108] θ μ′ ←τθ μ +(1-τ)θ μ′ ;
[0109] Among them, θ Q is the parameter of the current value network, θ Q′ is the parameter of the target value network, θ μ is the parameter of the current policy network, θ μ′is the parameter of the target policy network, τ is the soft update coefficient, which is usually a small value to ensure a smooth and slow update process;
[0110] Step 62: Evaluate the performance of the strategy in a dense obstacle environment, and adjust hyperparameters such as the learning rate, noise parameter, and weight of the intrinsic reward based on the evaluation results.
[0111] Step 7: When the predetermined number of training rounds or the total reward converges, the training is terminated to obtain the parameters of the final policy network and value network, that is, the model parameters for the drone to perform navigation tasks.
[0112] More specifically, the specific process of step 7 is:
[0113] When the training process reaches the predetermined number of training rounds or the total reward converges stably within several consecutive rounds, the training ends. In this process, we first need to pay attention to the change trend of the total reward. When the change range of the total reward is within the set convergence threshold and lasts for a predetermined period of time or number of rounds, the algorithm can be considered to have converged. At this point, the weights of the final policy network and value network can be saved as model parameters used by the drone in navigation tasks.
[0114] This embodiment provides a simulation environment for the algorithm, such as Figure 2 The simulation environment is set to a complex and dense obstacle scene of 100×100. These obstacles are densely distributed, forming multiple narrow channels, requiring the UAV to have efficient navigation and obstacle avoidance capabilities. The UAV is equipped with a laser radar in this environment, and the scanning frequency is set to 20Hz, covering a scanning range of -180° to +180° to ensure real-time perception of the surrounding environment. The initial state of the UAV includes position [x, y, z] = [0, 0, 0], speed [v x ,v y ,v z ]=[0,0,0],acceleration[a x ,a y ,a z ]=[0,0,0], and the initial direction θ=[0,0,0]. In the algorithm design, the deep deterministic policy gradient (DDPG) algorithm is used for training.
[0115] During the training process, state entropy is introduced as an intrinsic reward to enhance the UAV’s ability to explore high uncertainty areas. Specifically, at each time step, the current state s is calculated. j The Euclidean distance to all state representations in the tuple sampled from the experience replay pool. Choose the distance s j The nearest 5th neighbor state, calculate the nearest neighbor state s k-nn Distance from current state||s j -sk-nn || 2 , and by the formula Obtain the state entropy intrinsic reward. This intrinsic reward encourages the drone to explore areas with high state entropy in depth, improving its adaptability and exploration efficiency in complex environments. Then, combine the external reward and the intrinsic reward to calculate the total reward The weight parameter λ is set to 0.5 to balance the impact of the two rewards on navigation decisions.
[0116] In terms of experience replay and model update, after each action is executed, the state, action, external reward, next state, and whether it is ended are stored in the experience replay pool. When the amount of data in the experience replay pool reaches a certain amount, 1024 tuple samples are randomly extracted from it for batch update of the neural network. This strategy helps to break the time correlation and improve the stability and efficiency of the learning process. The learning rate parameter of the model is set to 0.001 and the discount factor is set to 0.99 to ensure the stability and convergence of the training. The training process continues until it reaches 2×10 7 time steps.
[0117] After the training is completed, the weight parameters of the final policy network and value network are saved. Through verification in multiple different complex obstacle environments, the results show that Figure 3 As shown in the figure, the solid line represents the training result of the deep deterministic policy gradient (DDPG) algorithm, and the dotted line represents the training result of the deep deterministic policy gradient (DDPG) algorithm with the introduction of state entropy (SE) incentive; obviously, the algorithm after the introduction of state entropy can obtain more total rewards than the original algorithm, which shows that the reinforcement learning method based on state entropy incentive proposed in the present invention significantly improves the navigation efficiency and adaptability of the UAV. In the verification process, the UAV demonstrated excellent autonomous navigation capabilities, was able to efficiently navigate between dense obstacles, successfully avoid all obstacles and reach the target point. Compared with the traditional navigation method, the optimization strategy proposed in the present invention not only improves the exploration efficiency of the UAV, but also significantly enhances its navigation ability and adaptability in actual complex environments; moreover, because the state entropy incentive improves the universality of the learned strategy by prompting the UAV to explore different areas and states during the training process, the optimization method proposed in the present invention has stronger generalization. In general, the navigation strategy optimization method of the UAV in dense obstacle environment based on state entropy incentive of the present invention shows significant superiority, which verifies its effectiveness and practicality in the field of autonomous navigation of UAVs.
[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.
[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.
[0120] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for optimizing navigation strategy of UAV in dense obstacle environment based on reinforcement learning based on state entropy incentive, characterized by: The specific steps include: Step 1: Initialize the environment, including initializing the physical and dynamic parameters of the drone, setting the map information of the dense obstacle environment, and configuring the neural network model for the reinforcement learning algorithm, wherein the neural network model includes a policy network and a value network; Step 2: At time step j, obtain the current state s of the drone from the initialization environment j , input it into the policy network to get the action vector a j ′, and add random noise to determine the execution action a j ; Step 3: Execute action a j , update the next state s of the drone j+1 , calculate the external reward And determine whether the termination condition of the environment is met, and record the corresponding end mark; Step 4: At time step j, the current state s j , perform action a j , external rewards j ext , next state s j+1 The end mark is stored in the experience replay pool; the end of the round is judged according to the end mark: if the current round ends, return to step 1; if the current round does not end, samples are drawn from the experience replay pool and the state entropy is calculated to obtain the intrinsic reward r j int , and external rewards Combined into total reward Step 5: Update the parameters of the value network with the total reward obtained in step 4, and optimize the policy network through policy gradient; Step 6: Update the parameters of the target policy network and the target value network, and regularly evaluate the performance of the strategy in a dense obstacle environment; adjust the parameters based on the evaluation results and substitute back to step 2, and repeat steps 2-6 for iterative training; Step 7: When the predetermined number of training rounds or the total reward converges, the training is terminated to obtain the parameters of the final policy network and value network, that is, the model parameters for the drone to perform navigation tasks.
2. According to the state entropy incentive-based reinforcement learning method for optimizing navigation strategy in dense obstacle environments of unmanned aerial vehicles, the method is characterized in that: The step 1 includes the following sub-steps: Step 11: Set the initial position and attitude of the drone, including the (x, y, z) position coordinates and the direction angles (θ x ,θ y ,θ z ); configure the initial speed and acceleration parameters of the drone (v x ,v y ,v z ) and (a x ,a y ,a z ) ; define the UAV's dynamic model, including torque, thrust and air resistance parameters; Step 12: Set map information of dense obstacle environment, such as location, size and shape of obstacles; Step 13. Configure the neural network model for the reinforcement learning algorithm; define the structure of the policy network, including the number and connection method of neurons in the input layer, hidden layer, and output layer, and generate the actions that the drone should perform in a given state; configure the architecture of the value network to evaluate the value under a specific state and action combination; initialize the weights and biases of the neural network and set the training parameters of the network, including the learning rate, batch size, and discount factor, for subsequent training and optimization.
3. The method for optimizing navigation strategy of a UAV in a dense obstacle environment based on state entropy excitation by reinforcement learning according to claim 1 is characterized in that: The step 2 includes the following sub-steps: Step 21: At time step j, obtain the current state s of the drone from the environment j =[x,y,z,v x ,v y ,v z ,d1,d2,…,d 20 ] T , where [x, y, z] represents the position coordinates of the drone, [v x ,v y ,v z ] represents the speed of the drone along the X-axis, Y-axis and Z-axis, [d1, d2, …, d 20 ] represents the distance between the point detected by the laser radar ray carried by the drone and the drone; Step 22: Input the vector into the policy network to obtain the action vector a at the current time step j j ′=[v x ,v y ,v z ] T , v x ,v y ,v z Represents the speed of the drone along the X-axis, Y-axis, and Z-axis respectively; Step 23: Increase the exploration capability. In the above step 22, the action vector a j Add random noise on the basis of ′ to determine the final execution action a j .
4. The method for optimizing navigation strategy of a UAV in a dense obstacle environment based on state entropy excitation by reinforcement learning according to claim 1 is characterized in that: The step 3 comprises the following sub-steps: Step 31: Execute action a j Input into the UAV's dynamic model to update and calculate the next state s of the UAV j+1 ; Step 32: Calculate external reward r ext =r1+r2+r3, r1 is the distance reward, the formula is: Where d0 represents the distance between the current drone and the target point; r2 is the arrival reward. When the drone reaches the target point, a larger reward item is given: r3 is the collision penalty. When the drone collides with an obstacle, a larger penalty item is given: Step 33: determine whether the termination condition is met and record the corresponding end mark; if the drone reaches the target point or collides, it is determined that the current round is over, otherwise the current round is not over.
5. The method for optimizing navigation strategy of a UAV in a dense obstacle environment based on state entropy excitation by reinforcement learning according to claim 1 is characterized in that: The step 4 comprises the following sub-steps: Step 41: At each time step j, the current state s j , perform action a j , external rewards obtained from the environment r j ext , next state s j+1 and whether the identifier is finished are combined into a tuple, and the tuple is stored in the experience replay pool B; Step 42: When the amount of data in the experience replay pool reaches a certain amount or the accumulated number of rounds reaches a preset value, a batch of tuple samples are randomly extracted from the experience replay pool B, and these extracted samples are used for batch update in the training process; Step 43: The extracted tuple samples are recorded as For each tuple in the current state s j , calculate it and the set The distance between all states s in the calculation; among all the calculated distances, select the one with the current state s j The k nearest states are used as neighbors, and the state s closest to the current state is used. k-nn Calculate distance||s j -s k-nn || 2 , and the intrinsic reward is denoted as r j int , the formula is: Step 44: Calculate intrinsic rewards Then, combine it with external rewards Combine to get the final total reward The calculation formula is: Among them, λ is a weight parameter.
6. The method for optimizing navigation strategy of a UAV in a dense obstacle environment based on state entropy excitation by reinforcement learning according to claim 1 is characterized in that: The step 5 comprises the following sub-steps: Step 51: Use the total reward obtained to minimize the TD error δ j To update the parameters of the value network, the formula is expressed as: Among them, δ j is the TD error, Q is the value network, θ Q is its parameter, μ is the policy network, θ μ is the parameter of the policy network, γ is the discount factor, Q ′ and μ′ correspond to the target value network and the target strategy network respectively, θ Q′ and θ μ′ are the parameters of the target value network and the target strategy network respectively; Step 52: Use the policy gradient method to update the parameters of the policy network to maximize the long-term cumulative reward. The formula is: in, Represents relative to the policy network parameter θ μ The objective function J represents the expected value of the long-term cumulative reward. s is the state and a is the action. This policy gradient formula takes into account the total reward. Updated value network.
7. The method for optimizing navigation strategy of a UAV in a dense obstacle environment based on state entropy excitation by reinforcement learning according to claim 1 is characterized in that: The step 6 comprises the following sub-steps: Step 61: Regularly update the parameters of the target strategy network and the target value network, using a soft update strategy, and the formula is: i Q′ ←tth Q +(1-τ)θ Q′ ; i μ′ ←tth μ +(1-τ)θ μ′ ; Among them, θ Q is the parameter of the current value network, θ Q′ is the parameter of the target value network, θ μ is the parameter of the current policy network, θ μ′ is the parameter of the target policy network, τ is the soft update coefficient, which is a small value; Step 62: Evaluate the performance of the strategy in a dense obstacle environment, and adjust the learning rate, noise parameter, and weight hyperparameters of the intrinsic reward based on the evaluation results.
Citation Information
Cited By
Multi-unmanned aerial vehicle cooperation strategy learning method based on inter-unmanned aerial vehicle subsequent influence internal reward
CN120122702A
Robot motion planning method and system based on delay adaptive exploration
CN120576761A
A robot motion planning method and system based on delay adaptive exploration
CN120576761B
AUV hull docking method based on deep reinforcement learning
CN120745739A
AUV Hull Navigation Method Based on Deep Reinforcement Learning
CN120745739B
